To have a Kubernetes cluster up and running is pretty easy these days. However, when you start to use it and deploy some applications, you might expect some issues over time. Kubernetes being a distributed system isn't easy to troubleshoot. You need a proper monitoring solution, and because the Prometheus and fluentd is part CNCF project as Kubernetes, it is probably the best fit. In this post, I will show how to get the Prometheus and EFK stack running and start monitoring + logging your Kubernetes cluster.

This presentation aim to cover about how to deploy solution of Monitoring and centralized logging of k8s cluster. What my team really needed was something lean we could spin up in a docker container and then ‘grow’ by extending the configuration or adding components as and when my needs change. With those requirements in hand we soon came across Prometheus, a monitoring system and time series database, with its de-facto graphical front-end Grafana for monitoring. And we found EFK stack, with fluentd which is an open source data collector for unified logging layer.

The combination of Prometheus and Grafana is becoming a more and more common monitoring stack used by DevOps teams for storing and visualizing time series data. Prometheus acts as the storage backend and Grafana as the interface for analysis and visualization. Prometheus collects metrics from monitored targets by scraping metrics from HTTP endpoints on these targets. By adding Grafana as a visualization layer, we can easily set up a monitoring stack for our monitoring stack.

When running multiple services and applications on a Kubernetes cluster, a centralized, cluster-level logging stack can help you quickly sort through and analyze the heavy volume of log data produced by your Pods. One popular centralized logging solution is the Elasticsearch, Fluentd, and Kibana (EFK) stack. Elasticsearch is a real-time, distributed, and scalable search engine which allows for full-text and structured search, as well as analytics. It is commonly used to index and search through large volumes of log data, but can also be used to search many different kinds of documents. Elasticsearch is commonly deployed alongside Kibana, a powerful data visualization frontend and dashboard for Elasticsearch. Kibana allows you to explore your Elasticsearch log data through a web interface, and build dashboards and queries to quickly answer questions and gain insight into your Kubernetes applications. In this presentation we'll use Fluentd to collect, transform, and ship log data to the Elasticsearch backend. Fluentd is a popular open-source data collector that we'll set up on our Kubernetes nodes to tail container log files, filter and transform the log data, and deliver it to the Elasticsearch cluster, where it will be indexed and stored.

This presentation will help you to understanding what we need to monitoring and logging, and maybe we can help to contribute to the project of the application we used.