Centralized logging refers to collecting logs of many systems across multiple hosts in one central logging system. With the logs in a common log system, debugging issues with distributed systems becomes a lot easier because the logs can be analyzed efficiently.
In this demo, we collect the logs of only one host, a Kubernetes node, which will be created with Minikube. Minikube is a tool that makes it easy to run Kubernetes locally. It runs a single-node Kubernetes cluster inside a virtual machine.
Fluentd is a log agent that runs on each host. The agents collect, parse and forward the logs to Elasticsearch. Elasticsearch is a distributed search and analytics engine by elastic. It is commonly used for full text search use cases. We will use it as our central log system. It stores the logs and provides query capabilities. Kibana is also developed by elastic. It's a web based graphical frontend that allows querying Elasticsearch, visualizing the query results and creating dashboards. We will use it to query the logs comfortably in a graphical interface.
The goal of this blog post is to create a local lab environment that can be used to experiment with Fluentd, Elasticsearch and Kibana. The setup is completely unsuitable for production usage.
Kubernetes custer setup with Minikube
I use minikube 1.3.1-1 from Arch Linux community package repository in this blog post.
sudo pacman -S minikube virtualbox minikube start --disk-size='20000mb' --memory='6000mb' --vm-driver='virtualbox' minikube dashboard
We will use Helm to deploy Elasticsearch, Kibana and Fluentd on Kubernetes.
Let's install Helm.
# Install helm curl -LO https://git.io/get_helm.sh chmod 700 get_helm.sh ./get_helm.sh # Install tiller into the Kubernetes cluster helm init
Save the Helm chart value overrides to a file named elasticsearch-values.yml.
# Permit co-located instances for solitary minikube virtual machines. antiAffinity: "soft" # Shrink default JVM heap. esJavaOpts: "-Xmx128m -Xms128m" # Allocate smaller chunks of memory per pod. resources: requests: cpu: "100m" memory: "512M" limits: cpu: "1000m" memory: "512M" # Request smaller persistent volumes. volumeClaimTemplate: accessModes: [ "ReadWriteOnce" ] storageClassName: "standard" resources: requests: storage: 100M
Deploy Elasticsearch with the following commands.
helm repo add elastic https://helm.elastic.co helm upgrade --install --values elasticsearch-values.yml elasticsearch elastic/elasticsearch
Deploy Kibana with the following command.
helm upgrade --install kibana elastic/kibana
As we want Fluentd to run on each Kubernetes node, it is deployed as a DaemonSet. Save the Helm chart value overrides to a file named fluentd-values.yml.
elasticsearch: host: elasticsearch-master.default.svc.cluster.local configMaps: useDefaults: systemInputConf: false forward.input.conf: false monitoring.conf: false
Deploy Fluentd with the following commands.
helm repo add kiwigrid https://kiwigrid.github.io helm upgrade --install --values helm-values/fluentd-values.yml fluentd kiwigrid/fluentd-elasticsearch
Kubernetes nodes write all container logs to files in /var/log/.
The helm chart configures Fluentd to forward these logs by default.
The volume mounts are defined in this file:
This file contains the Fluentd config including parsing rules:
Forward traffic to Kibana with the following command.
kubectl port-forward svc/kibana-kibana 5601
Next, go to localhost:5601 to access Kibana. Create an index pattern logstash-* and choose the time filter field name. You can start exploring the logs in the Discover section.
We created a local Kubernetes cluster with Minikube, installed Helm and deployed Elasticsearch + Fluentd + Kibana with Helm. Have fun playing around with this centralized logging demo.