Centralized logging refers to collecting logs of many systems across multiple hosts in one central logging system. With the logs in a common log system, debugging issues with distributed systems becomes a lot easier because the logs can be analyzed efficiently.

The main idea behind centralized logging.

The main idea behind centralized logging.

In this demo, we collect the logs of only one host, a Kubernetes node, which will be created with Minikube. Minikube is a tool that makes it easy to run Kubernetes locally. It runs a single-node Kubernetes cluster inside a virtual machine.

Fluentd is a log agent that runs on each host. The agents collect, parse and forward the logs to Elasticsearch. Elasticsearch is a distributed search and analytics engine by elastic. It is commonly used for full text search use cases. We will use it as our central log system. It stores the logs and provides query capabilities. Kibana is also developed by elastic. It’s a web based graphical frontend that allows querying Elasticsearch, visualizing the query results and creating dashboards. We will use it to query the logs comfortably in a graphical interface.

The goal of this blog post is to create a local lab environment that can be used to experiment with Fluentd, Elasticsearch and Kibana. The setup is completely unsuitable for production usage.

Kubernetes custer setup with Minikube

I use minikube 1.3.1-1 from Arch Linux community package repository in this blog post.

1
2
3
sudo pacman -S minikube virtualbox
minikube start --disk-size='20000mb' --memory='6000mb' --vm-driver='virtualbox'
minikube dashboard

Helm setup

We will use Helm to deploy Elasticsearch, Kibana and  Fluentd on Kubernetes.

Let’s install Helm.

1
2
3
4
5
6
# Install helm
curl -LO https://git.io/get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
# Install tiller into the Kubernetes cluster
helm init

Elasticsearch setup

Save the Helm chart value overrides to a file named elasticsearch-values.yml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Permit co-located instances for solitary minikube virtual machines.
antiAffinity: "soft"

# Shrink default JVM heap.
esJavaOpts: "-Xmx128m -Xms128m"

# Allocate smaller chunks of memory per pod.
resources:
  requests:
    cpu: "100m"
    memory: "512M"
  limits:
    cpu: "1000m"
    memory: "512M"

# Request smaller persistent volumes.
volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "standard"
  resources:
    requests:
      storage: 100M

Deploy Elasticsearch with the following commands.

1
2
helm repo add elastic https://helm.elastic.co
helm upgrade --install --values elasticsearch-values.yml elasticsearch elastic/elasticsearch

Kibana setup

Deploy Kibana with the following command.

1
helm upgrade --install kibana elastic/kibana

Fluentd setup

As we want Fluentd to run on each Kubernetes node, it is deployed as a DaemonSet. Save the Helm chart value overrides to a file named fluentd-values.yml.

1
2
3
4
5
6
7
8
elasticsearch:
  host: elasticsearch-master.default.svc.cluster.local

configMaps:
  useDefaults:
    systemInputConf: false
    forward.input.conf: false
    monitoring.conf: false

Deploy Fluentd with the following commands.

1
2
helm repo add kiwigrid https://kiwigrid.github.io
helm upgrade --install --values helm-values/fluentd-values.yml fluentd kiwigrid/fluentd-elasticsearch

Kubernetes nodes write all container logs to files in /var/log/. The helm chart configures Fluentd to forward these logs by default.

The volume mounts are defined in this file:

This file contains the Fluentd config including parsing rules:

Access Kibana

Forward traffic to Kibana with the following command.

1
kubectl port-forward svc/kibana-kibana 5601

Next, go to localhost:5601 to access Kibana. Create an index pattern logstash-* and choose the time filter field name. You can start exploring the logs in the Discover section.

Summary

We created a local Kubernetes cluster with Minikube, installed Helm and deployed Elasticsearch + Fluentd + Kibana with Helm. Have fun playing around with this centralized logging demo.