Logging Using ELK Stack

Kubernetes Logging Using ELK Stack

Logging is a basic but critical need in any Kubernetes environment. When containers crash or pods get rescheduled, native logs often vanish. Developers and operators need a system that can collect, process, store, and visualize logs in one place. ELK Stack provides this workflow in a structured way.

What is ELK Stack?

ELK stands for Elasticsearch, Logstash, and Kibana. It is often extended with Beats to form a complete logging pipeline. Each component handles a specific stage in the log lifecycle:

Beats collects data from the source system.

Logstash retrieves and processes this data.

Elasticsearch stores the data in an indexed format.

Kibana visualizes the data for analysis.

This structure is simple and modular, making it suitable for dynamic environments like Kubernetes.

Breakdown of Components

The Elastic Stack groups its tools into three areas of responsibility.

1. Ingest: Handles collection, preprocessing, and shipping

Beats, APM, and Elastic Agent collect and ship logs.

Logstash and Ingest Pipelines process and enrich data.

2. Store: Handles indexing, storing, and searching

Elasticsearch keeps logs in a queryable format.

3. Consume: Handles visualization and analysis

Kibana is used to explore, search, and alert on logs.

Shipping Logs with Filebeat Inside Kubernetes

Filebeat runs as a DaemonSet and handles log collection on each node. Here's how it works:

Pod Watcher inside Filebeat connects to the Kubernetes API Server to watch for pod events and pull metadata like pod ID, name, namespace, and labels.

It also reads container logs from /var/log/containers/ in real time.

Each log line is enriched with Kubernetes metadata, allowing you to filter logs in Elasticsearch by pod, namespace, or label.

Logs are optionally parsed to extract fields or structured JSON.

Finally, enriched logs are pushed to Elasticsearch, making them ready for search and visualization in Kibana.

Now that we’ve covered the basics of the Elastic Stack, let’s look into how to practically implement inside a Kubernetes environment

Practical Implementation

This setup includes both logs and metrics, shipped via Filebeat and Metricbeat agents running inside your Kubernetes cluster.

1. Deploy Filebeat as a DaemonSet

Filebeat will collect logs from /var/log/containers/*.log across all nodes.

Create the DaemonSet using the official Helm chart or custom YAML.

helm repo add elastic https://helm.elastic.co

helm install filebeat elastic/filebeat -n logging --create-namespace --set daemonset.enabled=true --set output.elasticsearch.hosts=["http://elasticsearch.logging.svc:9200"]

2. Deploy Metricbeat as a DaemonSet

Metricbeat collects resource metrics like CPU, memory, network, etc., from nodes, pods, and containers.

helm install metricbeat elastic/metricbeat -n logging --set daemonset.enabled=true --set output.elasticsearch.hosts=["http://elasticsearch.logging.svc:9200"]

Key modules to enable in Metricbeat:

metricbeat.modules:

- module: kubernetes

metricsets:

- container

- node

- pod

- system

- volume

period: 10s

hosts: ["https://${NODE_NAME}:10250"]

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

ssl.verification_mode: "none"

3. Expose Elasticsearch and Kibana

Use NodePort or Ingress to access Kibana UI:

kubectl port-forward service/kibana-kibana 5601:5601 -n logging

Login and create index patterns:

For logs: filebeat-*

For metrics: metricbeat-*

4. APM Integration (Optional)

To collect application-level traces:

Install Elastic APM agent in your app (e.g. Java, Node.js).

Set ELASTIC_APM_SERVER_URL and ELASTIC_APM_SECRET_TOKEN in your pod env.

Deploy APM Server in cluster.

Here are a few important caveats to keep in mind before starting the implementation:

Avoid sending logs directly to the internet

Keep Elasticsearch access internal or behind proper authentication to avoid exposing logs publicly.

Control log volume at source

Use drop_event or filter noisy logs in Filebeat to avoid excessive ingestion costs and storage overload.

Resource limits are critical

Filebeat and Metricbeat can consume CPU on busy nodes. Always set resources.limits and test with load.

Index lifecycle policies matter

Without retention settings, logs and metrics can fill up disk quickly. Use ILM or manual cleanup.

Kibana won’t show data unless indices exist

You must wait until logs or metrics are ingested before setting up index patterns in Kibana.

Ensure node labels and metadata are consistent

Enrichment relies on proper Kubernetes metadata. If labels are missing, log context may be lost.

Elasticsearch needs persistence

Use a proper StorageClass and volume size for Elasticsearch StatefulSet or risk data loss.

Recap:

Filebeat runs as a DaemonSet to collect container logs.

Metricbeat runs as a DaemonSet to collect node and pod metrics.

Elasticsearch runs as a StatefulSet to store and index logs and metrics.

Kibana runs as a Deployment to visualize logs and metrics.

APM Server runs as a Deployment to collect application traces (optional).

This setup gives you end-to-end visibility across logs, metrics, all accessible in Kibana.