Understanding Kubernetes Logs

Understanding Kubernetes Logs - A Comprehensive Guide

Not every day you would be setting up clusters; however, there is a huge possibility that every day you would be involved in the operational and troubleshooting aspects of Kubernetes.

Understand the logs plays a crucial role in any K8S practitioner's life.

You can’t miss it or skip it.

I broke down the typical Kubernetes logs directory for easy understanding.

Like other directories, you can park the application logs under '/var/log/', which can further be organized by 'namespace’ and provide insights into the internal statuses of your application.

They are especially valuable for troubleshooting issues and tracking cluster events.

The log directory structure is quite straightforward; rather than spending time going through each one, let’s talk about ‘cluster-level logging’—a less discussed and crucial aspect.

Cluster-level logging architectures

Kubernetes natively doesn’t provide log storage, so cluster-level logging requires independent dedicated backend storage and lifecycle for logs, separate from nodes, pods, and containers.

Here are a few options:

Run a logging agent on each node to collect logs.

Add a sidecar container in the application pod specifically for logging.

Send logs directly from the application to a logging backend.

1. Node logging Agent

To implement cluster-level logging, deploy a node-level logging agent on each node, typically as a DaemonSet, to collect and forward logs to a backend.

This agent, running as a container, accesses log directories from all application containers on the node.

Node-level logging creates one agent per node without needing changes to applications.

Containers log to stdout and stderr in varied formats, which the node-level agent gathers for aggregation.

2. Streaming Sidecar Container

By configuring sidecar containers to write to their own stdout and stderr streams, you can leverage the kubelet and node-level logging agent already running on each node.

Sidecars read logs from sources like files, sockets, or journald and output to separate streams, enabling multiple log streams for different application parts.

This setup supports components that don’t natively log to stdout or stderr, with minimal redirection overhead.

Since kubelet manages stdout and stderr, you can easily access logs using tools like kubectl logs.

3. Sidecar Container With a Logging Agent

If the node-level logging agent doesn’t meet your needs, you can add a sidecar container with a logging agent set up specifically for your application.

4. Pushing Logs Directly From The Application

Ref: Kubernetes Documentation

Logging that involves directly exposing or pushing logs from each application is outside the scope of Kubernetes, though it can be an option for specific cases, like when an application requires direct integration with an external logging service.

Hope this was useful in clarifying a few questions related to Kubernetes logs.