ETCD

Understanding Kubernetes etcd

In Kubernetes, etcd is a distributed, consistent key value store that holds the entire state of the cluster. Every component in Kubernetes relies on etcd through the API server to know what should be running and what is actually running.

What etcd Stores

Pod specifications, Deployments, Services, ConfigMaps, Secrets, and other Kubernetes resources.

Status information such as pod conditions, node health, and workloads running state.

Policies and cluster wide metadata including RBAC roles, quotas, and namespaces.

Role in Pod Lifecycle

When you create a Pod using kubectl, the API server validates the request and writes the Pod specification into etcd. The scheduler then assigns a node and updates etcd with this decision.

The kubelet on the chosen node reads the assigned Pod and reports its status back through the API server, which again updates etcd.

This loop of desired state and observed state is always reconciled against the data in etcd.

High Availability and etcd

In production, etcd is run as a cluster to ensure fault tolerance. There are two common setups:

Stacked etcd cluster - etcd instances run on the same nodes as the Kubernetes control plane components. This setup is simple but offers less resilience in the event of node failures.

This is generally suitable for smaller environments or development clusters where ease of setup and management is prioritized over high availability.

External etcd cluster - etcd runs on dedicated nodes separate from the control plane, offering enhanced resilience and fault tolerance.

This setup enhances resilience and fault tolerance, as failures in the control plane do not directly impact etcd, and vice versa.

It provides a higher level of availability, making it the preferred choice for production environments where maintaining cluster stability is crucial.

Why etcd is Critical

If etcd is lost, the cluster loses its memory. Control plane components cannot function without it. This is why backups of etcd are essential for disaster recovery.

Kubernetes provides tools like etcdctl snapshot save to back up and restore etcd data.

Check out etcd-backup-restore github repo for the Collection of components to backup and restore the etcd of a Kubernetes cluster.

Key Takeaways

etcd is the single source of truth for Kubernetes.

Every cluster action goes through the API server and is persisted into etcd.

Running etcd in HA mode and backing it up regularly is mandatory for production clusters.