Troubleshooting with Ephemeral Containers in Kubernetes

Understanding the Use Case

When deploying containers in production environments, many projects opt for distroless or minimal base images. This strategy enhances security and minimizes resource consumption. Nonetheless, these images typically do not include a shell or the necessary utilities for troubleshooting. Consequently, using `kubectl exec` for debugging becomes ineffective. So, what are the methods to troubleshoot an application when issues arise?

For instance, you might need to:

• Check network connectivity

• Troubleshoot DNS issues

• Inspect files or configurations

  
  

This is where ephemeral containers come into play.

What Are Ephemeral Containers?

Ephemeral containers are temporary containers that run in the same pod namespace as your application. You can add them even after the pod has started. They are ideal for troubleshooting.

Imagine your pod runs on a minimal base image containing only essential application binaries and dependencies. If something goes wrong, you need to debug quickly. With ephemeral containers, you can add a debug container directly into the running pod. This debug container can have all the necessary utilities like a shell, `curl`, or other custom networking tools for troubleshooting your application.

Here’s what happens when you create a debug container:

1. Kubernetes adds a temporary (ephemeral) container directly inside the existing pod.

2. It shares the same pod's network namespace, allowing direct access to localhost services.

3. It shares volumes, storage, and resources with existing containers in the pod.

   
   

Let’s look at this with practical examples.

Debugging a Pod

How to Debug a Pod

Suppose you have an application running inside a pod that's unable to connect to a database pod. You've already checked configurations, endpoints, and done general troubleshooting. Now, you want to use the `nc` (netcat) command to verify network connectivity from your application pod to the database. Or maybe you need to run `strace` to identify issues with a slow process.

In these cases, you can use the `kubectl debug` command to add an ephemeral debug container to your existing pod. For example, the following command shows how to add an ephemeral debug container (using the `nicolaka/netshoot` image) to `nginx-pod` and test the database endpoint using `nc`:

```bash

kubectl debug nginx-pod --image=nicolaka/netshoot --target=nginx -- /bin/sh

```

Although this command adds a debug container inside the pod, Kubernetes won't show the ephemeral container when listing pods. For example, after deploying the debug container, the `nginx-pod` still shows (1/1) containers running. This means it only counts the original application container, not the ephemeral one.

```bash

$ kubectl get po

NAME READY STATUS RESTARTS AGE

mariadb-pod 1/1 Running 0 4m19s

nginx-pod 1/1 Running 0 2m46s

```

However, if you describe the `nginx-pod`, you will see the ephemeral container in the output. (I have truncated the output for clarity.)

Isolating Production Pods for Debugging

Let’s say you have a live application running in production, and you're facing performance issues. You want to troubleshoot or experiment without affecting the live pod. For example, you might want to change parameters or use a new image version with the current production configuration.

In such cases, you can copy the production pod using `kubectl debug`. This helps you:

• Reproduce and isolate issues outside the production pod.

• Run diagnostic tools (such as `strace`, `tcpdump`, or other debugging utilities) without risking or adding overhead to the original pod.

  
  

Copying a Pod

To copy a pod, use the `--copy-to` parameter as shown below:

```bash

kubectl debug mariadb-pod --copy-to=mariadb-debug-pod --image=nicolaka/netshoot

```

The above command creates a copy of the `mariadb-pod` with an additional debug container inside it.

```bash

$ kubectl get po

NAME READY STATUS RESTARTS AGE

mariadb-debug-pod 2/2 Running 0 7s

mariadb-pod 1/1 Running 0 25m

```

If you check the output, you'll see two containers inside the debug pod. This is different from regular debugging. The previous command just copies the pod and opens an exec session.

But you can also copy a pod and attach a debug container at the same time. For example, if you want to copy `mariadb-pod` and add a debug image (`nicolaka/netshoot`) for troubleshooting, you can use:

```bash

kubectl debug mariadb-pod --copy-to=mariadb-debug-pod --image=nicolaka/netshoot

```

This command creates a completely new pod. It copies your original container (`mariadb`) and adds the new debug container (`netshoot`).

Conclusion

In conclusion, ephemeral containers are a powerful tool for troubleshooting in Kubernetes. They allow you to debug applications running in minimal base images without disrupting the production environment. By leveraging the `kubectl debug` command, you can quickly add a debug container to your existing pods or create copies of your production pods for safe experimentation.

By following these practices, you can enhance your troubleshooting capabilities and maintain the stability of your applications. Embrace the power of ephemeral containers and take your debugging skills to the next level!

Remember, the key to effective debugging is to remain calm and methodical. Happy troubleshooting!