Ephemeral Containers — For a More Civilized Debugging Age

Bruno Teixeira
6 min readJun 27, 2022


Have you ever faced a scenario where you are deploying a new application, maybe for the first time, and things are not working as expected? Maybe the Pod keeps restarting, or enters CrashLoopBackOff or simply does not boot with the expected parameters? I am sure you have!

Well, the traditional solution was to exec into the failing pod to try and diagnose the issue and get it running properly. However, most containers nowadays, if you are following best practices, are distroless or scratch.

Distroless containers do not have sh (shell) or bash, or any of the core gnu tools used for troubleshooting, making this process tedious, painful and annoying at best.

Enter Ephemeral Containers!

Ephemeral Containers — The Silver Bullet?

Ephemeral Containers is not a new feature and has been around for a while. However, it has finally graduated to BETA in the Kubernetes v1.23 release and is enabled by default.

Up until now you were told that Pods are mostly immutable, intended to be created and disposed of when no longer needed.

With this new capability, the user is able to attach an ephemeral container to an already existent Pod. The motivation behind this feature is the ability to attach a debugging or troubleshooting container without having to add all the required tooling into Application Image itself, or having to change the deployment specs.

The ephemeral container can have access to the filesystem of the Pod, along with its namespaces (process, network etc).

Practical Use Cases

Network Trace

Let’s look at a traditional use case of performing a packet trace inside a particular Pod. The old process was tedious and error prone to automate as it relied on running this dump in the host machine and attaching to the proper interfaces.

Bruno Teixeira

Principal Cloud Engineer with a distributed system’s background, a passion for working with the bleeding edge and an unhealthy obsession for automation.