Where are Kubernetes' pods logfiles?

Short Answer:

If you're using Docker, the stdout from each container are stored in /var/lib/docker/containers. But Kubernetes also creates a directory structure to help you find logs based on Pods, so you can find the container logs for each Pod running on a node at /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/.

Longer Answer:

Docker traps the stdout logs from each container and stores them in /var/lib/docker/containers on the host. If Kubernetes uses Docker as the container runtime, Docker will also store the containers logs in that location on the Kubernetes node. But since we don't run containers directly in Kubernetes (we run Pods), Kubernetes also creates the /var/log/pods/ and /var/log/containers directories to help us better organize the log files based on Pods.

Each directory within /var/log/pods/ stores the logs for a single Pod, and each are named using the structure <namespace>_<pod_name>_<pod_id>.

You can get the ID of a Pod by running kubectl get pod -n core gloo-76dffbd956-rmvdz -o jsonpath='{.metadata.uid}'. If you're used to using yq, you may find running kubectl get pod <pod_name> -o yaml | yq r - metadata.uid more straight-forward.

Within each /var/log/pods/<namespace>_<pod_name>_<pod_id>/ directory are more directories, each representing a container within the Pod. The name of these directories is equal to the name of the container. Lastly, when we look inside a /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/ directory, we'll find symbolic links to the log files stored by Docker inside /var/lib/docker/containers.

Similarly, inside the /var/log/containers/ directory are symlinks to a /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/ directory. These symlinks are named using the structure <pod_name>_<namespace>_<container_id>.

Do you see anything in those directories?

In my clusters, the stdout/stderr logs from each pod are in /var/log/containers, however there is some linking/redirection:

/var/log/containers/<pod-name>_<namespace>_<container-name-container-id>.log -> /var/log/pods/<some-uuid>/<container-name>_0.log

And that log is actually linked into /var/lib/docker:

<container-name>_0.log -> /var/lib/docker/containers/<container-id>/<container-id>-json.log

The on-disk filename comes from

docker inspect $pod_name_or_sha | jq -r '.[0].LogPath'

assuming the docker daemon's configuration is the default {"log-driver": "json-file"}, which is almost guaranteed to be true if kubectl logs behaves correctly.

This may also go without saying, but you must be on the Node upon which the Pod was scheduled for either docker inspect, or sniffing around for the presence of log files on disk, to do anything helpful. kubectl describe pod $pod_name will render the Node name, or as you might suspect it'll be in kubectl get -o json pod $pod_name if you wish to acquire it programmatically.