FluentD log unreadable. it is excluded and would be examined next time

I faced a similar issue.So, what actually happens is -

1. Fluentd creates a symbolic links in /var/log/containers/ which are further a symbolic link of files that are in /var/log/pods/ like -

root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log lrwxrwxrwx. 1 root root 83 Oct 30 07:42 /var/log/containers/consul-0_default_consul-c4dbf47bf46b4cacfb0db67885fdba73835e05b45b14ec7dc746cc2d5ed92ea3.log -> /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1628.log

2. /var/log/pods are symbolic links of the log files mounted at host. In my setup I am using /data/ directory of host/node to store docker data.

root@fluentd-forwarders-5bfzm:/home/fluent# ls -ltr /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log lrwxrwxrwx. 1 root root 162 Oct 30 07:47 /var/log/pods/default_consul-0_2a206546-73b3-4d05-bd7a-0b307c8b24d1/consul/1629.log -> /data/docker/containers/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b/478642a56a6e15e7398391a2526ec52ad1aa24341e95aa32063163da11f4cc8b-json.log

So, in my deployment.yaml I had to mount /data/docker/containers rather /var/lib/containers/ to solve the issue i.e

        volumeMounts:        - mountPath: /var/log          name: varlog        - mountPath: /data/docker/containers          name: datadockercontainers          readOnly: true        - mountPath: /fluentd/etc          name: config-path

May colachg suggestion help you:

I think that kubelet create some symbolic links in '/var/log/containers'(just links not real file), so you must mount both links and real files or only mount real file with right fluentd.conf.

As you defined /var/log in the list, the others /var/log/... are duplicated.

Remove /var/log

Check with kubectl describe pod fluentd-...whether all volumes were mounted properly.