Unable to collect all kubernetes container/pod logs via fluentd/elasticsearch Unable to collect all kubernetes container/pod logs via fluentd/elasticsearch kubernetes kubernetes

Unable to collect all kubernetes container/pod logs via fluentd/elasticsearch


I am using fluent bit for the same purpose and I met exactly the same problem quite a few days back. Fluent bit is a light weight version of fluentd, and what worked for me might work for you as well.

What was wrong with my fluent bit was the input configuration. For the tail plugins that tail into large log files, there was some issue with the log rotation. So I lowered my refresh_interval to something like 5 secs (time period over which list of watched files are updated). Then I lowered the mem_buf_limit to something like 5MB (the total size of logs fluent bit takes into memory before flushing that out to the output plugin).

By these changes I was able to get more logs which were earlier not being collected for god knows reason.

I have asked this as an issue. Will update my answer if I get to know the reason.

Hope this helps in anyway. Mainly I suggest you to tweak your input configurations and then see the changes.