Docker daemon fails to process logs

Issue=Docker Engine Error json-file: fix sporadic unexpected EOF errors

I had spent some time researching and trying to find errors in the dockerlogs.

There is a bug in docker engine 20.10.5 which is the version running in docker. A Github PR fixes this. https://github.com/moby/moby/pull/42104

I found the error while running

$ journalctl -u docker.servicelevel=warning msg="got error while decoding json" error="unexpected EOF" retries=19999

Docker driver for logs has a bug and it wont handle some log formatting. The bug then floods the system's log with retries messages. This situation generates a very annoying CPU load.

Docker engine latest is 20.10.7. The issue has been fixed in release Docker engine 20.10.6 https://docs.docker.com/engine/release-notes/#20106

The aforementioned issue will cause an abnormal spike in CPU usage of the docker (version <= 19) process and constantly stay at that level until you fix the logs.

To fix the logs & cpu usage, you've to truncate all the container logs (since we are unable to determine which container has faulty logs). It works only if your logs are in json format:

truncate -s 0 /var/lib/docker/containers/*/*-json.log

Please see Andre's answer for a permanent fix.