Best practices: log management for microservices based on java, running on k8s
This is an opinion question so, I'll answer in an opinionated way. We've had this issue. The problem is that log4j (or log4j2) logs are very chatty and everytime you get a Java exception, for example, it gets printed in many lines. This combined with the regular stdout logs of the container seems convoluted and would be a pain to separate all from stdout to send to Fluentd or a something like logspout
You can allocate disk space in the container and send the log4j2 logs to a file inside the container. Then you can prune your existed containers or truncate your logs on a regular schedule. With [Kubernetes](You can also read up on Kubernetes logging you can set up a Fluentd or Filebeat sidecar to ship your logs too.
But the best solution IMO, in this case, would be just to ship the log4j logs directly to an Elasticsearch cluster and possibly storing the data in its own index. This is something that you might be able to use for that.