Java GC runs cause increase of mem usage on OS
While setting memory and CPU limits in one of our K8s cluster running Java pods, we faced some issue where the pod memory usage was shown as a lot higher in Prometheus (and even some pods crashed due to probe errors).
While further analyzing, we figured out that the GC was working hard to free up the memory as the limits were too small compared to the object creation; causing the CPU cycles to be spend on GC; rather than on actual work. We avoided that by increasing the pod memory limit.
So maybe you can increase the maximum memory limits provided to pods and see if this goes away. I am not exactly sure whether this will help you, but thought of sharing in case it might.