Kubernetes Pod memory usage does not fall when jvm runs garbage collection
It looks like you are creating a pod with a resource limit of 1GB Memory.You are setting -XX:MaxRAMFraction=2 which means you are allocating 50% of available memory to the JVM which seem to match what you are graphing as Memory Limit.
JVM then reserves around 80% of that which is what you are graphing in Memory Consumed.
When you look at Memory Consumed you will not see internal garbage collection (as in your second graph), because that GC memory is released back to JVM but is still reserved by it.
Is it possible that there is a memory leak in your java application? it is possibly causing more memory to get reserved over time, until the JVM limit (512MB) is met and your pod gets OOM killed.