private bytes increase for a javaw process in java 8

You may try another GC implementation like G1 introduced in Java 7 and probably the default GC in Java 9. To do so just launch your Java apps with:

-XX:+UseG1GC

There's also an interesting functionality with G1 GC in Java 8u20 that can look for duplicated Strings in the heap and "deduplicate" them (this only works if you activate G1, not with the default Java 8's GC).

-XX:+UseStringDeduplication

Be aware to test thoroughly your system before going to production with such a change!!!

Here you can find a nice description of the diferent GCs you can use

consider optimising the JVM options

1. Parallel Collector(throughput collector)

   -XX:+UseParallelGC

2. concurrent collectors (low-latency collectors)

   -XX:+UseConcMarkSweepGC

3. use String Duplicates remover

   -XX:+UseStringDeduplication

4. optimise compact ratio

   -XXcompactRatio:and refer link1link2

In this my answer you can see information and references how to profile native memory of JVM to find memory leaks. Shortly, see this.

UPDATE

Did you use -XX:NativeMemoryTracking=detail option? The results are straightforward, they show that the most memory allocated by malloc. :) It's a little bit obviously. Your next step is to profile your application. To analyze native methods and Java I use (and we use on production) flame graphs with perf_events. Look at this blog post for a good start.

Note, that your memory increased for threads, likely your threads grow in application. Before perf I recommend analyze thread dumps before/after to check does Java threads number grow and why. Thread dumps you can get with jstack/jvisualvm/jmc, etc.