Multiple threads stuck in native calls (Java)

My initial suspicion would be that you are experiencing some sort of class-loader realted dead lock. I imagine, that class loading needs to be synchronized at some level because class information will become available for the entire VM, not just the thread where it was initially loaded.

The fact that the methods on top of the stack are native methods seems to be pure coincidence, since part of the class loading mechanism happens to implemented that way.

I would investigate further what is going on class-loading wise. Maybe some thread uses the class loader to load a class from a network location which is slow/unavailable and thus blocks for a really long time, not yielding the monitor to other threads that want to load a class. Investigating the output when starting the JVM with -verbose:class might be one thing to try.

I was having similar problems a few months ago and found the jthread(?) utility to be invaluable. You give it the process ID for your Java application and it will dump the entire stack for each thread in your process.

From the output of jthread, I could see one thread was trying to obtain a lock after having entered a monitor and another thread was trying to enter the monitor after obtaining the lock. A recipe for deadlock.

I was also wondering if your application was running into a garbage collection issue. You say it runs for a couple days before it stops like this. How long have you let it sit in the stuck state to see if maybe the GC ever finishes?

Can you find out which thread is actually synchronizing on the monitor on which the native method is waiting?At least the thread-dump you get from the VM when you send it a SIGQUIT (kill -3) should show this information, as in

"Thread-0" prio=5 tid=0x0100b060 nid=0x84c000 waiting for monitor entry [0xb0c8a000..0xb0c8ad90]    at Deadlock$1.run(Deadlock.java:8)    - waiting to lock <0x255e5b38> (a java.lang.Object)..."main" prio=5 tid=0x01001350 nid=0xb0801000 waiting on condition [0xb07ff000..0xb0800148]    at java.lang.Thread.sleep(Native Method)    at Deadlock.main(Deadlock.java:21)- locked <0x255e5b38> (a java.lang.Object)

In the dumps you've posted so far, I can't see any thread that is actually waiting to lock a specific monitor...