Unable to create new native thread error - but very few threads are in use [duplicate]

Finally figured this out.

There were a couple of data streams that we processed (4 out of 10 million at this site) that wound up creating a ton of DeflaterOutputStream objects. A 3rd party library we were using was calling finish() on the stream instead of close(). The underlying Deflater finalizer was cleaning things up, so as long as the load wasn't too high, there were no problems. But past a tipping point, we started running into this:

http://jira.pentaho.com/browse/PRD-3211

which led us to this:

http://bugs.sun.com/view_bug.do?bug_id=4797189

Several hours after that happened, the system finally got itself into a corner that it couldn't get out of and was unable to create a native thread when we needed.

The fix was to get the 3rd party library to close the DeflaterOutputStream.

So definitely a native resource leak. If anyone else is ever hitting something like this, the VMMap tool was indispensable for eventually tracking down which data streams were causing the problem.

I suspect, though it is clearly difficult to prove, that you are running into a 32bit memory allocation problem.

Threads are allocated native memory, not heap memory which has to be contiguous, in order to run. Whilst I am sure that WOW64 allows 32 bit processes to operate in the region above 4gb I'm not so sure about allocating native memory for a new thread above the 4gb limit should the intervening space be used.

Hence your application and the heap are in lowish mem, other processes are taking the intervening 3.07gigs ( if memory serves )and then attempting to allocate a native memory block 4gb above the initial caller in order to create a new thread.

Could you confirm that this issue only occurs when memory use is around or above the 4gb mark?