How to detect the cause of OutofMemoryError?

According to this post:

There are two possible causes of the java.lang.OutOfMemoryError: Failed to create a thread message:

• There are too many threads running and the system has run out of internal resources to create new threads.
• The system has run out of native memory to use for the new thread. Threads require a native memory for internal JVM structures, a Java™ stack, and a native stack.

So this error may well be completely unrelated to memory, just that too many threads are created...

EDIT:

As you've got 695 threads, you would need 695 times the stack size as memory. Considering this post on thread limits, I suspect that you are trying to create too many threads for the available virtual memory space.

You should start the JVM with the -XX:+HeapDumpOnOutOfMemoryError flag. This will produce a heap dump when the OutOfMemoryError is generated.

Then, as @Steve said, you can use a tool like MAT to analyze the dump and see which objects are allocated, and who is keeping references to them. This usually will give you some insight on why your JVM is exhausting its memory.

I know what you mean, it can be confusing to find somewhere to begin.

Have a look at Eclipse Memory Analyzer (MAT). It will use JHat to dump a memory snapshot of your program into a file, which you can re-open and analyze.

The browser for this file outlines all the objects created by the program very neatly, and you can look into various levels to find if something is suspicious.

Appending my comments to answer...

Right when your executable webapp crashes, dump it to MAT. MAT will tell you what object is being created a bunch of times. If it's a custom object, and it often is, it's easy to find. If not, you can see its parent, amputate it, and dribble down from there (sorry for the graphic example, I'm not entirely focused on SO at the moment :).

Oh, and I forgot to mention, you can run the program several times under several conditions, and make a dump each time. Then, you can analyze each dump for the trend.

But in my case what should I use?I have a web app running in Tomcat

Sorry, missed this too. If I'm not mistaken, MAT dumps the JVM process, so as long as the VM is running on your box you can dump its process and see what's going on.

Another comment mutated into partial solution...

This is becoming more difficult than it actually is. Seriously, it's pretty easy, after you run MAT once or twice to get the hang of things. Run your app until the thing crashes. Dump it. Change something. Run, crash, dump. Repeat. Then, open the dumps in MAT, and compare what looks suspicious.

The trickiest part when I was learning this was finding the process ID to dump - which is still not too mind-numbing.