What happens when there's insufficient memory to throw an OutOfMemoryError?

The JVM never really runs out of memory. It does memory computation of the heap stack in advance.

The Structure of the JVM, Chapter 3, section 3.5.2 states:

• If Java virtual machine stacks can be dynamically expanded, and expansion is attempted but insufficient memory can be made available to effect the expansion, or if insufficient memory can be made available to create the initial Java virtual machine stack for a new thread, the Java virtual machine throws an OutOfMemoryError.

For Heap, Section 3.5.3.

• If a computation requires more heap than can be made available by the automatic storage management system, the Java virtual machine throws an OutOfMemoryError.

So, it does a computation in advance before doing allocation of the object.

What happens is that the JVM tries to allocate memory for an object in the memory called Permanent Generation region (or PermSpace). If allocation fails (even after the JVM invokes the Garbage Collector to try & allocate free space), it throws an OutOfMemoryError. Even exceptions requires a memory space so the error will be thrown indefinitely.

Further reading.? Furthermore, OutOfMemoryError can occur in different JVM structure.

Graham Borland seems to be right: at least my JVM apparently re-uses OutOfMemoryErrors. To test this, I wrote a simple test program:

class OOMTest {    private static void test (OutOfMemoryError o) {        try {            for (int n = 1; true; n += n) {                int[] foo = new int[n];            }        } catch (OutOfMemoryError e) {            if (e == o)                System.out.println("Got the same OutOfMemoryError twice: " + e);            else test(e);        }    }    public static void main (String[] args) {        test(null);    }}

Running it produces this output:

$ javac OOMTest.java && java -Xmx10m OOMTest Got the same OutOfMemoryError twice: java.lang.OutOfMemoryError: Java heap space

BTW, the JVM I'm running (on Ubuntu 10.04) is this:

$ java -versionjava version "1.6.0_26"Java(TM) SE Runtime Environment (build 1.6.0_26-b03)Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Edit: I tried to see what would happen if I forced the JVM to run completely out of memory using the following program:

class OOMTest2 {    private static void test (int n) {        int[] foo;        try {            foo = new int[n];            test(n * 2);        }        catch (OutOfMemoryError e) {            test((n+1) / 2);        }    }    public static void main (String[] args) {        test(1);    }}

As it turns out, it seems to loop forever. However, curiously, trying to terminate the program with Ctrl+C doesn't work, but only gives the following message:

Java HotSpot(TM) 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGINT to handler- the VM may need to be forcibly terminated

Most runtime environments will pre-allocate on startup, or otherwise reserve, enough memory to deal with memory starvation situations. I imagine most sane JVM implementations would do this.