How to solve "java.io.IOException: error=12, Cannot allocate memory" calling Runtime#exec()?
What's the memory profile of your machine ? e.g. if you run top, how much free memory do you have ?
I suspect UnixProcess performs a fork() and it's simply not getting enough memory from the OS (if memory serves, it'll fork() to duplicate the process and then exec() to run the ls in the new memory process, and it's not getting as far as that)
EDIT: Re. your overcommit solution, it permits overcommitting of system memory, possibly allowing processes to allocate (but not use) more memory than is actually available. So I guess that the fork() duplicates the Java process memory as discussed in the comments below. Of course you don't use the memory since the 'ls' replaces the duplicate Java process.
This is solved in Java version 1.6.0_23 and upwards.
See more details at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7034935