PHP, shell commands, and jar files
Using exec() would normally work properly; however, in your case it looks like Java is unable to allocate enough memory to start (hence the error message). If you're running in a virtual machine with limited RAM and no swap space, you'll find it all but impossible to run even the simplest java programs.
The first step is to see if java runs at all on your machine. Try to just run
java -versionIf that fails with the same error, you'll have to increase the amount of memory available somehow (eg, buying a plan with more memory or swap space). If you can't, you might be able to get by if you limit the maximum heap size with -Xmx16m (sets the maximum heap size to 16 megabytes) or even less, but in that case your jar file may not have enough memory to run!