See java exit code in Unix
The value returned by System.exit can be read by the operating system, and it's OS-specific how it's interpreted. In most UNIX-like systems, a program's exit code can be captured by other processes if they so choose, but nonzero return values don't cause the OS to print any error message. For example, the javac compiler will return a nonzero status code if the program fails to compile, but the OS won't print out any special error messages in this case. If you want to bring up some sort of error dialog, you will need to do so manually.
Hope this helps!