Why does max number of threads decrease when I increase max heap size?
Each thread requires a stack. The more memory you allocate to the heap, the less memory is left for the stacks.
This will be particularly acute if you're using a 32-bit JVM, since the process will have no more than 4GB of address space for everything (the code, the heap, the stacks etc). I cannot reproduce this behaviour on my 64-bit box, where "Max threads" remains the same regardless of how much memory I allocate to the heap.
It is worth noting that many operating systems enable you to tweak the size of the stack. On Unix this is done using ulimit -s.
I'm guessing that increasing the heap size (and using all of it) lowers the amount of memory the OS has available for creating new threads, so it can't create as many threads as before.