Old I/O thread per client model or NIO reactor pattern?

I do not want to design a complicated NIO system that uses multiple threads just to take full advantage of all the CPU power, but I also cringe at the idea of having a single application use 300+ threads. Which design is right for my purpose?

Our JVMs run with a lot more that 500 threads continually (right now they are sitting at ~700) with peaks in the 1000s. I see no reason to think that 800 threads is in some way "cringe" worthy. I'd start to worry when you reach 10,000 threads (as a ball park number) -- maybe less if you are running under 32bit. You certainly are going to have to allocate more memory as you get into the 1000s but anything under 1k threads should not be a problem. Here's a good page on thread creation numbers.

NIO is most efficient when you have a large number of connections with infrequent IO. It solves a lot of problems when it comes to asynchronous communication and there are things you can do with NIO that the "old IO" cannot do from a functional standpoint (interruptible channels and non-blocking IO for example) but the single thread handler is a much simpler model and I'm not surprised that it can outperform NIO implementations in many configurations. With NIO you are doing a lot of operations in Java code that are done for you in the JVM or even the kernel in native code. Multiplexing of streams and handling of ready IO are all things that you get "for free" (in terms of complexity) with the "old IO" model.

I would keep it simple and stick with the thread-per-client pattern until you have good reason to take the complexity hit.