What are good InstallAnywhere replacements for installing a Java EE application?

In this case, unfortunately, SO didn't tell us much that we didn't know already (and indeed the tool we ultimately chose was not mentioned in any answer). However, I'm posting our experiences here, as they might be helpful to others later on.

We (quickly) evaluated the following installer tools, mostly by looking at their websites and browsing for other information on the web: Actual Installer, Advanced Installer, BitRock InstallBuilder, Inno Setup, Install Creator, Installer VISE for Windows, InstallShield, install4j, IzPack, NSIS, openInstaller, Scriptlogic MSI Studio, Smart Install Maker, Symantec Wise Installation Studio, and WiX.

We screened out most of them early on, and ended up shortlisting two options for closer evaluation: BitRock InstallBuilder and install4j. For these tools, we downloaded evaluation versions and did some prototyping, to see if the things that are most important to us are really supported, and how easy or hard it is to get things working.

Both of the options were good in many things (and both seemed good alternatives to InstallAnywhere):

• They produce completely native and pretty Windows .exe installers that areeasy to customise with your own graphics etc.
• Both tools could easily be automated so that installer building is triggered from Ant. (With install4j it literally took me just five minutes to learn it and then implement it.)
• Both companies seem to have good support (well, at least for prospects evaluating their products ;-) Especially from BitRock we got very quick replies.

In the following things install4j seemed better than BitRock for our needs (many of these are subjective, of course, so YMMV):

• install4j definitely has better support for running custom Java code - it can be done at any point during the installation, and regardless of whether there's any preinstalled JRE on the system.
• BitRock uses a more hard-coded sequence of installation steps while install4j is more flexible. In install4j, adding custom screens and forms (with or without custom Java code), asking user for input, etc., can be done at any point, before or after installing any files.
• Also some basic things like defining the filesets that are to be copied to the target system, and adding an installation step to replace certain strings in configuration files seemed somewhat easier in install4j.
• install4j has better support for JRE bundling
• When creating installers on Linux, the look & feel of install4j IDE was nicer (more "native") than that of BitRock
• (install4j's licensing options were better for us - we strongly preferred a couple of floating licenses to named licenses for all developers or an expensive "site license")

So ultimately we went with install4j. Here are some more details on why it was impressive:

• Its IDE, where you put the installer together, is very simple and easy to use - I could figure out how to do most things I wanted quickly, even without looking at documentation. And when I did have to check something in the documentation (e.g. how to refer to installer variables; how to get started writing custom Java code against the install4j API), it didn't take long to find what I needed.
• You can completely customise the screens and actions during the installation procedure, and also add custom screens and actions (coded against their Java API) at any point. This was important to us because we need to reuse existing custom Java code from the old InstallAnywhere installer.
• In some small details, install4j seems ideal for Java developers. For example, if you want to add a validation script to check some user input, you can code that very quickly in the install4j IDE itself, using plain old Java, with coding assistance resembling that of IntelliJ IDEA.
• We deemed the cost of install4j floating licenses reasonable, considering how good the tool is (and downright bargain compared to the inflated pricing of InstallAnywhere...)
• In short, it seemed like the best installer tool available for deploying Java applications.

We created BitRock InstallBuilder, a crossplatform installation tool after some frustrating experiences with InstallAnywhere and specially its cousin (now defunct) InstallShield MP. More than half of our clients use our tool to package Java-based software but the tool is native, not Java-based. That means we have a number of advantages such as native look and feel (incl. Vista, GTK, Qt, etc.) and no need to do self-extraction previous to installation (faster startup, less space requirements) to mention a couple. To answer your specific questions:

• Supports calling Java code as part of the installation at multiple points in time (such as when a certain page is displayed, files being copied, at uninstall time, etc.). The code is not compiled in the installer, but it can be called externally and the result used in the installer. If you are more specific about which kind of code that you need I can provide specific sample code. We have a lot of built-in actions for functionality that needs to be manually added to other installers, so you may not need to write that code in the first place!
• We support all versions of Windows, including 64bit and Windows 2008 (multiple customers have certified on those platforms using our installers)
• We support bundled applications servers, including Tomcat, Resin, JBoss and others. Take a look at BitNami for examples of such stacks that we have created.
• We support copying and moving files, substituting values in config files, unpacking files and most other common installation functionality. One of our principles is that the installer should provide built-in support for common installation actions and have an easy, clean interface for invoking external code that is specific to a client's product.
• We support bundling JREs, as you mention is often enough just to bundle the JRE directory and setting the appropriate value for JAVA_HOME in the launching scripts

You can take a look at some of our customers such as MySQL, Samsung, Motorola, GitHub, etc. InstallBuilder is a commercial product, but we offer free licenses for open source projects and discounts for small business and microISVs.

I can vouch for izPack. It's Ant-based so you can package up an Ant install in it, and that Ant install can contain custom tasks, and/or scripts written within your Ant build.xml (e.g. using Groovy, Beanshell or any BSF-enabled scripting language). So it'll cover your customisable requirement.

I've used it in the past for all the above - unzipping/services/custom code. It's multi-platform, although I can't vouch for that specific instance above (64bit Windows). But if it runs Java, you should be fine.

The one headache is the JRE. You'd obviously require some JRE to run this. However I think you should be able to pack up and execute a specific JRE installation within izPack.