Selenium vs. SimpleTest vs. WatiN

There is a major difference between SimpleTest's web-tester and the Selenium suite :

• Selenium works with a real browser :
  • Each time a test is launched, a real Firefox, or a real Internet Explorer, is launched
  • All the browsing is done in that real browser
  • Which means you get all the Javascript features from the browser -- i.e. you can test fully-dynamic webpages.
• SimpleTest's web-tester simulates a browser :
  • An HTTP request is sent, the HTML content is fetched
  • And that HTML content is parsed
  • There is some level of features to test cookies and forms ; but nothing about Javascript

A couple of consequences and thoughs :

• Selenium tests take a lot of time to execute : launching a browser, and surfing (which includes downloading the JS/CSS/images, display everything including the ads, ...), is long and slow
• SimpleTest tests should be faster : only send the HTTP request, parse the HTML, and that's it
• Selenium requires a machine with a graphical interface, to launch the browser, which is a graphical software ; which also means it's harder to install/configure
• Selenium allows you to test rich/dynamic/javascript-based applications much better that only fetching and parsing HTML
• If you are testing a PHP application, PHPUnit is generally considered much better (it's activelly maintained and developped, at least) that SimpleTest ; and PHPUnit has support for Selenium tests

I would recommend a combinaison of both, if you can :

• Test HTML-related stuff with SimpleTest (Or Zend_Test, if using Zend Framework) :
  • it'll be faster
  • those tests will not depend on a browser
• Test dynamic pages with Selenium

i.e. use the best of both tools ;-)

Since SimpleTest only deals with the HTML content of a page you can't test pages with it that rely on JavaScript behavior. At the end of the day it's a speed and functionality tradeoff.

• If you need JavaScript functionality or want to test browser-specific behavior, use Selenium.
• If you want speed and have static HTML pages, use SimpleTest.

BTW, Selenium can be integrated into a PHPUnit test suite: http://www.phpunit.de/manual/3.1/en/selenium.html

Just to add another option, TestPlan works with both the Selenium back-end and HTMLUnit, so it can be used with our without a browser. The scripting language is simple and allows fast creation of automation tasks.

The browserless backend supports JavaScript very well, but for those cases where it just doesn't work you just switch to the Selenium mode and use a real browser.