Unit tests sometimes failing, sometimes passing

Tests can fail intermittently for a number of reasons and identifying why they fail is often revealing about your codebase and environment.

Here are a few possible causes:

• Shared objects - singletons that hold state can cause problems between tests if the test environment isn't reset to a well known state.if if your test runner executes tests in a non-deterministic order you may see random errors that are actually exposing corrupted state issues

• Environmental and external dependencies - any external object that can hold state can cause unpredictable results

• Timing - sometimes tests are written with timeouts or thread sleeps that are too specific. If the build server is operating under heavy load these timeouts may not be long enough

As general guidance, tests must be:

• isolated: tests focus on one unit at a time
• repeatable: produces the same results each time
• independent: the order in which tests are executed should not matter

I would try to narrow down the problem by reducing the number of executors to 1. If the tests are still failing intermittently, then you have a Test Run War, otherwise (given they run fine locally) it looks like a Resource Leakage.

More info at http://xunitpatterns.com/Erratic%20Test.html

I would geuss you are reusing the test fixtures for multiple tests. In some cases the order of the tests is changed from the order on your local machine and you do not have a correct fixture when you start a test. To fix it, you should give each test a clean fixture.

An other cause can be that you reuse a database from multiple locations when runnen tests (two build servers with the same test configuration). Give each build server its own database. Or sub/mock your database layer. Your test should not depend on external resources except when you are testing the connections to your external resources.