Automatically detect test coupling in Protractor (randomizing test execution order)

You can run tests randomly (at the file level) by setting the random property in your config . You can also set your salt/seed so it's reproducibly random.

/** * If true, run specs in semi-random order */random?: boolean,/** * Set the randomization seed if randomization is turned on */seed?: string,

You could also turn on shardTestFiles (parallel test runs), which should also be very telling in how coupled your tests are.

Did you try shuffling "it" blocks like below:

var shuffle = function (items) {  var item, randomIndex;        for(var i = 0; i < items.length; i++){    randomIndex= (Math.random() * items.length) | 0;    item = items[i];    items[i] = items[randomIndex];    items[randomIndex] = item;  }}describe('Suite', function() {  it("should a", function () {      console.log("execute a");  });  it("should b", function () {      console.log("execute b");  });  it("should c", function () {      console.log("execute c");  });  shuffle(this.children);    // shuffle the 'it' blocks});

Source: Can protractor tests be run in a random order?

One problem is you likely have no idea how tests might be coupled. If one test referenced some variables from another test, you might be able to find those automatically but that's only one way tests might be coupled and probably not a likely scenario.

My first thought was to just run them individually and see which ones fail. The problem is that if you aren't cleaning state between tests, you might change the order (randomizing them, as you suggested) but if test 50 expects the data that test 20 set up but in the new order, test 20 still runs before test 50... test 50 will still pass. You will find some but probably not all until you run all of them in a random order several times.

You don't describe your application but my second thought was if there was a way to get back to a clean slate between tests, you should be able to find the tests that rely on other tests to set up data. I'm kinda surprised you aren't doing that already but if there's a long setup process that has to run to install a clean slate, etc. that might be an issue. Depending on your system, you might be able to snapshot a VM after a clean install and restore it to "quickly" get back to clean or you may be able to roll back SQL tables, etc. It really depends on your system and without more details on the system, it's hard to offer advice.

Another option is to go to those that wrote or maintain the tests and have them self-identify the tests they own that are coupled and fix them. This likely won't find them all but it might be a semi-quick start.

Oh... I just thought of one more thing. If you could reverse the order of test execution, that should be better than randomizing the execution. With reverse order, NO test would run after it's former predecessor and you should be able to find them all in one go.