Coverage analysis for Functional Tests

There are at least two (commercial but cheap) tools that I know of that allow you to attach to the IIS process to capture coverage data for IIS applications.

NCover:

NCover includes the //iis command line switch. This switch sets up the coverage environment within IIS and restarts the web server. You’ll run NCover like this to analyze coverage for your web applications:

NCover.Console.exe nunit-console.exe TestAssembly.dll //iis When you run NCover in this way, IIS will be restarted to allow NCover to monitor your coverage, and your tests will be run. Once finished, NCover will stop IIS and detach itself.

See: http://docs.ncover.com/how-to/code-coverage-of-asp-net-applications-on-iis/

DotCover by jetbrains:

Dotcover has visual studio integration which allows you to attach to an IIS application in the same way you would do if you'd want to trace your IIS application. This probably can also be started with the commandline dotCover tool although i've never actually tried this.

See http://www.jetbrains.com/dotcover/

I think Rational and Microsoft Teamsystem also have solutions but they're a bit more expensive.

We use a system for testing that involves a person creating a text narrative - a test script - for manual testing of functionality. This is enumerated in some fashion (e.g. [functionality]-001). Then our Selenium tests are noted as covering one or more of these enumerations.

When new functionality is constructed, new test scripts are written and enumerated. Then with Selenium testing we are able to compare its list of what's automated relative to the enumerated test scripts - the delta is what must be tested manually.

Some of our Test Coverage tools (at present, Java, C# and COBOL) are designed to handle this kind of thing.

If you run your application and exercise a particular function, you can use these test coverage tools to collect code coverage data for that particular functionality. In essence, this is a record of all code which the functionality exercises. With some minor scripting, you can arrange for each functinonality test to run and obtain code coverage data for that test.

The collected test coverage vectors can be combined into a summary vector by the tool, that will give you code coverage number for your code based on the entire set of functionality tests.

If you change the code base, the test coverage tool will tell you which blocks of code have changed (it compares at the method level for differences). This in turn can be applied to the test coverage vectors already collected for individual functionalities; if there's an intersection, you need to run the functional test again, as the code on which it depended has changed.

This way you can decide which functionalities need to be re-tested after a change.