Proper way to automatically test performance in Python (for all developers)?

Check out funkload - it's a way of running your unit tests as either functional or load tests to gauge how well your site is performing.

Another interesting project which can be used in conjunction with funkload is codespeed. This is an internal dashboard that measures the "speed" of your codebase for every commit you make to your code, presenting graphs with trends over time. This assumes you have a number of automatic benchmarks you can run - but it could be a useful way to have an authoritative account of performance over time. The best use of codespeed I've seen so far is the speed.pypy.org site.

As to your requirement for determinism - perhaps the best approach to that is to use statistics to your advantage? Automatically run the test N times, produce the min, max, average and standard deviation of all your runs? Check out this article on benchmarking for some pointers on this.

I want the test to be DETERMINISTIC - regardless of what is happening on the machine running the tests, I want multiple runs of the test to return the same results.

Fail. More or less by definition this is utterly impossible in a multi-processing system with multiple users.

Either rethink this requirement or find a new environment in which to run tests that doesn't involve any of the modern multi-processing operating systems.

Further, your running web application is not deterministic, so imposing some kind of "deterministic" performance testing doesn't help much.

When we did time-critical processing (in radar, where "real time" actually meant real time) we did not attempt deterministic testing. We did code inspections and ran simple performance tests that involved simple averages and maximums.

Use cProfile to instrument the interpreter to ignore "outside noise". I'm not sure I know how to read the pstats structure yet, but I'm sure it is doable.

The Stats object created by the profiler is what you're looking for.

http://docs.python.org/library/profile.html#the-stats-class

Focus on 'pcalls', primitive call count, in the profile statistics and you'll have something that's approximately deterministic.