Identifying "sensitive" code in your application

You problem is similar to the one I answered over at SQA https://sqa.stackexchange.com/a/3082. This problem was associated with Java which made the tooling a bit easier, but I have a number of suggestions below.

A number of other answers suggest that there is no good runtime tools for Python. I disagree on this in several ways:

1. Coverage tools work very well
2. Based on my experience in tooling in Java, static and dynamic analysis tools in Python are weaker than in a strongly typed less dynamic language but will work more than well enough to give good heuristics for you here. Unless you use an unusually large pathological number of dynamic features (including adding and removing methods, intercepting method and property invocations, playing with import, manually modifying the namespace) - in which case any problems you have may well be associated with this dynamism...
3. Pylint picks up simpler problems, and will not detect problems with dynamic class/instance modifications and decorators - so it doesn't matter that the metric tools don't measure these
4. In any case, where you can usefully focus can be determined by much more than a dependency graph.

Heuristics for selecting code

I find that there are a number of different considerations for selecting code for improvement which work both individually and together. Remember that, at first, all you need to do is find a productive seam of work - you don't need to find the absolutely worst code before you start.

Use your judgement.

After a few cycles through the codebase, you will have a huge amount of information and be much better positioned to continue your work - if indeed more needs to be done.

That said, here are my suggestions:

High value to the business: For example any code that could cost your company a lot of money. Many of these may be obvious or widely known (because they are important), or they may be detected by running the important use cases on a system with the run-time profiler enabled. I use Coverage.

Static code metrics: There are a lot of metrics, but the ones that concern us are:

• High afferent couplings. This is code that a lot of other files depends on. While I don't have a tool that directly outputs this, snakefood is a good way to dump the dependencies directly to file, one line per dependency, each being a tuple of afferent and efferent file. I hate to say it, but computing the afferent coupling value from this file is a simple exercise left to the reader.
• High McCabe (cyclomatic) complexity: This is more complex code. PyMetrics seems to produce this measure although I have not used the tool.
• Size: You can get a surprising amount of information by viewing the size of your project using a visualiser (eg https://superuser.com/questions/8248/how-can-i-visualize-the-file-system-usage-on-windows or https://superuser.com/questions/86194/good-program-to-visualize-file-system-usage-on-mac?lq=1. Linux has KDirStat at Filelight). Large files are a good place to start as fixing one file fixes many warnings.

Note that these tools are file-based. This is probably fine-enough resolution since you mention the project is itself has hundreds of modules (files).

Changes frequently: Code that changes frequently is highly suspect. The code may:

• Historically have had many defects, and empirically may continue to do so
• Be undergoing changes from feature development (high number of revisions in your VCS)

Find areas of change using a VCS visualisation tool such as those discussed later in this answer.

Uncovered code: Code not covered by tests.

If you run (or can run) your unit tests, your other automated tests and typical user tests with coverage, take a look at the packages and files with next to no coverage. There are two logical reasons why there is no coverage:

• The code is needed (and important) but not tested at all (at least automatically). These areas are extremely high risk
• The code may be unused and is a candidate for removal.

Ask other developers

You may be surprised at the 'smell' metrics you can gather by having a coffee with the longer-serving developers. I bet they will be very happy if someone cleans up a dirty area of the codebase where only the bravest souls will venture.

Visibility - detecting changes over time

I am assuming that your environment has a DVCS (such as Git or Mercurial) or at least a VCS (eg SVN). I hope that you are also using an issue or bug tracker of some kind. If so, there is a huge amount of information available. It's even better if developers have reliably checked in with comments and issue numbers. But how do you visualise it and use it?

While you can tackle the problem on a single desktop, it is probably a good idea to set up a Continuous Integration (CI) environment, perhaps using a tool like Jenkins. To keep the answer short, I will assume Jenkins from now on. Jenkins comes with a large number of plugins that really help with code analysis. I use:

• py.test with JUnit test output picked up by the JUnit test report Jenkins plugin
• Coverage with the Cobertura plugin
• SLOCCount and SLOCCount plugin
• Pylint and Violations plugin
• Apparently there is a plugin for McCabe (cyclometric) complexity for Python, although I have not used it. It certainly looks interesting.

This gives me visibility of changes over time, and I can drill in from there. For example, suppose PyLint violations start increasing in a module - I have evidence of the increase, and I know the package or file in which this is occurring, so I can find out who's involved and go speak with them.

If you need historic data and you have just installed Jenkins, see if you can run a few manual builds that start at the beginning of the project and take a series of jumps forward in time until the present. You can choose milestone release tags (or dates) from the VCS.

Another important area, as mentioned above, is detecting the loci of changes in the code base. I have really liked Atlassian Fisheye for this. Besides being really good at searching for commit messages (eg bug id) or file contents at any point in time, it allows me to easily see metrics:

• Linecount by directory and subdirectory
• Committers at any point in time or in specific directories and/or files
• Patterns of committal, both by time and also location in the source code

I'm afraid you are mostly on your own.

If you have decent set of tests, look at code coverage and dead code.

If you have a decent profiling setup, use that to get a glimpse of what's used more.

In the end, it seems you are more interested in fan-in/fan-out analysis, I'm not aware of any good tools for Python, primarily because static analysis is horribly unreliable against a dynamic language, and so far I didn't see any statistical analysis tools.

I reckon that this information is sort of available in JIT compilers -- whatever (function, argument types) is in cache (compiled) those are used the most. Whether or not you can get this data out of e.g. PyPy I really don't have a clue.

Source control tools can give a good indication of frequently updated modules - often indicating trouble spots.

If you don't have source control but the project is run from a shared location delete all the pycache folders or .pyc files. Over time/under use watch which files get recreated to indicate their use.

Analysing the Python imports printed when running from particular entry points with

python -v entry_point

may give some insight into which modules are being used. Although if you have known access points you should try the coverage module.

For a more intrusive solution, consider setting up project wide logging. You can log metrics easy enough, even over distributed programs.