How To Reduce Python Script Memory Usage
Organizing:
Your python script seems indeed to be huge, maybe you should consider reorganizing your code first, to split it into several modules or packages. It will probably make easier the code profiling and the optimization tasks.
You may want to have a look there:
And possibly:
- SO: Python: What is the common header format?
- How do you organize Python modules?
- The Hitchiker's Guide to Packaging
Optimizing:
There is a lot of things that can be done for optimizing your code ...
For instance, regarding your data structures ... If you make big use of lists or lists comprehensions, you could try to figure out where do you really need lists, and where they might be replaced by non-mutable data structures like tuples or by "volatile" objects, "lazy" containers, like generator expressions.
See:
- SO: Are tuples more efficient than lists in Python?
- SO: Generator Expressions vs. List Comprehension
- PEP 255 - Simple Generators and PEP 289 - Generator Expressions
On these pages, you could find some useful information and tips:
- http://wiki.python.org/moin/PythonSpeed
- http://wiki.python.org/moin/PythonSpeed/PerformanceTips
- http://wiki.python.org/moin/TimeComplexity
- http://scipy.org/PerformancePython
Also, you should study your ways of doing things and wonder whether there is a way to do that less greedily, a way that it's better to do it in Python (you will find some tips in the tag pythonic) ... That is especially true in Python, since, in Python, there is often one "obvious" way (and only one) to do things which are better than the others (see The Zen of Python), which is said to be pythonic. It's not especially related to the shape of your code, but also - and above all - to the performances. Unlike many languages, which promote the idea that there should be many ways to do anything, Python prefers to focus on the best way only. So obviously, there are many ways for doing something, but often, one is really better.
Now, you should also verify whether you are using the best methods for doing things because pythonicality won't arrange your algorithms for you.
But at last, it varies depending on your code and it's difficult to answer without having seen it.
And, make sure to take into account the comments made by eumiro and Amr.
This video might give you some good ideas: http://pyvideo.org/video/451/pycon-2011---quot-dude--where--39-s-my-ram--quot-
The advice on generator expressions and making use of modules is good. Premature optimization causes problems, but you should always spend a few minutes thinking about your design before sitting down to write code. Particularly if that code is meant to be reused.
Incidentally, you mention that you have a lot of data structures defined at the top of your script, which implies that they're all loaded into memory at the start. If this is a very large dataset, consider moving specific datasets to separate files, and loading it in only as needed. (using the csv module, or numpy.loadtxt(), etc)
Separate from using less memory, also look into ways to use memory more efficiently. For example, for large sets of numeric data, numpy arrays are a way of storing information that will provide better performance in your calculations. There is some slightly dated advice at http://wiki.python.org/moin/PythonSpeed/PerformanceTips