Running a test suite with over a million test cases

I don't know any specifics about Python, but you might want to try spinning up each test case as a separate process. This will allow you to recover the memory from each process as it completes, and may allow you to run your million test cases safely on one machine.

Generally, when a program needs more memory from the OS, it takes it, but it cannot give it back until it exits. This is likely why your process falls over.

Quick processes that you can spin up and then kill, will alleviate this memory problem.... and then it is a pretty short step from there to running each of these test cases through rsh on a different machine entirely.

I hope that helps a little.

OK. First, I do not program in Python, and I also have no idea what kind of computers you are using or how much memory the computer has or anything else about your situation with your computer(s). So this is just general information gleaned from writing system programs and working across multiple computers.

With that said: If you are going to run huge datasets or a huge number of programs you will have to partition up what you want to do so it will fit into the system(s) that you are using. That is the first part.

The second part is that if you want to talk across programs to other programs or systems then you need to set up some type of common memory usage. I know that PHP has such a thing built in to it so I have to assume that Python also has such a thing. You set up the common memory area and then send information back and forth between the various programs. Usually using some form of tokens to identify each program and/or system. In this way, you can set things up so once one suite of tests completes, it can tell the next set to begin and thus not take over or bog down your system.

If you are really good - you could also create some kind of controller program that starts up a test, watches it until the test ends/exits, and then start the next test. (So instead of a daisy chain way of doing this you have a master/slave way of doing this just like a client/server relationship.)

If you are using a Windows box/OS I would suggest looking at AutoIt. AutoIt can easily handle watching a program run and starting up a new task once an old task has finished. It also allows you full control over both your system as well as the remote system and was developed to help automate tasks such as the one you are trying to do. I found (by experimenting) that AutoIt can also be used with Unix/Linux boxes such as Macs and others. Although you only then have the commands you send over rather than access to the Windows features.

If you are more proficient with Python that you can do the above using it - then more power to you. Even though it has been two years since you asked this - I hope this helps you out in your endeavors. :-)