How do I run another script in Python without waiting for it to finish? [duplicate]
p = subprocess.Popen([sys.executable, '/path/to/script.py'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)That will start the subprocess in background. Your script will keep running normally.
Read the documentation here.
Running this through a message queue is definitely the way to go if you're thinking about long-term scaling. Send a message to the queue who's running constantly in the background, and write job handlers to deal with the different sorts of messages.
Since you're using Django, I think Beanstalkd is a pretty good fit. Here's a pretty nice tutorial on the subject. The first comment in that article also has some good tips.
Personally I've rolled with a custom in-memory queue server written in Erlang, with Python-bindings written in C. But redis looks like it might work out as a great contender for future queuing/messaging-needs. Hope this helps!