Python/Django polling of database has memory leak

You need to regularly reset a list of queries that Django keeps for debugging purposes. Normally it is cleared after every request, but since your application is not request based, you need to do this manually:

from django import dbdb.reset_queries()

See also:

• "Debugging Django memory leak with TrackRefs and Guppy" by MikkoOhtamaa:

  Django keeps track of all queries for debugging purposes (connection.queries). This list is reseted at the end of HTTP request. But in standalone mode, there are no requests. So you need to manually reset to queries list after each working cycle

• "Why is Django leaking memory?" in Django FAQ - it talks bothabout setting DEBUG to False, which is always important, andabout clearing the list of queries using db.reset_queries(),important in applications like yours.

Does the settings.py file for the daemon process have DEBUG = True? If so, Django keeps in memory a record of all the SQL it has run so far, which can lead to a memory leak.

Apart from db.reset_queries() and DEBUG = False tricks, here is another approach:Just spawn another process that performs the django query and feeds the queue. This process will work in its own memory context, and after performing your task it will release back your memory.

I believe that sometimes (if not always) it's inevitable to control memory issues with a long running process that performs heavy django transactions.