Flask: Background thread sees a non-empty queue as empty
From the Things to Know documenation page:
uWSGI tries to (ab)use the Copy On Write semantics of the
fork()
call whenever possible. By default it will fork after having loaded your applications to share as much of their memory as possible. If this behavior is undesirable for some reason, use thelazy
option. This will instruct uWSGI to load the applications after each worker’s fork(). Lazy mode changes the way graceful reloading works: instead of reloading the whole instance, each worker is reloaded in chain. If you want “lazy app loading”, but want to maintain the standard uWSGI reloading behaviour, starting from 1.3 you can use thelazy-apps
option.
Your Flask app is started when uWSGI starts, then the one worker process is forked. On forking, the Queue
object is empty, and no longer shared with the original process. The thread isn't taken along.
Try setting the lazy-apps
option to delay the loading of the Flask app until the worker is started.
The documentation link in @Martijn Pieters' answer notes that lazy-apps
may consume more memory than preforking. If you're concerned about this, you may also wish to consider the @postfork
decorator to have more granular control over what gets run after forking. You could create your Queue inside a @postfork
-decorated function and it will get created in each worker.