How to pass a video file to worker function in Heroku (Flask) How to pass a video file to worker function in Heroku (Flask) flask flask

How to pass a video file to worker function in Heroku (Flask)


Heroku dynos are completely isolated containers, that is why they cannot share file system as you want. But if you'll host them on another hosting like DigitalOcean or Amazon, you'll be able to access files you stored by Flask from the other workers instantly (or almost instantly - don't forget to create a copy of temp-file, Flask or WSGI should delete it after response sent).

Another option is to find the fastest way of "transporting" video-data (not always file) to a worker. You can do it using:

  • queue - put whole file to a queue. Method is not recommended, but still ok, if video files are really small.
  • in-memory database - save file to some in-memory database (e.g. Redis). They have a lot of mechanisms to quickly transport data between servers or processes (cons: expensive on Heroku)
  • database - save file to a general purpose database (e.g. Postgresql), that will do the same as Redis, but it is able to work with bigger data cheaper, though a bit slower.
  • WebSockets or even Unix-socket - you can have one worker to which Flask can connect to, send file and return http-response. That "listener" will actually start task. It can either save video to a file and provide path to еру next worker (but it should always be on the same dyno as the rest workers are) or provide data directly using args, "fork", threading, subprocessing, etc.
    You can start with Flask-SocketIO.
    But remember, that you need to configure server-to-server connection - between webapp and worker that should list it in loop or separate thread. Not Javascript in browser, which potentially is also option - start task and upload file directly to worker.
    P.S. there are no 30sec. timeouts on Heroku for WebSockets :)