celery-beat KeyError: 'scheduler'

This is weird, I haven't got the solution right now, but I found a way to circumnavigate this.

Why we are getting the issue :

Here are some thoughts on celery docs which explains what is happening here :

Beat needs to store the last run times of the tasks in a local database file (named celerybeat-schedule by default), so it needs access to write in the current directory, or alternatively you can specify a custom location for this file:

Basicaly celery is trying to read the file named celerybeat-schedule but form some reason it's failing.

Why is it failing to read it on docker?

I have no clue for now...

However this comment on give some lights

It's something related to files storage.

Here is my workaround.

I decided to use Redis as scheduler run times of my task instead of file storage and luckily I found this package which helped me to achieve that.

What you can do is this:

Update your celery app config using :

app.conf.redbeat_redis_url = your redis url

Then in your docker file you need to tell celery which scheduler it should use.

celery:    build: .    command: celery worker -l info -A canopact.blueprints.contact.tasks     env_file:      - '.env'    volumes:      - '.:/canopact'  celery_beat:    build: .    command: celery beat -l info -A canopact.blueprints.contact.tasks -S redbeat.RedBeatScheduler    env_file:    - '.env'    volumes:    - '.:/canopact'

Thank you all for pointing in the direction that this is related to the scheduler files and to container / docker setup.

It seems the issue can also be mitigated by providing a dedicated location for the scheduler file, for exmaple in a separate volume.

Example (with beat running in the same worker process).

services:    ...    worker:        ...        volumes:          - scheduler-data:/some/directory               command: celery -A proj worker -B --schedule /some/directory/celerybeat-schedule        ...volumes:  scheduler-data: