Celery workers unable to connect to redis on docker instances
When you use docker-compose, you aren't going to be using localhost
for inter-container communication, you would be using the compose-assigned hostname of the container. In this case, the hostname of your redis container is redis
. The top level elements under services:
are your default host names.
So for celery to connect to redis, you should try redis://redis:6379/0
. Since the protocol and the service name are the same, I'll elaborate a little more: if you named your redis service "butter-pecan-redis" in your docker-compose, you would instead use redis://butter-pecan-redis:6379/0
.
Also, docker-compose.dev.yml doesn't appear to have celery and redis on a common network, which might cause them not to be able to see each other. I believe they need to share at least one network in common to be able to resolve their respective host names.
Networking in docker-compose has an example in the first handful of paragraphs, with a docker-compose.yml to look at.
You may need to add the link and depends_on sections to your docker compose file, and then reference the containers by their hostname.
Updated docker-compose.yml:
version: '2.1'services: db: image: postgres memcached: image: memcached redis: image: redis ports: - '6379:6379' backend-base: build: context: . dockerfile: backend/Dockerfile-base image: "/backend:base" backend: build: context: . dockerfile: backend/Dockerfile image: "/backend:${ENV:-local}" command: ./wait-for-it.sh db:5432 -- gunicorn backend.wsgi:application -b 0.0.0.0:8000 -k gevent -w 3 ports: - 8000 links: - db - redis - memcached depends_on: - db - redis - memcached celery: image: "/backend:${ENV:-local}" command: ./wait-for-it.sh db:5432 -- celery worker -E -B --loglevel=INFO --concurrency=1 environment: C_FORCE_ROOT: "yes" links: - db - redis - memcached depends_on: - db - redis - memcached frontend-base: build: context: . dockerfile: frontend/Dockerfile-base args: NPM_REGISTRY: http://.view.build PACKAGE_INSTALLER: yarn image: "/frontend:base" links: - db - redis - memcached depends_on: - db - redis - memcached frontend: build: context: . dockerfile: frontend/Dockerfile image: "/frontend:${ENV:-local}" command: 'bash -c ''gulp''' working_dir: /app/user environment: PORT: 3000 links: - db - redis - memcached depends_on: - db - redis - memcached
Then configure the urls to redis, postgres, memcached, etc. with:
redis://redis:6379/0
postgres://user:pass@db:5432/database
The issue for me was that all of the containers, including celery had a network argument specified. If this is the case the redis container must also have the same argument otherwise you will get this error. See below, the fix was adding 'networks':
redis: image: redis:alpine ports: - '6379:6379' networks: - server