Docker Stack tasks Rejected because of Permission denied error

(Note: only relevant to users who may have installed docker via snap)

I experienced this issue because I had snap installed docker, which meant that docker-compose was confused about my docker installation. The /var/lib/docker was giving file permission errors because my system was actually using docker at /snap/bin/docker

To check whether this is the case for you, you can run which docker. If it returns /usr/bin/docker, you've done the normal app install. If it returns something at the path /snap/ that means you've installed from the snap docker package (which is no longer maintained. Ref: https://github.com/docker/docker-snap)

To fix the issue, I first uninstalled the snap docker install using snap remove docker and then installed it through the docker setup instructions at https://docs.docker.com/install/linux/docker-ce/ubuntu/#docker-ee-customers. Once I did that, I could deploy stacks without issue.

I can't remember where the idea came into my head to snap install docker instead, but installing by setting up repositories and then running sudo apt install docker.io seems to be the way to go.

I was troubleshooting this issue for a few hours earlier where the containers would display as being rejected and shutdown due to the same Permission Denied error. I also had the same context of being able to manually create+scale a service but not being able to achieve the same results with my .yml.

Following the getting started guide, in their docker-compose.yml example they use version: "3" but do not go on to say that's supposed to match your corresponding Compose File Format number. My number ended up being "3.7" based on the version of Docker I installed and after I changed it in docker-compose.yml accordingly my containers were able to replicate as intended via docker stack deploy -c docker-compose.yml some-name.

Compose File Format (CFF) : Docker Engine Release (DCR)

CFF     DCR3.7     18.06.0+3.6     18.02.0+3.5     17.12.0+3.4     17.09.0+3.3     17.06.0+3.2     17.04.0+3.1     1.13.1+3.0     1.13.0+2.4     17.12.0+2.3     17.06.0+2.2     1.13.0+2.1     1.12.0+2.0     1.10.0+1.0     1.9.1.+

Reference doc available (here)

Example of Invalid to Valid Change

# From the Exampleversion: "3"services:  web:    image: username/repo:tag# What's actually required (in my case, because I'm using docker version 18.09.2)version: "3.7"services:  web:    image: username/repo:tag

Results After Change

owner@test_server:~/myyml$ docker service lsID                  NAME                MODE                REPLICAS            IMAGE                               PORTSbaxip2g6xgzy        first-swarm_web     replicated          5/5                 [username]/getting-started:my-serv   *:4000->80/tcpowner@test_server:~/myyml$ docker service ps first-swarm_webID                  NAME                IMAGE                               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTSss6jn07htkhp        first-swarm_web.1   [username]/getting-started:my-serv   test_server         Running             Running 37 minutes ago                       n5eael5ae3jl        first-swarm_web.2   [username]/getting-started:my-serv   test_server         Running             Running 37 minutes ago                       kzb3l1hzdpet        first-swarm_web.3   [username]/getting-started:my-serv   test_server         Running             Running 37 minutes ago                       6ijgoebvhhdg        first-swarm_web.4   [username]/getting-started:my-serv   test_server         Running             Running 37 minutes ago                       h0mbz1fp1id6        first-swarm_web.5   [username]/getting-started:my-serv   test_server         Running             Running 37 minutes ago