Is it safe to clean docker/overlay2/

Docker uses /var/lib/docker to store your images, containers, and local named volumes. Deleting this can result in data loss and possibly stop the engine from running. The overlay2 subdirectory specifically contains the various filesystem layers for images and containers.

To cleanup unused containers and images, see docker system prune. There are also options to remove volumes and even tagged images, but they aren't enabled by default due to the possibility of data loss:

$ docker system prune --helpUsage:  docker system prune [OPTIONS]Remove unused dataOptions:  -a, --all             Remove all unused images not just dangling ones      --filter filter   Provide filter values (e.g. 'label=<key>=<value>')  -f, --force           Do not prompt for confirmation      --volumes         Prune volumes

What a prune will never delete includes running containers, logs on those containers, and filesystem changes made by those containers. Additionally, anything created outside of the normal docker folders may not be seen by docker during this garbage collection. This could be from some other app writing to this directory, or a previous configuration of the docker engine (e.g. switching from AUFS to overlay2, or possibly after enabling user namespaces).

What would happen if this advice is ignored and you deleted a single folder like overlay2 out from this filesystem? The container filesystems are assembled from a collection of filesystem layers, and the overlay2 folder is where docker is performing some of these mounts (you'll see them in the output of mount when a container is running). Deleting some of these when they are in use would delete chunks of the filesystem out from a running container, and likely break the ability to start a new container from an impacted image.

To completely refresh docker to a clean state, stop the docker engine (systemctl stop docker) and delete the entire directory (not just the overlay2 folder) with rm -rf /var/lib/docker, and restart docker (systemctl start docker). The engine will restart without any images, containers, volumes, user created networks, or swarm state.

I found this worked best for me:

docker image prune --all

By default Docker will not remove named images, even if they are unused. This command will remove unused images.

Note each layer in an image is a folder inside the /usr/lib/docker/overlay2/ folder.

I had this issue... It was the log that was huge. Logs are here :

/var/lib/docker/containers/<container id>/<container id>-json.log

You can manage this in the run command line or in the compose file. See there : Configure logging drivers

I personally added these 3 lines to my docker-compose.yml file :

my_container:  logging:    options:      max-size: 10m