Correctly keeping docker VSTS / Azure Devops build agent clean yet cached

Probably you've already found a solution, but it might be useful for the rest of the community to have an answer here.

docker prune has a limited purpose. It was created to address the issue with cleaning up all local Docker images. (As it was mentioned by thaJeztah here)

To remove images in the more precise way it's better to divide this task into two parts:1. select/filter images to delete2. delete the list of selected images

E.g:

docker image rm $(docker image ls --filter reference=docker --quiet)docker image rm $(sudo docker image ls | grep 1.14 | awk '{print $3}')docker image ls --filter reference=docker --quiet | xargs docker image rm

It is possible to combine filters clauses to get exactly what you what:
(I'm using Kubernetes master node as an example environment)

$ docker imagesREPOSITORY                           TAG                 IMAGE ID            CREATED             SIZEk8s.gcr.io/kube-proxy                v1.14.2             5c24210246bb        3 months ago        82.1MBk8s.gcr.io/kube-apiserver            v1.14.2             5eeff402b659        3 months ago        210MBk8s.gcr.io/kube-controller-manager   v1.14.2             8be94bdae139        3 months ago        158MBk8s.gcr.io/kube-scheduler            v1.14.2             ee18f350636d        3 months ago        81.6MB  # beforequay.io/coreos/flannel               v0.11.0-amd64       ff281650a721        6 months ago        52.6MBk8s.gcr.io/coredns                   1.3.1               eb516548c180        7 months ago        40.3MB  # sincek8s.gcr.io/etcd                      3.3.10              2c4adeb21b4f        8 months ago        258MBk8s.gcr.io/pause                     3.1                 da86e6ba6ca1        20 months ago       742kB$ docker images --filter "since=eb516548c180" --filter "before=ee18f350636d" REPOSITORY               TAG                 IMAGE ID            CREATED             SIZEquay.io/coreos/flannel   v0.11.0-amd64       ff281650a721        6 months ago        52.6MB$ docker images --filter "since=eb516548c180" --filter "reference=quay.io/coreos/flannel" REPOSITORY               TAG                 IMAGE ID            CREATED             SIZEquay.io/coreos/flannel   v0.11.0-amd64       ff281650a721        6 months ago        52.6MB$ docker images --filter "since=eb516548c180" --filter "reference=quay*/*/*" REPOSITORY               TAG                 IMAGE ID            CREATED             SIZEquay.io/coreos/flannel   v0.11.0-amd64       ff281650a721        6 months ago        52.6MB$ docker images --filter "since=eb516548c180" --filter "reference=*/*/flan*" REPOSITORY               TAG                 IMAGE ID            CREATED             SIZEquay.io/coreos/flannel   v0.11.0-amd64       ff281650a721        6 months ago        52.6MB

As mentioned in the documentation, images / image ls filter is much better than docker prune filter, which supports until clause only:

The currently supported filters are:• dangling (boolean - true or false)  • label (label=<key> or label=<key>=<value>)  • before (<image-name>[:<tag>], <image id> or <image@digest>) - filter images created before given id or references• since (<image-name>[:<tag>], <image id> or <image@digest>) - filter images created since given id or references

If you need more than one filter, then pass multiple flags (e.g., --filter "foo=bar" --filter "bif=baz")

You can use other linux cli commands to filter docker images output:

grep "something"      # to include only specified imagesgrep -v "something"   # to exclude images you want to savesort [-k colN] [-r] [-g]] | head/tail -nX  # to select X oldest or newest images

Combining them and putting the result to CI/CD pipeline allows you to leave only required images in the local cache without collecting a lot of garbage on your build server.

I've copied here a good example of using that approach provided by strajansebastian in the comment:

#example of deleting all builds except last 2 for each kind of image #(the image kind is based on the Repository value.)#If you want to preserve just last build modify to tail -n+2.# delete dead containersdocker container prune -f# keep last 2 builds for each image from the repositoryfor diru in `docker images --format "{{.Repository}}" | sort | uniq`; do    for dimr in `docker images --format "{{.ID}};{{.Repository}}:{{.Tag}};'{{.CreatedAt}}'" --filter reference="$diru" | sed -r "s/\s+/~/g" | tail -n+3`; do         img_tag=`echo $dimr | cut -d";" -f2`;         docker rmi $img_tag;    done;done# clean dangling images if anydocker image prune -f