Docker build not using cache when copying Gemfile while using --cache-from

I've been scratching my head over issues with Docker build and --cache-from for the last few days, and it's a bit frustrating the lack of documentation for the proper behavior of --cache-from, while there is some misinformation in the wild.

I think I've finally managed to fix the issues I had on my side, after a few insights which I'm going to share here in the hopes it will be useful to someone else.

When providing multiple --cache-from, the order matters!

The order is very important, because at the first match, Docker will stop looking for other matches and it will use that one for all the rest of the commands.

This is explained by the fellow who implemented the feature in the Github PR:

When using multiple --cache-from they are checked for a cache hit in the order that user specified. If one of the images produces a cache hit for a command only that image is used for the rest of the build.

There is also a lenghtier explanation in the initial ticket proposal:

Specifying multiple --cache-from images is bit problematic. If both images match there is no way(without doing multiple passes) to figure out what image to use. So we pick the first one(let user control the priority) but that may not be the longest chain we could have matched in the end. If we allow matching against one image for some commands and later switch to a different image that had a longer chain we risk in leaking some information between images as we only validate history and layers for cache. Currently I left it so that if we get a match we only use this target image for rest of the commands.

Using --cache-from is exclusive: the local Docker cache won't be used

This means that it doesn't add new caching sources, the image tags you provide will be the only caching sources for the Docker build.

Even if you just built the same image locally, the next time you run docker build for it, in order to benefit from the cache, you need to either:

1. provide the correct tag with --cache-from (and with the correct precedence); or

2. not use --cache-from at all (so that it will use the local build cache)

If the parent image changes, the cache will be invalidated

For example, if you have an image based on docker:stable, and docker:stable gets updated, the cached builds of your image will not be valid anymore as the layers of the base image were changed.

This is why, if you're configuring a CI build, it can be useful to docker pull the base image as well and include it in the --cache-from, as mentioned in this comment in yet another Github discussion.

For whoever is fighting with DockerHub automated builds and --cache-from. I realized images built from DockerHub would always lead to cache bust on COPY commands when pulled and used as build cache source. It seems to be also the case for @Marcelo (refs his comment).

I investigated by creating a very simple image doing a couple of RUN commands and later COPY. Everything is using the cache except the COPY. Even though content and permissions of the file being copied is the same on both the pulled image and the one built locally (verified via sha1sum and ls -l).

The solution for me was to publish the image to the registry from the CI (Travis in my case) rather than letting DockerHub automated build doing it. Let me emphasis here that I'm talking here about a specific case where files are definitely the same and should not cache bust, but you're using DockerHub automated builds.

I'm not sure why is that, but I know for instance old docker-engine version e.g. prior 1.8.0 didn't ignore file timestamp to decide whether to use the cache or not, refs https://docs.docker.com/release-notes/docker-engine/#180-2015-08-11 and https://github.com/moby/moby/pull/12031.

I struggled with this problem, and in my case I used COPY when the checksum might have changed (but only technically, the content was functionally identical). So, I worked around this way:

Dockerfile:

ARG builder_image=base-builder# Compilation/build stageFROM golang:1.16 AS base-builderRUN echo "build the app" > /go/app# This step is required to facilitate docker cache. With the definition of a `builder_image` build tag# we can essentially skip the build stage and use a prebuilt-image directly.FROM $builder_image AS builder# myapp docker imageFROM ubuntu:20.04 AS myappCOPY --from=builder /go/app /opt/my-app/bin/

Then, I can run the following:

# build cacheDOCKER_BUILDKIT=1 docker build --target base-builder -t myapp-builder .docker push myapp-builder# use cacheDOCKER_BUILDKIT=1 docker build --target myapp --build-arg=builder_image=myapp-builder -t myapp .docker push myapp

This way we can force Docker to use a prebuilt image as a cache.