Using a pip cache directory in docker builds

I suggest you to use buildkit, also see this.

Dockerfile:

# syntax = docker/dockerfile:experimentalFROM python:3.6-alpineRUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml

NOTE: # syntax = docker/dockerfile:experimental is a must，you have to add it at the beginning of Dockerfile to enable this feature.

1.

The first execute build:

export DOCKER_BUILDKIT=1docker build --progress=plain -t abc:1 . --no-cache

The first log:

#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...#9   digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"#9  started: 2019-09-20 03:11:35.296107357 +0000 UTC#9 1.955 Collecting pyyaml#9 3.050   Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)#9 5.006 Building wheels for collected packages: pyyaml#9 5.007   Building wheel for pyyaml (setup.py): started#9 5.249   Building wheel for pyyaml (setup.py): finished with status 'done'#9 5.250   Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=867daf35eab43c2d047ad737ea1e9eaeb4168b87501cd4d62c533f671208acaa#9 5.250   Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030#9 5.267 Successfully built pyyaml#9 5.274 Installing collected packages: pyyaml#9 5.309 Successfully installed pyyaml-5.1.2#9completed: 2019-09-20 03:11:42.221146294 +0000 UTC#9 duration: 6.925038937s

From above, you can see the first time, the build will download pyyaml from internet.

2.

The second execute build:

docker build --progress=plain -t abc:1 . --no-cache

The second log:

#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...#9   digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"#9  started: 2019-09-20 03:16:58.588157354 +0000 UTC#9 1.786 Collecting pyyaml#9 2.234 Installing collected packages: pyyaml#9 2.270 Successfully installed pyyaml-5.1.2#9completed: 2019-09-20 03:17:01.933398002 +0000 UTC#9 duration: 3.345240648s

From above, you can see the build no longer download package from internet, just use the cache. NOTE, this is not the traditional docker build cache as I have use --no-cache, it's /root/.cache/pip which I mount into build.

3.

The third execute build which delete buildkit cache:

docker builder prunedocker build --progress=plain -t abc:1 . --no-cache

The third log:

#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...#9   digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"#9  started: 2019-09-20 03:19:07.434792944 +0000 UTC#9 1.894 Collecting pyyaml#9 2.740   Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)#9 3.319 Building wheels for collected packages: pyyaml#9 3.319   Building wheel for pyyaml (setup.py): started#9 3.560   Building wheel for pyyaml (setup.py): finished with status 'done'#9 3.560   Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=cea5bc4689e231df7915c2fc3abca225d4ee2e869a7540682aacb6d42eb17053#9 3.560   Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030#9 3.580 Successfully built pyyaml#9 3.585 Installing collected packages: pyyaml#9 3.622 Successfully installed pyyaml-5.1.2#9completed: 2019-09-20 03:19:12.530742712 +0000 UTC#9 duration: 5.095949768s

From above, you can see if delete buildkit cache, the package download again.

In a word, it will give you a shared cache between several times build, and this cache will only be mounted when image build. But, the image self will not have these cache, so avoid a lots of intermediate layer in image.

EDIT for folks who are using docker compose and are lazy to read the comments...:

You can also do this with docker-compose if you setCOMPOSE_DOCKER_CLI_BUILD=1. For example: COMPOSE_DOCKER_CLI_BUILD=1DOCKER_BUILDKIT=1 docker-compose build –

UPDATE according to folk's question 2020/09/02:

I don't know from which version (my version now is 19.03.11), if not specify mode for cache directory, the cache won't be reused by next time build.

Don't know the detail reason, but you could add mode=0755, to Dockerfile to make it work again:

Dockerfile:

# syntax = docker/dockerfile:experimentalFROM python:3.6-alpineRUN --mount=type=cache,mode=0755,target=/root/.cache/pip pip install pyyaml