Using ccache in automated builds on Docker cloud

You should try saving and restoring your cache data from a third party service:- an online object storage like Amazon S3- a simple FTP server- an Internet available machine with ssh to make a scp

I'm assuming that your cache data is stored inside the ´~/.ccache´ directory

Using Docker multistage build

From some time, Docker supports Multi-stage builds and you can try using it to implement the solution with a single Dockerfile:

Warning: I've not tested it

# STAGE 1 - YOUR ORIGINAL DOCKER FILE CUSTOMIZED# CACHE_TAG is provided by Docker cloud# see https://docs.docker.com/docker-cloud/builds/advanced/# using ARG in FROM requires min v17.05.0-ceARG  CACHE_TAG=latestFROM  qgis/qgis3-build-deps:${CACHE_TAG} as builderMAINTAINER Denis Rouzaud <denis.rouzaud@gmail.com>ENV CC=/usr/lib/ccache/clangENV CXX=/usr/lib/ccache/clang++ENV QT_SELECT=5COPY  . /usr/src/QGISWORKDIR /usr/src/QGIS/build# restore cacheRUN curl -o ccache.tar.bz2 http://my-object-storage/ccache.tar.bz2RUN tar -xjvf ccache.tar.bz2COPY --from=downloader /.ccache ~/.ccacheRUN cmake \ -GNinja \ -DCMAKE_INSTALL_PREFIX=/usr \ -DBINDINGS_GLOBAL_INSTALL=ON \ -DWITH_STAGED_PLUGINS=ON \ -DWITH_GRASS=ON \ -DSUPPRESS_QT_WARNINGS=ON \ -DENABLE_TESTS=OFF \ -DWITH_QSPATIALITE=ON \ -DWITH_QWTPOLAR=OFF \ -DWITH_APIDOC=OFF \ -DWITH_ASTYLE=OFF \ -DWITH_DESKTOP=ON \ -DWITH_BINDINGS=ON \ -DDISABLE_DEPRECATED=ON \ .. \ && ninja install# save the current cache onlineWORKDIR ~/RUN tar -cvjSf ccache.tar.bz2 .ccacheRUN curl -T ccache.tar.bz2 -X PUT http://my-object-storage/ccache.tar.bz2# STAGE 2FROM alpine:latest# YOUR CUSTOM LOGIC TO CREATE THE FINAL IMAGE WITH ONLY REQUIRED BINARIES# USE THE FROM IMAGE YOU NEED, this is only an example# E.g.:# COPY --from=builder /usr/src/QGIS/build/YOUR_EXECUTABLE /usr/bin# ...

In the stage 2 you will build the final image that will be pushed to your repository.

 Using Docker cloud hooks

Another, but less clear, approach could be using a Docker Cloud pre_build hook file to download cache data:

#!/bin/bashecho "=> Downloading build cache data"curl -o ccache.tar.bz2 http://my-object-storage/ccache.tar.bz2 # e.g. Amazon S3 like servicecd /tar -xjvf ccache.tar.bz2

Obviously you can use dedicate docker images to run curl or tar mounting the local directory as a volume in this script.

Then, copy the .ccache extracted folder inside your container during the build, using a COPY command before your cmake call:

WORKDIR /usr/src/QGIS/buildCOPY /.ccache ~/.ccache    RUN cmake ...

In order to make this you should find a way to upload your cache data after the build and you could make this easily using a post_build hook file:

#!/bin/bashecho "=> Uploading build cache data"tar -cvjSf ccache.tar.bz2 ~/.ccachecurl -T ccache.tar.bz2 -X PUT http://my-object-storage/ccache.tar.bz2

But your compilation data aren't available from the outside, because they live inside the container. So you should upload the cache after the cmake command inside your main Dockerfile:

RUN cmake...  && tar ...  && curl ...  && ninja ...  && rm ...

If curl or tar aren't available, just add them to your container using the package manager (qgis/qgis3-build-deps is based on Ubuntu 16.04, so they should be available).