Docker bundle install cache issues when updating gems

I found two possible solutions that use external data volume for gem storage: one and two.

Briefly,

• you specify an image that is used to store gems only
• in your app images, in docker-compose.yml you specify the mount point for BUNDLE_PATH via volumes_from.
• when your app container starts up, it executes bundle check || bundle install and things are good to go.

This is one possible solution, however to me it feels like it goes slightly against the docker way. Specifically, bundle install to me sounds like it should be part of the build process and shouldn't be part of the runtime. Other things, that depend on the bundle install like asset:precompile are now a runtime task as well.

This is a vaiable solution but I'm looking forward to something a little more robust.

I cache the gems to a tar file in the application tmp directory. Then I copy the gems into a layer using the ADD command before doing the bundle install. From my Dockerfile.yml:

WORKDIR /home/app# restore the gem cache. This only runs when# gemcache.tar.bz2 changes, so usually it takes# no timeADD tmp/gemcache.tar.bz2 /var/lib/gems/COPY Gemfile /home/app/GemfileCOPY Gemfile.lock /home/app/Gemfile.lockRUN gem update --system && \gem update bundler && \bundle install --jobs 4 --retry 5

Be sure you are sending the gem cache to your docker machine. My gemcache is 118MB, but since I am building locally it copies fast. My .dockerignore:

tmp!tmp/gemcache.tar.bz2

You need to cache the gems from a built image, but initially you may not have an image. Create an empty cache like so (I have this in a rake task):

task :clear_cache do  sh "tar -jcf tmp/gemcache.tar.bz2 -T /dev/null"end

After the image is built copy the gems to the gem cache. My image is tagged app. I create a docker container from the image, copy /var/lib/gems/2.2.0 into my gemcache using the docker cp command, and then delete the container. Here's my rake task:

task :cache_gems do  id = `docker create app`.strip  begin    sh "docker cp #{id}:/var/lib/gems/2.2.0/ - | bzip2 > tmp/gemcache.tar.bz2"  ensure    sh "docker rm -v #{id}"  endend

On the subsequent image build the gemcache is copied to a layer before the bundle install is called. This takes some time, but it is faster than a bundle install from scratch.

Builds after that are even faster because the docker has cached the ADD tmp/gemcache.tar.bz2 /var/lib/gems/ layer. If there are any changes to Gemfile.lock only those changes are built.

There is no reason to rebuild the gem cache on each Gemfile.lock change. Once there are enough differences between the cache and the Gemfile.lock that a bundle install is slow you can rebuild the gem cache. When I do want to rebuild the gem cache it is a simple rake cache_gems command.

The "copy local dependencies" approach (accepted answer) is a bad idea IMO. The whole point of dockerizing your environment is to have an isolated, reproducible environment.

Here's how we are doing it.

# .docker/docker-compose.dev.ymlversion: '3.7'services:  web:    build: .    command: 'bash -c "wait-for-it cache:1337 && bin/rails server"'    depends_on:      - cache    volumes:      - cache:/bundle    environment:      BUNDLE_PATH: '/bundle'  cache:    build:      context: ../      dockerfile: .docker/cache.Dockerfile    volumes:      - bundle:/bundle    environment:      BUNDLE_PATH: '/bundle'    ports:      - "1337:1337"volumes:  cache:

# .docker/cache.DockerfileFROM ruby:2.6.3RUN apt-get update -qq && apt-get install -y netcat-openbsdCOPY Gemfile* ./COPY .docker/cache-entrypoint.sh ./RUN chmod +x cache-entrypoint.shENTRYPOINT ./cache-entrypoint.sh

# .docker/cache-entrypoint.sh#!/bin/bashbundle check || bundle installnc -l -k -p 1337

# web.dev.DockerfileFROM ruby:2.6.3RUN apt-get update -qq && apt-get install -y nodejs wait-for-itWORKDIR ${GITHUB_WORKSPACE:-/app}# Note: bundle install step removedCOPY . ./

This is similar to the concept explained by @EightyEight but it doesn't put bundle install into the main service's startup, instead, the update is managed by a different service. Either way, don't use this approach in production. Running services without their dependencies being installed in the build step will at the very least cause more downtime than necessary.