docker-compose volume on node_modules but is empty

TL;DR Working example, clone and try: https://github.com/xbx/base-server

You need a node_modules in your computer (outside image) for debugging purposes first (before run the container).

If you want debug only node_modules:

volumes:    - /path/to/node_modules:/usr/src/app/node_modules

If you want debug both your code and the node_modules:

volumes:    - .:/usr/src/app/

Remember that you will need run npm install at least one time outside the container (or copy the node_modules directory that the docker build generates). Let me now if you need more details.

Edit. So, without the need of npm in OSX, you can:

1. docker build and then docker cp <container-id>:/path/to/node-modules ./local-node-modules/. Then in your docker-compose.yml mount those files and troubleshot whatever you want.
2. Or, docker build and there (Dockerfile) do the npm install in another directory. Then in your command (CMD or docker-compose command) do the copy (cp) to the right directory, but this directory is mounted empty from your computer (a volume in the docker-compose.yml) and then troubleshot whatever you want.

Edit 2. (Option 2) Working example, clone and try: https://github.com/xbx/base-serverI did it all automatically in this repo forked from the yours.

Dockerfile

FROM node:6.3# Install app dependenciesRUN mkdir /build-dirWORKDIR /build-dirCOPY package.json /build-dirRUN npm install -g babel babel-runtime babel-register mocha nodemonRUN npm install# Create app directoryRUN mkdir -p /usr/src/appWORKDIR /usr/src/appRUN ln -s /build-dir/node_modules node_modules# Bundle app sourceCOPY . /usr/src/appEXPOSE 1234CMD [ "npm", "start" ]

docker-compose.yml

web:  build: .  ports:    - "1234:1234"  links:    - db # liaison avec la DB  environment:    PORT: 1234  command: /command.sh  volumes:    - ./src/:/usr/src/app/src/    - ./node_modules:/usr/src/app/node_modules    - ./command.sh:/command.shdb:  image: mongo:3.3  ports:    - "27017:27017"  command: "--smallfiles --logpath=/dev/null"

command.sh

#!/bin/bashcp -r /build-dir/node_modules/ /usr/src/app/exec npm start

Please, clone my repo and do docker-compose up. It does what you want.PS: It can be improved to do the same in a better way (ie best practices, etc)

I'm in OSX and it works for me.

First, there's an order of operations. When you build your image, volumes are not mounted, they only get mounted when you run the container. So when you are finished with the build, all the changes will only exist inside the image, not in any volume. If you mount a volume on a directory, it overlays whatever was from the image at that location, hiding those contents from view (with one initialization exception, see below).

Next is the volume syntax:

  volumes:    - .:/usr/src/app    - /usr/src/app/node_modules

tells docker-compose to create a host volume from the current directory to /usr/src/app inside the container, and then to map /usr/src/app/node_modules to an anonymous volume maintained by docker. The latter will appear as a volume in docker volume ls with a long uuid string that is relatively useless.

To map /usr/src/app/node_modules to a folder on your host, you'll need to include a folder name and colon in front of that like you have on the line above. E.g. /host/dir/node_modules:/usr/src/app/node_modules.

Named volumes are a bit different than host volumes in that docker maintains them with a name you can see in docker volume ls. You reference these volumes with just a name instead of a path. So node_modules:/usr/src/app/node_modules would create a volume called node_modules that you can mount in a container with just that name.

I diverged to describe named volumes because they come with a feature that turns into a gotcha with host volumes. Docker helps you out with named volumes by initializing them with the contents of the image at that location. So in the above example, if the named volume node_modules is empty (or new), it will first copy the contents of the image at /usr/src/app/node_modules` to this volume and then mount it inside your container.

With host volumes, you will never see any initialization, whatever is at that location, even an empty directory, is all you see in the container. There's no way to get contents from the image at that directory location to first copy out to the host volume at that location. This also means that directory permissions needed inside the container are not inherited automatically, you need to manually set the permissions on the host directory that will work inside the container.

Finally, there's a small gotcha with docker for windows and mac, they run inside a VM, and your host volumes are mounted to the VM. To get the volume mounted to the host, you have to configure the application to share the folder in your host to the VM, and then mount the volume in the VM into the container. By default, on Mac, the /Users folder is included, but if you use other directories, e.g. a /Projects directory, or even a lower case /users (unix and bsd are case sensitive), you won't see the contents from your Mac inside the container.

With that base knowledge covered, one possible solution is to redesign your workflow to get the directory contents from the image copied out to the host. First you need to copy the files to a different location inside your image. Then you need to copy the files from that saved image location to the volume mount location on container startup. When you do the latter, you should note that you are defeating the purpose of having a volume (persistence) and may want to consider adding some logic to be more selective about when you run the copy. To start, add an entrypoint.sh to your build that looks like:

#!/bin/sh# copy from the image backup location to the volume mountcp -a /usr/src/app_backup/node_modules/* /usr/src/app/node_modules/# this next line runs the docker commandexec "$@"

Then update your Dockerfile to include the entrypoint and a backup command:

FROM node:6.3# Create app directoryRUN mkdir -p /usr/src/appWORKDIR /usr/src/app# Install app dependenciesCOPY package.json /usr/src/app/RUN npm install -g babel babel-runtime babel-register mocha nodemonRUN npm install# Bundle app sourceCOPY . /usr/src/appRUN cp -a /usr/src/app/. /usr/src/app_backupEXPOSE 1234ENTRYPOINT [ "/usr/src/app/entrypoint.sh" ]CMD [ "npm", "start" ]

And then drop the extra volume from your docker-compose.yml:

  volumes:    - .:/usr/src/app

I added upon @Robert's answer, as there were a couple of things not taken into consideration with it; namely:

• cp takes too long and the user can't view the progress.
• I want node_modules to be overwritten if it were installed through the host machine.
• I want to be able to git pull while the container is running and not running and update node_modules accordingly, should there be any changes.
• I only want this behavior during the development environment.

To tackle the first issue, I installed rsync on my image, as well as pv (because I want to view the progress while deleting as well). Since I'm using alpine, I used apk add in the Dockerfile:

# Install rsync and pv to view progress of moving and deletion of node_modules onto host volume.RUN apk add rsync && apk add pv

I then changed the entrypoint.sh to look like so (you may substitute yarn.lock with package-lock.json):

#!/bin/ash# Declaring variables.buildDir=/home/node/build-dirworkDir=/home/node/work-dirpackage=package.jsonlock=yarn.locknm=node_modules########################## Begin Functions#########################copy_modules () { # Copy all files of build directory to that of the working directory.  echo "Calculating build folder size..."  buildFolderSize=$( du -a $buildDir/$nm | wc -l )  echo "Copying files from build directory to working directory..."  rsync -avI $buildDir/$nm/. $workDir/$nm/ | pv -lfpes "$buildFolderSize" > /dev/null  echo "Creating flag to indicate $nm is in sync..."  touch $workDir/$nm/.docked # Docked file is a flag that tells the files were copied already from the build directory.}delete_modules () { # Delete old module files.    echo "Calculating incompatible $1 direcotry $nm folder size..."    folderSize=$( du -a $2/$nm | wc -l )    echo "Deleting incompatible $1 directory $nm folder..."    rm -rfv $2/$nm/* | pv -lfpes "$folderSize" > /dev/null # Delete all files in node_modules.    rm -rf $2/$nm/.* 2> /dev/null # Delete all hidden files in node_modules.node_modules.}########################## End Functions# Begin Script#########################if cmp -s $buildDir/$lock $workDir/$lock >/dev/null 2>&1 # Compare lock files.  then    # Delete old modules.    delete_modules "build" "$buildDir"    # Remove old build package.    rm -rf $buildDir/$package 2> /dev/null    rm -rf $buildDir/$lock 2> /dev/null    # Copy package.json from working directory to build directory.    rsync --info=progress2 $workDir/$package $buildDir/$package    rsync --info=progress2 $workDir/$lock $buildDir/$lock    cd $buildDir/ || return    yarn    delete_modules "working" "$workDir"    copy_modules# Check if the directory is empty, as it is when it is mounted for the first time.elif [ -z "$(ls -A $workDir/$nm)" ]  then    copy_moduleselif [ ! -f "$workDir/$nm/.docked" ] # Check if modules were copied from build directory.  then    # Delete old modules.    delete_modules "working" "$workDir"    # Copy modules from build directory to working directory.    copy_moduleselse    echo "The node_modules folder is good to go; skipping copying."fi########################## End Script#########################if [ "$1" != "git" ] # Check if script was not run by git-merge hook.  then    # Change to working directory.    cd $workDir/ || return    # Run yarn start command to start development.    exec yarn start:debugfi

I added pv to, at least, show the user the progress of what is happening. Also, I added a flag to appear to indicate that node_modules was installed through a container.

Whenever a package is installed, I utilized the postinstall and postuninstall hooks of the package.json file to copy the package.json and yarn.lock files from the working directory to the build directory to keep them up to date. I also installed the postinstall-postinstall package to make sure the postuninstall hook works.

"postinstall"  : "if test $DOCKER_FLAG = 1; then rsync -I --info=progress2 /home/node/work-dir/package.json /home/node/build-dir/package.json && rsync -I --info=progress2 /home/node/work-dir/yarn.lock /home/node/build-dir/yarn.lock && echo 'Build directory files updated.' && touch /home/node/work-dir/node_modules/.docked; else rm -rf ./node_modules/.docked && echo 'Warning: files installed outside container; deleting docker flag file.'; fi","postuninstall": "if test $DOCKER_FLAG = 1; then rsync -I --info=progress2 /home/node/work-dir/package.json /home/node/build-dir/package.json && rsync -I --info=progress2 /home/node/work-dir/yarn.lock /home/node/build-dir/yarn.lock && echo 'Build directory files updated.' && touch /home/node/work-dir/node_modules/.docked; else rm -rf ./node_modules/.docked && echo 'Warning: files installed outside container; deleting docker flag file.'; fi",

I used an environment variable called DOCKER_FLAG and set it to 1 in the docker-compose.yml file. That way, it won't run when someone installs outside a container. Also, I made sure to remove the .docked flag file so the script knows it has been installed using host commands.

As for the issue of synchronizing node_modules every time a pull occurs, I used a git hook; namely, the post-merge hook. Every time I pull, it will attempt to run the entrypoint.sh script if the container is running. It will also give an argument to the script git that the script checks to not run exec yarn:debug, as the container is already running. Here is my script at .git/hooks/post-merge:

#!/bin/bashif [ -x "$(command -v docker)" ] && [ "$(docker ps -a | grep <container_name>)" ]then  exec docker exec <container_name> sh -c "/home/node/build-dir/entrypoint.sh git"  exit 1fi

If the container is not running, and I fetched the changes, then the entrypoint.sh script will first check if there are any differences between the lock files, and if there are, it will reinstall in the build directory, and do what it did when the image was created and container run in the first time. This tutorial may be used to be able to share hooks with teammates.

Note: Be sure to use docker-compose run..., as docker-compose up... won't allow for the progress indicators to appear.