How to connect to a remote Jupyter Notebook running on a docker container?

This is an excellent question. My day to day development workflow involves connecting to a jupyter notebook running on a docker container on a gpu accelerated server from my laptop.

There are three parts to the problem:

1. Container

2. Host

3. Remote

Part 1: Configuring the Container.

You need to make sure that you can ssh to your container. I modified the Dockerfile from Docker's sshd example. You should modify the Docker example to fit your needs; e.g. does your use case allow for you to connect as root?

I have my containers setup for passwordless ssh. Note you can pre-make the files needed for passwordless ssh and just copy them over to the appropriate directories when you build your image.

If you want to save yourself some typing later, I strongly recommend modifying your jupyter config file as well.

Part 2: Configuring the Host

I'm assuming you're running linux on the host machine. You'll have to create a docker network. You should only need the --attachable option. I like to have an entry in my /etc/hosts to make life easier, for example:

172.21.0.2  container-1 container-1

You will also have to modify your /home/<user>/.ssh/config file on your host system.

Host container-1    Hostname container-1    IdentityFile ~/.ssh/containers_rsa    User rootHost tunnel-container-1    Hostname container-1    IdentityFile ~/.ssh/containers_rsa    User root    LocalForward 8888 localhost:8888    LocalForward 8000 localhost:8000

Don't you get sick of having to type nvidia-docker?
Here is a sample /etc/docker/daemon.json:

  {    "runtimes": {        "nvidia": {            "path": "nvidia-container-runtime",            "runtimeArgs": []        }    },    "graph":"/docker-files"  }

Part 3: Configuring the Local System.

You're almost there. You need to be able to establish an ssh tunnel to the remote system. Example ~/.ssh/config for a local system:

Host remote_host    Hostname remote_host    IdentityFile ~/.ssh/remote_host/remote_rsaHost tunnel-remote_host    Hostname remote_host    IdentityFile ~/.ssh/remote_host/remote_rsa    LocalForward 10000 localhost:8888    LocalForward 10006 localhost:8000

An important note. I have an entry in my local /etc/hosts for my remote host. Your instance may not have a static ip. So you can either modify your local /etc/hosts every time you spin up your instance or have a script for sshing to your remote system with all the correct LocalForward directives.

Putting it all together.

Assumes your remote instance has successfully launched and Docker is running correctly. In addition it assumes that you have done the convenience steps I mentions.

1. SSH to remote.
2. launch docker container. Note the container name or even better specify it.
3. Connect your container to your network. e.g. docker network connect skynet container-1. You can check to see if your container is attached by running docker network inspect skynet. Note your network will most likely be name something other then skynet.
4. SSH to your docker container e.g. ssh container-1.
5. Start your juypyter server. jupyter notebook &. I background the process so I don't have to open another session.
6. While on your remote/host, system: ssh -N -f tunnel-container-1. Ok we now have an ssh tunnel from our host system to our docker container.
7. On your local system: ssh -N -f tunnel-remote_host.
8. On your local system navigate to http://localhost:10000.

If everything works you should be connected to your jupyter server and in the root directory you specified in your Dockerfile!

How to trouble shoot.

First of all confirm that you can launch your docker container manually with the correct run time environment. Confirm that you can ssh and launch jupyter notebook. Yes you won't be able to connect to it but you will know that all those pieces are working. Next confirm that you can establish an ssh tunnel with your container from remote to container. You can confirm the tunnel is working correctly by using wget to get the html for the landing page for your jupyter server. Next check the forwarding from local to remote/host. It took me some trial an error to get the order correct for the LocalForward directive.

I hope this helps.