ssh-agent does not remember identities when running inside a docker container in DC/OS

The key file contents being passed via PRIVATE_KEY originally contain line breaks. After echoing the PRIVATE_KEY variable content to ~/.ssh/id_rsa the line breaks will be gone. You can fix that issue by wrapping the $PRIVATE_KEY variable with double quotes.

Another issue arises when the container is started without attached TTY, typically via -i -t command line parameters to docker run. The password request will fail and won't add the ssh key to the ssh-agent. For the container being run in DC/OS, the interaction probably won't make sense, so you should change your entrypoint script accordingly. That will require your ssh key to be passwordless.

This changed Dockerfile should work:

ENTRYPOINT eval "$(ssh-agent -s)" && \           mkdir -p .ssh && \           echo "$PRIVATE_KEY" > .ssh/id_rsa && \           chmod 600 /root/.ssh/id_rsa && \           ssh-add /root/.ssh/id_rsa && \           while true; do ssh-add -l; sleep 2; done

Docker Version

Check that your local version of Docker matches the version installed on the DC/OS agents. By default, the DC/OS 1.9.3 AWS CloudFormation templates uses CoreOS 1235.12.0, which comes with Docker 1.12.6. It's possible that the entrypoint behavior has changed since then.

Docker Command

Check the Mesos task logs for the Marathon app in question and see what docker run command was executed. You might be passing it slightly different arguments when testing locally.

Script Errors

As mentioned in another answer, the script you provided has several errors that may or may not be related to the failure.

1. echo $PRIVATE_KEY should be echo "$PRIVATE_KEY" to preserve line breaks. Otherwise key decryption will fail with Bad passphrase, try again for /root/.ssh/id_rsa:.
2. expect -c "spawn ssh-add /root/.ssh/id_rsa; expect \"Enter passphrase for /root/.ssh/id_rsa:\" send \"\"; interact " should be expect -c "spawn ssh-add /root/.ssh/id_rsa; expect \"Enter passphrase for /root/.ssh/id_rsa:\"; send \"\n\"; interact ". It's missing a semi-colon and a line break. Otherwise the expect command fails without executing.

File Based Secrets

Enterprise DC/OS 1.10 (1.10.0-rc1 out now) has a new feature named File Based Secrets which allows for injecting files (like id_rsa files) without including their contents in the Marathon app definition, storing them securely in Vault using DC/OS Secrets.

• Creation: https://docs.mesosphere.com/1.10/security/secrets/create-secrets/
• Usage: https://docs.mesosphere.com/1.10/security/secrets/use-secrets/

File based secrets wont do the ssh-add for you, but it should make it easier and more secure to get the file into the container.

Mesos Bug

Mesos 1.2.0 switched to using Docker --env_file instead of -e to pass in environment variables. This triggers a Docker env_file bug that it doesn't support line breaks. A workaround was put into Mesos and DC/OS, but the fix may not be in the minor version you are using.

A manual workaround is to convert the rsa_id to base64 for the Marathon definition and back in your entrypoint script.