Setting up private Github access with AWS Elastic Beanstalk and Ruby container

After a good day of effort, I finally enabled use of my organization's private GitHub repos with Elastic Beanstalk by just using a .config file. I am using Python and pip, but it should also work for other package installers on EB.

rhetonik's ssh-agent+ssh-add approach did not work for me at all, so I elected to set up an ssh configuration file instead.

Here is my .ebextensions/3-pip-install-from-github.config file:

files:    "/root/.ssh/config":        owner: root        group: root        mode: "000600"        content: |            Host github.com                User git                Hostname github.com                IdentityFile /root/.ssh/githubcommands:    01-command:        command: sudo ssh-keyscan -H github.com >> /root/.ssh/known_hosts    02-command:        command: sudo chmod 644 /root/.ssh/known_hosts    03-command:        command: sudo aws s3 cp s3://bucket-with-your-github-ssh-key/github /root/.ssh    04-command:        command: sudo chmod 600 /root/.ssh/github

Rough instructions:

• Set up an S3 bucket accessible by your EB instance. Inside of that bucket, store the SSH key allowing access to the GitHub repository you want to access via pip, npm, bundle, etc. Use sudo aws s3 cp to copy that key onto your EB instance on deploy. sudo is necessary because EB scripts use root and not ec2-user.

• This ebextensions config file also creates 2 files on your EB instance. /root/.ssh/config tells ssh (invoked by pip and git) to use the key you copied from S3. Storing the output of ssh-keyscan -H github.com into /root/.ssh/known_hosts will pre-verify that ssh on your EB instance is actually communicating with GitHub to avoid MITM attacks. This is better than disabling StrictHostKeyChecking in /root/.ssh/config.

Here is my requirements.txt file for pip:

Beaker==1.7.0Flask==0.10.1Jinja2==2.7.3MarkupSafe==0.23# [...]git+ssh://git@github.com/myorganization/myprivaterepo.git@0.0.142

While running eb-deploy, you can tail -f /var/log/eb-activity.log to make sure everything runs smoothly.

Here's how I finally did it. It's all about setting up an SSH Key for the user which is responsible for bundle install phase.

1. Start an environment for an application in AWS Elastic Beanstalk
2. Optional - Login to Amazon EC2 console and change instance type to a desired value
3. Update SSH Key pair name to enable remote SSH login. (I'm sure there must be a way to specify instance type and SSH key pair name while starting an environment)
4. Look for the newly launched instance either in EC2 console or through CLI, note the Fully Qualified Domain Name (FQDN) for this instance. EB instances are like any other instance you would create with Amazon EC2. Login via SSH to this instance.
5. Execute the following commands to create an SSH key for root user

   $ sudo su - root

   $ ssh-keygen -t rsa -C "some-email@yourdomain.com"

6. Edit .bash_profile to explicitly start ssh-agent and add the newly generated SSH Key. Add the following lines (This might seem unnecessary, I did it just to be sure)

   eval `ssh-agent

   eval ssh-add ~/.ssh/id_rsa

7. Note the SSH public key E.g.: ~/.ssh/id_rsa.pub and add it to the set of SSH Keys for Github account which has access to private repositories

8. At this point, your instance has access to your private Github repositories. You could test this by issuing a git clone on those repositories by logging in as root user.

9. Create an AMI out of this instance using standard methods

10. Come back to your AWS Elastic Beanstalk Dashboard and look for Edit Configuration option in your application's environment. In the Server tab, look for an option which lets you specify a Custom AMI. Update this field with the newly created AMI ID E.g.: ami-4324fd4.

11. Save configuration by hitting Apply Changes. AWS Elastic Beanstalk would start deploying new instances across your environment and terminating the old ones. This is to ensure all your auto-scaled instances have the whitelisted SSH Key required for private Github access.

After the above steps are done, you could go ahead and deploy your Rails application with git aws.push

Hope this helps others who are stuck. I'd be glad to see a more graceful solution than this one though.

If you're in a hurry, and your application repo is also private, you can create an additional Github user account and assign it read-only privileges to the repo containing the gem.

Then give bundler the https url with the new account's credentials:

gem 'somegemname', git: "https://username:password@github.com/example/privaterepository"