How do I use GitHub with SSH without having to run ssh-agent?

You can use config file in .ssh directory of your user in order to configure key for particular server:

# ~/.ssh/configHost github.com    IdentityFile ~/.ssh/git_id_rsa

CodeHunter