You can set a timeout period for gpg-agent in ~/.gnupg/gpg-agent.conf with this line:
default-cache-ttl 3600
That would tell gpg-agent to store the passphrase for one hour. You wouldn't want it to be indefinite, but not constantly typing it is of benefit too.