How do I use my gpg key to decrypt a file in a docker container without saving it in the image?

You are correct in saying, that adding the key during docker build is insecure.

The env var is a valid way. If you use kubernetes later, you can safely save the your key as a secret in kubernetes. When deploying you can provide a secret via env to a container.

You could also provide a secret as a file from a kubernetes secret by using a volume, and mount a volume locally and provide the key file.