How to deal with temporary apt-key adv failures?

The Issue

The often used keyserver pool.sks-keyserver.net in fact is not a single server, but a whole pool of keyservers of which you get a random selection through DNS. They synchronize with each other, and because of the vast number of servers, you can assume any of those are available. But not all of them are very stable, lots of servers are offered by individuals or even running at home. Choosing a single, fixed server the the MIT's you quoted works out if the server is very reliable, but often the key servers are down from time to time.

keys.gnupg.net in fact points to pool.sks-keyservers.net (to be more speccific, a CNAME record is set up).

Although a whole bunch of servers is returned whenever the pool is queried, GnuPG does only select one of them. If the server does not respond, GnuPG just fails instead of trying another one (or even trying multiple in parallel).

Possible Solutions

All of those solutions try to either not query the key server network, or deal with the reduced stability of the network.

• Add a local copy of the key to the repository

  This is the way I chose whenever reworking my Dockerfiles. I add a minimal copy of the signing key to the git repository holding the Dockerfile, and then import it into the image. An example from the prosody Dockerfile I wrote:

  COPY 107D65A0A148C237FDF00AB47393D7E674D9DBB5.gpg /rootRUN apt-key add /root/107D65A0A148C237FDF00AB47393D7E674D9DBB5.gpg && \    echo deb http://packages.prosody.im/debian jessie main >>/etc/apt/sources.list && \    apt-get update

  This strategy has the advantage of not relying on any external resources and not pulling all the signatures which could heavily enlarge the image (OpenPGP keys can easily grow to multiple MB if having a lot of signatures on them). You don't need the signatures anyway, if you verify the key out-of-band and add a verified key (in fact, pulling a key from the key server network using it's fingerprint is not much different, anyway). A possible disadvantage is that you will not be fetching revocations, so watch the key, mailing list, security notifications, ... for the rare case the key is leaked and revoked afterwards.

• Fetch from the repository server

  If you pull the key from the repository server (given they offer it for download), you can also pull directly pull it from there. I don't see any relevant advantage if compared to storing a local copy. You have to make sure you pulled the right key (ie., verify the fingerprint). If the server is compromised (and manipulated packages are uploaded), they will be able to manipulate the key anyway (ie., suppress revocation certificates).

• Use a specific key server

  If you know a specific, very reliable key server, you can always choose a fixed server through the --keyserver option. For example, if you expect Ubuntu's server to be super stable (or host your own, but make sure to include it in the keyserver network), consider choosing this one.

• Retry.

  Retry fetching the key until you succeed. One of the servers will be fine, for sure. Downside: building the image is even more erratic with respect to the build time, and network failures will result in endless loops (unless you add some maximum retry count). You'll have to script that on your own.

• Choose the high-availability pool

  There is ha.pool.sks-keyservers.net, which requires servers contained to be a highly available cluster setup. If the reverse proxy/load balancer is down, you'll get stuck anyway; but expect the general availability to be much higher than the usual pool.sks-keyservers.net. If people care about providing a cluster setup, they're probably monitoring their systems properly.

  If you want to stick with the key server network, this is probably the best way to go. Mixing a local copy with this pool (and fall back to a local copy) might also be viable, providing high chances of catching the rare event of a revoked key while still not having build failures because of key server timeouts from time to time.