How perform a rolling update with Vault HA in Kubernetes?

After some more digging I learned, what I want is impossible.Whenever a Vault instance is restarted it will be unsealed first and there is no way to unseal it automatically using Vault-own techniques.

You can find a lot of "vault-unsealer" implementations in GitHub and Docker store which try to fill this gap by regularly checking the Vault pods state and unsealing it if necessary.

It is suggested to use an K8s readinessprobe to avoid that services access a sealed Vault pod.

As there is no official "vault-unsealer" image, the public implementations must be used with caution. I ended up writing my own "vault-unsealer" to avoid security flaws and licensing problems.

My solution is a sidecar-container with each Vault pod. The unseal keys first have to be entered once manually with kubectl exec ... at one sidecar.The sidecars regularly check all Vault pods and communicate the unseal keys to the other sidecar if sealed. If a sidecar receives unseal keys, they are stored in memory and are used to unseal its own Vault instance.

1. kubect aply -f vault.yaml -> vault-0 starting
2. kubectl exec vault-0 -c sidecar ... to enter unseal keys -> vault-0 sidecar unseals vault-0 and is ready
3. vault-1 starting
4. vault-0 sidecar detects vault-1 unsealed and calls the vault-1 sidecar to transmit the unseal keys. -> vault-1 sidecar unseals vault-0 and is ready
5. and so on...