Ensure Kubernetes Deployment has completed and all pods are updated and available
Update #2: Kubernetes 1.5 will ship with a much better version of kubectl rollout status
and improve even further in 1.6, possibly replacing my custom solution/script laid out below.
Update #1: I have turned my answer into a script hosted on Github which has received a small number of improving PRs by now.
Original answer:
First of all, I believe the kubectl
command you got is not correct: It replaces all white spaces by commas but then tries to get the 4th field after separating by white spaces.
In order to validate that a deployment (or upgrade thereof) made it to all pods, I think you should check whether the number of available replicas matches the number of desired replicas. That is, whether the AVAILABLE
and DESIRED
columns in the kubectl
output are equal. While you could get the number of available replicas (the 5th column) through
kubectl get deployment nginx | tail -n +2 | awk '{print $5}'
and the number of desired replicas (2nd column) through
kubectl get deployment nginx | tail -n +2 | awk '{print $2}'
a cleaner way is to use kubectl
's jsonpath output, especially if you want to take the generation requirement that the official documentation mentions into account as well.
Here's a quick bash script I wrote that expects to be given the deployment name on the command line, waits for the observed generation to become the specified one, and then waits for the available replicas to reach the number of the specified ones:
#!/bin/bashset -o errexitset -o pipefailset -o nounsetdeployment=get_generation() { get_deployment_jsonpath '{.metadata.generation}'}get_observed_generation() { get_deployment_jsonpath '{.status.observedGeneration}'}get_replicas() { get_deployment_jsonpath '{.spec.replicas}'}get_available_replicas() { get_deployment_jsonpath '{.status.availableReplicas}'}get_deployment_jsonpath() { local readonly _jsonpath="$1" kubectl get deployment "${deployment}" -o "jsonpath=${_jsonpath}"}if [[ $# != 1 ]]; then echo "usage: $(basename $0) <deployment>" >&2 exit 1fireadonly deployment="$1"readonly generation=$(get_generation)echo "waiting for specified generation ${generation} to be observed"while [[ $(get_observed_generation) -lt ${generation} ]]; do sleep .5doneecho "specified generation observed."readonly replicas="$(get_replicas)"echo "specified replicas: ${replicas}"available=-1while [[ ${available} -ne ${replicas} ]]; do sleep .5 available=$(get_available_replicas) echo "available replicas: ${available}"doneecho "deployment complete."
Just use a rollout status:
kubectl rollout status deployment/<deployment-name>
This will run in foreground, it waits and displays the status, and exits when rollout is complete on success or failure.If you're writing a shell script, then check the return code right after the command, something like this.
kubectl rollout status deployment/<deployment-name>if [[ "$?" -ne 0 ]] then echo "deployment failed!" exit 1fi
To even further automate your script:
deployment_name=$(kubectl get deployment -n <your namespace> | awk '!/NAME/{print $1}') kubectl rollout status deployment/"${deployment_name}" -n <your namespace>if [[ "$?" -ne 0 ]] then echo "deployment failed!" #exit 1else echo "deployment succeeded"fi
If you're running in default namespace then you could leave out the -n <your namespace>.The command awk '!/NAME/{print $1}') extracts the first field (deployment name), while ignoring the first row which is the header(NAME READY UP-TO-DATE AVAILABLE AGE).If you have more than one deployment files then you could also add more regex or pattern to awk: e.g. awk '!/NAME/<pattern to parse>/{print $1}')