How to verify that a Kubernetes deployment update has been successful?

You have not mentioned your deployment strategy. But one generic problem I have seen with k8s deployments is that if the application fails to boot up, it will be restarted infinitely. So you might have to kubectl delete deploy/****** explicitly after detecting the deployment failed status. (There is also failureThreshold for probes, but I didn't try yet).

Case Recreate:

You can use the combination of progressDeadlineSeconds and readinessProbe. Let's say your application needs 60 seconds to boot-up/spin-up. You need to configure progressDeadlineSeconds a bit more than 60 seconds just be in the safer side. Now, after running your kubectl apply -f my-deploy.yaml, run the kubectl rollout status deploy/my-deployment command. For me it looks like this:

12:03:37 kubectl apply -f deploy.yaml12:03:38 deployment "my-deployment" configured12:04:18 kubectl rollout status deploy/my-deployment12:04:18 Waiting for rollout to finish: 0 of 1 updated replicas are available (minimum required: 1)...12:04:44 deployment "my-deployment" successfully rolled out

Once you execute the rollout command, kubectl will keep waiting till it has some answer. Also it returns with a proper exit code echo $? - you can check this programmatically and delete the deployment.

Case rollingUpdate:

If you have multiple replicas, then the above mentioned trick should work.If you have just one replica, then use maxUnavailable: 0 and maxSurge: 1 along with the above config.