Running integration/e2e tests on top of a Kubernetes stack

Interesting question and something that I have worked on over the last couple of months for my current employer. Essentially we ship a product as docker images with manifests. When writing e2e tests I want to run the product as close to the customer environment as possible.

Essentially to solve this we have built scripts that interact with our standard cloud provider (GCloud) to create a cluster, deploy the product and then run the tests against it.

For the major cloud providers this is not a difficult tasks but can be time consuming. There are a couple of things that we have learnt the hard way to keep in mind while developing the tests.

1. Concurrency, this may sound obvious but do think about the number of concurrent builds your CI can run.
2. Latency from the cloud, don't assume that you will get an instant response to every command that you run in the cloud. Also think about the timeouts. If you bring up a product with lots of pods and services what is the acceptable start up time?
3. Errors causing build failures, this is an interesting one. We have seen errors in the build due to network errors when communicating with our test deployment. These are nearly always transitive. It is best to avoid these making the build fail.

One thing to look at is GitLab are providing some documentation on how to build and test images in their CI pipeline.

1. On my side I use travis-ci. I build my container image inside it, then run k8s with kind (https://kind.sigs.k8s.io/) inside travis-CI, and then launch my e2e tests.

• Here is some additional information on this blog post: https://k8s-school.fr/resources/en/blog/k8s-ci/
• And the scripts to install kind inside travis-ci in 2 lines: https://github.com/k8s-school/kind-travis-ci.git. It allows lots of customization on the k8s side (enable psp, change CNI plugin)
• Here is an example: https://github.com/lsst/qserv-operator

2. Or I use Github Actions CI, which allows to install kind easily: https://github.com/helm/kind-action and provide plenty of features, and free worker nodes for open-source projects.

• Here is an example: https://github.com/xrootd/xrootd-k8s-operator

Please note that Github action workers may not scale for large build/e2e tests. Travis-CI scales pretty well.

In my understanding, this workflow coud be moved to an on-premise gitlab CI where your application can interact with other services located inside your network.

One interesting thing is that you do not have to maitain a k8s cluster for your CI, kind will do it for you!