Ignite ReadinessProbe

Update:

After posting on the ignite mailing list, it looks like StatefulSets are the way to go. (Thanks Dmitry!)

I think I'm going to leave in the below logic to self-heal any segmentation issues although hopefully it won't be triggered often.

Original answer:

We are having the same issue and I think we have a workable solution. The Kubernetes discovery spi lists services as they become ready.

This means that if there are no ready pods at startup time, ignite instances all think that they are the first and create their own grid.

The cluster should be able to self heal if we have a deterministic way to fail pods if they aren't part of an 'authoritative' grid.

In order to do this, we keep a reference to the TcpDiscoveryKubernetesIpFinder and use it to periodically check the list of ignite pods.

If the instance is part of a cluster that doesn't contain the alphabetical first ip in the list, we know we have a segmented topology. Killing the pods that get into that state should cause them to come up again, look at service list and join the correct topology.

I am facing the same issue, using Ignite embedded within a Java spring application.

As you said the readinessProbe: on the Kubernetes Deployment spec.template.spec.container has the side effect to prevent the Kubernetes Pods from being listed on the related Kubernetes Service as Endpoints

Trying without any readinessProbe, it seems to indeed works better (Ignite nodes are all joinging the same Ignite cluster)

Yet this have the undesired side effect of exposing the Kubernetes Pods when not yet ready, as Spring has not yet fully started ...