Apache Ignite: 1000s of warnings "Unable to perform handshake within timeout" get added to the log

If you've defined a service and opened port 10800 then K8 will perform a health check through kube-proxy. This causes Ignite to receive an incomplete handshake on that port log the "unable to perform handshake" message.

ClientListenerNioListener: Site: WARN - Unable to perform handshake within timeout [timeout=10000, remoteAddr=/10.250.0.4:58050]

Here the client connector listener(ClientListenerNioListener) is saying that it was not able to establish a successful handshake within 10 seconds to remoteAddr=/10.250.0.4:58050

config client connector: https://apacheignite.readme.io/docs/binary-client-protocol#connectivity
client connector handshake: https://apacheignite.readme.io/docs/binary-client-protocol#connection-handshake
 
 

example of service w/port 10800 opened:

kind: Servicemetadata:   # The name must be equal to TcpDiscoveryKubernetesIpFinder.serviceName  name: ignite  # The name must be equal to TcpDiscoveryKubernetesIpFinder.namespaceName  namespace: ignitespec:  type: LoadBalancer  ports:    - name: rest      port: 8080      targetPort: 8080    - name: sql      port: 10800      targetPort: 10800

You can redefine the service to not open the port or update the service definition to use different ports for the healthcheck:https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip

from the doc:
service.spec.healthCheckNodePort - specifies the health check node port (numeric port number) for the service. If healthCheckNodePort isn’t specified, the service controller allocates a port from your cluster’s NodePort range. You can configure that range by setting an API server command line option, --service-node-port-range. It will use the user-specified healthCheckNodePort value if specified by the client. It only has an effect when type is set to LoadBalancer and externalTrafficPolicy is set to Local.