Communication in k8s cluster
Thus, I believe this is because of LoadBalancer service which distributes requests equally among target pods and that there is something wrong with this set-up.
Yes, it is true.
how should I configure services in k8s so that client apps can talk to all members?
Service can only balance requests between available pods.
Or this is not the case and it should work in a different way?
Yes, that is a good idea. The simplest way is to create one service per hazelcast replica. And that is practically the only way to do that.
If your hazelcast client is inside the kubernetes cluster, you dont really need LoadBalancer type. A simple service ClusterIP or headless would suffice. Hazelcast supports kubernetes discovery mode. I suggest try using ClusterIP or none.