Changing hostname breaks Rabbitmq when running on Kubernetes

The idea is to use a different 'service' and 'deployment' for each of the node you want to create.

As you said, you have to create a custom NODENAME for each i.e:

RABBITMQ_NODENAME=rabbit@rabbitmq-1

Also rabbitmq-1,rabbitmq-2,rabbitmq-3 have to be resolved from each nodes. For that you can use kubedns. The /etc/resolv.conf will look like:

search rmq.svc.cluster.local 

and /etc/hosts must contains:

127.0.0.1 rabbitmq-1  # or rabbitmq-2 on node 2...

The services are here to create a stable network identity for each nodes

rabbitmq-1.svc.cluster.localrabbitmq-2.svc.cluster.localrabbitmq-3.svc.cluster.local

The different deployments resources will allow you to mount a different volume on each node.

I'm working on a deployment tool to simplify those actions:I've done a demo on how I scale and deploy rabbitmq from 1 to 3 nodes on kubernetes:https://asciinema.org/a/2ktj7kr2d2m3w25xrpz7mjkbu?speed=1.5

More generally, the complexity your facing to deploy a clustered application is addressed in the 'petset proposal': https://github.com/kubernetes/kubernetes/pull/18016

In addition to the first reply by @ant31:

Kubernetes now allows to setup a hostname, e.g. in yaml:

template:  metadata:    annotations:      "pod.beta.kubernetes.io/hostname": rabbit-rc1

See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns # A Records and hostname Based on Pod Annotations - A Beta Feature in Kubernetes v1.2

It seems that the whole configuration alive multiple restarts or re-schedules. I've not setup a cluster however I'm going to follow the tutorial for mongodb, see https://www.mongodb.com/blog/post/running-mongodb-as-a-microservice-with-docker-and-kubernetes

The approach will be probably almost same from kubernetes point of view.