How do you set-up Mongo replica set on Kubernetes?

This answer is out of date. I wrote a detailed step-by-step tutorial here using more up to date methods. I highly recommend reading it all.

In a nutshell, you run a sidecar app to configure the replica set for you, and either use a service per instance or ping the K8s API for the pod IP addresses.

Example: This will only work in Google Cloud. You will need to make modifications for other platforms, particularly around the volumes:

1. Follow the example in https://github.com/leportlabs/mongo-k8s-sidecar.git
   • git clone https://github.com/leportlabs/mongo-k8s-sidecar.git
   • cd mongo-k8s-sidecar/example/
   • make add-replica ENV=GoogleCloudPlatform (do this three times)
2. Connect to the replica set via services.
   • mongodb://mongo-1,mongo-2,mongo-3:27017/dbname_?
3. You can also use the raw pod IP addresses instead of creating a service per pod
   • Use this https://github.com/thesandlord/kubernetes-pod-ip-finder.git

Typically, to set up a clustered set of nodes like mongo with replicas sets, you would create a Service that tracks the pods under the service name (so for example, create a MongoDB replication controller with a tag mongodb, and a Service tracking those instances)The Service can then be queried for its members (using the API server, you can look up the nodes with

curl -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes/api/v1/namespaces/default/endpoints/mongodb

where mongodb is your selector on the name of the service.

that returns a JSON object with a bunch of fields, so a good way to parse these easily is to use jq https://stedolan.github.io/jq/

piping the curl command into a jq query like

jq '.subsets[].addresses[]' | jq '{ip: .ip, host:.targetRef.name}' will return the IP and hostnames of the mongodb instances in your cluster.

So now you know who is in the cluster and you can create the replica set in your init script. Obviously here that means you need to start the Service first, your startup script needs to wait for all the nodes to be up and registered with the service, and then you can proceed. If you use one image, with one script, it will run n each node, so you need to check that the replica set does not exists already or handle errors. The first pod to register should do the work.Another option is to run all nodes as single nodes, then run a separate bootstrapping script that will create the replica set.

Finally, then you call the mongodb cluster, you will need to make sure you specify the url with replica set name as an option:

mongodb://mongodb:27017/database?replicaSet=replicaSetName

Since you don't know the IP of the master, you would call it through the service mongodb which will load balance the requests to one of the nodes, and if you don't specify the replica set name, you will end up with connection errors as only the master can get write requests.

Obviously this is not a step by step tutorial, but i hope that gets you started.

This is the example I'm currently running.

apiVersion: v1kind: Servicemetadata:  labels:    name: mongo  name: mongo-svc1spec:  ports:    - port: 27017      targetPort: 27017  selector:    type: mongo-rs-A---apiVersion: v1kind: Servicemetadata:  labels:    name: mongo  name: mongo-svc2spec:  ports:    - port: 27017      targetPort: 27017  selector:    type: mongo-rs-B---apiVersion: v1kind: Servicemetadata:  labels:    name: mongo  name: mongo-svc3spec:  ports:    - port: 27017      targetPort: 27017  selector:    type: mongo-rs-C---apiVersion: v1kind: ReplicationControllermetadata:  name: mongospec:  replicas: 1  selector:    name: mongo-nodea    role: mongo    environment: test  template:    metadata:      labels:        name: mongo-nodea        role: mongo        environment: test        type: mongo-rs-A    spec:      containers:        - name: mongo-nodea          image: mongo          command:            - mongod            - "--replSet"            - rsABC            - "--smallfiles"            - "--noprealloc"          ports:            - containerPort: 27017          volumeMounts:            - name: mongo-persistent-storage              mountPath: /data/db      volumes:        - name: mongo-persistent-storage          flocker:            datasetName: FlockerMongoVolSetA---apiVersion: v1kind: ReplicationControllermetadata:  name: mongo-1spec:  replicas: 1  selector:    name: mongo-nodeb    role: mongo    environment: test  template:    metadata:      labels:        name: mongo-nodeb        role: mongo        environment: test        type: mongo-rs-B    spec:      containers:        - name: mongo-nodeb          image: mongo          command:            - mongod            - "--replSet"            - rsABC            - "--smallfiles"            - "--noprealloc"          ports:            - containerPort: 27017          volumeMounts:            - name: mongo-persistent-storage              mountPath: /data/db      volumes:        - name: mongo-persistent-storage          flocker:            datasetName: FlockerMongoVolSetB---apiVersion: v1kind: ReplicationControllermetadata:  name: mongo-2spec:  replicas: 1  selector:    name: mongo-nodec    role: mongo    environment: test  template:    metadata:      labels:        name: mongo-nodec        role: mongo        environment: test        type: mongo-rs-C    spec:      containers:        - name: mongo-nodec          image: mongo          command:            - mongod            - "--replSet"            - rsABC            - "--smallfiles"            - "--noprealloc"          ports:            - containerPort: 27017          volumeMounts:            - name: mongo-persistent-storage              mountPath: /data/db      volumes:        - name: mongo-persistent-storage          flocker:            datasetName: FlockerMongoVolSetCkubectl --kubeconfig=clusters/k8s-mongo/kubeconfig get po,svc -L type,role,nameNAME            READY     STATUS    RESTARTS   AGE       TYPE         ROLE      NAMEmongo-1-39nuw   1/1       Running   0          1m        mongo-rs-B   mongo     mongo-nodebmongo-2-4tgho   1/1       Running   0          1m        mongo-rs-C   mongo     mongo-nodecmongo-rk9n8     1/1       Running   0          1m        mongo-rs-A   mongo     mongo-nodeaNAME         CLUSTER_IP   EXTERNAL_IP   PORT(S)     SELECTOR          AGE       TYPE      ROLE      NAMEkubernetes   10.3.0.1     <none>        443/TCP     <none>            21h       <none>    <none>    <none>mongo-svc1   10.3.0.28    <none>        27017/TCP   type=mongo-rs-A   1m        <none>    <none>    mongomongo-svc2   10.3.0.56    <none>        27017/TCP   type=mongo-rs-B   1m        <none>    <none>    mongomongo-svc3   10.3.0.47    <none>        27017/TCP   type=mongo-rs-C   1m        <none>    <none>    mongo

On the Primary node I am going into mongo shell

rs.status() rs.initiate() rs.add("10.3.0.56:27017")

I'm currently running into this issue where I'm stuck in Secondary and Startup statuses for the two nodes without a primary.

rs.status(){    "set" : "rsABC",    "date" : ISODate("2016-01-21T22:51:33.216Z"),    "myState" : 2,    "term" : NumberLong(1),    "heartbeatIntervalMillis" : NumberLong(2000),    "members" : [        {            "_id" : 0,            "name" : "mongo-rk9n8:27017",            "health" : 1,            "state" : 2,            "stateStr" : "SECONDARY",            "uptime" : 242,            "optime" : {                "ts" : Timestamp(1453416638, 1),                "t" : NumberLong(1)            },            "optimeDate" : ISODate("2016-01-21T22:50:38Z"),            "infoMessage" : "could not find member to sync from",            "configVersion" : 2,            "self" : true        },        {            "_id" : 1,            "name" : "10.3.0.56:27017",            "health" : 1,            "state" : 0,            "stateStr" : "STARTUP",            "uptime" : 45,            "optime" : {                "ts" : Timestamp(0, 0),                "t" : NumberLong(-1)            },            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),            "lastHeartbeat" : ISODate("2016-01-21T22:51:28.639Z"),            "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),            "pingMs" : NumberLong(40),            "configVersion" : -2        }    ],    "ok" : 1}