CephFS Volume Mount Fails

AFAIK you might have 2 problems here:

• Ceph required the ip addresses of the machines to work
• The OS you are running the container on, is the one which mounts the storage: The ceph tooling needs to be installed on that machine. The container is completely unaware of the mounted disks

There were a couple of issues that needed fixed in order to successfully mount a CephFS volume in kubernetes. Keep in mind I've deployed Kubernetes 1.4.6 using the kube-deploy docker multinode configuration.

Issue #1: Mount command Fails using Kubernetes secrets

When examining the error above more closely, I found that Kubernetes encrypts my Ceph secret with characters that are interpreted as newlines. As a result, the kubelet fails when attempting to mount the file system.

To workaround, I configured my YAML to use a Ceph secretfile instead of a Kubernetes secret:

apiVersion: v1kind: Podmetadata:  name: cephfs-testspec:  containers:  - name: cephfs-rw    image: kubernetes/pause    volumeMounts:    - mountPath: "/mnt/cephfs"      name: cephfs  volumes:  - name: cephfs    cephfs:      monitors:      - "<monitor1>:6789"      - "<monitor2>:6789"      - "<<monitor3>:6789"      user: admin      # Omit for CephFS mounting error      # secretRef:      #   name: ceph-secret      secretFile: "/etc/ceph/user.secret"      readOnly: false

Issue #2: Kubelet Missing Ceph Packages and Configuration

The kubelets were all missing the ceph-fs-common and ceph-common packages required to mount CephFS volumes to containers as well as the necessary configuration files. The following script should applies the necessary updates to the kubelet master/worker agents:

docker exec $KUBELET_ID apt-get updatedocker exec $KUBELET_ID apt-get install -y wget lsb-release apt-transport-httpsdocker exec $KUBELET_ID /bin/bash -c "wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -"RELEASE=$(docker exec $KUBELET_ID lsb_release -sc)docker exec $KUBELET_ID /bin/bash -c "echo deb https://download.ceph.com/debian-jewel/ $RELEASE main | tee /etc/apt/sources.list.d/ceph.list"docker exec $KUBELET_ID apt-get updatedocker exec $KUBELET_ID apt-get install -y ceph-fs-common ceph-commondocker exec $KUBELET_ID mkdir -p /etc/cephdocker exec $KUBELET_ID /bin/bash -c "echo $CEPH_SECRET > /etc/ceph/admin.secret"

Full gist here