How to mount kerberised NFS on kubernetes?

This is how I do it.

For my approach you need:

• Workers joined to the kerberso realm. (NFS is mounted bu the worker with the host TGT)
• Keytabs stored in a safe place, I use vault and vault-agent-injector
• Sidecar container that will manage the credentials
• Shared KCM between server and pods sharing the KCM socket, avoiding other pods to have access to the stored TGTs
• Krb5 stored as configmap in the namespace
• Proper rights in the NFS exported data, owned by the users that exist in the IPA and which uids/gids will be used for running the containers in the pod.

The reasons for this approach are:

• Information secured in vault
• Secrets removed from storage and stored only in memory
• Each container manages a single process or task.

With all this in mind you first write down your Dockerfile for the krb5-sidecar.command: ["/bin/sh"]args: ["-c", "/usr/bin/sleep 3600000"]

    FROM centos:centos7        # install the kerberos client tools    RUN yum install -y krb5-workstation && \     mkdir /krb5 && chmod 755 /krb5        # add resources, the kinit script and the default krb5 configuration    ADD entrypoint.sh /entrypoint.sh    RUN chmod +x /krb-sidecar-entrypoint.sh    # Little trick here that will allow my container to remove     # the vault secrets without root    RUN chmod u+s /usr/bin/rm    ENTRYPOINT ["/entrypoint.sh"]

And this is the entrypoint script that manages

1. Keytab load into KCM memory
2. Removal from shared /vault/secrets file of the keytab
3. Renewal of the kerberos ticket depending on your Krb5 policy

# Default value for renewing the TGT ticketKERBEROS_RENEWAL_TIME=86400 # One day# Move the keytab into keytabfileecho  "Generating keytab file"cat /vault/secrets/${USERNAME}.keytab  | cut -d' ' -f2 | base64 -d > /etc/${USERNAME}.keytab# Get the TGTecho "Loading keytab"kinit -kt /etc/${USERNAME}.keytab ${USERNAME}@${REALM}# Remove secrets for security reasonsrm -rf /vault/secrets/*rm -rf /etc/${USERNAME}.keytabecho "Secrets removed from tmpfs"while :;do  kinit -R  sleep ${KERBEROS_RENEWAL_TIME}done

Of course you need to create PersistentVolumes and PersistentVolumeClaims for the deployment.

PersistentVolume

apiVersion: v1kind: PersistentVolumemetadata:  name: NFS-volspec:  volumeMode: Filesystem  accessModes:    - ReadWriteMany  persistentVolumeReclaimPolicy: Recycle  storageClassName: slow  mountOptions:    - sec=krb5  nfs:    path: /exports    server: nfs.server.test

PersistentVolumeClaim:

apiVersion: v1kind: PersistentVolumeClaimmetadata:  name: nfsvolspec:  storageClassName: manual  accessModes:    - ReadWriteMany  resources:    requests:      storage: 3Gi

And finally the Deployment:

apiVersion: apps/v1kind: Deploymentmetadata:  name: deployment-userspec:  selector:    matchLabels:      test: test  template:    metadata:      labels:        test: test      annotations:        vault.hashicorp.com/agent-inject: 'true'        vault.hashicorp.com/agent-inject-secret-userKeytab: 'user/keytabs/user'        vault.hashicorp.com/role: 'nfs'        vault.hashicorp.com/ca-cert: 'certs/ca.crt'        vault.hashicorp.com/tls-secret: 'tls-ca'        vault.hashicorp.com/agent-pre-populate-only: "true"    spec:      securityContext:        # Here we defined the user uid, this user must be present in the NFS server        runAsUser: 2500        runAsGroup: 2500      # This may be needed or not depending on your DNS setup      hostAliases:        - ip: "192.168.111.130"          hostnames:            - "IPA"            - "IPA.server"        - ip: "192.168.111.131"          hostnames:            - "nfs"            - "nfs.serer"      restartPolicy: Always      volumes:      - name: nfs-user        persistentVolumeClaim:          claimName: nfs-vol      - name: krb5        configMap:          name: keos-kerberos-config      - name: kcmsocket        hostPath:          path: /var/run/.heim_org.h5l.kcm-socket          type: File      containers:      - name: krb5-sidecar        image: krb5-sidecar:0.1.0        env:        - name: KRB5CCNAME          value: "KCM:"        - name: USERNAME          value: user        - name: REALM          value: server        volumeMounts:        - name: krb5          mountPath: "/etc/krb5.conf"          subPath: "krb5.conf"        - name: kcmsocket          mountPath: "/var/run/.heim_org.h5l.kcm-socket"        lifecycle:          preStop:            exec:              command: ["/usr/bin/kdestroy"]      - name: mount-nfs-container        image: nfs-centos:0.2.0        env:        - name: KRB5CCNAME          value: "KCM:"        volumeMounts:        - name: nfs-user          mountPath: "/nfs"        - name: krb5          mountPath: "/etc/krb5.conf"          subPath: "krb5.conf"        - name: kcmsocket          mountPath: "/var/run/.heim_org.h5l.kcm-socket"