Kubernetes deployment read-only filesystem error

Since kubernetes version 1.9 and forth, volumeMounts behavior on secret, configMap, downwardAPI and projected have changed to Read-Only by default.

A workaround to the problem is to create an emtpyDir volume and copy the contents into it and execute/write whatever you need.

this is a small snippet to demonstrate.

    initContainers:    - name: copy-ro-scripts      image: busybox      command: ['sh', '-c', 'cp /scripts/* /etc/pre-install/']      volumeMounts:        - name: scripts          mountPath: /scripts        - name: pre-install          mountPath: /etc/pre-install   volumes:      - name: pre-install        emptyDir: {}      - name: scripts        configMap:          name: bla

Merged PR which causes this break :( https://github.com/kubernetes/kubernetes/pull/58720

    volumeMounts:      - name: airflow-config-volume        mountPath: /usr/local/airflow  volumes:  - name: airflow-config-volume    secret:      secretName: airflow-config-secret

Is the source of your problems, for two reasons: first, you have smashed the airflow user's home directory by volume mounting your secret onto the image directly into a place where the image expects a directory owned by airflow.

Separately, while I would have to fire up a cluster to confirm 100%, I am pretty sure that Secret volume mounts -- and I think their ConfigMap friends -- are read-only projections into the Pod filesystems; that suspicion certainly appears to match your experience. There is certainly no expectation that changes to those volumes propagate back up into the kubernetes cluster, so why pretend otherwise.

If you want to continue to attempt such a thing, you do actually have influence over the defaultMode of the files projected into that volumeMount, so you could set them to 0666, but caveat emptor for sure. The short version is, by far, not to smash $AIRFLOW_HOME with a volume mount.