Unable to execute Airflow KubernetesExecutor

Since no answer yet, I'll share my findings so far. In my airflow.conf under kubernetes section, we are to pass the following values

dags_volume_claim = airflow-pvcdags_volume_subpath = airflow/development/dagslogs_volume_claim = airflow-pvclogs_volume_subpath = airflow/development/logs

the way how scheduler creates a new pod from the above configs is as follows (only mentioning the volumes and volumeMounts);

"volumes": [  {    "name": "airflow-dags",    "persistentVolumeClaim": {      "claimName": "airflow-pvc"    }  },  {    "name": "airflow-logs",    "persistentVolumeClaim": {      "claimName": "airflow-pvc"    }  }],"containers": [    { ...  "volumeMounts": [      {        "name": "airflow-dags",        "readOnly": true,        "mountPath": "/usr/local/airflow/dags",        "subPath": "airflow/development/dags"      },      {        "name": "airflow-logs",        "mountPath": "/usr/local/airflow/logs",        "subPath": "airflow/development/logs"      }]...}]

K8s DOESN'T likes multiple volumes pointing to same pvc (airflow-pvc). To fix this, I'd to create two PVC (and PV) for dags and logs dags_volume_claim = airflow-dags-pvc and logs_volume_claim = airflow-log-pvc which works fine.

I don't kow if this has already been addressed in newer version of airflow (I am using 1.10.3). The airflow scheduler should handle this case when ppl using same PVC then create a pod with single volume and 2 volumeMounts referring to that Volume e.g.

"volumes": [  {    "name": "airflow-dags-logs",    <--just an example name    "persistentVolumeClaim": {      "claimName": "airflow-pvc"    }  }"containers": [    { ...  "volumeMounts": [      {        "name": "airflow-dags-logs",        "readOnly": true,        "mountPath": "/usr/local/airflow/dags",        "subPath": "airflow/development/dags"     <--taken from configs      },      {        "name": "airflow-dags-logs",        "mountPath": "/usr/local/airflow/logs",        "subPath": "airflow/development/logs"     <--taken from configs      }]...}]

I deployed a pod with above configurations and it works!