How can I give grafana user appropriate permission so that it can start successfully?

I recreated your deployment with appropriate PVC and noticed that grafana pod was failing.

Output of command: $ kubectl get pods -n monitoring

NAME READY STATUS RESTARTS AGEgrafana-6466cd95b5-4g95f 0/1 Error  2  65s

Further investigation pointed the same errors as yours:

mkdir: can't create directory '/var/lib/grafana/plugins': Permission deniedGF_PATHS_DATA='/var/lib/grafana' is not writable.You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migration-from-a-previous-version-of-the-docker-container-to-5-1-or-later

This error showed on first creation of a pod and the deployment. There was no need to recreate any pods.

What I did to make it work was to edit your deployment:

apiVersion: apps/v1kind: Deploymentmetadata:  name: grafana  namespace: monitoringspec:  replicas: 1  selector:    matchLabels:      app: grafana  template:    metadata:      name: grafana      labels:        app: grafana    spec:      securityContext:          runAsUser: 472          fsGroup: 472      containers:      - name: grafana        image: grafana/grafana:6.6.2        ports:        - name: grafana          containerPort: 3000        resources:          limits:            memory: "1Gi"            cpu: "500m"          requests:            memory: "500Mi"            cpu: "100m"        volumeMounts:          - mountPath: /var/lib/grafana            name: grafana-storage      volumes:        - name: grafana-storage          persistentVolumeClaim:              claimName: grafana-pvc

Please take a specific look on part:

      securityContext:          runAsUser: 472          fsGroup: 472

It is a setting described in official documentation: Kubernetes.io: set the security context for a pod

Please take a look on this Github issue which is similar to yours and pointed me to solution that allowed pod to spawn correctly:

• https://github.com/grafana/grafana-docker/issues/167

Grafana had some major updates starting from version 5.1. Please take a look: Grafana.com: Docs: Migrate to v5.1 or later

Please let me know if this helps.