How to monitor disk usage of persistent volumes?

Yes, in newest version of Kubernetes you cannot monitor metric such as kubelet_volume_stats_capacity_bytes, but there are some workarounds. Unfortunately this is a bit fragmented in Kubernetes today. PVCs may have capacity and usage metrics, depending on the volume provider, and it seems that any CSI based volume doesn't have these at all. We can do this on a best effort basis butit is simple to quickly hit cases where these metrics are not available.

First, just simply write your own script which will be every time values of metric like container_fs_usage_bytes are gathered will be count difference between capacity before measurement and container usage in bytes (metric will container_fs_usage_bytes be helpful).

Prometheus is quite popular solution but to monitor capacity especially disk usage you can use Heapster, now he is about to "retire", but just for this special case you can use it, but you will have to implement script too. Take look on repository: heapster-memory

"res.Containers = append(res.Containers, metrics.ContainerMetrics{Name: c.Name, Usage: usage})"

I hope it helps.

I have a job like the following in my prom config:

  - job_name: 'kubernetes-nodes'    kubernetes_sd_configs:    - role: node    scheme: https    tls_config:      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token    relabel_configs:    - action: labelmap      regex: __meta_kubernetes_node_label_(.+)    - target_label: __address__      replacement: kubernetes.default.svc:443    - source_labels: [__meta_kubernetes_node_name]      regex: (.+)      target_label: __metrics_path__      replacement: /api/v1/nodes/${1}/proxy/metrics

With this job in place I see the following metrics available in Prometheus:

kubelet_volume_stats_available_byteskubelet_volume_stats_capacity_byteskubelet_volume_stats_inodeskubelet_volume_stats_inodes_freekubelet_volume_stats_inodes_usedkubelet_volume_stats_used_bytes

More here: https://github.com/google/cadvisor/issues/1702

I have a prometheus exporter for monitoring pvc usage and provides the mapping between pod and pvc. If you are interested you can try it.

https://github.com/kais271/pvc-exporter

Prometheus metrics:

pvc_usagepvc_mapping