How can I get metrics in Kubernetes for the write layer disk usage of pod containers?

The only solution that I could come up with is to use BaseUsage definition instead of fs.Usage. This would unfortunately require to to compile that stuff in one kubelet binary compatible with your desired k8s version.

fs_usage_bytes reflect the amount of data stored in /var/lib/docker for particular container. It comes from Usage property of FsStats object and it is compatible with different CRI. HoweverFsStats has another property called BaseUsage which compatible only with Docker, and considering Docker builtin support future deprecation there might be a good reason why it's not used. According to description in the code, BaseUsage contains the value that comes from Docker and it's likely to be the same with the size shown in docker ps -s output.

So you want to use BaseUsage type definition instead of fs.Usage:

// Number of bytes that is consumed by the container on this filesystem.         Usage uint64 `json:"usage"` // Base Usage that is consumed by the container's writable layer.         // This field is only applicable for docker container's as of now.         BaseUsage uint64 `json:"base_usage"

Reference in code can be found here.

This is what is returned as container_fs_usage_bytes:

getValues: func(s *info.ContainerStats) metricValues {        return fsValues(s.Filesystem, func(fs *info.FsStats) float64 {                return float64(fs.Usage)

Reference in the code can be found here.

To avoid breaking any existing process with swapping the return value you might want to add whole section like this:

}, {          name: "container_fs_base_usage_bytes",          help: "Docker writable layer size",          valueType: prometheus.GaugeValue,          extraLabels: []string{"device"},          getValues: func(s *info.ContainerStats) metricValues {                  return fsValues(s.Filesystem, func(fs *info.FsStats) float64 {                                          return float64(fs.BaseUsage)                  }, s.Timestamp)          },  },