Monitoring Kubernetes persistent volumes usage with Prometheus

Question

I have a K8s 1.7 cluster using vSphere as persistent storage provider. I have also deployed Prometheus, node_exporter and kube-state-metrics.

I'm trying to find a way to monitor a persistent volume's usage using Prometheus. I have added custom labels to some PVs, eg. app=rabbitmq-0, etc.

How can I combine kube_persistentvolume_labels with node_filesystem_size metrics so that I can query PV usage using my custom label?

PS.
I know that K8s 1.8 directly exposes these metrics from kubelet as mentioned in How to monitor disk usage of kubernetes persistent volumes? but currently a cluster upgrade is not an option.

Answer 1

Starting from (v1.3.0-rc.0 / 2018-03-23) in the kube-state-metrics, two metrics that can convert PersistentVolume and PersistenVolumeClaims labels to Prometheus labels accordingly were added:

kube_persistentvolume_labels
kube_persistentvolumeclaim_lables

To get more details about implementing aggregation of metrics based on labels, consider reading these articles:

• Aggregating Pod resource (CPU, memory) usage by arbitrary labels with Prometheus
• Exposing the software version to Prometheus