Skip to main content

Prometheus and Grafana

Requires the API option

/api/v1/public/metrics is part of the public API, unlocked by the ProxCenter API Access add-on on an Enterprise edition. Without it every call on this path answers 403 Feature not licensed, and a Grafana dashboard built on it stays empty.

One scrape job covers your whole estate. ProxCenter answers with its own aggregated view of every cluster it manages, so there is nothing to install on any Proxmox node, and the response is served from ProxCenter's inventory cache rather than fanning out to the hypervisor on each scrape.

The exposition carries 53 metric families. Everything the five dashboards chart is in there, and the full list is in the series reference below.

The five dashboards​

Each answers one question rather than scrolling through everything, and they link to one another from a ProxCenter dashboards menu that carries your time range and filters across.

DashboardAnswers
ProxCenter Fleet OverviewIs every cluster reachable, and has it been? Ceph health, high availability state, and the changes an instantaneous count hides, such as a node that bounced overnight
ProxCenter Node PerformanceHow hard are the hosts working, and what runs out first? Load average, CPU, memory headroom, I/O wait, swap, and the host root filesystem with a seven day projection
ProxCenter Guest WorkloadWhat are the guests doing? Running state over time, the biggest CPU and memory consumers, and disk and network throughput
ProxCenter StorageWhere is the space going? Capacity per storage, with shared storages counted once for the cluster and non-shared ones broken down per node
ProxCenter Backup ComplianceWhich guests are actually protected, how stale their backups are, and whether the Proxmox Backup Server datastores behind them have room left

All five ship with the image and import into Grafana 11.0 or newer, using core panels only, so nothing has to be installed alongside them:

FilePurpose
/integrations/prometheus-scrape-config.ymlScrape job to paste into your Prometheus configuration
/integrations/grafana-dashboard-proxcenter.jsonFleet Overview
/integrations/grafana-dashboard-proxcenter-nodes.jsonNode Performance
/integrations/grafana-dashboard-proxcenter-guests.jsonGuest Workload
/integrations/grafana-dashboard-proxcenter-storage.jsonStorage
/integrations/grafana-dashboard-proxcenter-backups.jsonBackup Compliance

The Backup Compliance dashboard counts guests that have never been backed up.

Why that count is a series of its own

That number cannot be derived from backup age alone, because a guest with no backup has no age to report, so a dashboard built on age silently leaves the least protected guests out of its own compliance figure.

The scrape job​

Seeing the whole exposition takes four scopes: nodes:read, vms:read, storage:read and backups:read. Each governs its own families, and a token short of one still answers 200 without them. storage:read alone does not open the endpoint, so grant it alongside at least one of the other three:

scrape_configs:
- job_name: proxcenter
scheme: https
metrics_path: /api/v1/public/metrics
scrape_interval: 60s
scrape_timeout: 30s
authorization:
type: Bearer
credentials: pxc_REPLACE_WITH_YOUR_TOKEN
static_configs:
- targets:
- proxcenter.example.com
A partial token looks like a broken dashboard

A token missing one of the four scopes still answers 200. The families it may not see are filtered out of the response rather than refused, so the panels that need them render empty with no error anywhere. If a whole row of a dashboard is blank, or the Storage dashboard is empty end to end, check the token's scopes first.

Keep the 60 second interval. The values come from a cache refreshed on its own schedule, so a tighter scrape buys no fresher data.

Series reference​

Every series is a gauge unless marked as a counter, and every family is filtered independently by the calling token's scopes.

Counters need rate()

The four guest byte totals are cumulative since the guest started, which is why their names end in _total. Read them with rate(), never as a value: rate(proxcenter_vm_network_receive_bytes_total[5m]) gives bytes per second. Declared as counters, Prometheus handles the reset when a guest restarts instead of reading it as an enormous negative rate.

Build (no scope, any valid token)​

SeriesLabelsMeaning
proxcenter_build_infoversionProxCenter build information

Clusters (scope nodes:read)​

SeriesLabelsMeaning
proxcenter_cluster_upconnection, typeCluster or standalone node reachability (1 online, 0 otherwise)
proxcenter_cluster_degradedconnectionCluster in a degraded state (1 degraded, 0 otherwise)
proxcenter_cluster_ceph_healthconnection, healthCeph health as a state set; health is one of ok, warn, err, unknown. Clusters without Ceph emit nothing

Nodes (scope nodes:read)​

SeriesLabelsMeaning
proxcenter_node_onlineconnection, nodeNode online state (1 online, 0 otherwise)
proxcenter_node_cpu_usage_ratioconnection, nodeNode CPU usage ratio (0 to 1)
proxcenter_node_mem_usage_ratioconnection, nodeNode memory usage ratio (0 to 1)
proxcenter_node_mem_bytesconnection, nodeNode memory in use, in bytes
proxcenter_node_mem_total_bytesconnection, nodeNode memory capacity, in bytes
proxcenter_node_rootfs_usage_ratioconnection, nodeNode root filesystem usage ratio (0 to 1). This is the host root filesystem, NOT cluster storage capacity
proxcenter_node_uptime_secondsconnection, nodeNode uptime in seconds
proxcenter_node_maintenanceconnection, nodeNode in maintenance mode (1 in maintenance, 0 otherwise)
proxcenter_node_load1connection, nodeNode load average over one minute
proxcenter_node_load5connection, nodeNode load average over five minutes
proxcenter_node_load15connection, nodeNode load average over fifteen minutes
proxcenter_node_iowait_ratioconnection, nodeShare of node CPU time spent waiting on I/O (0 to 1)
proxcenter_node_swap_bytesconnection, nodeNode swap in use, in bytes
proxcenter_node_swap_total_bytesconnection, nodeNode swap capacity, in bytes; 0 when the node has no swap
proxcenter_node_rootfs_bytesconnection, nodeNode root filesystem in use, in bytes
proxcenter_node_rootfs_total_bytesconnection, nodeNode root filesystem capacity, in bytes
proxcenter_node_cpu_coresconnection, nodeCPU cores the node reports
proxcenter_node_infoconnection, node, pve_version, kernelNode build information: Proxmox VE version and running kernel

Guests (scope vms:read)​

SeriesLabelsMeaning
proxcenter_vm_statusconnection, node, vmid, name, typeGuest running state (1 running, 0 otherwise)
proxcenter_vm_cpu_usage_ratioconnection, node, vmid, name, typeGuest CPU usage ratio (0 to 1)
proxcenter_vm_agent_enabledconnection, node, vmid, nameGuest agent config flag (1 enabled, 0 otherwise)
proxcenter_vm_mem_usage_ratioconnection, node, vmid, name, typeGuest memory usage ratio (0 to 1)
proxcenter_vm_mem_bytesconnection, node, vmid, name, typeGuest memory in use, in bytes
proxcenter_vm_mem_total_bytesconnection, node, vmid, name, typeGuest memory allocation, in bytes
proxcenter_vm_uptime_secondsconnection, node, vmid, name, typeGuest uptime in seconds
proxcenter_vm_ha_stateconnection, node, vmid, name, type, stateGuest HA state as a state set; guests not managed by HA emit nothing
proxcenter_vm_cpu_coresconnection, node, vmid, name, typeVirtual CPU cores allocated to the guest
proxcenter_vm_mem_host_bytesconnection, node, vmid, name, typeGuest memory as the PVE 9 host accounts it, distinct from the guest-side figure; 0 for a container, which reports none
proxcenter_vm_network_receive_bytes_total (counter)connection, node, vmid, name, typeBytes received by the guest since it started
proxcenter_vm_network_transmit_bytes_total (counter)connection, node, vmid, name, typeBytes sent by the guest since it started
proxcenter_vm_disk_read_bytes_total (counter)connection, node, vmid, name, typeBytes read from the guest's disks since it started
proxcenter_vm_disk_written_bytes_total (counter)connection, node, vmid, name, typeBytes written to the guest's disks since it started

Storage (scope storage:read)​

SeriesLabelsMeaning
proxcenter_storage_total_bytesconnection, storage, type, sharedStorage capacity in bytes. A shared storage is reported ONCE for the cluster, never once per node
proxcenter_storage_used_bytesconnection, storage, type, sharedStorage space in use, in bytes. A shared storage is reported ONCE for the cluster, never once per node
proxcenter_storage_usage_ratioconnection, storage, type, sharedStorage usage ratio (0 to 1), served pre-computed so a consumer never divides by a zero capacity
proxcenter_storage_enabledconnection, storageStorage enabled state (1 enabled, 0 disabled); a disabled storage still reports its capacity
proxcenter_storage_node_total_bytesconnection, storage, nodeCapacity of a non-shared storage on one node. Shared storages emit nothing here, since their capacity is not per node
proxcenter_storage_node_used_bytesconnection, storage, nodeSpace in use of a non-shared storage on one node
proxcenter_storage_node_usage_ratioconnection, storage, nodeUsage ratio (0 to 1) of a non-shared storage on one node, served pre-computed

Backups (scope backups:read)​

SeriesLabelsMeaning
proxcenter_backup_age_secondsconnection, node, vmid, name, type, datastoreSeconds since the most recent backup of this guest
proxcenter_backup_protectedconnection, node, vmid, name, typeGuest backup coverage (1 has at least one backup, 0 has none)
Aggregate proxcenter_backup_age_seconds over the guest

The metric carries node, name and type next to connection, vmid and datastore, so a stale backup can be read without a join. In Prometheus a label is part of the series identity: a guest that migrates to another node starts a new series while the old one goes stale.

Aggregate over the guest explicitly rather than over the bare metric:

max by (vmid) (proxcenter_backup_age_seconds)

max by (connection, vmid) if several connections may carry the same VMID. The same applies to any recording rule or alerting rule built on this metric.

Proxmox Backup Server (scope backups:read)​

SeriesLabelsMeaning
proxcenter_pbs_upconnectionPBS server reachability (1 online, 0 otherwise)
proxcenter_pbs_infoconnection, versionPBS server build information
proxcenter_pbs_datastore_total_bytesconnection, datastoreDatastore capacity in bytes; 0 for a backend that does not report one, such as S3
proxcenter_pbs_datastore_used_bytesconnection, datastoreDatastore space in use, in bytes
proxcenter_pbs_datastore_available_bytesconnection, datastoreDatastore space available, in bytes
proxcenter_pbs_datastore_usage_ratioconnection, datastoreDatastore usage ratio (0 to 1), served pre-computed so a consumer never divides by a zero capacity
proxcenter_pbs_datastore_snapshotsconnection, datastoreSnapshots held in this datastore
proxcenter_pbs_datastore_guestsconnection, datastore, kindDistinct backup sources in this datastore, by kind (vm, ct, host)

Three families are absent rather than wrong, and every case is deliberate:

  • proxcenter_vm_agent_enabled is missing on most installs. ProxCenter builds its inventory from the Proxmox cluster/resources endpoint, which carries no guest agent configuration flag. The flag is therefore unknown, and an unknown is omitted rather than published as a misleading 0 that would report a working agent as absent.
  • proxcenter_vm_ha_state covers only the HA estate. A guest that HA does not manage emits nothing at all, so counting this family gives you the size of the HA estate, not of the fleet.
  • proxcenter_backup_age_seconds skips a guest that has never been backed up, because it has no age to report. That is exactly why proxcenter_backup_protected exists: it publishes a 0 for those guests so they can be counted.

Three values are surprising but correct:

  • A PBS datastore capacity of 0 means the storage backend does not report one, which is normal for a datastore backed by S3. The space in use is still real, which is why proxcenter_pbs_datastore_usage_ratio is served pre-computed: a dashboard dividing used by capacity would divide by zero.
  • A shared storage emits no per-node series. A Ceph pool or an NFS export has one capacity for the whole cluster, and Proxmox reports it once per node. Summing the raw figures would triple an RBD pool on a three node cluster, so proxcenter_storage_* carries shared storages once and only breaks non-shared ones down per node.
  • proxcenter_vm_mem_host_bytes reads 0 for a container. It is the PVE 9 host-side accounting of a guest's memory, which containers do not report.