Expose database/table/volume/node storage sizes as gauges

[sleekmountaincat]

What

Surface Harper's storage-size analytics as Prometheus gauges. Harper already records database-size, table-size, and storage-volume into hdb_analytics on every aggregation cycle (v4 and v5; plus node-storage on v5), but the exporter never mapped those records, so no size metric exists anywhere in its output today. New gauges:

Metric	Labels	Source record / field
harperdb_database_size_bytes	database	database-size .size (RocksDB: sum of .sst; LMDB: data file size)
harperdb_database_audit_size_bytes	database	database-size .transactionLog (RocksDB) / .audit (LMDB)
harperdb_database_used_bytes	database	database-size .used (LMDB only; set only when present)
harperdb_database_free_bytes	database	database-size .free (LMDB only; set only when present)
harperdb_table_size_bytes	database, table	table-size .size
harperdb_node_storage_bytes	—	node-storage .size (whole HDB dir)
harperdb_database_volume_size_bytes	database	storage-volume .size (statfs of the DB's volume)
harperdb_database_volume_free_bytes	database	storage-volume .free
harperdb_database_volume_available_bytes	database	storage-volume .available

Mechanically: gauge definitions beside the existing harperdb_database_* block, resets in get() with the others (the stuck-value guard), and four new cases in generateMetricsFromAnalytics's switch ahead of default:.

Why not customMetrics

These records cannot be surfaced through the existing customMetrics setting: that path renders the quantile-summary shape (p1/p10/median/mean/count), and the size records are plain {database, size} snapshots — configured that way it emits all zeros, silently. Plain gauges are the correct representation. (A follow-up worth its own issue: teach customMetrics a gauge mode so record shapes like these are configurable without code changes. Also noticed in passing: the filesystem_*_bytes gauges are defined and reset but never set anywhere — dead code from an earlier iteration; left untouched here.)

Notes

• Field-name variance between storage engines is handled: transactionLog ?? audit, and used/free set only when present, so RocksDB databases don't emit fake zero series.
• harperdb_node_storage_bytes needs special treatment: node-storage analytics are interval-gated (analytics.storageInterval, off by default), and a registered-but-never-set unlabeled prom-client gauge renders 0 — a fake "node storage: 0 bytes" on every instance with the interval disabled. The gauge is therefore removed from the registry each scrape and re-registered only when a record is present, so it is absent rather than zero when the data doesn't exist. (Labeled gauges get this behavior for free; verified both behaviors against prom-client directly.)
• Values inherit the file's existing freshness semantics (records searched over the last 1.5 aggregation windows, earliest-in-window wins), so a value can be up to ~one window old — fine for sizes.
• Cardinality is bounded by schema size (one series per database, one per table).

Verification (live, harpersystems/harper container + this component mounted)

• node --check resources.js clean.
• Seeded a data.widgets table with 5 records, waited an aggregation cycle, scraped with the admin credential and Accept: text/plain (note for anyone testing by hand: without a text Accept, the REST layer JSON-encodes the whole exposition as one string):
  • harperdb_database_size_bytes{database="data"} 151552 / {database="system"} 532480
  • harperdb_database_used_bytes / _free_bytes populated (LMDB branch), harperdb_database_audit_size_bytes matches the analytics records exactly
  • harperdb_table_size_bytes{database="data",table="widgets"} 24576 (and one series per system table)
  • harperdb_database_volume_{size,free,available}_bytes match the container volume's statfs
• The container run doubles as a v4 compatibility check: harpersystems/harper:latest is 4.7.38, so the LMDB-branch fields (used/free/audit) above are v4's own writer output.
• harperdb_node_storage_bytes: the 4.7.38 image has no node-storage writer (v5 feature), which exercised the absent case end to end — the gauge is absent, not a fake zero. The present case is covered two ways: the remove/re-register idiom verified directly against prom-client (register -> set -> renders the value), and the record shape it consumes ({metric: "node-storage", size}) verified against live production analytics on a v5 fabric instance (fresh records, size 1.46GB).
• Same production instance (Walmart usgm-er-teflon) confirms the RocksDB-branch database-size shape ({database, size, transactionLog}) maps onto the same cases.

Motivation

Walmart asked for per-database size visibility. With this change the series flow automatically through the existing fleet scrape of /prometheus_exporter/metrics (no pipeline changes) on every cluster where the exporter is converged — including both Walmart clusters — and become alertable/trendable in Mimir. (Their other route, the Grafana Connector's get_analytics panels, already charts table-size live per cluster; this PR is the fleet-metrics half.)

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request adds several Prometheus gauges to track storage-size snapshots (such as database, table, node storage, and volume sizes) populated from analytics records. The feedback suggests a more efficient and idiomatic approach to handling the unlabeled node_storage_gauge by using its built-in remove() method to clear the series instead of dynamically unregistering and re-registering the entire metric from the Prometheus registry.

[gemini-code-assist]

Unregistering and re-registering a metric on every scrape is an anti-pattern in Prometheus client libraries and can lead to performance overhead or registry pollution. Instead of removing the entire metric from the registry, you can use the built-in remove() method on the gauge itself. For an unlabeled gauge, calling node_storage_gauge.remove() will remove the unlabeled series from the gauge's internal map, preventing it from being exported when no data is present, while keeping the metric registered.

    // node_storage_gauge is unlabeled, and an unlabeled prom-client gauge that
    // has been registered renders `0` even when never set. node-storage
    // analytics are interval-gated (analytics.storageInterval) and absent on
    // instances that disable them, so a registered-but-unset gauge would report
    // a fake "0 bytes". Remove the unlabeled series from the gauge instead of
    // unregistering/re-registering it, which is more efficient and idiomatic.
    node_storage_gauge.remove();

[gemini-code-assist]

Since we are now using node_storage_gauge.remove() to clear the unlabeled series during resets, we no longer need to dynamically check and re-register the metric in the registry when processing the analytics record.

        case "node-storage":
          gaugeSet(node_storage_gauge, {}, metric.size);
          break;

[sleekmountaincat]

v5 verification (harper 5.2.7 via npm, RocksDB default, TLS endpoints)

Ran the same live test against v5.2.7 (npm i harper@5.2.7, component symlinked, prod-default config). Results, plus two upstream findings the test surfaced:

Working on v5 / RocksDB branch:

• harperdb_database_size_bytes{database="system"} 47185, harperdb_database_audit_size_bytes from the {size, transactionLog} record shape; harperdb_database_volume_{size,free,available}_bytes populated. used/free correctly absent (LMDB-only fields, conditional sets).
• harperdb_node_storage_bytes 1717640 present on every consecutive scrape after the follow-up commit (see below), including immediately after a restart with the newest record ~20 min old.

Design change in the follow-up commit (node-storage): the windowed search can't carry this metric — node-storage records are interval-gated upstream (default every 10th aggregation cycle) so their cadence is far longer than the 1.5-window search, and the gauge flapped absent between writes (would be present on only ~15% of fleet scrapes). It now fetches the newest record directly (id-descending, first match) and renders the latest known measurement every scrape; absent only when no record exists at all.

Upstream findings (Harper core, not this component):

1. analytics.storageInterval: 1 disables node-storage entirely: the gate is ++count % interval !== 1, and x % 1 is never 1. Verified live (interval=1 wrote nothing; interval=2 and the default 10 write on cycle 1). Off-by-one worth a Harper issue.
2. table-size records are written only by the LMDB branch of storeDBSizeMetrics, so harperdb_table_size_bytes will not populate on RocksDB-backed databases (verified: zero table-size records on both local 5.2.7 and a production harper-pro 5.2.0 instance). It populates fully on v4/LMDB (34 series in the v4 container test). Per-database size works on both engines.
3. RocksDB database-size counts only flushed .sst bytes — a young database reads size 0 while its data sits in WAL/memtable (the local data db reported 0 with 5 records inserted; the record itself says 0, faithfully rendered). Worth knowing before anyone alerts on small absolute values.

Combined coverage: v4.7.38/LMDB e2e (all gauges incl. per-table) + v5.2.7/RocksDB e2e (all applicable gauges incl. node-storage present-case) + prod record-shape cross-checks on harper-pro 5.2.0.