Exporter sends stale values instead of no value, from bursty source

[chadmiller]

I have data source, power meters, who report at random times. I put their values into a Gauge.

Some meters report often, every 30 seconds. Some report every 12 hours.

When the data arrives at least a frequently as the prometheus time-frame interval, there is no weirdness. When the time between data points is longer, the exporter reports the last reported value, which looks like it's asserting that the value hasn't changed, when in fact it has no idea if the value has changed.

I am kind of new to Prometheus and I may be doing something wrong, I admit, but I think I can get a better idea from truthful time reports

1000,null,null,null,null,null,null,3000,null,null,null,null,null,null,null,null,null,null,null,null,8000

than a series of mostly lies

1000,1000,1000,1000,1000,1000,1000,3000,3000,3000,3000,3000,3000,3000,3000,3000,3000,3000,3000,3000,8000

from prometheus_client import start_http_server, Counter, Gauge

power_gauge = Gauge('powermeter_readings_kwh', 'power consumption', ["meterid"])

start_http_server(8000)

for reading in source():
    power_gauge.labels(meterid=reading.get("ID")).set(reading.get("Consumption"))

[csmarchbanks]

Hello, I converted this to a discussion as this behavior is by design, not an issue with the client.

Typically, when you want to create an exporter collecting data from another source you will typically want to use a Custom Collector to create the values at scrape time so the information is correct for that moment. You can also specifically call power_gauge.remove('<some-meterid>) to remove a metric that has disappeared, but keeping the state is usually harder than populating the state correctly during each scrape using a custom collector.

Hope that helps!

[chadmiller]

Hi, csmarchbanks. Thank you. Is it a bug that gaugeinstance.remove(string) isn't documented? It's news to me. And the parameter is a mystery so far.

I guess I'll answer this by trying it: I can remove a metric "that has disappeared", but does that mean something more? Essentially, I have an unscrapable system that emits gauge-values at random intervals. Nothing has "disappeared". I'm searching for the best way to represent these values.

[csmarchbanks]

I think it is an oversight that remove is not documented, though its usage is rare in most code. Adding/removing series constantly can cause a lot of churn in a Prometheus server. I think it is certainly reasonable to add documentation under the Labels section in the README though.

The best solution is a bit up to you. As you noticed by default the last value is kept, which can be helpful in many operational cases and means your time series will remain connected, but is misleading if the value has not been reported in a long time. If you remove a metric, Prometheus will mark that series as stale, and stop returning data for the series when queried, providing closer to the null solution you mention, though in reality there would just be no data at all for that timestamp, not a null. You can read more about staleness in the docs: https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness.

[Sandwich1699975]

Having this same problem as well. Were you ever able to find a nice solution or was there ever some documentation released @csmarchbanks ?

I am not sure how to approach this problem almost 3 years later. If I was to use the power_gauge.remove('<some-meterid>') would you be able to point me towards an example? I can't find any documentation on it. Thank you