You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: proposals/2025-03-25_otel-delta-temporality-support.md
+13-7Lines changed: 13 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -129,6 +129,12 @@ No scraped metrics should have delta temporality as there is no additional benef
129
129
130
130
Delta metrics will be filtered out from metrics being federated. If the current value of the delta series is exposed directly, data can be incorrectly collected if the ingestion interval is not the same as the scrape interval for the federate endpoint. The alternative is to convert the delta metric to a cumulative one, which has issues detailed above.
131
131
132
+
### Handling missing StartTimeUnixNano
133
+
134
+
StartTimeUnixNano is optional in the OTEL spec. To ensure compatibility with the OTEL spec, this case should be supported. Also note that before implementing CT-per-sample, every sample will be missing StartTimeUnixNano.
135
+
136
+
For functions that require an interval to operate (e.g. `rate()`/`increase()`), assume the spacing between samples is the ingestion interval when StartTimeUnixNano is missing.
137
+
132
138
### Querying deltas
133
139
134
140
*Note: this section likely needs the most discussion. I'm not 100% about the proposal because of issues guessing the start and end of series. The main alternatives can be found in [Querying deltas alternatives](#querying-deltas-alternatives), and a more detailed doc with additional context and options is [here](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.3zt1m2ezcl1s).*
@@ -147,7 +153,9 @@ While the intention is to eventually use `rate()`/`increase()` etc. for both del
147
153
148
154
TODO: write some code to make this clearer
149
155
150
-
In general: `sum of all sample values / (last sample ts - first sample start ts)) * range`. If the start time of the first sample is outside the range, truncate the first sample.
156
+
In general:
157
+
* If CT-per-sample is available: `sum of all sample values / (last sample ts - first sample start ts)) * range`. If the start time of the first sample is outside the range, truncate the first sample.
158
+
* If CT-per-sample is not available: `sum of second to last sample values / (last sample ts - first sample ts)) * range`. We skip the value of the first sample value as we do not know its interval.
151
159
152
160
The current `rate()`/`increase()` implementations guess if the series starts or ends within the range, and if so, reduces the interval it extrapolates to. The guess is based on the gaps between gaps and the boundaries on the range.
153
161
@@ -157,6 +165,10 @@ We could just not try and predict the start/end of the series and assume the ser
157
165
158
166
Assuming `rate()` only has information about the sample within the range, guessing the start and end of series is probably the least worst option - this will at least work in delta cases where the samples are continuously ingested. To predict if a series has started ended in the range, check if the timestamp of the last sample are within 1.1x of an interval between their respective boundaries (aligns with the cumulative check for start/end of a series).
159
167
168
+
To calculate the interval:
169
+
* If CT-per-sample is available, use the average of the intervals of the samples (i.e. TimeUnixNano - StartTimeUnixNano).
170
+
* If CT-per-sample is not available, use the average spacing between samples.
171
+
160
172
As part of the implementation process, experiment with heuristics to try and improve this (e.g. if intervals between samples are regular and there are than X samples, assume the samples are continuously ingested and therefore a gap would mean the series ended). This would make the calculation more complex, however.
161
173
162
174
With CT-per-sample, we do not need to predict the start of the series, as if a sample is before the range start, it can't be within in the range at all.
@@ -181,12 +193,6 @@ As the samples are written at TimeUnixNano, only S1 and S2 are inside the query
181
193
182
194
One possible solution would to have a function that does `sum_over_time()` for deltas and the cumulative equivalent too (this requires subtracting the latest sample before the start of the range with the last sample in the range). This is outside the scope of this design, however.
183
195
184
-
### Handling missing StartTimeUnixNano
185
-
186
-
StartTimeUnixNano is optional in the OTEL spec. To ensure compatibility with the OTEL spec, this case should be supported. Also note that before implementing CT-per-sample, every sample will be missing StartTimeUnixNano.
187
-
188
-
For functions that require an interval to operate (e.g. rate()/increase()), assume the spacing between samples is the ingestion interval when StartTimeUnixNano is missing.
0 commit comments