'preserveTags' flag for divideSeries, divideSeriesLists, asPercent

[shanson7]

Is your feature request related to a problem? Please describe.

Several functions (at least divideSeries, divideSeriesLists, and asPercent) trash the tags of the inputs. This makes it difficult to use features in grafana that require tags (e.g. "Labels to Fields" transformation) or other functions like groupByTags post division.

It would be nice to have an option to preserve tags where the dividend and divisor have the same values.

Describe the solution you'd like

Add a flag (default to false for backwards compatibility) to preserve overlapping tags

Describe alternatives you've considered

N/A

[shanson7]

Perhaps even easier (more useful?) an option to keep the tags of the dividend for these functions, rather than strictly overlapping tags.

[deniszh]

Hi @shanson7,
Sorry, but could you please elaborate your request a bit? What do you mean "trash the tags of the inputs" ?
Maybe add some useful example which can be used in tests?

[shanson7]

What do you mean "trash the tags of the inputs" ?

Tags go in but they don't come out.

Maybe add some useful example which can be used in tests?

Sure, what format?

[deniszh]

Tags go in but they don't come out.

Sorry, I still not sure that I understand what that means. I do not use tags in production, though.

Sure, what format?

Example input, current output, desired output ?

[shanson7]

Example input, current output, desired output ?

Haha, yeah I meant what format do tests take for series/tags. I'll dig it out of the code.

[shanson7]

Ok so let's say we have 2 input series representing the number of requests and the total time taken (divisor and dividend respectively):

divisor.name = 'requests.count'
divisor.tags = {
  'host' : 'h1',
  'instanceName': 'instance1'
}

dividend.name = 'requests.seconds'
dividend.tags = {
  'host' : 'h1',
  'instanceName': 'instance1'
}

Currently, if we do divideSeries(dividend, divisor) we get a new series:

result.name = 'divideSeries(requests.seconds,requests.count)'
result.tags = {}

It would be desirable to maintain the tags like:

result.name = 'divideSeries(requests.seconds,requests.count)'
result.tags = {  
  'host' : 'h1',
  'instanceName': 'instance1'
}

You can expand this to divideSeriesLists where losing the tags makes the output series indistiguishable and we need to rely on functions like aliasByTags before the division to make it work. However, grafana is becoming more and more tag aware and features that rely on tags become inaccessible.

[DanCech]

imho the fact that the resulting series have no tags should be considered a bug, and I agree that we should propagate tags in roughly the same way as we do for other aggregation functions. My suggestion would be that by default all tags of the dividend series should be propagated, along with a new tag to be consistent with the behavior of other aggregation-type functions. We may want to consider also propagating tags on the divisor series that are not present on the dividend, whether by default or with an option flag. Does that seem reasonable?

[shanson7]

My suggestion would be that by default all tags of the dividend series should be propagated, along with a new tag to be consistent with the behavior of other aggregation-type functions. We may want to consider also propagating tags on the divisor series that are not present on the dividend, whether by default or with an option flag. Does that seem reasonable?

I get a little nervous about changing this type of behavior after the fact (bug or not) because some people might be actively working around it and it could break existing dashboards. However, the behavior you describe does sound reasonable to me (backward compatibility aside).

[Dieterbe]

imho the fact that the resulting series have no tags should be considered a bug, and I agree that we should propagate tags in roughly the same way as we do for other aggregation functions.

+1. I'm not too worried about accommodating people who may have worked around this lack of tags, as it is such as obvious bug; also tags are not commonly used yet. But ultimately this is a question for the graphite maintainers (which i'm not part of).

Note the asPercent signature: asPercent(seriesList, total=None, *nodes)
since *nodes swallows all remaining args, i don't think we can just add an arg at the end, we would probably add it in the middle (before or after total), because it would be an optional flag, this should be fine (in both graphite and metrictank)

Note that Dan suggests setting a new tag aggregatedBy: <func-name> as well as either of these:

• all dividend tags only
• all dividend tags + tags that are set only on divisor (this is effectively the union)

In that proposal, it is implied that if the same tag keys are set between dividend and divisor, but with different values, dividend wins. Keeping a tag value of one of the series as is, when the other series directly contradicts is, seems like a bad idea....

E.g. imagine a division between:

http_responses{service=A,code=200}

and

http_responses{service=A,code=total}

we have the choice between:

1. let dividend or divisor overrule the other. per Dan (service=A,code=200)
2. drop tag (service=A)
3. generate a new tag that's the unison of them (service=A,code=200/total)

2 is probably not that useful (we may drop precisely what distinguishes different series), Dan's idea is actually not so bad (we typically look at a series "with respect to" another, where the "another" is implied and keeping only the former explicit may be good enough.), the 3rd one seems most factually correct because it acknowledges the contribution of both components of the division. but the naive formatting may be hard to decipher if either of the tag values already contain a slash.

Anyway, that tangent aside, I note that there's two more options to consider to dan's proposal:

• intersection between dividend and divisor (key-wise intersection that is, value is subject to the same question above)
• divisor tags only

I agree the latter isn't all that useful e.g. in asPercent() when we have multiple distinct dividends and a single divisor, we would want to distinguish based on divident tags; whereas in other cases where we map dividends and divisors 1:1 (this happens in divideSeries, divideSeriesList and certain asPercent invocations), I think "divisor tags only" is just as defensible as "dividend tags only".

Thus I think all these modes make sense:

• dividend only
• divisor only
• union
• intersection

and then i think it would make sense to have an additional flag to control the behavior in the latter 2 cases, in case of tag-matches with value mismatches:

• drop
• merge
• divisor wins
• dividend wins

[deniszh]

Agreed with @DanCech here - tags should not be swallowed by default.
Not sure do we need to implement full spectrum of modes/flags, though. But I think we need listen what users of tags functionality want - otherwise indeed implement full fledged solution as mostly flexible, as @Dieterbe described.

[shanson7]

We have been using tags for a while now, so I expect many dashboards currently "work" with the existing behavior. I expect many of those will be negatively impacted by a backwards incompatible change (not to mention any code that is making queries using these functions).

That being said, I'm not strictly against a backwards incompatible change, it just shouldn't be done lightly. The issue might be significantly more prevalent if you expand it to all functions that aggregate/combine multiple series, e.g multiplySeries or even sum/avg/etc.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Dieterbe]

We plan to provide a concrete proposal for this feature after the holidays