Add setting for filtering empty containers on Prometheus queries

[jweak]

Adds a setting for applying the old Prometheus queries which were changed in this PR #6679

This change made it so that some setups are observing metrics as doubled. container!="" does filter those values.

We have this change only for operator provider since it's the only one that had that change. We'll show the setting for Auto Detect or Operator values since those are the ones that could lead into using the operator Prometheus provider.

#7299

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[dragoangel]

Can this PR get merged?

[ErikLundJensen]

Any progress? Would be great to get it into the next release.

[Nova-Logic]

@jweak please resolve the conflicts, community really need this

[Nova-Logic]

@jim-docker @ixrock please review this PR

[pasztorl]

Hi! There is any update on this please?

[2martens]

It appears that this repo is no longer maintained.

[Nova-Logic]

very sad to see that this project is dead

[roman-parkhunovskyi]

This is a must have.

[pasztorl]

lens is not dead, official site links to this repository, but I can't see any communication about how this repo maintained now and the future. I hope lens crew will announce something about this.

[circlingthesun]

I've got kube-prometheus-stack configured and it has container!="" configured on all the graphana dashboards. Would be good to have this on lens.

[circlingthesun]

The bundled lens metrics doesn't have this issue, so I did some more digging and found:

https://stackoverflow.com/questions/69281327/why-container-memory-usage-is-doubled-in-cadvisor-metrics/69282328#69282328

So I modified my helm values for kube-prometheus-stack with:

kubelet:  
  serviceMonitor:
    cAdvisorMetricRelabelings:
      - sourceLabels: [container, __name__, pod]
        regex: ";container_.+;.+"
        action: drop

And that took care of the duplicate metrics for kube-prometheus-stack as the above thread suggested.

[dragoangel]

The bundled lens metrics doesn't have this issue, so I did some more digging and found:

https://stackoverflow.com/questions/69281327/why-container-memory-usage-is-doubled-in-cadvisor-metrics/69282328#69282328

So I modified my helm values for kube-prometheus-stack with:

kubelet:  
  serviceMonitor:
    cAdvisorMetricRelabelings:
      - sourceLabels: [container, __name__, pod]
        regex: ";container_.+;.+"
        action: drop

And that took care of the duplicate metrics for kube-prometheus-stack as the above thread suggested.

This is the point, that's a proper way is just to use container!="" and do not change metrics in way they work.

[circlingthesun]

@dragoangel yeah. This seems to be a pretty simple thing to fix, but it's been over a year and a half. So I prefer to work around this rather than look at bad metrics for the foreseeable future.