Skip to content

Conversation

@morenod
Copy link

@morenod morenod commented Oct 15, 2025

As part of the openstack [watcher](https://opendev.org/openstack/watcher) tests we are injecting metrics on prometheus to simulate CPU and memory load on openstack instances.

we need to delete old metrics to avoid errors calculating node resource utilizations.

To inject metrics, only remotewritter is needed, but to delete them, we need the adminAPI.

We are enabling it by patching prometheus on post_scripting, after controlplane is created, using:

oc patch prometheuses.monitoring.rhobs metric-storage --namespace=openstack --type=merge -p '{"spec":{"enableAdminAPI":true}}' here

Having the possibility of adding this parameter to the control-plane kustomize variables at the installation time would avoid to need the post scripting, making edpm jobs easier to maintain (no need to patch, undeploy prometheus, waiting for it to be available...)

I have followed the same logic than used to add enableRemoteWriteReceiver parameter, which is also used by our tests to inject metrics.

I'm also creating a similar PR on telemetry-operator project, not sure which will require the other to be first merged

@openshift-ci
Copy link

openshift-ci bot commented Oct 15, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: morenod
Once this PR has been reviewed and has the lgtm label, please assign danielmellado for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
Copy link

openshift-ci bot commented Oct 15, 2025

Hi @morenod. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@morenod
Copy link
Author

morenod commented Oct 21, 2025

cc @danielmellado

@danielmellado
Copy link
Contributor

/ok-to-test

},
RemoteWrite: config.RemoteWrite,
ExternalLabels: config.ExternalLabels,
EnableAdminAPI: config.EnableAdminAPI,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a deeper look, this just won't work. If you check in https://github.com/rhobs/observability-operator/blob/main/pkg/controllers/monitoring/monitoring-stack/components.go#L150 you'll notice that there's a mapping fom this API to the prometheus-operator one. As prom-op doesn't have anything about that, it'll just fail. You'd need to couple this change with another one adding the feature to prometheus-operator (specifically, to the downstream fork used here)

@morenod morenod force-pushed the prometheus_enable_admin_api branch 2 times, most recently from 2e42e2b to 6130562 Compare October 21, 2025 16:01
@morenod morenod force-pushed the prometheus_enable_admin_api branch from 6130562 to 8ca0b03 Compare October 21, 2025 16:05
@morenod morenod force-pushed the prometheus_enable_admin_api branch 2 times, most recently from 8be8e95 to 3552875 Compare October 22, 2025 08:52
go.mod Outdated

replace github.com/openshift/api => github.com/openshift/api v0.0.0-20240404200104-96ed2d49b255

replace github.com/rhobs/observability-operator/pkg/apis => ./pkg/apis
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this may bring issues as there's a mapping in between the apis and the observability operator ones. Let's see if the images build but I'll take a deeper look.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've just discussed it with @danielmellado. Since we (ok it was me) split the API (into a new Go module), I think we need to split this into two PRs (so that this replacement is not needed):

  1. introducing/updating the API changes
  2. introducing the "impl" changes with API dependency update (previous commit updating the API)

Does it make sense?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so I have updated the files here to only contain the changes on API

@morenod morenod changed the title Add parameter enableAdminAPI to prometheus configuration [DRAFT] Add parameter enableAdminAPI to prometheus configuration Oct 22, 2025
@danielmellado
Copy link
Contributor

After checking this a bit more in detail,two things here. Yes, you can rely on the fork but I'd split this into two. Get the api in first so you can get the new pseudo commit API version there in. Thanks!

@morenod morenod force-pushed the prometheus_enable_admin_api branch from 3552875 to 012e3cb Compare October 22, 2025 14:15
@morenod
Copy link
Author

morenod commented Oct 23, 2025

After checking this a bit more in detail,two things here. Yes, you can rely on the fork but I'd split this into two. Get the api in first so you can get the new pseudo commit API version there in. Thanks!

CI is passing now just with the changes in the API

@morenod morenod changed the title [DRAFT] Add parameter enableAdminAPI to prometheus configuration Add parameter enableAdminAPI to prometheus configuration Oct 23, 2025
@morenod
Copy link
Author

morenod commented Oct 27, 2025

@tremes @danielmellado anything else missing to merge this?

Add the EnableAdminAPI boolean field to the PrometheusConfig struct
to allow users to configure whether the Prometheus Admin API should
be enabled.

This commit only adds the API definition without implementing its
usage in the controller. The field includes appropriate validation
and documentation warning about security implications.

Signed-off-by: morenod <[email protected]>
@morenod morenod force-pushed the prometheus_enable_admin_api branch from 012e3cb to 181d599 Compare October 27, 2025 09:44
@simonpasquier
Copy link
Contributor

Since #941 we don't need to split an API change into 2 PRs. Can you rebase on main?
Having said that, I'm not sure that we want to land the change

  1. as you've mentioned, server-side apply allows to override the field.
  2. enabling it without proper authn/authz is a security risk.

@openshift-merge-robot
Copy link
Collaborator

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants