SDK Error: the namespace lacks label \"katib.kubeflow.org/metrics-collector-injection: enabled\"

[helenxie-bit]

What happened?

When I try to use the tune API in katib python SDK to run the get started example, it returned the following error message:

kubernetes.client.exceptions.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7775619d-f51b-40c5-a978-9f605e902582', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '0982521a-d333-4c33-a7f1-0e7d2819c6e9', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'ddf2fccb-7ad4-44ef-9a0c-b725ea4099a9', 'Date': 'Fri, 17 Jan 2025 03:46:16 GMT', 'Content-Length': '336'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"validator.experiment.katib.kubeflow.org\" denied the request: Cannot create the Experiment \"tune-experiment\" in namespace \"kubeflow\": the namespace lacks label \"katib.kubeflow.org/metrics-collector-injection: enabled\"","code":400}

What did you expect to happen?

Successfully create and finish the Experiment.

Environment

Kubernetes version:

$ kubectl version

Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

Katib controller version:

$ kubectl get pods -n kubeflow -l katib.kubeflow.org/component=controller -o jsonpath="{.items[*].spec.containers[*].image}"

docker.io/kubeflowkatib/katib-controller:latest% 

Katib Python SDK version:

$ pip show kubeflow-katib

Name: kubeflow-katib
Version: 0.17.0
Summary: Katib Python SDK for APIVersion v1beta1
Home-page: https://github.com/kubeflow/katib/tree/master/sdk/python/v1beta1
Author: Kubeflow Authors
Author-email: [email protected]
License: Apache License Version 2.0
Location: /opt/homebrew/anaconda3/envs/katib-llm-test/lib/python3.12/site-packages
Requires: certifi, grpcio, kubernetes, protobuf, setuptools, six, urllib3
Required-by: 

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[helenxie-bit]

/remove-label lifecycle/needs-triage
/good-first-issue
/area api

[google-oss-prow]

@helenxie-bit:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/remove-label lifecycle/needs-triage
/good-first-issue
/area api

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[helenxie-bit]

/assign