feature: allow custom pod labels in notebooks, pipeline components, serving pods

[shalberd]

/kind feature
@thesuperzapper @juliusvonkohout @franciscojavierarceo FYI, copying you here because you seem to have been in environments where corporate-level requirements are common.

see also opendatahub-io/kubeflow#494

Why you need this feature:
Corporate-level admission controllers like https://kyverno.io that do not allow any pod to be created unless it has certain corporate policy-based labels.

Is there a way in kubeflow notebooks to set common labels for all workbench pods created?
We have this requirement that all pods created dynamically and in general need to have certain corporate label key-value combinations set.
At notebook / statefulSet spec.template.metadata.labels.

I am currently patching derived statefulsets' podSpec with our labels by means of a cronjob so our label enforcement tool / admission controller Kyverno allows pods to be created. That is just a workaround, though.

Apache Airflow has thought of that topic ( common labels) for all its KubernetesExecutor related resources and even for the static Helm Deployment. However, I think this additional / custom pod labels requirement is currently not implemented in notebooks, or even kubeflow pipelines, for that matter, globally.

At kubeflow upstream https://www.kubeflow.org/docs/components/notebooks/quickstart-guide/, I see mention of PodDefault Resource and profiles that could well do the job ... what do you think? Is something like this on the radar for ODH in terms of kind:PodDefault i.e. in ODH kubeflow notebook 2.0 ?

BUT I am not sure though if that concept applies to setting additional labels in a pod metadata as well.
Cause the similar concept of podPreset says: "Note: A Pod Preset is capable of modifying the .spec.containers field in a Pod spec when appropriate. No resource definition from the Pod Preset will be applied to the initContainers field."
https://cjyabraham.gitlab.io/docs/concepts/workloads/pods/podpreset/
In any case, that requirement for additional labels is often implemented in software not just Airflow ... so I'd be surprised if there were no way to achieve this by means of kubeflow.

Describe the solution you'd like:
I need to have a way to centrally define and configure additional pod (not notebook) labels for pods derived from a notebook spec / statefulset. This should be the case for pods spawned by kserve, for component pods related to pipelines, pods related to notebooks ....
Then, notebook controller needs to add those labels to the notebook podSpec and to the statefulSet podSpec on reconciliation.

Anything else you would like to add:
Same issue applied to spawned objects / pods in kubeflow pipelines as well.
Is there a way somewhere to specifiy additional pod labels including their values that are set when pods are spawned dynamically.
There is a discussion on how to do this programmatically in pipelines API,
kubeflow/pipelines#11014
but I am talking here about a centralized, CR entry and controller-driven approach instead.

[thesuperzapper]

For notebooks 2.0, yes it is possible to add common labels for Notebook Pods created under a specific WorkspaceKind, see comment here:

• Kubeflow Notebooks 2.0 (Kubeflow Workspaces) notebooks#85 (comment)

[shalberd]

Hi @thesuperzapper that is very nice to see regarding notebooks 2.0. Do you also have a hint regarding support for universal templates, i.e. for labels, for KF pipelines, serving ...?

[juliusvonkohout]

Hi @thesuperzapper that is very nice to see regarding notebooks 2.0. Do you also have a hint regarding support for universal templates, i.e. for labels, for KF pipelines, serving ...?

You might want to take a look at kyverno for such tasks.

[shalberd]

@juliusvonkohout meaning there are no plans to include a template kind of CR/CRD, similar to the notebooks 2.0 approach, in kubeflow pipelines? It is not Kyverno's main task to do such overall config, yes, sure, it can patch before admission, but that is merely a workaround. See also discussion at opendatahub-io/kubeflow#494 (comment)

Instead if splitting up configuration and running instances / CRs, as notebooks does in 2.0:
Another option might be, as Airflow does, to include a podTemplate yaml File, mapped-in via ConfigMap, in whatever controller pipelines uses, and then when spawning new pipeline components, that podTemplate file (which includes the metadata section) is used as a global basis for all pipelines and pipeline components / tasks. Not just for the podSpec, for which I think there are already solutions, but also for pod metadata like labels and annotations.

Of course, in the API, additional labels can always be added per-pipeline or per task/component, in addition to what is defined globally.

Either way, a way to configure globally things like labels and annotations for pipeline components would be great.