Enhancement: Allow Argo Label customization

[Capiru]

If we could add labels to argo workflows, we could more easily group workflows in the Argo UI.

Current state:
I see that here the labels are only being built using _base_labels:

metaflow/metaflow/plugins/argo/argo_workflows.py

Line 892 in 13a7cad

.labels(self._base_labels)

However, _base_labels is only being built from _base_kubernetes_labels: https://github.com/Netflix/metaflow/blob/13a7cad8081aff1f34db8d4e663d8940e65ebbdc/metaflow/plugins/argo/argo_workflows.py#L189C34-L189C57
Since _base_kubernetes_labels is effectively hardcoded, there's no customization possible in the labels, i see also that there's a TODO exactly what i would suggest here, that there is an environment variable to use here:

metaflow/metaflow/plugins/argo/argo_workflows.py

Lines 402 to 409 in 13a7cad

	def _base_kubernetes_labels(self):
	"""
	Get shared Kubernetes labels for Argo resources.
	"""
	# TODO: Add configuration through an environment variable or Metaflow config in the future if required.
	labels = {"app.kubernetes.io/part-of": "metaflow"}
	return labels

Proposed Solution:
Add METAFLOW_ARGO_WORKFLOWS_LABELS. Then use it to set the labels.

Example PR: #2781

[Sridhar1030]

Hi, I’m interested in working on this issue.

From my understanding, the current implementation only applies labels from _base_kubernetes_labels, which is effectively static. Adding support for a configurable environment variable like METAFLOW_ARGO_WORKFLOWS_LABELS would allow users to attach custom labels to Argo Workflows and make grouping/filtering in the Argo UI much easier.

My proposed approach:

• Introduce support for a new environment variable METAFLOW_ARGO_WORKFLOWS_LABELS.
• Accept labels in a key=value comma-separated format (e.g., team=data,env=prod).
• Parse and merge these labels with _base_kubernetes_labels, ensuring base labels are preserved.
• Add validation to ensure Kubernetes label format compliance.
• Include documentation and tests for the new configuration.

Before starting, I’d like to confirm:

1. Does the comma-separated key=value format work, or would JSON/YAML be preferred?
2. Should custom labels override base labels if the same key is provided?
3. Are there any existing configuration patterns in Metaflow that I should follow for environment-based settings?

If this sounds good, I’ll go ahead and start working on a PR.

[Sridhar1030]

/assign