PVO11Y-5347 Deploy cardinality exporter to production (ring 1) by kubasikus · Pull Request #13021 · redhat-appstudio/infra-deployments

kubasikus · 2026-07-14T06:41:48Z

Summary

Deploy cardinality exporter to production ring 1 clusters (stone-prod-p01, kflux-prd-rh02):

Create production base, empty-base (no-op overlay), and per-cluster overlays for ring 1 clusters. Remaining production clusters are onboarded in the ApplicationSet with empty-base until promoted in later rings.
Remove $patch: delete for monitoring-cardinality from both production ArgoCD overlays.

Follows the same gradual rollout pattern as konflux-rbac using empty-base for clusters not yet promoted.

Risk Assessment

Level: Low
Description: The cardinality exporter is a read-only tool that queries Prometheus TSDB status API every 6 minutes. Validated in staging since March 2026. Deploying to 2 production clusters in ring 1.
Rollback: Point ring 1 clusters to empty-base in the ApplicationSet list generator.

Validation

Cardinality exporter validated in staging for 4+ months
kustomize build for ring 1 overlays renders correctly; empty-base renders 0 resources
Follows the same deployment pattern as kaexporter and konflux-rbac

github-actions · 2026-07-14T06:42:15Z

Kustomize Render Diff

Comparing 8ad61ac0c → 2d3feefa7

No render differences detected.

qodo-for-redhat-appstudio · 2026-07-14T06:43:58Z

Qodo is busy working

Check back in a few minutes. Qodo's code review agents are on it.

qodo-for-redhat-appstudio · 2026-07-14T06:47:41Z

PR Summary by Qodo

Deploy cardinality exporter to production clusters

✨ Enhancement ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add production ApplicationSet entries for all 9 production clusters.
• Enable monitoring-cardinality deployment by removing production overlay delete patches.
• Introduce production kustomize base (RBAC + Deployment + ServiceMonitor) for cardinality exporter.

Diagram

graph TD
  A["monitoring-cardinality ApplicationSet"] --> B["Argo CD"] --> C["Prod cluster overlays"] --> D["Prod base manifests"] --> E["Kubernetes clusters"]
  D --> F["Cardinality exporter"] --> G["Cluster Prometheus"]
  G --> F

High-Level Assessment

The following are alternative approaches to this PR:

1. Single prod overlay + cluster inventory (no per-cluster dirs)

➕ Fewer directories/files to maintain for new clusters
➕ Centralizes production configuration changes
➖ Requires additional generator variables or inventory plumbing
➖ Harder to apply cluster-specific patches when they inevitably arise

2. ArgoCD matrix generator (env × cluster) from a shared inventory source

➕ Reduces manual list maintenance in the ApplicationSet
➕ Scales better as cluster count grows
➖ More moving parts (inventory source-of-truth, generator templating)
➖ Potentially inconsistent with existing repo patterns (e.g., kaexporter-style overlays)

Recommendation: Proceed with the PR’s approach: it follows the established per-cluster overlay pattern, keeps production enablement explicit (removing $patch: delete), and minimizes rollout risk for a new production deployment. Alternatives improve scalability but add complexity and are better handled as a dedicated follow-up refactor if cluster inventory management becomes a recurring burden.

Files changed (22) +246 / -14

Other (22) +246 / -14

monitoring-cardinality.yamlAdd production clusters to monitoring-cardinality ApplicationSet generator +18/-0

Add production clusters to monitoring-cardinality ApplicationSet generator
• Extends the ApplicationSet list generator to include nine production cluster entries. This allows Argo CD to render per-cluster Applications for production overlays.
argo-cd-apps/base/member/infra-deployments/monitoring-cardinality/monitoring-cardinality.yaml

delete-applications.yamlStop deleting monitoring-cardinality ApplicationSet in konflux-public-production +0/-7

Stop deleting monitoring-cardinality ApplicationSet in konflux-public-production
• Removes the staging-only $patch: delete block for the monitoring-cardinality ApplicationSet. This enables the ApplicationSet to be deployed in this production overlay.
argo-cd-apps/overlays/konflux-public-production/delete-applications.yaml

delete-applications.yamlStop deleting monitoring-cardinality ApplicationSet in production-downstream +0/-7

Stop deleting monitoring-cardinality ApplicationSet in production-downstream
• Removes the staging-only $patch: delete block for the monitoring-cardinality ApplicationSet. This enables deployment to downstream production clusters.
argo-cd-apps/overlays/production-downstream/delete-applications.yaml

kustomization.yamlDefine production base kustomization and image override +8/-0

Define production base kustomization and image override
• Creates a production base that composes RBAC and runtime resources. Overrides the exporter image name and tag for production deployment.
components/monitoring/exporters/cardinality/production/base/kustomization.yaml

cardinality-exporter.yamlAdd service account and cluster-level RBAC for exporter discovery/scrape +55/-0

Add service account and cluster-level RBAC for exporter discovery/scrape
• Introduces a ServiceAccount plus ClusterRole/Binding for listing namespaces/endpoints (service discovery). Also binds cluster-monitoring-view to allow access through kube-rbac-proxy-protected Prometheus endpoints.
components/monitoring/exporters/cardinality/production/base/rbac/cardinality-exporter.yaml

cardinality-maintainers.yamlGrant konflux-o11y maintainer access to exporter namespace +12/-0

Grant konflux-o11y maintainer access to exporter namespace
• Adds a RoleBinding granting the konflux-o11y group component-maintainer access. This supports admin/ops workflows in the exporter namespace.
components/monitoring/exporters/cardinality/production/base/rbac/cardinality-maintainers.yaml

kustomization.yamlAssemble RBAC resources under appstudio-cardinality-exporter namespace +5/-0

Assemble RBAC resources under appstudio-cardinality-exporter namespace
• Defines the RBAC kustomization, including exporter RBAC and maintainer bindings. Sets the namespace for namespaced RBAC objects.
components/monitoring/exporters/cardinality/production/base/rbac/kustomization.yaml

auth-config.yamlAdd auth ConfigMap for kube-rbac-proxy protected Prometheus +20/-0

Add auth ConfigMap for kube-rbac-proxy protected Prometheus
• Creates a ConfigMap providing auth.yaml configuration for service-account-based access and CA configuration. Documents that user-workload monitoring is currently blocked from the TSDB status endpoint.
components/monitoring/exporters/cardinality/production/base/resources/auth-config.yaml

deployment.yamlDeploy cardinality exporter with probes, resource limits, and auth mount +66/-0

Deploy cardinality exporter with probes, resource limits, and auth mount
• Adds a single-replica Deployment running the exporter with read-only root FS and non-root security context. Configures service discovery, namespace/regex targeting, and mounts auth config for Prometheus access.
components/monitoring/exporters/cardinality/production/base/resources/deployment.yaml

kustomization.yamlAssemble runtime resources (namespace, service, ServiceMonitor, deployment) +8/-0

Assemble runtime resources (namespace, service, ServiceMonitor, deployment)
• Defines the production runtime resource set for the exporter. Ensures objects are created in the appstudio-cardinality-exporter namespace.
components/monitoring/exporters/cardinality/production/base/resources/kustomization.yaml

namespace.yamlCreate appstudio-cardinality-exporter namespace with early sync wave +7/-0

Create appstudio-cardinality-exporter namespace with early sync wave
• Adds a Namespace manifest for the exporter with ArgoCD sync-wave -1 to ensure it exists before namespaced resources are applied.
components/monitoring/exporters/cardinality/production/base/resources/namespace.yaml

service.yamlExpose exporter HTTP port via Service +14/-0

Expose exporter HTTP port via Service
• Creates a Service targeting the exporter pod label and exposing port 9090 as http. This is used for scraping by the ServiceMonitor.
components/monitoring/exporters/cardinality/production/base/resources/service.yaml

servicemonitor.yamlEnable Prometheus scraping via ServiceMonitor +15/-0

Enable Prometheus scraping via ServiceMonitor
• Adds a ServiceMonitor that scrapes /metrics over the http port at a 2-minute interval. Selects the Service by app label.
components/monitoring/exporters/cardinality/production/base/resources/servicemonitor.yaml

kustomization.yamlAdd per-cluster production overlay for kflux-fedora-01 +2/-0

Add per-cluster production overlay for kflux-fedora-01
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/kflux-fedora-01/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for kflux-ocp-p01 +2/-0

Add per-cluster production overlay for kflux-ocp-p01
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/kflux-ocp-p01/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for kflux-osp-p01 +2/-0

Add per-cluster production overlay for kflux-osp-p01
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/kflux-osp-p01/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for kflux-prd-rh02 +2/-0

Add per-cluster production overlay for kflux-prd-rh02
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/kflux-prd-rh02/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for kflux-prd-rh03 +2/-0

Add per-cluster production overlay for kflux-prd-rh03
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/kflux-prd-rh03/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for kflux-rhel-p01 +2/-0

Add per-cluster production overlay for kflux-rhel-p01
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/kflux-rhel-p01/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for stone-prd-rh01 +2/-0

Add per-cluster production overlay for stone-prd-rh01
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/stone-prd-rh01/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for stone-prod-p01 +2/-0

Add per-cluster production overlay for stone-prod-p01
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/stone-prod-p01/kustomization.yaml

kustomization.yamlAdd per-cluster production overlay for stone-prod-p02 +2/-0

Add per-cluster production overlay for stone-prod-p02
• Creates a minimal per-cluster overlay that includes the shared production base. Used by the ApplicationSet clusterDir mapping.
components/monitoring/exporters/cardinality/production/stone-prod-p02/kustomization.yaml

Create production base, empty-base, and per-cluster overlays for ring 1 clusters (stone-prod-p01, kflux-prd-rh02). Remaining production clusters are onboarded in the ApplicationSet with empty-base so they deploy nothing until promoted in later rings. Assisted-by: Claude Code

Remove $patch: delete for monitoring-cardinality from both production ArgoCD overlays so the ApplicationSet deploys to production clusters. Assisted-by: Claude Code

openshift-ci · 2026-07-14T07:36:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kubasikus
Once this PR has been reviewed and has the lgtm label, please ask for approval from filariow. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

konflux-ci-qe-bot · 2026-07-14T07:43:49Z

🤖 Pipeline Failure Analysis

Category: Infrastructure

The pipeline failed due to an ArgoCD application timeout caused by a degraded redhat-oadp-operator subscription, which prevented the deployment of critical backup infrastructure components required for Konflux installation.

📋 Technical Details

Immediate Cause

The appstudio-e2e-tests/konflux-ci-install-konflux step failed because the backup-in-cluster-local ArgoCD application did not sync within the allotted 45 minutes, reporting an OutOfSync and Degraded health status.

Contributing Factors

The root cause for the ArgoCD application's failure to sync was a degraded redhat-oadp-operator subscription. This degradation was characterized by CatalogSourcesUnhealthy and ResolutionFailed conditions, indicating issues with the operator's ability to resolve and deploy its components. As a result, several essential backup infrastructure resources like ConfigMap/image-repository-resource-modifier-config, Secret/minio-storage-configuration, Job/create-velero-bucket, Tenant/oadp-minio-storage, DataProtectionApplication/velero-aws, and Schedule/backup-tenants were missing or unable to be provisioned.

Impact

The inability to deploy the necessary backup infrastructure components due to the degraded OADP operator subscription directly blocked the successful installation of Konflux, leading to the overall failure of the appstudio-e2e-tests/konflux-ci-install-konflux step and the subsequent pipeline termination.

🔍 Evidence

appstudio-e2e-tests/konflux-ci-install-konflux

Category: infrastructure
Root Cause: The backup-in-cluster-local ArgoCD application timed out because the redhat-oadp-operator subscription was degraded, exhibiting CatalogSourcesUnhealthy and ResolutionFailed conditions, which prevented the successful deployment of required backup infrastructure components.

Logs: