Try out if kro could be feasible as deployment-tool

[frewilhelm]

Description

As described in #149, the ocm-controllers require something to deploy OCM resources. The v1 controllers used a custom operator FluxDeployer to achieve this.

We want to try out if kro could be our deployer for the v2 controllers, as kro is an resource orchestration tool and offers several configuration possibilities to alter and configure values in deployments. In combination with FluxCD, we could use

We need the following use-cases to work:

• simple helm: Use FluxCD to deploy a helm-chart inside an OCM resource

• simple kustomize: Use FluxCD to deploy a kustomization inside an OCM resource

• simple configuration helm: Use FluxCD to deploy a helm-chart and configure values inside the helm-chart

• simple configuration kustomization: Use FluxCD to deploy a kustomization and configure values inside the manifests

• A complicated one containing

  • configuration
  • localisation
  • instance-spec parameters
  • passing a secret
  • (Also include the replication)

• RGD in CV

  • Requires new operator, e.g. OCMDeployer that deploys the RGD

How would we recommend to use the kro-resources?

• ???

How can resources be upgraded/update?

• see Try out if kro could be feasible as deployment-tool #164 (comment)

Timebox: 3 day(s)

Done Criteria

• Estimation of impact on existing code incl. tests
• Estimation of impact on Enduser Documentation updated (if applicable)
• Estimation of impact on Internal technical Documentation created/updated (if applicable)
• Created refinable tasks for the actual implementation

[frewilhelm]

Estimations

Estimation of impact on existing code incl. tests

• Configuration and localisation could be replaced by kro ResourceGraphDefintion CEL statements
  • Configuration + localisation while deploying (no intermediate results are stored anymore)
    • Values for configuration and localisation (e.g. image locations, ...) can be set hardcoded, passed through kros instance, or dynamically referencing other k8s resources (as long as they are known in the cluster and accessed inside the graph).
  • to localise we deploy a CRD ocm resource and store its access information about the source OCI reference in its status. Accordingly, we do not need an intermediate layer to provide the resources as the original source is used.
    • works only for OCI artifacts
    • kro does not allow to reference json apiextension.RAW (problem: dynamic fields that are not know previously). This raises an error in the ResourceGraphDefinition as its dry-run (checking if fields exists) is accordingly failing.
      • API changes are required
  • Omitting all CRDs for configuration and localisation, e.g. localisation-rules.
• OCI storage backend implementation + zot-registry could be removed, assuming we don't need to store any resources from localisation or configuration (compatibility layer)
  • OCM component descriptor (lists) cannot be stored anymore. Adjustments are needed
  • Omitting the storage backend means that localBlob resources cannot be deployed as every resources needs a source OCI registry from which it can be fetched.
• e2e-tests must be adjusted
  • Deployment of kro
  • Replace current resources with ResourceGraphDefinitions
• "unit" controller tests must be adjusted to work without the storage
• Config inheritance could be omitted as it can be specified in the ResourceGraphDefinition directly. Would reduce complexity towards propagation-policy of each config.
• Probably requires the possibility to pack a ResourceGraphDefinition in an ocm component version and use it for deployment
  • Requires a new operator

Estimation of impact on Enduser Documentation updated (if applicable)

• Enduser Documentation is already outdated and requires some work either way
• Would/Can be part of EPIC: Documentation of v2 controllers (Repository documentation) ocm-project#370

Estimation of impact on Internal technical Documentation created/updated (if applicable)

• Technical documentation is already outdated and requires some work either way

Created refinable tasks for the actual implementation

• As this spike is part of an ADR (Create a Deployment ADR #136) the refinable tasks will be part of the decision of this ADR. A quick overview is presented in the "Estimation of impact on exists code incl. tests"

[frewilhelm]

Current progress is saved in https://github.com/frewilhelm/ocm-k8s-toolkit/tree/spike_kro (based on #98)

[frewilhelm]

A potential blocker could be that instances are not reconciled when their graph is updated. Thus, changes will not be propagated to the resources.

However, there are at least two issues, one of them a feature request from the maintainers themselves, that address this issue and want to fix it.

[ikhandamirov]

Wow, sounds like a huge simplification!

[frewilhelm]

In a scenario in which the ResourceGraphDefinition is part of the CV there is a possibility of a potential race condition:

• CV contains
  • OCM Resource (e.g. HelmChart to-be-deployed)
  • ResourceGraphDefinition that contains
    • k8s OCM resource (refers to OCM Resource in CV)
    • FluxCD OCI Repository (refers to location of CV stored in OCI registry)
    • FluxCD HelmRelease (points to FluxCD OCI Repository)

To deploy the resource, the user has to deploy the k8s resources OCMRepository (points to OCI registry in which the CV is stored), component (points to the component name and OCMRepository), resource (points to ResourceGraphDefinition in the CV and component), and a new CRD OCMDeployer (or the like) that references the resource for the ResourceGraphDefinition. The OCMDeployer takes the manifest of the ResourceGraphDefinition and deploys it. After creating an instance for the new kind from the ResourceGraphDefinition, kro will deploy the OCM resource (the HelmChart) using the resources for OCM and Flux from the ResourceGraphDefinition.

The problem arises when the CV is updated with a new version. This triggers an update of the k8s resource component. However, the resource component is watched by (a) the RGD resource that is also watched by the OCMDeployer and (b) the k8s resource resource that contains the HelmChart. As an example, assume that the k8s resource resource for the HelmChart is removed from the updated CV. The trigger/watch for the RGD resource is fine as this will just deploy the new graph. But what happens with the original k8s resource resource for the HelmChart. If it reconciles before the RGD is updated, then it will fail as the resource cannot be find anymore in the component.

This is not necessarily a problem in the context of eventually consistency as we expect that the k8s resource resource for the helmChart can fail at first, but will then be removed, when the RGD is updated (we assume the RGD is updated as well as one resource was deleted). But we should keep such scenarios in mind.

[Skarlso]

Also, let's consider the considerable setup complication ( needing to install and configure kro ) plus the overhead of people learning kro and maintenance of kro version ( unless we farm this out to the infra maintainers which is as additional burden on them in that case ).

OCI storage backend implementation + zot-registry could be removed, assuming we don't need to store any resources from localisation or configuration (compatibility layer)

I don't understand this one. :) The registry is a cache and a sync point. It's not just there to share results, but it's also there so that we don't have to re-download a 6 gigabyte image when it's being fetched from somewhere over and over dealing with the same component or resource. Also, it's not explained how you would work with Flux then? Like, how do you present it with the created artifact that it needs to deploy?

[frewilhelm]

The registry is a cache and a sync point. It's not just there to share results, but it's also there so that we don't have to re-download a 6 gigabyte image when it's being fetched from somewhere over and over dealing with the same component or resource

If we omit the configuration and localisation (or rather move it to kros ResourceGraphDefinition), then we do not have to download any image at all.

If the image must be available in a specific environment, one could use the replication controller to move the component version to that environment.

Also, it's not explained how you would work with Flux then? Like, how do you present it with the created artifact that it needs to deploy?

Assuming we omit the OCI registry, then we would need to publish the original source (= OCI registry, HelmRepository, git Repository, or the like) in the status of the resource. This information can then be taken as CEL to pass the location for example to a FluxCD OCI Repository. Consider the following example of an ResourceGraphDefinition:

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: complicated-deployment
spec:
  schema:
    apiVersion: v1alpha1
    kind: ComplicatedDeployment
    spec:
      podinfo:
        releaseName: string
        message: string | default="hello, world"
  resources:
    - id: resourceChart
      template:
        apiVersion: delivery.ocm.software/v1alpha1
        kind: Resource
        metadata:
          name: static-resource-chart-name
        spec:
          componentRef:
            name: static-component-name # should be referenced/passed
          resource:
            byReference:
              resource:
                name: helm-resource
          interval: 10m
    - id: resourceImage
      template:
        apiVersion: delivery.ocm.software/v1alpha1
        kind: Resource
        metadata:
          name: static-resource-image-name
        spec:
          componentRef:
            name: static-component-name # should be referenced/passed
          resource:
            byReference:
              resource:
                name: image
          interval: 10m
    - id: ocirepository
      template:
        apiVersion: source.toolkit.fluxcd.io/v1beta2
        kind: OCIRepository
        metadata:
          name: helm-podinfo-config
        spec:
          interval: 1m0s
          layerSelector:
            mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
            operation: copy
          url: ${resourceChart.status.ociArtifact.sourceReference.registry}/${resourceChart.status.ociArtifact.sourceReference.repository} 
          ref:
            digest: ${resourceChart.status.ociArtifact.digest}
    - id: helmrelease
      template:
        apiVersion: helm.toolkit.fluxcd.io/v2
        kind: HelmRelease
        metadata:
          name: ${schema.spec.podinfo.releaseName}
        spec:
          releaseName: ${schema.spec.podinfo.releaseName}
          interval: 1m
          timeout: 5m
          chartRef:
            kind: OCIRepository
            name: ${ocirepository.metadata.name}
            namespace: default
          values:
            # Localisation
            image:
              repository: ${resourceImage.status.ociArtifact.sourceReference.registry}/${resourceImage.status.ociArtifact.sourceReference.repository}
              tag: ${resourceImage.status.ociArtifact.sourceReference.reference}
            # Configuration
            ui:
              message: ${schema.spec.podinfo.message}

[Skarlso]

If the image must be available in a specific environment, one could use the replication controller to move the component version to that environment.

The replication controller was all but archived.

This information can then be taken as CEL to pass the location for example to a FluxCD OCI Repository. Consider the following example of an ResourceGraphDefinition:

How would that work with modified resources by the ocm client during transfer? Which step would that be? So you declare a component version, you declare a target, and then deploy that all with Kro, and by the end there would be an end registry with a Status updated having the location of the endresult in a registry I assume?

Keep in mind that this all needs to work offline.

[frewilhelm]

Note: A replication cannot be part of a RGD in the same CV that should be replicated with that replication.

[Skarlso]

We talked about this on slack. Outcome:

• since loc/conf won't be part of the main flow anymore, the reason for the registry becomes not that great anymore
• shared some historical reasons behind the local registry including DMZs where the registry was the target ( this can, of course be mitigated if the infrastructure maintainers run their own registry )
• shared two concerns:
  • increase in burden for the infra maintainers ( kro + local registry in case of DMZ )
  • kro is really new in the game being about a year old might raise concerns with some clients who have strict environment policies

[frewilhelm]

Spike is closed and the implementation will be implemented here #172