Communicate with stakeholders if the deployer using kro is viable for their usecases

[frewilhelm]

Description

Get in touch with current and potential stakeholders and present our idea using kro as a deployment tool for our OCM controllers.

Done Criteria

• Present a demo
  • in the internal review
  • external stakeholders
  • community call (07.05.2025)

[frewilhelm]

There were no feedback or comments from the internal review other than: "Looks interesting/promising"

[fabianburth]

We had a rather long meeting with Dan, discussing the concept thoroughly.
As a TLDR;

• In the new concept, the service owners / developers would be responsible for maintaining the deployer specific CRDs (such as OCIRepository, HelmRelease or Kustomization in Flux).
• In the old concept, the service owners / developers only have to know the deployment technology (Helm, Kustomize) but not the deployer itself.

Dan claims this is a significant issue for them. He will provide us with a simplified version of their use case and we will try to see whether we can somehow adjust the concept to fulfill this requirement.

[fabianburth]

Blocked. Waiting for example use case from @dee0.

[dee0]

Hey @fabianburth

I'll try to have something to you by end of this week.

That said, I want to point out here we did call out the importance of not exposing the teams owning the services to the details of the gitops config. ( i.e. The OCM deployment objects )

[fabianburth]

Hey @dee0,

I'll try to have something to you by end of this week.

thanks, that would be quite valuable for us to make a decision!

---

That said, I want to point out here we did call out the importance of not exposing the teams owning the services to the details of the gitops config. ( i.e. The OCM deployment objects )

You are right, you did point that out. And I'm also very sorry that we are bothering you with yet another concept proposal! Especially because you have been such a valuable stakeholder to us, continuously giving us feedback, validating our ideas and pushing adoption!

But the issue we see with the concept of the current controllers with all its indirections (that make it very powerful) is, it is:

• complex for complex use cases (might be acceptable)
• complex even for simple use cases (might be a problem --> leads to adoption issues)
• hard to maintain and debug

Thereby, we also have to face the reality that service development teams cannot be shielded entirely from operations and vice versa. While I agree that in an ideal world such a strict separation of concerns seems to be clean and desirable, in our non-ideal world these indirections make it much harder for individual developers to understand the whole picture and to debug and investigate issues.

Even with the current concept, there are several hidden assumptions that break that clear separation (e.g. when writing the localization rules, your service development teams rely on the assumption that the operations team will transfer the component in a way where your resource will end up as an OCI artifact in an OCI Registry).

The KRO concept is certainly not perfect either, but I think it is a solid compromise! And in my opinion, it seems to be the best solution we came up with so far.

So I guess, even more than asking "can we map the KRO concept exactly to your use case?", I'd encourage you to ask:

• Are you confident that your service development teams would prefer (and be sustainable with) the current concept where:
  • they have to treat the deployment process including the application of the localization and configuration rules as a black box
  • they have to deal with a proprietary/custom configuration mechanisms

over the KRO concept?

• Are you confident that your operations teams would prefer (and be sustainable with) the current concept where:
  • they have to maintain 4 additional custom resources
  • a cluster internal oci registry

over the KRO concept?

[dee0sap]

Hey @fabianburth

First, sorry it is taking me longer to make the use case example than I had hoped.

That said, we're absolutely certain the service teams would prefer something like more like the v1 configdata than the kro concept. That is why we called it out from the very beginning like we did, cause it was critical aspect of our use case.

That is also why I asked that we look at the resource graph file and the v1 configdata file side by side, just to highlight all the extra code that they will be confronted with.

Regarding the devops teams, first they are expected to be doing everything possible to make the service teams lives easier. There is a great deal of pressure to make sure that the service teams, and their managers, are happy. In other words if anyone saw that we did something that moved work off of the devops teams and on to the service teams there would be some hard conversations.

Regarding managing 4 additional custom resources, we are anyway planning to create an SACK8sComponventVersion ( name not final ) composition using Crossplane. In addition to creating the OCM deploy objects be making sure that the destination namespace(s), RBAC objects for deployment, deployment ServiceAccount and ExternalSecrets that are required for the deployment of the service. Also, when I look here in the KRO proposal aren't devops responsible for 4 objects because of the bootstrapping?

Regarding the internal registry, yes it would be nice if the complexity of having an additional deployment in the cluster could be removed. However I don't see how it relates to whether service teams are exposed to deployment objects or not.

Long story short, kro may be fine for use on the devops side of things but it absolutely isn't something that the service teams should be looking at. The existing ConfigData is more like what they should be seeing. NOTE: While the content that goes into the ConfigData isn't really a problem that doesn't mean there is no room for improvement. e.g. Debuggability. However the kro proposal really doesn't seem to do anything to help with that. On the other hand, a command line tool that took as input

• The name of a file to mutate
• The path to the ConfigData
• Either a URI, for testing localization, or a ConfigMap.yaml, for configuration
  and then spit out the mutated result would be nice. And of course it would be good if it performed the schema checks had some logging to help provide insight into what was happening.

[jakobmoellerdev]

I would appreciate if instead of going back and forth if something is being pushed down the service teams we could get an actual example of a delivery with concrete directives on what is being maintained by whom.

I still do not believe it is possible for DevOps and Service Teams to be completely separated because the configuration for deployment is opinionated per service. (e.g. one service might need an ingress or a cache config that another service doesnt).

In the last call we agreed to jointly work out how we can make this work by getting a prototype and then using that prototype to iterate. Im still of the opinion this is missing.

That said, we're absolutely certain the service teams would prefer something like more like the v1 configdata than the kro concept. That is why we called it out from the very beginning like we did, cause it was critical aspect of our use case.

@dee0 Where does this confidence stem from? When you showed us the examples in our joint meeting, it was evident to me that you acknowledged several times that currently devops and service configuration are intermingled already. How would a developer now prefer an abstracted localization without a schema over a Resource Graph?

[frewilhelm]

I will prepare another demonstration for the community call. As we already have shown the bootstrap example, I thought I could prepare another example with a "GitOps" approach:

• RGD is part of a GitHub repository as a kustomization
• FluxCDs kustomization is watching the GitHub repository and deploying the RGD on changes