Order of resources to be applied

[maxkokocom]

Hi, how should I manage the order of resources being applied?

I am trying to deploy rook ceph operator with one instance of ceph cluster and it fails because CRDs are created via helm chart release. How can I make flux order so helm chart is deployed first? Right now it is not even getting to helm chart as it fails on a missing CRD.

My error code:

CephBlockPool/rook-ceph/replicapool dry-run failed, error: no matches for kind "CephBlockPool" in version "ceph.rook.io/v1

Related issue which is closed: fluxcd/flux#2170

[stefanprodan]

You need to use two Flux Kustomizations, one that deploys the HelmRelease with wait: true, and the 2nd one that dependsOn the first one. See here the docs: https://fluxcd.io/docs/components/kustomize/kustomization/#kustomization-dependencies

[kingdonb]

To maintain this structure, you can continue to follow the pattern described in https://github.com/fluxcd/flux2-kustomize-helm-example. I don't see how anything about this structure dictates you must have only one Flux Kustomization.

When you have only two Flux Kustomizations here, "apps" and "infrastructure" and you want apps to depend on Ceph, there are simply not enough touch points with Helm involved. Helm is unfortunately not very good at installing and managing CRDs.

In order to do X -> Y -> Z where X is the Ceph/Rook operator, Y is the instances of Ceph storage, and Z is your app that depends on Y, with Helm involved you will need 3 Kustomizations arranged in dependency order, but with CRDs managed separately you will only need two, because Kustomization controller can apply the resources in the correct sequence so the Rook Storage CRD instances will be valid at once. Kustomize Controller cannot reach through Helm and wait for the CRDs to be ready (although Helm Controller can wait for the CRDs and everything else, and Kustomize controller can wait for the HelmRelease, it will apply the other CRD instances at the same time as the HelmRelease, and so you find these instances are invalid until the HelmRelease is actually installed...)

If you can find a way to keep the version installed by Kustomize Controller correctly in sync, I would recommend grabbing the CRDs from here and using Kustomize Controller to deploy them, while crds.enabled: false is set in the values. (Be careful, on a running cluster changing this setting will likely delete your CRDs!)

It does not look like Rook have made their manifests particularly amenable to this configuration, but if you're still having trouble I'm not sure I understood the question and might need to be clarified further 🙃

[maxkokocom]

Yes I was thinking about deploying CRDs outside of helm, sadly rook-ceph is not great for that.

My issue with the structure is that I currently have rook-ceph in /infrastructure/base/rook, my /infrastructure/stage/ is referencing the base for rook, but also all other applications.

The flux kustomization having depends_on is actually used in /clusters/stage_template/infrastructure.yaml where it's referencing non-flux kustomization type (just pure kustomization) listing resources from base.

So If I split ceph into two flux kustomization I don't really know how to use those templates for clusters or where to put it. Right now each cluster has following kustomizations with references:

apps.yaml ref-> /apps/cluster_type_or_name ref-> /apps/base + some values overwrites
infrastructure.yaml ref-> /infrastructure/cluster_type_or_name ref-> /infrastructure/base + some values overwrites
secrets.yaml ref-> /secrets/cluster_type_or_name ref-> /secrets/base + some values overwrites
sources.yaml ref-> /sources/cluster_type_or_name ref-> /sources/base + some values overwrites

If I would want to add another kustomization only for ceph I need to split infrastructure one into each infra app. This would make sense, but this means I cannot have those simple template clusters anymore.

Maybe I overcomplicated in some way I don't see it, I also see more kustomizations may be better for debugging etc. but I still wonder what's the good structure to have it.

I am following the pattern in the mentioned repo, but it also is referenced in the same way. /clusters/staging/apps.yaml is one global kustomization referencing a ./apps/staging as well. It's not referencing each single app there. Both apps.yaml and infrastructure.yaml are referencing a single directory. Either base /apps|infrastructure or some overwrites /apps|infrastructure/staging

Once I start using multiple kustomizes to simply have depends_on I am losing generic benefits of base/staging/personal etc. or at least it's not as a simple setup anymore for users.

[kingdonb]

If I would want to add another kustomization only for ceph I need to split infrastructure one into each infra app.

Not really, you just use the same pattern for all Helm apps with CRDs. You can have one crds which is depended on by infrastructure and only pull problem CRDs out so they can be loaded ahead of infrastructure. This does not mean you need another Flux kustomization for each app (or even for each problem app). You just need another Flux kustomization "to the left" of the one that installs the ceph/rook chart, so that the CRDs can be prepared in advance of it, and the HelmRelease can be kept alongside of your custom resource instances. All CRDs in this situation can be kept in the same Flux Kustomization.

(The other option would be to take your Custom Resources and push them "to the right" -- where they can depend on infrastructure.)

This would be the same on all clusters as I understand it, so I don't see how this strategy compromises on that design intention at all.

[maxkokocom]

Okey so I did fix it (I assume according to your recommendations), tbh I am not very happy with it tho, but it does work. It seems the best way with flux and the cleanest would be to get CRDs out of helm charts, sadly rook doesn't allow it. I know it's not a flux team issue, but up there with helm itself so you probably don't want to deal with it and figure out which helm has CRDs to prioritize it, but I would still consider some ordering overwrite or helm template before to put helmrelese up in the order if it has CRDs, namespaces etc.

My current setup is not really that clean in my mind cause of that.

[kingdonb]

I point at cert-manager as a good "good example" because they publish their instructions with strong admonishments about how Helm users will run into trouble if they're not aware of Helm's behavior around CRDs, they publish alternate instructions for people who ARE aware of those issues who would like to simply avoid Helm altogether, and they publish the CRD bundle as a plaintext YAML file with every release, so they are basically setting Flux users up for success, to be able to set up cert-manager and associated CRs just once, then forget about it:

https://github.com/kingdonb/bootstrap-repo/tree/staging/apps/cert-manager
https://github.com/kingdonb/bootstrap-repo/blob/staging/apps/cert-manager/kustomization.yaml#L4
https://github.com/kingdonb/bootstrap-repo/blob/staging/clusters/moo-cluster/cert-manager/flux-kustomization.yaml

One Flux Kustomization, one SIG-CLI overlay file, ... it would be ideal to be able to do a similar thing here.

Another option that I hadn't considered, you could add the Custom Resources to the chart using a Post-Render patch:

https://fluxcd.io/docs/components/helm/helmreleases/#post-renderers

I think that would work and is probably the better option that you're looking for, for now anyway.

We do seek out and look for this kind of trouble issues with common helm charts in the wild, as we aim to make Helm more usable and that's a problem that includes charts of many shapes, and when we talk about usability problems with dependsOn now, I use Rook as my canonical example of a chart publisher not doing us any favors. (No offense to the fine folks making Rook, I'm sure the chart has many good qualities, but when doing examples one must have good and bad examples, and they make a good "bad example" here.)

This solution I'm offering is still something that I would consider a work-around. The only negative thing about postrender kustomizations is that they live inline in the HelmRelease content, so it's easy to wind up with something like this if you're not careful:

https://github.com/kingdonb/flux2/blob/32e56b0d8a455db587f3f7b3e867f2bee4b7198c/manifests/monitoring/kube-prometheus-stack/release.yaml#L123

[o6uoq]

3+ years later, is there anything in the more recent versions of Flux, especially the Flux Operator, that providers feature-driven ordering/dependency management of reconcilations?

[matheuscscp]

What do you mean by feature-driven? Everything is feature-driven.

.spec.dependsOn is the feature you're looking for, and it's available in the Kustomization and HelmRelease Flux APIs and in the ResourceSet Flux Operator API.