cluster: ObjectStore helm hook breaks backups under ArgoCD

[yama6a]

What's wrong

With the plugin backup method, the chart renders the ObjectStore with: "helm.sh/hook": pre-install,pre-upgrade,pre-rollback

(in backup-objectstore.yaml and recovery-objectstore.yaml, from #924).

Fine in plain Helm, it jsut orders the ObjectStore before the Cluster. But under ArgoCD a Helm hook becomes an ephemeral PreSync resource, so ArgoCD creates it once and then prunes it. It never comes back: WAL archiving stops and the Cluster goes Ready=False with ContinuousArchivingFailing: ObjectStore ... not found, wedging the whole app sync.

Repro

backups.method: plugin synced by ArgoCD (auto-sync + prune) -> ObjectStore applied as a PreSync hook, then pruned, and archiving breaks.

For now, I've worked around it by rendering the CNPG CRs myself instead of using this chart directly, so the ObjectStore has no helm hook and ArgoCD keeps it as a normal tracked resource, wtih an argocd.argoproj.io/sync-wave: "-1" for ordering.

Suggested fix

Make the hook optional, (default enabled, to not causea BC break of course)

    backups:
      objectStore:
        helmHook: true      # set false for ArgoCD/Flux
        annotations: {}      # e.g. argocd.argoproj.io/sync-wave: "-1"

When false, drop the hook annotation. The annotations map lets GitOps users add their own ordering without leaking it onto the Cluster like the existing cluster.annotations passthrough would. I'd apply it to both the backup and recovery ObjectStores.

Happy to open a PR (templates + values + schema + docs + chainsaw test) if the approach works for you. just wanted to check the approach first.

// Edit: it was kindof a small change, so I already created the PR. No hard feelings if you reject it or prefer a different approach :)
#965

[itay-grudev]

I haven't used ArgoCD in a while, but I just went through their documentation and it seems like it should work and your description of the CR being deleted shouldn't be true.

Both pre-install and pre-upgrade map to PreSync. It's possible pre-rollback causes some kind of an issue, because it's not supported.

The documentation also clarifies that it honors the helm.sh/hook-delete-policy and before-hook-creation is the default.

Investigate whether the pre-rollback is the reason of the failure. It may make sense to remove it anyway.

[yama6a]

Thanks for looking!

I dug a bit more and re-tested it again in different configurations.

I think pre-rollback is a red herring tbh. Per the helm hooks docs it's "Not supported... Unsupported hooks are ignored", so argo never did anything with it in the first place. The only mapping that actually does something is pre-install,pre-upgrade to PreSync. I tried dropping pre-rollback anyway just to be sure and it breaks exactly the same.

The way I understand it there's two things going on. First, the chart doesn't set a hook-delete-policy, and per the sync phases docs the default is BeforeHookCreation, which deletes the old objecstore before making the new one every sync. And second, because its a hook argo never tracks it as a live resource, so once its gone the app still shows Synced/Healthy, it's not counted as drift, and selfheal never puts it back. I checked this by deleting it by hand and yeah, app stayed green and never recreated it.

And the whole time the cluster's WAL archiver is looking for that objectstore, so the second it's gone you get objectstores.barmancloud.cnpg.io "<name>-backups" not found.

Tested on k8s 1.36, single instance cluster with the barman plugin:

• stock (pre-install,pre-upgrade,pre-rollback): hook, selfheal doesnt restore it, archiving = not found
• your suggestion (pre-install,pre-upgrade): same, still a hook, still breaks
• no hook + sync-wave "-1": managed resource, selfheal restores it, archiving fine

So imo the fix isnt the hook list, its making the objectstore not be a hook at all, which is what #965 does.

Last thing: the objectstore also has to exist before the cluster starts, otherwise the cluster just hangs at BootstrapPending with no pods until it appears. So order matters, but thats easy, hook-weight maps to sync-wave so it can just carry sync-wave "-1" and go first.

[itay-grudev]

I'm open to ideas, but it needs to be a hook. It needs to be available before the cluster, especially if you're migrating from in-line backups to the plugin. It's a key feature allowing migrations to be done in a single step.

[yama6a]

hiya,

sorry for the slow reply. life got in the way.

My PR #965 doesn't remove the hook. It defaults to true, so nothing changes for plain Helm or for the migration case you're describing.

and yeah you need the hook for ordering. I looked at helm's sorter and unknown kinds go alphabetically by kind, so Cluster gets applied before ObjectStore. Nothing else in the chart can fix that I think.

Problem is that under Argo the hook doesn't actually order anything either. Argo maps the annotation but not the lifecycle (their docs say as much). The ObjectStore gets applied once as PreSync, then deleted before the next sync, and it's never a tracked resource after that, so selfheal won't bring it back and the app still shows green while archiving is dead. There's no way to work around it from values either, the only annotation passthrough is cluster.annotations and putting an argo hook annotation there disables all helm hooks in the chart and leaks onto the Cluster.

Since in-tree barmanObjectStore is deprecated as of 1.26, anyone on Argo eventually hits this.

Fine either way on naming, happy to move the flag or shape it differently if you have a different preference :)

[yama6a]

@itay-grudev poke 🙂