How to confirm S3 bucket reconciliation

[danieljamesscott]

Hi,

We are managing resources in an S3 bucket using a process which runs externally to our k8s cluster and are looking for a way to confirm that reconciliation has completed.

Is there a way to confirm that flux has reconciled the bucket contents using the event data that the source controller emits? I can see a digest and revision which is calculated using a list of objects and their etags - do I need to duplicate this logic myself? It looks like this code is in an internal package, so I can't re-use it.

Thanks.

[matheuscscp]

S3 is strongly consistent: all reads after a successful write will see the result of the write. So after writing all objects to the bucket you can trigger a reconciliation of the Flux Bucket resource and wait for it to complete. You need to replicate the code of flux reconcile source bucket:

https://github.com/fluxcd/flux2/blob/main/cmd/flux/reconcile_source_bucket.go

[matheuscscp]

The only thing missing is how I can confirm that the event data I'm receiving came from the reconciliation of the files I pushed to S3.

You got a hard problem there. That's why I don't like Bucket, the controller could reconcile between you writing one file and another. The event will then reflect a partial state of your write operations on the S3 bucket. Worst case, you will need to wait for the whole reconciliation interval to be sure. Even worse, if something else is also writing objects in the bucket, you may never observe the digest you expect! OCIRepository is much better, the write operation uploads a single tarball with everything.

[matheuscscp]

Actually, even with OCIRepository, if something else is pushing artifacts you could still never observe the digest you expect, as the controller could skip your digest and go directly to a more recent push. Listening to the events is not going to be very reliable...

[danieljamesscott]

Yes - I did wonder whether it was possible to mirror the OCIRepository structure where "versioned bundles" are uploaded to the bucket and then use version numbers instead.

For my use case, it's not essential that it's the exact files that were uploaded. As far as I'm concerned it's "these files or newer" that would do. My current plan is to wait for a source event timestamped after I uploaded the files, and then wait for a Kustomization event which references the source digest.

But I may consider switching to an OCI repository instead - then I can use semver ordering and ensure that the version is newer.

[stefanprodan]

OCI Repositories using semver are reliable, as you can use a semver range on the receiver end and validate that what's being rolled out is at least the version you expect.

[danieljamesscott]

Yep, thanks. I'll look at switching to OCI.