Runner pods and datastore logs cleaning

[seboudry]

I noticed that only the 5 last pods are kept.
But do we really need to keep them when datastore already keeps logs?
=> Can we just delete them after run?

Some other point, is the retention on datastore.
For an example, I have 13 layers and drift detection every 30 minutes.
After less than one day, our datastore bucket size is already at 500MB.
This is quite huge for just plans logs with no apply.
=> Could we keep only the last X run logs for plan and Y run logs for apply (maybe infinite), configurable by layers?

What do you think about it?

[AlanLonguet]

Hey @seboudry we used cloud provider buckets as a backend storage so users can setup their own lifecycle policy would that be something ok for you ?

For the number of logs you have on datastore, I think you can set the drift detection period to a bigger number, we're on a 12hours period for one of our current setup with around 500/600 layers

[AlanLonguet]

We wanted to make it configurable (Number of last runs/plans/apply) but didn't have time to ship it in a good condition so for now we compromised on 5

[seboudry]

Hi @Alan-pad !

datastore
Logs for plans are nice for drift detection but apply/destroy are more important for audit.
That's why we need to have different deletion policies.

So I'm OK for having responsabilities to underlying bucket store, but with the current layout path of logs in bucket it's not possible to do that.
On S3/MinIO (and also IBM Cloud) only a prefix for object path can be used (so, not a path with wildcards).

So I see two options:

1. have two buckets: one for plans, one for apply/destroy
2. have a top level directory to use as prefix

pods
My question was: does we really need to keep them?
Maybe only the last one to see variables/volumes/etc configured on the pod, but it's more debugging purpose.

[AlanLonguet]

The retention of pods is tied up to the retention of TerraformRun resources, we rely on ownerReferences for their deletion, if you want to keep less pods you can use this variable : https://github.com/padok-team/burrito/blob/main/api/v1alpha1/common.go#L90 which defaults to five, since for now we only support you seeing this number of run in the UI I'd advise you keep at least 2 or 3

[AlanLonguet]

But I agree with you we need a better control on the history capabilities of the datastore that's something that needs to be enhanced

[seboudry]

OK I understand the underlying considerations.

Thanks for the tips on runHistoryPolicy.runs.

I need to read the code more ;)