dynostore: policy deletion and compaction support

[shortishly]

Does it make sense to apply the cleanup.policy to dynostore (S3, memory/etc) based storage engines?

Yes, it makes sense to apply cleanup.policy (delete/compact) to dynostore-based backends like S3.

Tansu aims to be a drop in replacement for Apache Kafka. In Kafka, cleanup.policy is a core topic configuration that works regardless of the underlying storage (log segments on disk). Users expect the same behavior when they set cleanup.policy=delete or compact (or both) on a topic; especially for S3-backed setups, which are often used for cost-effective, long-term or high-volume storage.

One additional suggestion I want to add is in the docs to make this clear explicitly; during the time it is being built. It took me a issue search to figure this out; that the above support is not fully ready. Without this; I see that people using the S3 engine (or S3 compatible ones) will configure topics with retention and compaction policies; expecting it to work. Without enforcement, data will just keep accumulating in S3.

SQL engines (Postgres, SQLite) already implement this via maintenance operations (compaction via CTEs keeping latest offset per key, deletion via timestamp). Applying similar logic to dynostore keeps the behavior uniform and avoids surprises when switching storage engines.

Originally posted by @ErSauravAdhikari in #464

[shortishly]

probably makes sense to split this into:

• deletion support, which should be reasonably simple
• compaction support

[ggekos]

Hey,
happy to help on this subject around S3.

• cleanup delete
• cleanup compaction

It should be pretty simple for the retention.ms but the retention.bytes is not. Do you want to scope both config or focus on ms ?

Do you wann create 2 distinct issues ?

[OrHayat]

@ggekos I needed to implement something on Tansu with a new storage backend, found out it doesn't have GC, and eventually landed on this ticket.

Your solution looks REALLY incorrect found some by just looking at the code and more by running claude on it to verify and he caught alot of those right away.

• no guard against overlapping maintain() runs. the broker spawns it every tick without waiting for the previous one, so on a big bucket the runs pile up on each other
• compaction rewrites corrupt ListOffsets. earliest/latest are read from each object's last_modified, and rewriting a batch bumps it, so consumers can get
  wrong offsets and skip live data
• even if the multiple-workers issue gets solved, you need to ensure one of two things:
  • the maintain window is SHORTER than 10 mins (or whatever the guard is)
  • stuck maintainers' leases get cleaned up, or something of that type, otherwise a slow/stalled worker causes split-brain
• unguarded panic on malformed object names. the [0..20] slice parses safely but slices first, so any short/stray name under the prefix panics maintenance

and many more (combined compact+delete retention never fires, deleted data can get resurrected by a racing overwrite).

[ggekos]

Hey @OrHayat, thanks for your insights and happy that you are helping on this subject.
I already fix items 1, 4 on a fork.
For 2 it's I will have look :)
For 3 yeah, the multi-replica is a disctinct aspect. On our end we choose to have worker without maintain and one with maintain.

[OrHayat]

we choose to have worker without maintain and one with maintain.
is terrible idea in general

for example in my work we deploy tansu in an on prem system as attempt to have kafka api.
there is no way to make it reliable by not making at least 2-3 nodes "maintance"==true
u cant pre-select it or know if node or which node will crash and when ,

also u might want to split the job somehow across the topics/paritions to not stress the cpus , or not idk that will require adding communciaton between the node which is not exactly in the goal of this mode of deployment