Raw data cleanup after the retention period not working

[ashishvaishno]

Hello all,

I am trying to figure out why I see 14 days of raw data for metrics in my promethues + thanos setup instead of 7 days

• Thanos config uses

/bin/thanos compact --wait --log.level=debug --log.format=json --http-address=0.0.0.0:10902 --data-dir=/var/thanos/compact --objstore.config-file=/etc/thanos/objstore.yaml --deduplication.replica-label=prometheus_replica --retention.resolution-raw=7d --retention.resolution-5m=30d --retention.resolution-1h=180d --compact.concurrency=2 --downsample.concurrency=2 --block-files-concurrency=2 --delete-delay=48h

• Object store is aws s3 bucket
• Prometheus is running with --storage.tsdb.retention.time=1d
• Enabled debug logs on thanos. Nothing unusual
• also upgraded thanos to latest stable version i.e 0.37.2
• manually executed cleanup command as well
  thanos tools bucket cleanup --delete-delay=0s --objstore.config-file=/etc/thanos/objstore.yaml
• thanos_compact_todo_compactions, thanos_compact_todo_compaction_blocks, thanos_compact_todo_downsample_blocks, thanos_compact_todo_deletion_blocks are 0
• thanos_compact_iterations_total (increases at a steady rate) and thanos_compact_downsample_total (also increases at a steady rate)

What else should I investigate ?

Regards
Ashish

[dosubot[bot]]

Hello @ashishvaishno 🌟 I'm here to help you with your technical issues and questions. Let's figure this out together!

I found a similar closed issue that might be relevant to your situation. The suggested solution in that issue was to shard the compactor to do more work or to stop the compactor and use the tools bucket retention + clean command to delete all blocks that have exceeded their retention period. Additionally, setting the retention period to delete and running the cleanup command to delete old blocks was recommended. If using the cleanup command, configuring the delete delay flag is necessary, and setting it to 0 will delete blocks directly ^[1].

In your case, since you've already executed the cleanup command with --delete-delay=0s, it might be worth checking if the compactor is correctly configured to handle the retention settings and if there are any issues with the synchronization of metadata from the object store. Also, ensure that the compactor is not encountering any hidden errors during its operations, even if the logs appear normal.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[ashishvaishno]

running

thanos tools bucket retention --objstore.config-file=/etc/thanos/objstore.yaml
thanos tools bucket cleanup --delete-delay=0s --objstore.config-file=/etc/thanos/objstore.yaml

helped