TSDB Retention Period is not working in mimir 2.17.0

[vikramkumar-git]

Hi Team,

I have deployed Mimir 2.17.0 in EKS via the mimir-distributed Helm chart, and it's multitenant-enabled.

I have requirements to fetch metrics from ingester nodes till "12h" and after that fetch from the S3 bucket via the store-gateway. As per Grafana documentation, **blocks_storage.tsdb.retention_period: 12h** flag to keep block within the ingester node for "12h".
But somehow it is not working.

Grafana dashboard screenshot of missing data, where the data is coming from the ingester for the last 4 hours, approximately. Then the data is missing and coming from the store gateway as "query_store_after: 12h"

mimir UI page for ingester tsdb

result of query: sum(cortex_ingester_tsdb_blocks_loaded) by (pod)

Below is my Mimir Helm chart configuration.

mimir:
    usage_stats:
      enabled: true
    structuredConfig:
      multitenancy_enabled: true
      server:
        http_server_write_timeout: 600s
        grpc_server_max_recv_msg_size: 536870912
        grpc_server_max_send_msg_size: 536870912
      api:
        alertmanager_http_prefix: "/alertmanager"
        prometheus_http_prefix: "/prometheus"
      querier:
        query_engine: "mimir"
        max_concurrent: 32
        query_store_after: 12h
      memberlist:
        cluster_label: mimir
        join_members:
          - dns+mimir-application-gossip-ring.monitoring.svc.cluster.local:7946
      frontend:
        parallelize_shardable_queries: true
        split_queries_by_interval: 24h
        cache_results: true
        max_outstanding_per_tenant: 1024
        results_cache:
          backend: "memcached"
          compression: "snappy"
        grpc_client_config:
          grpc_compression: s2
          max_recv_msg_size: 268435456
          max_send_msg_size: 268435456
      frontend_worker:
        grpc_client_config:
          grpc_compression: s2
          max_recv_msg_size: 268435456
          max_send_msg_size: 268435456
      query_scheduler:
        grpc_client_config:
          grpc_compression: s2
          max_recv_msg_size: 104857600
          max_send_msg_size: 104857600
      ruler:
        tenant_federation:
          enabled: true
        ruler_client:
          grpc_compression: s2
          max_recv_msg_size: 104857600
          max_send_msg_size: 104857600
        query_frontend:
          grpc_client_config:
            grpc_compression: s2
            max_recv_msg_size: 104857600
            max_send_msg_size: 104857600
      ingester_client:
        grpc_client_config:
          grpc_compression: s2
          max_recv_msg_size: 104857600
          max_send_msg_size: 104857600
      alertmanager:
        alertmanager_client:
          grpc_compression: "s2"
      compactor:
        block_ranges:
          - 12h0m0s
          - 24h0m0s
      limits:
        request_rate: 0
        request_burst_size: 0
        ingestion_rate: 50000
        ingestion_burst_size: 100000
        max_label_names_per_series: 256
        max_global_series_per_user: 1073741824
        max_global_series_per_metric: 1073741824
        max_global_metadata_per_user: 1073741824
        max_global_metadata_per_metric: 1073741824
        max_global_exemplars_per_user: 1048576
        native_histograms_ingestion_enabled: true
        max_query_parallelism: 256
        query_sharding_total_shards: 256
        query_sharding_max_sharded_queries: 512
        cardinality_analysis_enabled: true
        out_of_order_time_window: 15m
        ruler_max_rules_per_rule_group: 0
        ruler_max_rule_groups_per_tenant: 0
        compactor_blocks_retention_period: 45d 
      common:
        storage:
          backend: s3
          s3:
            endpoint: s3.eu-west-1.amazonaws.com
            region: eu-west-1
            bucket_name: <bucket-name>
            insecure: false
      alertmanager_storage:
        backend: "s3"
        storage_prefix: "alertmanager"
        s3:
          endpoint: s3.eu-west-1.amazonaws.com
          region: eu-west-1
          bucket_name: <bucket-name>
      blocks_storage:
        backend: "s3"
        storage_prefix: "blocks"
        s3:
          endpoint: s3.eu-west-1.amazonaws.com
          region: eu-west-1
          bucket_name: <bucket-name>
        bucket_store:
          ignore_blocks_within: 10h
          index_cache:
            backend: "memcached"
            memcached:
              timeout: 60s
          chunks_cache:
            backend: "memcached"
            memcached:
              timeout: 60s
          metadata_cache:
            backend: "memcached"
            memcached:
              timeout: 60s
        tsdb:
          ship_interval: 30s
          wal_compression_enabled: true
          flush_blocks_on_shutdown: true
          retention_period: 12h
          ship_interval: 30s
      ruler_storage:
        backend: "s3"
        storage_prefix: "ruler"
        s3:
          endpoint: s3.eu-west-1.amazonaws.com
          region: eu-west-1
          bucket_name: <bucket-name>
      tenant_federation:
        enabled: true

runtimeConfig:
    overrides:
      tenanat-1:
        ingestion_rate: 100000 
        ingestion_burst_size: 200000
        compactor_blocks_retention_period: 15d
      tenanat-2:
        ingestion_rate: 150000             
        ingestion_burst_size: 400000     
        compactor_blocks_retention_period: 15d
      tenanat-3:
        compactor_block_upload_enabled: true
        compactor_blocks_retention_period: 45d
        ingestion_rate: 300000 
        ingestion_burst_size: 600000
      tenanat-4:
        compactor_blocks_retention_period: 45d
        ingestion_rate: 300000 
        ingestion_burst_size: 400000

Please help here to resolve the issue.

My requirements are that I want to serve the last 12 hours of metrics data from ingesters only, and the old data from the store-gateway.

And I can't downgrade to a lower version as I am using the Prometheus-remote-write V2 protocol, as it's supported in this Mimir 2.17.0 version only

[aknuds1]

Converting to a discussion, as it's more of a support request than a bug report.

[Nachtfalkeaw]

The Default configuration should be 12h so No need to Change anything for this in the Default config.

[vikramkumar-git]

default value is 13h https://grafana.com/docs/mimir/v2.15.x/configure/configuration-parameters/#blocks_storage as per grafana documetation but with default config its also not working. so i made it 12h to check but no luck..

mimir version 2.17.0

[vikramkumar-git]

@Nachtfalkeaw

[Nachtfalkeaw]

Whats your replication Factor?
Do you use persistent storage for ingesters?
Any ingesters rebooted within this time range?

[vikramkumar-git]

@Nachtfalkeaw

replication factor is default(3).

yes ingester is running as a statefulsets with individual PVC with each ingester.

and pvc is max used 29%.

none of ingester pod restarted since last 28h

as per ingester logs for "tenant-1" block (ulid=01K3XZ4Z96W2VKP38WNHT85Y03)

created at ts=2025-08-30T17:07:09.999472442Z
uploaded at ts=2025-08-30T17:07:30.997719899Z
deleted at ts=2025-08-30T19:09:04.290895253Z

`ts=2025-08-30T17:07:09.999472442Z caller=handler.go:87 level=info user=tenant-1 caller=compact.go:786 time=2025-08-30T17:07:09.999449722Z msg="write block" mint=1756569600000 maxt=1756576800000 ulid=01K3XZ4Z96W2VKP38WNHT85Y03 duration=265.023722ms ooo=true
ts=2025-08-30T17:07:09.999520853Z caller=handler.go:87 level=info user=tenant-1 caller=db.go:1564 time=2025-08-30T17:07:09.999512353Z msg="out-of-order compaction completed" duration=265.093753ms ulids=[01K3XZ4Z96W2VKP38WNHT85Y03]

ts=2025-08-30T17:07:30.997719899Z caller=shipper.go:161 user=tenant-1 method=Ingester.Shipper.Sync level=info msg="uploading new block to long-term storage" block=01K3XZ4Z96W2VKP38WNHT85Y03
ts=2025-08-30T17:07:31.269102317Z caller=shipper.go:169 user=tenant-1 method=Ingester.Shipper.Sync level=info msg="finished uploading new block to long-term storage" block=01K3XZ4Z96W2VKP38WNHT85Y03

ts=2025-08-30T19:09:04.290895253Z caller=handler.go:87 level=info user=tenant-1 caller=db.go:1939 time=2025-08-30T19:09:04.290873523Z msg="Deleting obsolete block" block=01K3XZ4Z96W2VKP38WNHT85Y03
`

just after 2hr block got deleted. according to config(retention_period: 12h) same block should be kept in ingester node for atleast 12h.

[vikramkumar-git]

Same has been fixed. some issues with configuration..

[alemsh]

what configuration changes had you made to resolve the issue? I am also running into a problem with ingester local disk constantly expanding