Open
Description
Describe the bug
Running cortex image with race detection it was found a condition when deleting blocks from a user tsdb and creating new tsdb. This seems to be an ok scenario as we can be cleaning tsdb while receiving data for a new tenant, but we should eliminate the race condition
To Reproduce
Steps to reproduce the behavior:
- Start Cortex (v.1.18) with -race enabled
- Perform Operations Write while waiting for delete loop to be executed
Expected behavior
No race condition detected
Environment:
- Infrastructure: kubernets
- Deployment tool: helm
Additional Context
@yeya24 initial test data
WARNING: DATA RACE
Read at 0x00c005626100 by goroutine 5919:
golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).blocksToDelete()
/local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester/ingester.go:510 +0x58
golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).blocksToDelete-fm()
<autogenerated>:1 +0x51
golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb.(*DB).reloadBlocks()
/local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb/db.go:1558 +0x310
golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb.(*DB).run()
/local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb/db.go:1078 +0x771
golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb.open.gowrap1()
/local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb/db.go:1022 +0x4f
Previous write at 0x00c005626100 by goroutine 2314:
golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*Ingester).createTSDB()
/local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester/ingester.go:2346 +0x2415
golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*Ingester).openExistingTSDB.func1()
/local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester/ingester.go:2456 +0x3e4
golang.a2z.com/Cortex/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
/local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x91