Flaky test Test_MultiLevelCache

[CharlieTLe]

Describe the bug

The following test is failing:

--- FAIL: Test_MultiLevelCache (0.08s)
    --- FAIL: Test_MultiLevelCache/[StoreExpandedPostings]_Should_store_on_m2_only (0.00s)
        multilevel_cache_test.go:473: 
            	Error Trace:	/__w/cortex/cortex/pkg/storage/tsdb/multilevel_cache_test.go:473
            	Error:      	Not equal: 
            	            	expected: map[string][][]interface {}{"StoreExpandedPostings":[][]interface {}{[]interface {}{ulid.ULID{0x1, 0x69, 0xd1, 0xdf, 0x90, 0x90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, []*labels.Matcher{(*labels.Matcher)(0xc000f26ea0)}, []uint8{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}}}
            	            	actual  : map[string][][]interface {}{}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,26 +1,2 @@
            	            	-(map[string][][]interface {}) (len=1) {
            	            	- (string) (len=21) "StoreExpandedPostings": ([][]interface {}) (len=1) {
            	            	-  ([]interface {}) (len=3) {
            	            	-   (ulid.ULID) (len=16) {
            	            	-    00000000  01 69 d1 df 90 90 00 00  00 00 00 00 00 00 00 00  |.i..............|
            	            	-   },
            	            	-   ([]*labels.Matcher) (len=1) {
            	            	-    (*labels.Matcher)({
            	            	-     Type: (labels.MatchType) 0,
            	            	-     Name: (string) (len=4) "name",
            	            	-     Value: (string) (len=5) "value",
            	            	-     re: (*labels.FastRegexMatcher)(<nil>)
            	            	-    })
            	            	-   },
            	            	-   ([]uint8) (len=100) {
            	            	-    00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
            	            	-    00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
            	            	-    00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
            	            	-    00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
            	            	-    00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
            	            	-    00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
            	            	-    00000060  00 00 00 00                                       |....|
            	            	-   }
            	            	-  }
            	            	- }
            	            	+(map[string][][]interface {}) {
            	            	 }
            	Test:       	Test_MultiLevelCache/[StoreExpandedPostings]_Should_store_on_m2_only
FAIL
FAIL	github.com/cortexproject/cortex/pkg/storage/tsdb	0.896s

To Reproduce
Steps to reproduce the behavior:

1. go test -timeout=10s -count=10000 -run=Test_MultiLevelCache ./pkg/storage/tsdb

[dsabsay]

If I run it with the race detector enabled, it works.

cortex [1]$ go test -timeout=60s -count=10000 -race -run=Test_MultiLevelCache ./pkg/storage/tsdb
ok      github.com/cortexproject/cortex/pkg/storage/tsdb        47.374s
cortex [0]$ go test -timeout=60s -count=10000 -race -run=Test_MultiLevelCache ./pkg/storage/tsdb
ok      github.com/cortexproject/cortex/pkg/storage/tsdb        47.845s
cortex [0]$ go test -timeout=60s -count=10000 -race -run=Test_MultiLevelCache ./pkg/storage/tsdb
ok      github.com/cortexproject/cortex/pkg/storage/tsdb        48.056s
cortex [0]$ 

[dsabsay]

Found it. It's a bug in Thanos: thanos-io/thanos#8044

I implemented a test, fix, and submitted PRs.