Repository navigation
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: djkazic The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/ok-to-test can you do me a favor and do a more longer running benchmark? If possible, even a run of rw-heatmap from etcd. We recently had some trouble with madvise in #940 which had some unfavorable effects on etcd downstream. |
|
Sure thing! Ran the same harness CI uses ( No real difference, which is what I'd expect: the bench never opens a db without a synced freelist, so it doesn't touch the changed code at all. On the madvise worry, I think this one's a different animal from #940.
So the mapping looks exactly the same afterwards, still one VMA with the same flags. That was on a local SSD by the way, no added latency. It's also only called while rebuilding the freelist on open, and only if the freelist wasn't synced. From what I can tell etcd doesn't set No rw-heatmap yet. My box is pretty slow, but I'll try to run it anyways. |
| return | ||
| } | ||
|
|
||
| _ = madviseWillNeed(db.data[start:end:end]) |
There was a problem hiding this comment.
why even return an error from the syscall when you're ignoring it here?
There was a problem hiding this comment.
Good point, no reason to. Dropped the return value, it's just a hint so there's nothing useful to do with a failure anyway.
| return | ||
| } | ||
|
|
||
| // Children that are next to each other in the file are requested with |
There was a problem hiding this comment.
how about overflow pages?
There was a problem hiding this comment.
Good catch, they weren't covered and it mattered more than I thought. Only the first page of each child was hinted, so for a leaf with big values the later keys (which sit after the values, in the overflow pages) were still faulted in one at a time.
Tried it on a 325 MB db with 6 KB values, same slow disk setup as in the description:
| cold open | |
|---|---|
| main | 104.2s |
| this PR before | 66s |
| this PR now | 2.2s |
What it does now: once the children of a branch page are requested, it looks at each child's header and also requests the overflow pages the scan will actually read. For a branch child that's all of them, for a leaf only the ones holding a key or a bucket, so we don't pull in value bytes nobody looks at. The db from the description is unchanged (5.3s).
| // Since nested buckets are looked up in the pages only, this must only be | ||
| // used with a read-only transaction, which has no in-memory nodes or buckets | ||
| // that differ from what is stored in the pages. | ||
| func (tx *Tx) recursivelyCheckBucketPages(rootPageId common.Pgid, reachable map[common.Pgid]*common.Page, |
There was a problem hiding this comment.
do we need both functions (recursivelyCheckBucket+recursivelyCheckBucketPages)? they look almost the same
There was a problem hiding this comment.
Mostly no! The page walking part was a copy of recursivelyCheckPageKeyOrderInternal, so I folded it into that one with two optional callbacks (one per page, one per bucket entry). Tx.Check passes nil for both and behaves as before.
The two bucket level functions I'd keep though. I actually tried using the single pass for Tx.Check first, and it breaks on writable transactions: it finds nested buckets by reading the leaf pages, so a bucket deleted earlier in the same tx is still found in the old page and its pages get reported as "reachable freed". The existing one goes through the cursor and sees the in-memory nodes, so it has to stay for that case. freepages() always runs on a fresh read-only tx so it doesn't have that problem.
| // madviseWillNeed advises the kernel that the given part of the mmap is | ||
| // going to be accessed soon. | ||
| func madviseWillNeed(b []byte) error { | ||
| return unix.Madvise(b, syscall.MADV_WILLNEED) |
There was a problem hiding this comment.
I actually wonder: why can't we set this on the entire database file when we're opening the database? is there any drawback?
It seems to me that will speedup the entire operation at almost no additional cost and without the additional code we're introducing here - and it is one syscall vs. a couple of thousands recursively issued.
There was a problem hiding this comment.
Good point. Let me investigate that. It's possible we can simplify the code a bit.
There was a problem hiding this comment.
Looked into it, and it's a bit of a mixed bag. Cold open of the 488 MB db from the description:
| fits in memory | 200 MB limit | 64 MB limit | |
|---|---|---|---|
| main | 224s | 650s | |
| this PR | 5.5s | 5.4s | 6.0s |
| one madvise over the whole file | 217s | ||
| whole file, in 128 KiB chunks | 0.3-0.6s | 200s | 196s |
| whole file in chunks + this PR | 0.3-0.9s | 5.7s | 5.8s |
A single call doesn't do anything useful. As far as I can tell the kernel caps one WILLNEED request at a readahead window, so it reads the first few hundred KB and that's it. It has to be done in chunks, so it ends up being a few thousand calls either way.
In chunks it's great as long as the file fits in the page cache, since it turns into big sequential reads. Once it doesn't fit, the pages get evicted before the scan gets to them and we're back to one fault at a time. It also reads free pages and values, which the scan never looks at.
Doing both gives the best of each, at the cost of reading the file twice when it doesn't fit. One caveat: my slow disk setup only adds latency and has no throughput limit, so the whole file numbers are on the optimistic side.
I kept the per-page hints since they hold up in both cases, but happy to add the whole file pass on top if you'd like that. It's only in the freelist rebuild path either way, so opens with a synced freelist wouldn't be affected.
|
Got the rw-heatmap run done. Took a while on my box, so I cut it down a bit: 4 of the 8 ratios (1/128, 1/8, 8/1, 128/1), full value size and connection grid, 3 repeats of 50k requests per cell instead of the defaults. etcd at dd2c518ff, bbolt main at 622eada vs this PR. Geomean of PR vs main across the 49 cells per ratio:
So nothing systematic that I can see. There are a few noisy cells in both directions (the 1/128 ratio especially, and one -22% cell at 8/1), which I'm putting down to the box not being idle and the reduced sample count rather than the change. Happy to share the raw CSVs or rerun specific cells with more repeats if something looks off to you. |
Finding the reachable pages took three passes over every bucket: one to mark its pages, one to check the key order and one to find the nested buckets. Do all of it in one pass when reconstructing the freelist. On a database that doesn't fit into memory, every page was read three times. Tx.Check is unchanged, it has to look up nested buckets through the in-memory nodes of a writable transaction. Signed-off-by: Kevin Cai <kevincai96@gmail.com>
The pages are read through the mmap one page fault at a time, so a cold open takes the number of pages times the latency of a read. Hint to the kernel with MADV_WILLNEED that all children of a branch page are about to be read, so that it starts reading them at once. That includes the overflow pages of the children that hold keys. Signed-off-by: Kevin Cai <kevincai96@gmail.com>
39fc284 to
6248acf
Compare


Opening a database without a synced freelist (
NoFreelistSync) rebuilds the freelist by scanning every page. On storage with a noticeable per-request latency this is very slow, and it gets worse when the database doesn't fit into memory. This PR addresses both, without changing what is verified during the scan.Related: #305
Problem
DB.freepages()callstx.recursivelyCheckBucket, which does three passes over each bucket from a single goroutine:tx.forEachPageto mark the pages as reachablerecursivelyCheckPageKeyOrderBucket.ForEachBucketwith a cursor to find the nested bucketsAll of it reads through the mmap, so every page that isn't cached is a synchronous page fault. The storage only ever sees one outstanding 4 KiB read, and a cold open takes roughly
pages * read latencyregardless of how many IOPS the device could serve.If the database is larger than the available memory, the pages read by one pass have been evicted by the time the next pass needs them, so every page is read three times.
Changes
Single pass. The first commit adds
recursivelyCheckBucketPages, which verifies reachability and key order of a page when it visits it, and descends into nested buckets from the leaf page they are stored in.DB.freepages()uses it instead ofrecursivelyCheckBucket. The same properties are verified and the same helpers report the errors.Tx.Checkis not changed. It can run in a writable transaction, where nested buckets have to be looked up through the in-memory nodes, which is what the cursor in the third pass does. The freelist is always reconstructed in a fresh read-only transaction, where the pages are all there is.Prefetch. The second commit hints to the kernel with
MADV_WILLNEEDthat all children of a branch page are about to be read when the branch page is visited, so that it starts reading them at once. Adjacent children are covered by one call, and ranges are aligned to the OS page size. This is a no-op on Windows.The hint only covers pages the scan is about to read anyway, so it doesn't need the file to fit into memory, unlike
MAP_POPULATE.Benchmarks
bbolt.OpenwithNoFreelistSyncon a 488 MB database (~101k pages) with a cold page cache, on a network block device with ~2 ms per read. The memory limits are cgroup limits on the process opening the database.main reads ~105k pages when the file fits and ~315k with the 200 MB limit. With this PR it is ~101k in all cases.
For comparison,
MmapFlags: syscall.MAP_POPULATE(measured on v1.4.3) takes 2.4s when the file fits in memory and 481.2s with the 200 MB limit.Testing
TestTx_recursivelyCheckBucketPageschecks that the single pass reaches exactly the same pages asrecursivelyCheckBucketon a database with multiple levels of branch pages, overflow pages, nested and inline buckets, for the default page size and for one smaller than an OS page.TestPrefetchRangecovers the alignment of the prefetched ranges.