Skip to content

Speed up freelist reconstruction on open - #1342

Open
djkazic wants to merge 2 commits into
etcd-io:mainfrom
djkazic:freelist-single-pass
Open

djkazic wants to merge 2 commits into
etcd-io:mainfrom
djkazic:freelist-single-pass

Conversation

@djkazic

@djkazic djkazic commented Nov 17, 2026

Copy link
Copy Markdown

Opening a database without a synced freelist (NoFreelistSync) rebuilds the freelist by scanning every page. On storage with a noticeable per-request latency this is very slow, and it gets worse when the database doesn't fit into memory. This PR addresses both, without changing what is verified during the scan.

Related: #305

Problem

DB.freepages() calls tx.recursivelyCheckBucket, which does three passes over each bucket from a single goroutine:

  1. tx.forEachPage to mark the pages as reachable
  2. recursivelyCheckPageKeyOrder
  3. Bucket.ForEachBucket with a cursor to find the nested buckets

All of it reads through the mmap, so every page that isn't cached is a synchronous page fault. The storage only ever sees one outstanding 4 KiB read, and a cold open takes roughly pages * read latency regardless of how many IOPS the device could serve.

If the database is larger than the available memory, the pages read by one pass have been evicted by the time the next pass needs them, so every page is read three times.

Changes

Single pass. The first commit adds recursivelyCheckBucketPages, which verifies reachability and key order of a page when it visits it, and descends into nested buckets from the leaf page they are stored in. DB.freepages() uses it instead of recursivelyCheckBucket. The same properties are verified and the same helpers report the errors.

Tx.Check is not changed. It can run in a writable transaction, where nested buckets have to be looked up through the in-memory nodes, which is what the cursor in the third pass does. The freelist is always reconstructed in a fresh read-only transaction, where the pages are all there is.

Prefetch. The second commit hints to the kernel with MADV_WILLNEED that all children of a branch page are about to be read when the branch page is visited, so that it starts reading them at once. Adjacent children are covered by one call, and ranges are aligned to the OS page size. This is a no-op on Windows.

The hint only covers pages the scan is about to read anyway, so it doesn't need the file to fit into memory, unlike MAP_POPULATE.

Benchmarks

bbolt.Open with NoFreelistSync on a 488 MB database (~101k pages) with a cold page cache, on a network block device with ~2 ms per read. The memory limits are cgroup limits on the process opening the database.

fits in memory 200 MB limit 64 MB limit warm cache
main 224.3s 649.9s 0.13s
this PR 5.1s 5.4s 5.5s 0.12-0.16s

main reads ~105k pages when the file fits and ~315k with the 200 MB limit. With this PR it is ~101k in all cases.

For comparison, MmapFlags: syscall.MAP_POPULATE (measured on v1.4.3) takes 2.4s when the file fits in memory and 481.2s with the 200 MB limit.

Testing

  • TestTx_recursivelyCheckBucketPages checks that the single pass reaches exactly the same pages as recursivelyCheckBucket on a database with multiple levels of branch pages, overflow pages, nested and inline buckets, for the default page size and for one smaller than an OS page.
  • TestPrefetchRange covers the alignment of the prefetched ranges.

@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: djkazic
Once this PR has been reviewed and has the lgtm label, please assign spzala for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tjungblu

Copy link
Copy Markdown
Contributor

/ok-to-test

can you do me a favor and do a more longer running benchmark? If possible, even a run of rw-heatmap from etcd.

We recently had some trouble with madvise in #940 which had some unfavorable effects on etcd downstream.

@djkazic

djkazic commented Nov 18, 2026

Copy link
Copy Markdown
Author

Sure thing! Ran the same harness CI uses (make test-benchmark-compare REF=main) but a good bit longer: 50M ops per run, 20 runs per side, about half an hour total.

        │    BASE     │                HEAD                │
        │   sec/op    │   sec/op     vs base               │
Write     905.4n ± 2%   904.2n ± 2%       ~ (p=0.931 n=20)
Read      44.30n ± 5%   43.92n ± 9%       ~ (p=0.820 n=20)
geomean   200.3n        199.3n       -0.50%

No real difference, which is what I'd expect: the bench never opens a db without a synced freelist, so it doesn't touch the changed code at all.

On the madvise worry, I think this one's a different animal from #940. MADV_RANDOM changes how the kernel treats the mapping from then on, while MADV_WILLNEED is just a one-off "please start reading these pages". To double check I opened the same 488 MB db on main and on this PR (kernel 6.12) and looked at /proc/self/smaps afterwards:

VMAs VmFlags major faults during open cold open warm open
main 1 rd sh mr mw me ms rr 104,700 3.94s 0.12s
this PR 1 rd sh mr mw me ms rr 6 0.49s 0.12s

So the mapping looks exactly the same afterwards, still one VMA with the same flags. That was on a local SSD by the way, no added latency.

It's also only called while rebuilding the freelist on open, and only if the freelist wasn't synced. From what I can tell etcd doesn't set NoFreelistSync, so it shouldn't hit this path at all, but you'd know better than me.

No rw-heatmap yet. My box is pretty slow, but I'll try to run it anyways.

Comment thread db.go Outdated
return
}

_ = madviseWillNeed(db.data[start:end:end])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why even return an error from the syscall when you're ignoring it here?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, no reason to. Dropped the return value, it's just a hint so there's nothing useful to do with a failure anyway.

Comment thread db.go
return
}

// Children that are next to each other in the file are requested with

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about overflow pages?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, they weren't covered and it mattered more than I thought. Only the first page of each child was hinted, so for a leaf with big values the later keys (which sit after the values, in the overflow pages) were still faulted in one at a time.

Tried it on a 325 MB db with 6 KB values, same slow disk setup as in the description:

cold open
main 104.2s
this PR before 66s
this PR now 2.2s

What it does now: once the children of a branch page are requested, it looks at each child's header and also requests the overflow pages the scan will actually read. For a branch child that's all of them, for a leaf only the ones holding a key or a bucket, so we don't pull in value bytes nobody looks at. The db from the description is unchanged (5.3s).

Comment thread tx_check.go
// Since nested buckets are looked up in the pages only, this must only be
// used with a read-only transaction, which has no in-memory nodes or buckets
// that differ from what is stored in the pages.
func (tx *Tx) recursivelyCheckBucketPages(rootPageId common.Pgid, reachable map[common.Pgid]*common.Page,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need both functions (recursivelyCheckBucket+recursivelyCheckBucketPages)? they look almost the same

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly no! The page walking part was a copy of recursivelyCheckPageKeyOrderInternal, so I folded it into that one with two optional callbacks (one per page, one per bucket entry). Tx.Check passes nil for both and behaves as before.

The two bucket level functions I'd keep though. I actually tried using the single pass for Tx.Check first, and it breaks on writable transactions: it finds nested buckets by reading the leaf pages, so a bucket deleted earlier in the same tx is still found in the old page and its pages get reported as "reachable freed". The existing one goes through the cursor and sees the in-memory nodes, so it has to stay for that case. freepages() always runs on a fresh read-only tx so it doesn't have that problem.

Comment thread bolt_unix.go Outdated
// madviseWillNeed advises the kernel that the given part of the mmap is
// going to be accessed soon.
func madviseWillNeed(b []byte) error {
return unix.Madvise(b, syscall.MADV_WILLNEED)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually wonder: why can't we set this on the entire database file when we're opening the database? is there any drawback?

It seems to me that will speedup the entire operation at almost no additional cost and without the additional code we're introducing here - and it is one syscall vs. a couple of thousands recursively issued.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Let me investigate that. It's possible we can simplify the code a bit.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looked into it, and it's a bit of a mixed bag. Cold open of the 488 MB db from the description:

fits in memory 200 MB limit 64 MB limit
main 224s 650s
this PR 5.5s 5.4s 6.0s
one madvise over the whole file 217s
whole file, in 128 KiB chunks 0.3-0.6s 200s 196s
whole file in chunks + this PR 0.3-0.9s 5.7s 5.8s

A single call doesn't do anything useful. As far as I can tell the kernel caps one WILLNEED request at a readahead window, so it reads the first few hundred KB and that's it. It has to be done in chunks, so it ends up being a few thousand calls either way.

In chunks it's great as long as the file fits in the page cache, since it turns into big sequential reads. Once it doesn't fit, the pages get evicted before the scan gets to them and we're back to one fault at a time. It also reads free pages and values, which the scan never looks at.

Doing both gives the best of each, at the cost of reading the file twice when it doesn't fit. One caveat: my slow disk setup only adds latency and has no throughput limit, so the whole file numbers are on the optimistic side.

I kept the per-page hints since they hold up in both cases, but happy to add the whole file pass on top if you'd like that. It's only in the freelist rebuild path either way, so opens with a synced freelist wouldn't be affected.

@djkazic

djkazic commented Nov 20, 2026

Copy link
Copy Markdown
Author

Got the rw-heatmap run done. Took a while on my box, so I cut it down a bit: 4 of the 8 ratios (1/128, 1/8, 8/1, 128/1), full value size and connection grid, 3 repeats of 50k requests per cell instead of the defaults. etcd at dd2c518ff, bbolt main at 622eada vs this PR.

Geomean of PR vs main across the 49 cells per ratio:

R/W ratio read write
1/128 +0.9% +0.8%
1/8 +1.6% +1.8%
8/1 -0.5% -0.4%
128/1 +0.0% +0.3%

So nothing systematic that I can see. There are a few noisy cells in both directions (the 1/128 ratio especially, and one -22% cell at 8/1), which I'm putting down to the box not being idle and the reduced sample count rather than the change.

Happy to share the raw CSVs or rerun specific cells with more repeats if something looks off to you.

@djkazic

djkazic commented Nov 20, 2026

Copy link
Copy Markdown
Author
cmp_read cmp_write

Finding the reachable pages took three passes over every bucket: one to
mark its pages, one to check the key order and one to find the nested
buckets. Do all of it in one pass when reconstructing the freelist. On a
database that doesn't fit into memory, every page was read three times.

Tx.Check is unchanged, it has to look up nested buckets through the
in-memory nodes of a writable transaction.

Signed-off-by: Kevin Cai <kevincai96@gmail.com>
The pages are read through the mmap one page fault at a time, so a cold
open takes the number of pages times the latency of a read. Hint to the
kernel with MADV_WILLNEED that all children of a branch page are about
to be read, so that it starts reading them at once. That includes the
overflow pages of the children that hold keys.

Signed-off-by: Kevin Cai <kevincai96@gmail.com>
@djkazic
djkazic force-pushed the freelist-single-pass branch from 39fc284 to 6248acf Compare November 20, 2026 16:22
@kubernetes-prow kubernetes-prow Bot added size/XL and removed size/L labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

2 participants