Skip to content

Keep the prefix range end when paging through a chunked get - #542

Open
lbarrosby wants to merge 1 commit into
etcd-io:mainfrom
lbarrosby:fix-chunked-get-range
Open

lbarrosby wants to merge 1 commit into
etcd-io:mainfrom
lbarrosby:fix-chunked-get-range

Conversation

@lbarrosby

Copy link
Copy Markdown
Contributor

client.Get pages through a prefix with clientv3.WithPrefix() and WithLimit(chunkSize), and for the next page it moves the key to the last key returned plus \x00. WithPrefix does not keep the range of the original prefix. It computes the range end from the key of each request (op.end = getPrefix(op.key)). So the second request asks for [/registry/pods/default/pod-2\x00, /registry/pods/default/pod-2\x01), gets nothing back with More: false, and the loop stops.

augerctl get pages with --chunk-size 500 by default, so any resource with more than 500 objects was listed only up to the first chunk, with no error. With a smaller --chunk-size the truncation shows up sooner.

The chunked branch now uses WithRange(clientv3.GetPrefixRangeEnd(path)) with the limit (and the pinned revision), so every page stays inside the prefix. The single-key and unpaged paths are unchanged.

TestGetChunkedReturnsEveryKey runs Get on a clientv3.Client whose KV is a small fake that serves [key, end) with limit and More the way etcd does. It covers one chunk, several chunks, and several chunks with --limit. On main the two multi-chunk cases return 3 of the 8 pods. With the fix all cases return every key in order. go test ./pkg/... ./cmd/... and golangci-lint run --config tools/.golangci.yaml ./... are clean.

AI disclosure: written with the help of Claude Code, reviewed and tested by me.

clientv3.WithPrefix computes the range end from the key of each request.
After the first chunk the key is the last key returned plus a NUL byte,
so the next request only covered keys under that key, returned nothing
and ended the loop. augerctl get, which pages by 500 by default, listed
at most one chunk of any resource. Use WithRange with the range end of
the whole prefix for every chunk.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lbarrosby
Once this PR has been reviewed and has the lgtm label, please assign ivanvc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant