Skip to content

Use the latest version of each key in key summaries - #543

Open
lbarrosby wants to merge 1 commit into
etcd-io:mainfrom
lbarrosby:list-summaries-latest-value
Open

lbarrosby wants to merge 1 commit into
etcd-io:mainfrom
lbarrosby:list-summaries-latest-value

Conversation

@lbarrosby

Copy link
Copy Markdown
Contributor

ListKeySummaries walks the key bucket in revision order and builds the summary of a key from the first record it sees, which is the oldest version still in the db file. For later versions it only updates Version, ValueSize and the counters. The value, the type meta and the filters are never looked at again, so:

  • auger extract --fields=value and --template '{{.Value...}}' print the oldest value of the key, not the one at the requested revision (or the latest);
  • --filter is matched against the oldest value: a key whose old value matched is listed even if the current value does not;
  • if the oldest version does not match, the key is not stored, the next version is treated as a new key, and VersionCount is off by the skipped versions.

This shows up on any db file that still has history for a key, for example a snapshot taken between two compactions.

Reproduction with a db that has two versions of one ConfigMap (data.mode is old at revision 2 and new at revision 3):

# main
$ auger extract -f db --fields=key,version-count,value
/registry/configmaps/default/app 2 {"apiVersion":"v1","data":{"mode":"old"},...}
$ auger extract -f db --template '{{.Key}} {{.Stats.VersionCount}}' --filter '.Value.data.mode=old'
/registry/configmaps/default/app 2
$ auger extract -f db --template '{{.Key}} {{.Stats.VersionCount}}' --filter '.Value.data.mode=new'
/registry/configmaps/default/app 1

# this branch
$ auger extract -f db --fields=key,version-count,value
/registry/configmaps/default/app 2 {"apiVersion":"v1","data":{"mode":"new"},...}
$ auger extract -f db --template '{{.Key}} {{.Stats.VersionCount}}' --filter '.Value.data.mode=old'
$ auger extract -f db --template '{{.Key}} {{.Stats.VersionCount}}' --filter '.Value.data.mode=new'
/registry/configmaps/default/app 2

The walk now only collects stats and keeps the raw value of the latest version of each key (tombstones still drop the key). After the walk each remaining value is decoded once and the filters run on it. As a side effect only one value per key is decoded instead of the oldest one of every key, so the cost does not grow.

TestListKeySummariesUsesLatestValue covers no filter, a filter on the latest value, a filter on an older value and --revision pointing at the older version. On main the first three cases fail; with the change all pass. go test ./pkg/... ./cmd/... and golangci-lint run --config tools/.golangci.yaml ./... are clean.

AI disclosure: written with the help of Claude Code, reviewed and tested by me.

ListKeySummaries decoded the value and ran the filters only for the
first version of a key it walked, which is the oldest one. Later
versions only updated the version and the sizes. So --fields=value and
--template printed the oldest value, a filter matched against the
oldest value, and a key whose oldest version did not match was counted
again from its next version.

Keep the raw value of the latest version while walking, then decode it
and apply the filters once per key.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lbarrosby
Once this PR has been reviewed and has the lgtm label, please assign ivanvc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant