feat(stores): add Valkey Search vector store backend#10770
Open
daric93 wants to merge 3 commits into
Open
Conversation
daric93
force-pushed
the
feature/valkey-store-backend
branch
from
f72d11f to
17c3e0a
Compare
Add a new built-in Go gRPC store backend 'valkey-store' that implements the four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*) using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the existing per-request 'backend' field on /stores, so there is no proto or HTTP API change, and it mirrors the in-memory local-store while adding persistence across restarts and opt-in HNSW. Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is created lazily on first Set (FLAT+COSINE by default), cosine similarity is derived as 1-distance, and namespaces get a collision-resistant token. Includes unit tests (valkey-go mock) and env-gated integration tests against valkey/valkey-bundle, plus build/matrix/gallery wiring and docs. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com>
- Load now recovers the persisted vector DIM from FT.INFO (not just index existence), so a post-restart Set/Find validates against the real DIM instead of silently re-learning a wrong one and dropping mismatched vectors from the index. This also restores Find's dimension check after a restart. - StoresFind treats a dropped/missing index as an empty store (empty result, no error) and clears the stale indexCreated flag, matching local-store's empty-store behaviour. - StoresSet reuses checkDims for its per-key length check so the four RPCs share one dimension-guard implementation. - Add unit tests for FT.INFO dimension recovery, loadIndexState, and the dropped-index Find path. Assisted-by: Kiro:claude-opus-4.8 Signed-off-by: Daria Korenieva <daric2612@gmail.com>
…il-fast Addresses external review comments on the valkey-store backend: - StoresFind now rejects a nil/empty query Key before dereferencing it, so a malformed gRPC request can no longer panic the backend. - TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT (custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs. - Config integer parsing now fails fast on a malformed value (e.g. VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the fail-fast behaviour of the index-algo/distance-metric validation. - Add VALKEY_DB (SELECT n) support for logical-DB isolation. - Cap the human-readable part of a namespace token at 64 chars so a very long model name cannot produce an unbounded key prefix / index name (the appended short hash keeps distinct namespaces collision-free). - Document the KNN-query injection-safety invariant (fields are constants) and why StoresGet uses a single aggregate DoMulti deadline for reads. - Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing, and VALKEY_DB parsing/validation; docs + .env updated for the new vars. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com>
daric93
force-pushed
the
feature/valkey-store-backend
branch
from
17c3e0a to
86ac171
Compare
richiejp
reviewed
Jul 11, 2026
| # VALKEY_HNSW_M=16 | ||
| # VALKEY_HNSW_EF_CONSTRUCTION=200 | ||
| # VALKEY_HNSW_EF_RUNTIME=10 | ||
| # VALKEY_REQUEST_TIMEOUT_MS=5000 |
Collaborator
There was a problem hiding this comment.
I think at least some of these should go in a model config. Then you can have multiple stores configs.
| } | ||
|
|
||
| BeforeEach(func() { | ||
| valkeyAddr = os.Getenv("VALKEY_ADDR") |
Collaborator
There was a problem hiding this comment.
Please avoid putting env accesses throughout the code. Model config is the natural place instead of env vars.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a new built-in Go gRPC store backend
valkey-storethat implements the fourStores*RPCs (Set / Get / Delete / Find) against the Valkey Search module (FT.*vector similarity), selectable via the existing per-requestbackendfield on the/storesendpoints. It mirrors the in-memorylocal-storeand adds persistence across restarts plus opt-in HNSW."backend": "valkey-store"(alias"valkey").github.com/valkey-io/valkey-gov1.0.76 (official, pure Go, no CGO).hex(little-endian float32); theFTindex is created lazily on firstSet(FLAT + COSINE by default), and cosine similarity is derived as1 - distanceto matchlocal-storesemantics exactly.sanitize(name) + short sha256).VALKEY_*env vars (addr, auth, TLS, index algo + HNSW knobs, distance metric, per-command request timeout, mandatoryClientName).Why
The stores subsystem is designed to be pluggable, but the only shipped backend (
local-store) is in-memory and loses all data on restart, and itsFindis an O(N) scan. Face/voice biometric registries and the router embedding cache all consume this seam, so a durable, scalable Valkey-backed backend benefits them with zero caller changes.Why
valkey-goand notvalkey-glide? The officialvalkey-glideGo client is built on a Rust core via FFI, which forces CGO and a per-arch native library — that breakslocal-store'sCGO_ENABLED=0static build and the Linux/Darwin backend matrix.valkey-gois pure Go with a typedFT.SEARCHbuilder, aVectorString32encoder, and a gomock mock package for unit tests.Testing
valkey-gomock, no container): 35 specs — RPC command-shape/wire contract, empty/len/dim rejects, omit-missing Get, tolerate-missing Delete,topK<1reject, distance→similarity conversion (0→1, 1→0, 2→−1), lazyFT.CREATE, HNSW arg-shape, key-encoding round-trip (incl.-0.0/NaN), namespace-token collision resistance, config parsing.make test/go run ...ginkgo -r backend/go/valkey-store.Label("valkey"), env-gated onVALKEY_ADDR, againstvalkey/valkey-bundle:9.1.0on :6379): 14 specs mirroring thelocal-storesuite one-for-one (set/get/delete/find, exact cosine for orthogonal/opposite unit + non-unit vectors, triangle inequality incl. random 768-d) plus a persistence/restart test and aCLIENT LISTclient-name assertion. Index back-fill is absorbed with a boundedfindpoll (notime.Sleep). Skipped automatically whenVALKEY_ADDRis unset, so unit CI needs no container.golangci-lintclean on the changed packages. All Valkey logic lives inbackend/go/valkey-store/(outsideCOVERAGE_COVERPKG), so the coverage ratchet is unaffected.Run integration locally:
Scope / follow-ups
Purely additive and opt-in per request. Deferred to follow-ups: hybrid / metadata-filter search (needs new API surface beyond the four RPCs), Valkey Cluster mode, and single-instance multi-namespace consolidation.
Notes for Reviewers
cc @mudler
Signed commits