Skip to content

AGENTS.md

Latest commit

 

History

History
113 lines (92 loc) · 6.38 KB

AGENTS.md

File metadata and controls

113 lines (92 loc) · 6.38 KB
Raw

AGENTS.md

Project Overview

valkey-datasketches is a Valkey module that provides probabilistic data structure commands (CPC, HLL, Theta, KLL, etc.) powered by the DataSketches family of algorithms.

Current Implementation Status

  • Fully implemented sketch families: CPC, HLL, Theta, KLL, REQ, Reservoir, LongSketch, ItemSketch
  • Placeholder command modules only: TDigest, Tuple, VarOpt
  • SKETCH.INFO reports the implemented algorithms from the registry in src/sketches/algorithms.cc

Build System

  • CMake 3.17+ with C++11 standard (strict — no C++14/17 features)
  • Build generator: Ninja (via build.sh) or Make
  • Main library target: libdatasketches (shared .so/.dylib)
  • Build: ./build.sh --release (fast, module only) or cmake -B build -G Ninja && ninja
  • Unit tests: ./build.sh --unit or ./build/tst/unit/unitTests
  • Integration tests: ./build.sh --integration or tst/integration/run.sh test
  • Lint: scripts/lint.sh

Environment Variables

Variable Default Description
VALKEY_SOURCE_DIR (none) Path to a local Valkey source tree. When set, skips GitHub clone and copies valkeymodule.h directly. Required for fast builds.
VALKEY_DS_DEPS_CACHE build/_deps Shared cache for Valkey/GoogleTest builds across worktrees.
SERVER_VERSION unstable Valkey version tag (e.g. 8.1).
ASAN_BUILD (none) Set to enable Address Sanitizer.
CFLAGS -g -O3 -fno-omit-frame-pointer -Wall -Werror -Wextra Override compiler flags.

CMake Options

Option Default Description
BUILD_RELEASE OFF Build only the module, skip tests and Valkey compilation.
ENABLE_UNIT_TESTS ON Build GoogleTest unit tests.
ENABLE_INTEGRATION_TESTS ON Build integration tests (requires Valkey compilation).
ENABLE_ASAN OFF Enable Address Sanitizer.
VALKEY_SOURCE_DIR (none) Path to local Valkey source (see env var above).
DEPS_CACHE_DIR build/_deps Override dependency cache location.

Directory Structure

src/
├── module.cc              # Module entry point (ValkeyModule_OnLoad)
├── include/               # Build-generated headers (valkeymodule.h) — DO NOT EDIT
├── commands/              # Command JSON metadata
├── modules/               # Per-algorithm Valkey module bindings (commands + type registration)
│   ├── cpc/               # CPC sketch commands + type registration
│   ├── hll/               # HLL sketch commands + type registration
│   ├── theta/             # Theta sketch commands + set operations
│   ├── tuple/             # Tuple sketch commands
│   ├── kll/               # KLL sketch commands + union support
│   ├── req/               # REQ sketch commands + union support
│   ├── tdigest/           # TDigest sketch commands
│   ├── longsketch/        # LongSketch frequent-items commands
│   ├── itemsketch/        # ItemSketch frequent-items commands
│   ├── countmin/          # CountMin sketch commands
│   ├── reservoir/         # Reservoir sampling commands + type registration
│   └── varopt/            # VarOpt sampling commands
└── sketches/              # Core sketch algorithm implementations / adapters
    ├── algorithms.h/cc    # Algorithm registry
    ├── cpc_sketch.h/cc    # CPC sketch algorithm
    ├── cpc_union.h/cc     # CPC union algorithm
    └── murmurhash3.h      # Third-party hash (DO NOT LINT)
tst/
├── unit/                  # GoogleTest unit tests
└── integration/           # Pytest integration tests (valkey-test-framework)

Coding Conventions

  • C++ Standard: C++11 only. No auto return types, no std::make_unique, no structured bindings.
  • Include Order: Own header first, then system headers (alphabetical), then project headers (alphabetical).
  • Access Specifiers: Indent with 1 space inside class body ( public:, private:).
  • Naming: PascalCase for classes/types, snake_case for functions/variables, UPPER_CASE for constants.
  • Command Registration: Each algorithm has XxxCommands_Register(ValkeyModuleCtx *ctx) in src/modules/{algo}/{algo}.cc.
  • Command Prefix: All commands use SKETCH.{ALGO}.{VERB} naming (e.g., SKETCH.CPC.ADD).
  • No comments unless explaining a non-obvious "why". No TODO/FIXME without a tracking issue.
  • Explicit includes: Include the header for every symbol you use directly (e.g. <cstddef> for size_t, <cstdio> for snprintf). Do not rely on transitive includes — macOS/libc++ pulls them in implicitly, but Linux CI (GCC/libstdc++) does not, causing CI-only build failures.

Lint

cpplint is configured via CPPLINT.cfg at project root. Disabled checks:

  • legal/copyright — no copyright headers
  • build/header_guard — project uses own naming
  • build/include_subdir — CMake -I handles paths
  • whitespace/line_length — limit is 120 (not 80)
  • readability/casting — third-party code uses C-style casts
  • runtime/int — ValkeyModule API requires long long

Run locally: scripts/lint.sh

Testing

  • Unit tests: GoogleTest, located in tst/unit/. Run via ./build/tst/unit/unitTests.
  • Unit coverage today: cpc, hll, theta, kll, req, reservoir, longsketch, itemsketch, plus module-level registry tests.
  • Integration tests: Pytest with valkey-test-framework. Each algorithm has tst/integration/test_{algo}.py.
    • Base class: DataSketchesTestCase (in tst/integration/datasketches_test_case.py)
    • Replication tests inherit ReplicationTestCase from valkeytests.valkey_test_case
    • Shared setup uses the setup_test autouse fixture defined on DataSketchesTestCase
    • tdigest, tuple, and varopt integration tests currently validate placeholder command behavior
  • Environment variables: MODULE_PATH, SERVER_VERSION, PYTHONPATH, VALKEY_SOURCE_DIR

Key Constraints

  • src/include/valkeymodule.h is generated during configure/build from the Valkey source tree; do not modify it manually.
  • Third-party src/sketches/murmurhash3.h should not be linted or modified.
  • Module name is valkey-datasketches, library output is libdatasketches.
  • For self-contained objects (no background tasks/external refs), only free callback is needed — skip unlink/unlink2.