This repository is the language-agnostic engine behind pypaginate. It exists so the computational heart of the library lives once, in Rust, and is shared by every host language through a thin native adapter — Python (PyO3) and Node/TypeScript (napi-rs) today, with WebAssembly an optional future target.
pypaginate's pure-Python implementation is already heavily optimized (38 optimizations across 7 rounds; #1 in most in-memory benchmarks). It has reached a local optimum for pure Python — the next step-change requires native code. The audit explicitly rejected mypyc/Cython for "build complexity, C extension distribution issues."
A Rust core gives one implementation reusable across runtimes — and native speed where the FFI boundary is cheap. A Python-only native extension would not be reusable; the same Rust crate drives a CPython extension (PyO3) and a Node/TS addon (napi-rs) without reimplementing a line of engine logic. One core, many adapters; behaviour can never drift between languages — which, as the benchmarks show, is the core's main value (raw in-memory speed mostly stays with the JIT'd host).
The boundary between a host runtime and the core is native by design:
Native addons are the simplest and fastest option at the I/O boundary: no serialization tax across an interpreter sandbox, direct access to host objects, and one core linked straight into the runtime. They are the foundation.
WASM is an optional, future-only target. wasm-bindgen
exists for browser / edge runtimes where a native addon cannot be loaded —
never as the foundation for Python or Node. The core has zero binding
dependencies and already compiles to wasm32-unknown-unknown at no extra cost,
so that door stays open if and when a browser/edge consumer needs it.
| Decision | Choice | Rationale |
|---|---|---|
| Boundary | Native-first | PyO3 for Python, napi-rs for Node/TS — native addons are simpler and faster at the I/O boundary than WASM. |
| WASM | Optional future target | wasm-bindgen for browser/edge only; the core compiles to wasm32 at no cost so the door stays open. |
| First scope | All pure-compute | Port the whole compute surface (cursor, pagination, normalize, filter, sort, search) rather than spike one module. |
| Repo layout | Separate repo | The core is its own crate; adapters and consumers depend on it. Keeps the polyglot core decoupled from any one language package. |
| Target | Tool | Output |
|---|---|---|
| Domain engine | Pure Rust | paginate-core crate (no bindings/ORM/DB) |
| Python module | PyO3 + maturin | paginate_core extension module |
| Node/TS addon | napi-rs | native .node addon |
| Browser / edge | WASM + wasm-bindgen (optional) | wasm32 module |
paginate-core/
├── Cargo.toml # workspace
├── rust-toolchain.toml # stable + clippy/rustfmt (wasm32 optional)
├── crates/
│ ├── core/ # PURE engine — NO bindings/ORM/DB/HTTP
│ │ ├── src/
│ │ │ ├── lib.rs # flat re-exports = the public API surface
│ │ │ ├── value.rs # Value: the JSON-like FFI data model
│ │ │ ├── error.rs # CoreError (#[non_exhaustive])
│ │ │ ├── accessor.rs # dotted-path resolve (shared, private)
│ │ │ ├── coerce.rs # Python-semantics compare/eq/str(x)
│ │ │ ├── cursor/ # keyset cursor codec: mod + encode + decode
│ │ │ ├── pagination.rs # offset / pages / has_next math
│ │ │ ├── keyset.rs # keyset predicate terms
│ │ │ ├── normalize.rs # text normalization
│ │ │ ├── filter/ # 20 operators, groups, like, regex (+ types)
│ │ │ ├── sort/ # stable multi-key sort, null placement
│ │ │ ├── search/ # spec + ranked + index + match_filter
│ │ │ ├── columnar/ # typed-column fast path: mod + build + filter
│ │ │ └── pipeline/ # filter → search → sort → paginate (mod + stages)
│ │ └── benches/ # one `main` criterion target + common/ + per-domain
│ ├── pyo3/ # PyO3 adapter -> `paginate_core` module
│ │ └── src/{lib.rs, conv.rs, specs.rs, engines/, dataset.rs}
│ └── node/ # napi-rs adapter -> Node/TS .node addon
│ └── src/{lib.rs, conv.rs, engines.rs, dataset.rs}
├── py/ # pypaginate — the Python package + its
│ # adapters (SQLAlchemy / Django / FastAPI)
└── ts/ # @cyblow/paginate — the TypeScript package
# + its adapters (Express / Prisma / Drizzle)
Each multi-concern module is a directory (mod.rs gateway + focused
submodules + co-located tests.rs); single-concern modules stay flat files.
Every source file is kept under the 250-line limit. crates/core, crates/pyo3,
and crates/node are built and validated; py/ and ts/ hold the published
language packages. The core engine is complete and the adapters layer on top
without engine changes.
The core owns the canonical domain contract so the bindings never redefine it and cannot drift:
- Enums / specs (
FilterLogic,FilterOp,SortDirection,NullsPosition,SearchFieldMode,FuzzyMode, and the*Specstructs) live once in the core and are re-exported flat fromlib.rs(paginate_core::FilterSpec, neverpaginate_core::filter::types::FilterSpec). - String ↔ enum parsing lives once, as
<Enum>::from_token(&str) -> Result. Both the PyO3 (crates/pyo3/src/specs.rs) and napi (crates/node/src/engines.rs) bindings delegate to it, so the wire vocabulary has a single home and an unknown token fails fast at the boundary instead of silently defaulting. - Error taxonomy is
CoreError(#[non_exhaustive]). Each binding maps it to a host exception (crates/pyo3/src/conv.rs→ thePaginateErrorfamily;crates/node/src/conv.rs→ a napi error); neither defines a parallel set.
The core is the domain engine: it has no knowledge of ORMs, databases, HTTP, or any host runtime. Responsibilities split cleanly:
| Concern | Owner |
|---|---|
| ORM (SQLAlchemy / Prisma / Drizzle / TypeORM) | language adapter |
| Business rules (cursor, pagination, filter, sort, search) | Rust core |
Serialization / object ↔ Value mapping |
adapter |
| DB transaction / session lifecycle | adapter |
The core only ever sees plain DTOs — the Value model — and never talks to
an ORM, a database, or the network. Each adapter (crates/pyo3, crates/node)
maps host objects to/from Value and hands the core nothing else.
The core speaks only Value — a small enum:
Null | Bool | Int | Float | Str | Bytes | List | Map, plus typed scalars
DateTime | Date | Decimal | Uuid that carry their canonical string so cursors
round-trip with full fidelity. Nothing host-specific crosses the boundary; the
adapter layer converts host objects to/from Value.
The cursor codec and pagination math are small-payload — marshalling is negligible. The in-memory engines are large-payload (every item), so:
- Index-based returns.
filter,sort, andsearchreturn indices (aVec<usize>/ permutation), never cloned items. The adapter selects from the original host objects by index — host objects never round-trip through Rust, so an ORM model never crosses the FFI boundary as data. - Projected extraction (planned for the engine adapters). Only the fields a spec references are extracted per item, not the whole record.
- Resident
Dataset(the big win). One-shot calls re-marshal every item per query, paying the FFI tax each time.Datasetmarshals the rows intoValueonce, then answers many queries natively — and fields that are the same scalar (int/float/str) in every row get a dense typed columnarVec, so the filter and sort stages skip the map lookup andValuedispatch. The whole pipeline (filter → sort → paginate) runs in one call returning the page's indices: ~36× faster than pure-Python on a 10K pipeline in CPython, all verified identical. This is the shape a host should use; see BENCHMARKS.md.
Each module mirrors the semantics of its Python counterpart, verified three ways:
- In-language unit tests — 65
cargo testcases + 8proptestproperties covering each module (incl. columnar == row-engine equivalence). - Cross-language golden vectors — the cursor wire format is asserted
byte-identical to output generated by the real Python codec
(
cursor::tests::golden_vectors_match_python_codec), including non-ASCII (ensure_ascii) and astral surrogate pairs. - Cross-language behavioural checks — filter/sort/search outputs were
compared against the real Python
FilterEngine/SortEngine/SearchEngineon shared fixtures: 22/22 cases matched (9 filter, 13 sort/search).
The cursor codec is also wire-compatible in both directions: a cursor minted by Python decodes here and vice-versa, so switching implementations never invalidates a client's existing cursors.
| Module | Core | Verified vs Python |
|---|---|---|
cursor |
✅ | ✅ byte-identical (11 golden vectors) |
pagination |
✅ | ✅ formula parity |
normalize |
✅ | ✅ ASCII + accented Latin (casefold edges documented) |
coerce |
✅ | ✅ used by filter/sort cross-checks |
filter (20 ops, groups) |
✅ | ✅ 9/9 cases |
sort (multi-key, nulls) |
✅ | ✅ 13/13 cases (with search) |
search (rank, weights, fuzzy) |
✅ | ✅ (rapidfuzz partial_ratio / token_sort_ratio) |
PyO3 adapter (crates/pyo3) |
cursor, normalize, pagination, filter, sort, search ✅ | — |
napi-rs adapter (crates/node) |
full surface ✅ | validated from Node (cursor wire-identical) |
Plus 8
proptestproperties (cursor round-trip, filter-never-adds, sort-permutation/monotonic/idempotent, search-subset, max_results cap).
As of v0.3 the native _core engine is mandatory — there is no pure-Python
fallback. Filtering, sorting, ranked search (incl. fuzzy / token-sort), the
cursor codec, pagination math, and text normalization all run in the core; the
Python package is a thin, typed adapter over it (see
MIGRATION.md).
Raw in-memory speed is not the headline, though: a full benchmark
(BENCHMARKS.md) shows that for single-shot calls the FFI
marshalling cost can dwarf the per-item work (in JS, naive V8 Array
operations beat the native engines by 40–230×), which is exactly why the
resident Dataset — marshal once, query many — is the shape hosts should use for
hot in-memory paths. The core's primary payoff is cross-language behaviour
consistency: identical cursor wire format and identical filter/sort/search
semantics across runtimes, which JIT'd hosts can't get from re-implementations.
The Node/TS adapter is a napi-rs crate (crates/node) that wraps the same
core — no engine reimplementation, the same index-based returns and Value
boundary as the Python adapter. It produces a native .node addon
(@cyblow/paginate-core) consumed by the @cyblow/paginate npm package (ts/)
and its ORM integrations (Express, Prisma, Drizzle).
WASM stays optional. Because crates/core carries zero binding
dependencies and already builds for wasm32-unknown-unknown (CI runs this as a
portability check, not a release artifact), a crates/wasm (wasm-bindgen)
adapter can be added later for browser/edge runtimes without touching the core.
It is a future option, never the foundation.
The packages are published and install prebuilt native binaries directly (the native engine is required — there is no pure-language fallback):
pypaginateon PyPI — wheels bundle the PyO3 extension.@cyblow/paginateon npm — wraps the@cyblow/paginate-corenapi addon (per-platform binaries).paginate-coreon crates.io.
cargo test --workspace # unit + golden-vector tests
cargo clippy --workspace --all-targets -- -D warnings
maturin build -r -m crates/pyo3/Cargo.toml # Python module (pypaginate._core)
# Node/TS adapter:
( cd crates/node && npm install && npm run build ) # -> .node + index.js/.d.ts
# Optional browser/edge portability check (not a release target):
cargo build -p paginate-core --target wasm32-unknown-unknown