Forked for production hardening — happy to discuss upstream

[hang-in]

Hi jaytoone,

I really liked the retrieval algorithm in CTX and forked it to harden the Claude Code hook implementation for production use:

https://github.com/hang-in/tunaCtx

Retrieval logic is unchanged. The work focused on:

• Decomposing bm25-memory.py (1837 LOC) into orchestrator + 11 sub-modules under _bm25/
• Unifying BM25 tokenizer/scoring between eval and production paths
• Adding 82 unit tests + 26 golden fixture regression guards
• Safety hardening (sqlite_vec fallback, deterministic sort, atomic install with hash-based update)

Original copyright (MIT) is preserved in LICENSE, and the README explicitly credits this as a fork.

If any of these changes look useful for upstream, I'd be happy to extract them as PRs. No pressure either way — happy to keep it as a downstream fork too.

Thanks for the great tool.

[jaytoone]

This is really impressive work — decomposing bm25-memory.py into 11 sub-modules with 82 unit tests is exactly what it needs. The original file grew organically and the 1837-LOC monolith is a real maintenance problem.

I would love to pull this upstream. A few questions before I open a tracking PR:

1. Tokenizer unification — which file controls the canonical tokenizer in your fork? That is the change I am most interested in merging first.
2. sqlite_vec fallback — is this a graceful degradation (BM25-only mode on import failure) or a full replacement?
3. Hash-based update in atomic install — is this in ctx_retriever/cli/install.py or a separate module?

Happy to take the 11 sub-modules as-is if you want to open a PR — or I can cherry-pick specific pieces. Either way works.

Thank you for preserving the MIT attribution and for the kind framing.

[hang-in]

Hi jaytoone,

Glad the audit was useful. To your three questions:

1. Tokenizer unification

Canonical entry point: src/hooks/_bm25/tokenizer.py (function tokenize). It does:

• Korean particle strip (handful of common 조사: 은/는/이/가/을/를/에/의/와/과/로/도/만)
• Lowercase + Porter stem (English)
• Stopword removal (combined Korean + English short list)

Callers that go through it post-unification:

• src/hooks/_bm25/ranker.py (BM25 scoring primitives — score_corpus_bm25, bm25_rank_decisions)
• src/retrieval/adaptive_trigger.py
• src/retrieval/bm25_retriever.py
• src/evaluator/coir_evaluator.py
• benchmarks/eval/doc_retrieval_eval_v2.py

Integration sanity check is scripts/verify_bm25_unified.py — verifies all callers route through _bm25/tokenizer.tokenize and produce identical token streams for a fixed input. The archival benchmark scripts (g1_*.py, g2_*.py, mab_*.py, nemotron_*.py) deliberately keep their own BM25 implementations to preserve historical A/B comparison points — I'd suggest leaving those alone.

Note (deliberate follow-up, not in this unification PR): the current tokenizer is NL-oriented — Porter stem + Korean particle strip. It's weak on code identifiers — getUserById is treated as a single token, and the _ in snake_case_function isn't a word boundary, so a query for ""user"" can't match getUserById. The natural next step is a subtoken splitter (camelCase / snake_case / dot.notation), but I'd keep that as a separate cycle. The unification PR prioritizes behavior-identical migration and carries today's tokenization as-is. (Not yet implemented in our fork either — flagging it for visibility.)

This is the cleanest piece to land first — single tokenizer file plus a few caller edits, no behavior change (token stream is byte-identical; BM25 score deltas are 0 on the regression suite).

2. sqlite_vec fallback

Graceful degradation, not a replacement.

Location: src/hooks/chat-memory.py:16

try:
    import sqlite_vec
    HAS_VEC0 = True
except ImportError:
    HAS_VEC0 = False

Downstream behavior:

• If HAS_VEC0: vault.db is opened with the vec0 extension loaded; chat-memory does FTS5 + vec0 hybrid (α=0.5 cosine + 0.5 BM25)
• Otherwise: falls back to FTS5-only search; the hook still produces output, just without dense rerank
• The hook also emits a one-line warning so users know the semantic layer is off

Motivation: keeps chat-memory functional on environments without sqlite_vec wheels (CI containers, fresh Windows installs, conda envs without a compile toolchain). The previous unguarded import sqlite_vec would silently turn the entire chat-memory hook into a no-op on those.

3. Hash-based update in install

Module: src/cli/install.py (the full ctx-install entry point lives here).

Split helpers:

• src/cli/settings_patcher.py — atomic write to ~/.claude/settings.json (temp file + os.replace) with timestamped .backup_<TS>.json per write. Idempotent: re-running ctx-install does not create duplicate hook entries.
• The hash-based update lives inside install.py itself: SHA256 of source vs. installed hook → if different, copy + create .backup_<TS>.py of the previous version.

Three new flags in install.py:

• --force-hooks — skip hash compare, overwrite unconditionally
• --no-update-hooks — never overwrite (preserve user edits)
• --uninstall — remove ctx-install's settings.json entries (hook files left in place by default; --force also removes them)

Test coverage: tests/unit/test_settings_patcher.py (22 tests) + tests/unit/test_install_cli.py (32 tests). Real filesystem rename is exercised, not mocked.

On PR shape

If cherry-picking sounds easier than pulling the whole decomposition, here's a split that lands cleanly with each piece independent:

1. Tokenizer unification (Q1) — smallest, safest first move. No public API change, behavior-identical.
2. sqlite_vec graceful degradation (Q2) — tiny patch, prevents the hook from going silently dark.
3. install.py hash-based update + atomic settings_patcher (Q3) — the install machinery; useful even without the rest.
4. bm25-memory.py decomposition into the 11 sub-modules — the largest piece; depends on (1) being landed since modules import the canonical tokenizer.
5. Telemetry instrumentation — opt-in via env var, zero-cost when off.

Outside the 5-piece scope (separate cycle candidate, not part of these PRs):

• Code-identifier subtoken splitting in the tokenizer — see the follow-up note under Q1 above. Not implemented in our fork yet either; keeping it separate preserves the deterministic <100ms posture of the unification PR. If you're already exploring this direction, I'd happily fold our work into a separate PR or issue rather than overlap.

Happy to open them in that order with the relevant unit tests each, or as one bundled PR if you'd rather review the whole thing at once. Both options pass golden + 105 unit tests against current upstream master.

Let me know which shape fits your review bandwidth best.

Thanks again for the open door on this.

[jaytoone]

Hey hang-in — this is exactly the kind of hardening CTX needs. Thank you for the detailed writeup.

I looked at tunaCtx and the work is substantial. A few specific things I'd love to pull upstream:

High priority for PRs:

1. Tokenizer unification — the eval/production BM25 scoring divergence is a known gap (it was causing misleading benchmark numbers). This is the one I want most.
2. 82 unit tests + 26 golden fixtures — CTX's CI is thin right now and this would directly fix that.
3. Deterministic sort — bm25-memory.py has a subtle non-determinism bug I've been meaning to address.

Already in 0.3.14:

• sqlite_vec fallback — landed this week after catching the hard import crash on fresh installs.

On the decomposition (1837 LOC → 11 sub-modules): I'd want to discuss the boundary design before merging — the monolithic file is a deliberate tradeoff (single-file hooks are easier to copy/audit), but your approach might be worth it if the module boundaries are clean.

Please open PRs for whichever pieces you're happy to extract — I'll review promptly. Even partial cherry-picks are welcome. And if you want to co-maintain upstream, that's also on the table.

Thanks again for preserving the MIT license and attribution.

[hang-in]

Hey jaytoone — thanks for the detailed triage. Here's a status update with PRs opened (Draft) and four points worth flagging from the audit cycle that produced them.

Three Draft PRs opened

PR	Mapping to your priorities
#3 — [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline	P0 (the one you wanted most)
#4 — [Docs Search] Dedup root README/CLAUDE/MEMORY against same-name docs/research/ files	P1 audit + an unrelated production-hook bug discovered along the way
#5 — [Determinism] Explicit tiebreak on 3 ranker.py sort sites	P2 (the "subtle non-determinism bug" you flagged)

All three are Draft because the change sites live inside the _bm25/ package layout (PR-4 territory, awaiting your boundary review). Once you've had a chance to look at the boundaries, I can either (a) cherry-pick onto the merged decomposition branch, or (b) re-author each change directly inside the upstream monolith and discard these Drafts. Either path is fine on my end.

I dropped sqlite_vec fallback from the original 5-stage plan since it's already in 0.3.14 (ba7df3d).

Four findings worth flagging from the audit

1. Your 08e262b already covers part of PR-1

Your fix: Korean tokenizer gap in eval pipeline + 6 regression tests (with the Related: hang-in/tunaCtx tokenizer.py attribution — appreciated) already addressed doc_retrieval_eval_v2.py. PR #3 covers the remaining three sites we found in the eval pipeline (g1_docs_bm25_eval.py, g1_longterm_baseline_eval.py, g2_docs_paraphrase_eval.py). Two further sites (src/cli/telemetry.py, src/retrieval/bm25_retriever.py) are intentionally kept divergent — rationale annotated inline (telemetry is identifier-frequency stats not BM25 ranking; bm25_retriever needs raw TF on code identifiers, and canonical's dict.fromkeys() dedup would flatten that).

2. Test count was 82, audit settled it at 80 + 26 golden

I quoted "82 unit tests + 26 golden" in the original issue body. After classifying tests for upstream relevance:

• 80 unit tests (upstream-ready: test_trigger_classifier_ko.py 45, test_ast_improvements.py 19, test_chat_memory_fallback.py 9, test_code_search_sort.py 7)
• 23 unit tests gated on PR-4 decomposition (test_bm25_init_reexport.py, test_bm25_memory_cache.py, test_bm25_memory_telemetry.py)
• 66 unit tests fork-only (atomic install machinery — test_install_cli.py, test_settings_patcher.py, test_uninstall_cleanup.py)

So the test-suite PR (when it comes) ships 80 + 26 golden, not 82 + 26. Apologies for the ±2 drift in the original number.

3. PR #4 carries an unrelated production-hook bug fix

While re-running the golden suite I caught a real bug: build_docs_bm25 indexes docs/research/*.md and root extras (CLAUDE.md, README.md, MEMORY.md) without name-collision dedup. When docs/research/README.md exists alongside root README.md, both get indexed under the same name, and the path that returns bm_filtered[:top_k] without rerank can emit both — surfaces as a duplicate > README.md line in the G2-DOCS output. The fix in PR #4 switches corpus build to a name-keyed dict with root extras winning on collision. Originally PR-2 was just the test suite; this audit finding got bundled in.

4. PR #5 ships 5 regression cases for determinism

The three ranker.py sort sites previously relied on CPython's stable-sort guarantee for equal-key ordering. PR #5 adds explicit tiebreak keys (matching the existing pattern at code_search.py:233) plus tests/regression/test_pr3_deterministic_sort.py covering: idempotency, equal-rank tiebreak independent of input order, equal-score tiebreak by hash, no-emb sanity, index tiebreak.

Co-maintain — yes, happy to

I use CTX daily so the maintenance overhead is already paid on my end. Concretely I can take:

• Issue triage on weekday afternoons KST
• Hook reviews (BM25 / tokenizer / install machinery / golden fixtures)
• Release notes + version bumps in coordination with you

I'd defer to you on retrieval algorithm direction, paper alignment, and anything benchmark-facing — those are your design decisions and I'd like to keep upstream's voice intact. If a MAINTAINERS.md with explicit areas of ownership works for you, I can draft one as a follow-up after the boundary discussion settles.

Order of operations I'd suggest

1. You take a look at the _bm25/ boundary in PR [Tokenizer] Unify _bm25.tokenizer canonical entry across eval pipeline (Draft, P0 of #1) #3 (or any of the three — same _bm25/ layout) and let me know whether it's broadly OK or needs redrawing.
2. Once boundaries are agreed, I either land PR-4 (decomposition) first and the three Drafts become clean diffs, or I re-author each into the upstream monolith and close the Drafts.
3. Subtoken splitting stays out of all four PRs (separate cycle, as agreed earlier in this thread).

Let me know if the order or scope needs adjusting.