Skip to content

HiveForensics-AI/knolo-core

Repository files navigation

🧠 Knolo

Knolo is a local-first knowledge base engine built around deterministic retrieval and portable .knolo packs.

It provides:

  • @knolo/core — pack format + deterministic retrieval engine, LivePack overlay, and Cortex memory layer
  • @knolo/cli — build workflows for .knolo artifacts
  • create-knolo-app — instant Next.js starter with playground
  • @knolo/langchain — LangChain-style retriever adapter
  • @knolo/llamaindex — LlamaIndex-style retriever adapter

Knolo prioritizes:

  • Deterministic lexical retrieval
  • Optional hybrid semantic reranking
  • Zero vector database requirement
  • Local-first execution (offline capable)
  • Portable binary knowledge packs
  • Strict runtime contracts (optional advanced features)

⚠️ knolo-core (unscoped) on npm is deprecated. Use @knolo/core.


📊 Retrieval Benchmark (March 2026)

Knolo was evaluated using a deterministic lexical-first + optional rerank configuration.

Run: 2026-03-01 TopK: 5

Aggregate Metrics

Metric Score
Precision@5 0.490
Recall@5 1.000
MRR@5 0.867
nDCG@5 0.900

Interpretation

  • Recall@5 = 1.0 → All relevant documents were retrieved in every test query.
  • High MRR (0.867) → Relevant documents appear near the top.
  • Strong nDCG (0.900) → Ranking quality is consistently high.
  • 🔍 Precision reflects lexical grounding before rerank — by design, Knolo prioritizes deterministic recall over aggressive pruning.

This benchmark demonstrates:

  • Deterministic lexical retrieval is highly reliable.
  • Hybrid reranking improves ranking quality without sacrificing grounding.
  • No vector database is required to achieve strong retrieval performance.

⚡ 5-Minute Quickstart

npx create-knolo-app@latest my-kb-chat
cd my-kb-chat
npm install
npm run knolo:build
npm run dev

For pack workflows, knolo dev is the watch/rebuild loop for configured sources. We are keeping that workflow instead of adding a separate build --watch command in this phase.

Open:

http://localhost:3000

Ask questions against the generated /docs corpus.


🔍 What Knolo Actually Is

Knolo is not a vector database wrapper. It is not a hosted retrieval service.

Knolo is:

  • A structured, versioned binary pack format
  • A deterministic lexical retrieval engine
  • An optional hybrid rerank layer
  • A local-first Knolo Cortex overlay memory layer for .knolo packs
  • A portable knowledge artifact you can ship anywhere

You build .knolo packs once. You mount them anywhere — Node, web, React Native, offline. When you need a deterministic mutable overlay on top of a mounted pack, use LivePack.

Retrieval is lexical-first and deterministic by default.

Hybrid semantic reranking is optional and never replaces lexical grounding.

🧪 Live KBs MVP

LivePack is the phase-1 mutable overlay for mounted packs. The base pack stays immutable, live docs are keyed by stable ids, and serialize() returns a standard .knolo snapshot.

For the rollout plan, implementation notes, and test matrix, see LIVE_KBS_MVP.md.

import { createLivePack, mountPack } from '@knolo/core';

const base = await mountPack({ src: './dist/knowledge.knolo' });
const live = await createLivePack(base, [
  { id: 'notes.alpha', text: 'alpha note', namespace: 'notes' },
]);

await live.updateDocument({ id: 'notes.alpha', text: 'alpha note v2' });
await live.removeDocument('notes.alpha');
await live.addDocument({ id: 'notes.alpha', text: 'alpha note restored' });

const snapshot = await live.serialize();
const rebuilt = await mountPack({ src: snapshot });

LivePack stays lexical/graph-only in v1. Cortex remains a separate append-only memory layer.


🧠 Knolo Cortex

Knolo Cortex adds a local-first overlay memory layer on top of @knolo/core. It is separate from LivePack: Cortex is append-only memory, while LivePack is a mutable document overlay for mounted packs.

It gives you:

  • Immutable createCortex(), remember(), forget(), labelMemory(), and linkMemories() writes
  • Deterministic lexical recall with labels, namespaces, source filters, and confidence/importance thresholds
  • Portable memory logs that can be serialized, merged, and replayed
  • consolidateMemories() to turn selected memories back into build docs
  • memoryToClaimOps() to export deterministic ClaimGraph ops without changing the existing graph builder
import {
  buildPack,
  consolidateMemories,
  createCortex,
  mountPack,
  recall,
  remember,
} from '@knolo/core';

const cortex = createCortex({ actor: 'notes-app' });
const { cortex: next, memory } = remember(cortex, {
  kind: 'note',
  text: 'Project alpha uses a local-first memory overlay.',
  labels: ['project.alpha'],
  namespace: 'project.alpha',
});

const hits = recall(next, 'project alpha');
const docs = consolidateMemories(next, { namespacePrefix: 'memory' });
const bytes = await buildPack(docs);
const pack = await mountPack({ src: bytes });

For the full API surface and memory-specific examples, see packages/core/README.md and examples/memory-overlay/README.md.


📦 Packages

Package Description
@knolo/core Pack builder, pack loader, deterministic retrieval engine, and Cortex memory layer
@knolo/cli CLI for building .knolo artifacts
create-knolo-app Next.js scaffolding with playground
@knolo/langchain LangChain-style retriever interface
@knolo/llamaindex LlamaIndex-style retriever interface
knolo-core-rust Native Rust pack mount + lexical query runtime

🦀 Rust Runtime Support (New)

Knolo now includes an initial Rust runtime in packages/core-rust.

Current Rust support includes:

  • Mounting .knolo packs from bytes
  • Parsing v1/v3-compatible core sections (meta, lexicon, postings, blocks)
  • Deterministic lexical querying with top_k, min_score, namespace, and source filters

Run Rust tests:

cd packages/core-rust
cargo test

🐍 Python Runtime Support (Phase 2)

Knolo also ships a pure-Python runtime in packages/core-python for mounting existing .knolo packs and running deterministic lexical queries locally.

It stays local-first, requires no vector database, and does not use embeddings on the default query path.

Install locally:

cd packages/core-python
python -m pip install -e ".[dev]"

Use it from Python:

from knolo import mount_pack, query

pack = mount_pack("tests/fixtures/simple.knolo")
hits = query(pack, "alpha beta", top_k=5)

For the release checklist and publishing notes, see packages/core-python/README.md and packages/core-python/RELEASE.md.


🌐 ICP Canister Adapter (New)

Knolo now ships a local-first ICP path that keeps retrieval lexical-first and talks to the canister directly, with no middleware and no vector database.

Status: the ICP integration is still under active testing and should be treated as experimental, not fully production-ready.

What it includes:

  • packages/icp-canister for the Rust canister adapter
  • examples/icp-knowledge-canister for a local dfx example
  • knolo icp CLI commands for init, build-pack, upload, and query
  • scripts/e2e-icp-local.sh for one-command local end-to-end verification

Local prerequisites:

  • dfx
  • Rust wasm target: rustup target add wasm32-unknown-unknown
  • npm install from the repo root

CLI path:

knolo icp init ./my-icp-canister
cd ./my-icp-canister
knolo icp build-pack ./knowledge --out ./dist/knowledge.knolo
knolo icp upload ./dist/knowledge.knolo --canister knolo_knowledge
knolo icp query "alpha beta" --canister knolo_knowledge

Run the example manually:

cd examples/icp-knowledge-canister
dfx start --background --clean
dfx deploy
node scripts/build-sample-pack.mjs
bash scripts/upload-pack.sh
bash scripts/query.sh "alpha beta"

If dfx is running in a minimal shell and complains about terminal colors, prefix the commands with TERM=xterm-256color.

Run the full end-to-end check:

bash scripts/e2e-icp-local.sh

The sample pack is built from the checked-in docs under examples/icp-knowledge-canister/knowledge, so the search results are deterministic and easy to verify locally.


🚀 10-Minute Ecosystem Path

From this repository:

npm install
npm run build

Run examples:

cd examples/langchain-basic && npm install && npm run start
cd ../llamaindex-basic && npm install && npm run start

🔌 LangChain-Style Usage

import { mountPack } from '@knolo/core/node';
import { KnoLoRetriever } from '@knolo/langchain';

const pack = await mountPack({ src: './dist/knowledge.knolo' });
const retriever = new KnoLoRetriever({ pack, topK: 5 });

const docs = await retriever.getRelevantDocuments(
  'How do I configure Knolo?'
);

for (const doc of docs) {
  console.log(doc.pageContent);
  console.log(doc.metadata); // { score, source, namespace, id }
}

🦙 LlamaIndex-Style Usage

import { mountPack } from '@knolo/core/node';
import { KnoLoRetriever } from '@knolo/llamaindex';

const pack = await mountPack({ src: './dist/knowledge.knolo' });
const retriever = new KnoLoRetriever({ pack, topK: 5 });

const nodes = await retriever.retrieve('Show me API usage examples');

for (const hit of nodes) {
  console.log(hit.node.text);
  console.log(hit.node.metadata);
}

📱 Expo / React Native Mounting

Use the runtime-safe entrypoint (@knolo/core) and pass URL/bytes. For local filesystem paths in Node.js, use @knolo/core/node.

import { mountPack } from '@knolo/core';

const ab = await (await fetch(PACK_URL)).arrayBuffer();
const pack = await mountPack({ src: new Uint8Array(ab) });

Node-only local path usage:

import { mountPack } from '@knolo/core/node';

const pack = await mountPack({ src: './dist/knowledge.knolo' });

🔀 Hybrid Retrieval (Optional)

Lexical-first. Semantic rerank second.

Build with embeddings

import { buildPack } from '@knolo/core';

const bytes = await buildPack(docs, {
  semantic: {
    enabled: true,
    modelId: 'text-embedding-3-small',
    embeddings,
    quantization: {
      type: 'int8_l2norm',
      perVectorScale: true
    }
  }
});

Query with rerank

import { mountPack, query, hasSemantic } from '@knolo/core';

const kb = await mountPack({ src: bytes });

const hits = query(kb, 'react native bridge throttling', {
  topK: 8,
  semantic: {
    enabled: hasSemantic(kb),
    mode: 'rerank',
    topN: 50,
    minLexConfidence: 0.35,
    blend: { enabled: true, wLex: 0.75, wSem: 0.25 },
    queryEmbedding
  }
});

Semantic sidecar workflow (Ollama, optional)

Lexical retrieval is still the first-pass and default. Sidecars add optional local reranking over lexical top-N candidates (no vector DB, no .knolo format migration).

# 1) Build deterministic lexical pack
knolo build

# 2) Generate local semantic sidecar (requires Ollama running)
knolo semantic:index --pack ./dist/knowledge.knolo --out ./dist/knowledge.knolo.semantic.json --model qwen3-embedding:4b

# 3) Inspect and validate sidecar before query-time use
knolo semantic:inspect --sidecar ./dist/knowledge.knolo.semantic.json
knolo semantic:validate --pack ./dist/knowledge.knolo --sidecar ./dist/knowledge.knolo.semantic.json --model qwen3-embedding:4b

Troubleshooting:

  • If Ollama is not running, start it and ensure http://localhost:11434 is reachable.
  • If model is missing, run ollama pull qwen3-embedding:4b.
  • If validate fails for fingerprint/model mismatch, regenerate sidecar with the current pack and exact model.

🧠 Optional: Agent Metadata & Routing

Knolo is a knowledge base first.

Packs may optionally embed structured metadata for:

  • System prompts
  • Namespace restrictions
  • Tool policies
  • Routing hints

Agent registries are validated once at mountPack() time.

Strict namespace binding ensures agents cannot escape configured domains.

These features are additive — they do not change the retrieval-first architecture.


🛠 Runtime Contracts (Optional Advanced Features)

Knolo defines strict validation contracts for deterministic workflows:

RouteDecisionV1

type RouteDecisionV1 = {
  type: 'route_decision';
  intent?: string;
  entities?: Record<string, unknown>;
  candidates: { agentId: string; score: number }[];
  selected: string;
};

ToolCallV1

type ToolCallV1 = {
  type: 'tool_call';
  callId: string;
  tool: string;
  args: Record<string, unknown>;
};

Helpers:

  • isRouteDecisionV1
  • validateRouteDecisionV1
  • isToolAllowed
  • assertToolCallAllowed

🗂 Repository Structure

.
├── packages/
│   ├── core
│   ├── cli
│   ├── langchain
│   ├── llamaindex
│   └── create-knolo-app
└── examples/

⚙️ Design Guarantees

  • Deterministic lexical retrieval
  • Deterministic hybrid rerank (fixed vectors)
  • No vector DB required
  • No cloud dependency required
  • Works offline
  • Works in React Native / Expo
  • Binary pack format is versioned

🛠 Pack Format

Binary layout:

[metaLen][meta]
[lexLen][lexicon]
[postCount][postings]
[blocksLen][blocks]
[semantic?]

Semantic section is optional and auto-detected.


🗺 Roadmap

  • Hybrid evaluation tooling
  • Incremental pack updates
  • Better diagnostics & introspection
  • Continued local-first performance tuning

🌐 Website

Docs & updates:

https://www.knolo.dev/



🕸 ClaimGraph (Deterministic Knowledge Graph + Delta Logs)

Knolo packs can now embed an optional ClaimGraph section built deterministically from source docs.

What it adds:

  • Deterministic node/edge extraction from markdown links, wiki links, headings, and conservative X is Y sentences.
  • Pack-embedded base graph (meta.claimGraph) with stable IDs and sorted ordering.
  • Agent-shareable append-only ClaimGraphLog overlays for offline collaboration.

Determinism guarantees:

  • Same docs + options → same graph JSON and pack bytes.
  • Stable hash IDs for nodes and edges.
  • Sorted nodes/edges, sorted evidence arrays, deterministic caps.

Build a pack with ClaimGraph

import { buildPack } from '@knolo/core';

const bytes = await buildPack(docs, {
  graph: {
    enabled: true,
    maxEdgesPerDoc: 500,
  },
});

Mount and inspect ClaimGraph

import { mountPack, getClaimGraph } from '@knolo/core';

const pack = await mountPack({ src: bytes });
const graph = getClaimGraph(pack);

console.log(pack.meta.claimGraph); // { version: 1, nodes, edges }
console.log(graph?.edges.slice(0, 3));

Agent-shared delta logs

import {
  createGraphLog,
  appendOp,
  mergeClaimGraphLogs,
  applyClaimGraphLog,
} from '@knolo/core';

let a = createGraphLog();
a = appendOp(a, {
  op: 'upsert_node',
  label: 'Delta Log',
  ts: 1710000000000,
  actor: 'agent.alpha',
});

let b = createGraphLog();
b = appendOp(b, {
  op: 'add_edge',
  from: 'n_1234abcd',
  p: 'mentions',
  to: 'n_7890ef12',
  ts: 1710000000100,
  actor: 'agent.beta',
});

const merged = mergeClaimGraphLogs(a, b);
const effectiveGraph = applyClaimGraphLog(graph ?? { version: 1, nodes: [], edges: [] }, merged);

Optional deterministic graph-based query expansion

import { query } from '@knolo/core';

const hits = query(pack, 'knolo determinism', {
  topK: 5,
  graph: {
    expand: true,
    maxExtraTerms: 12,
    predicates: ['defined_as', 'is', 'mentions', 'ref'],
  },
});

📄 License

Apache-2.0 — see LICENSE

About

KnoLo Core is a local-first knowledge base engine built for small language models (LLMs). It packages your documents into a compact .knolo file and enables fully deterministic querying — no embeddings, no vector databases, no cloud services required. Designed for on-device and edge LLM deployments.

Topics

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors