Skip to content

mvp-ingestion.md

Latest commit

 

History

History
130 lines (103 loc) · 5.63 KB

mvp-ingestion.md

File metadata and controls

130 lines (103 loc) · 5.63 KB
Raw

Ingestion Pipeline

The knowledge ingestion pipeline reads local source files, normalizes them into canonical documents, deduplicates by content hash, splits documents into token-budgeted chunks, and writes index-ready artifacts.

Run

Install (from the repo root):

pip install -e ".[dev]"

Dry run (validates config and lists planned input files without writing artifacts):

python -m llm_knowledge_ingestion.cli.main --dry-run

Execute (uses configs/ingestion.yaml by default):

python -m llm_knowledge_ingestion.cli.main
# or with a custom config
python -m llm_knowledge_ingestion.cli.main --config path/to/ingest.yaml

Config shape

ingestion:
  source_id: my-source           # stable identifier, non-empty
  input_path: ./examples         # directory scanned recursively
  max_documents: 100             # cap on processed files
chunking:
  strategy: fixed_tokens      # fixed_tokens | sentence_aware | heading_aware
  target_tokens: 400
  overlap_tokens: 40
output:
  normalized_documents_path: ./out/documents
  chunks_path: ./out/chunks
  lineage_path: ./out/lineage
  index_records_path: ./out/index
  run_result_path: ./out/run/ingestion_result.json

Relative paths are resolved against the config file's directory.

Chunking strategies

Selectable via chunking.strategy:

  • fixed_tokens (default) — sliding window over whitespace tokens. Fast and format-agnostic; may split mid-sentence.
  • sentence_aware — packs whole sentences up to target_tokens, carrying overlap_tokens worth of trailing sentences into the next chunk. Never splits mid-sentence. Best for prose.
  • heading_aware — splits on heading boundaries emitted by the Markdown/HTML parsers, then sentence-packs within each section; each chunk records its originating heading in section. Falls back to sentence_aware when the document has no headings.

Token counts use a real tokenizer when the optional ingestion-tokenizer extra (tiktoken) is installed, and fall back to a whitespace word count otherwise. Chunk IDs are deterministic under every strategy (keyed by document, index, and offsets), so artifacts stay diffable across runs.

Operational behavior

  • Error isolation — a document that fails to load/parse/process is recorded in IngestionResult.error_details ({source_uri, stage, exception_type, message}) and the run continues. A corrupt file in a large batch does not abort the run.
  • Failure thresholds — the run aborts if errors > fail_limit (default 100) or, once at least a few documents have been attempted, errors / attempted > fail_ratio (default 0.5). Both are configurable under ingestion:.
  • --strict aborts on the first error (use this in CI to keep runs strict); defaults are lenient for unattended production use.
  • Exit codes: 0 full success · 1 internal error (bad config / IO) · 2 partial success (some documents failed, within thresholds) · 3 aborted (threshold exceeded or --strict). The run result is always written, even on abort, with started_at, finished_at, and the error breakdown.
  • Structured logging — --log-format json emits parseable log records carrying run_id, source_id, and stage.
  • Observability hook — each run appends a summary line to ingestion_runs.jsonl next to the run result, so the observability pipeline can track ingestion runs alongside LLM interactions.

Supported inputs

  • .txt
  • .md / .markdown (frontmatter stripped, headings preserved)
  • .html / .htm (reduced to clean text)
  • .json (deterministic re-serialization, or configurable text-field selection)
  • .pdf (requires the optional ingestion-pdf extra)

Files are discovered recursively under ingestion.input_path and processed in stable sorted order.

Outputs

  • <normalized_documents_path>/documents.jsonl — one normalized document per line.
  • <chunks_path>/chunks.jsonl — one chunk per line, each carrying document_id.
  • <lineage_path>/lineage.jsonl — one lineage reference per chunk.
  • <index_records_path>/index_records.jsonl — one index-ready record per chunk.
  • <run_result_path> — JSON summary with counters (documents_processed, documents_skipped_unchanged, deduplicated_documents, chunks_generated) and warnings.
  • <run dir>/ingestion_state.json — per-source_id map of document_id -> content_hash from the last successful run, used to skip unchanged documents.
  • <run dir>/dedup_report.jsonl — one line per content-hash duplicate group (written only when duplicates are found).

Idempotency and deduplication

  • Re-running over unchanged inputs reports documents_processed=0 and documents_skipped_unchanged=N; modifying a file reprocesses only that file.
  • Two documents with identical content collapse to a single kept document (first-seen wins); the collapsed ones are counted in deduplicated_documents, recorded in dedup_report.jsonl, and linked back to the kept document in lineage via parent_document_id.
  • --force-reingest ignores prior state and reprocesses everything.
  • document_id is derived from source_id + the document's source URI (path/object key/URL), so it preserves provenance: identical content at two different URIs yields two IDs, and the duplicate is collapsed at the content -hash layer rather than by ID.

All JSON output is deterministic at the field level (sort_keys=True) so artifacts are diffable across runs.

Example

A worked config is constructed at runtime in tests/test_end_to_end_pipeline.py and sample inputs live under examples/.