Skip to content

Commit 80fb0bc

Browse files
committed
Mark production-ingestion-v1 epics as implemented in roadmap
1 parent f79f6df commit 80fb0bc

1 file changed

Lines changed: 9 additions & 1 deletion

File tree

ROADMAP.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,15 @@ Issues #22 and #23 can land in parallel. #24 lands last and flips the gate on.
4848

4949
[Milestone on GitHub](https://github.com/DiogoRibeiro7/llm-data-platform/milestone/2)
5050

51-
**Why:** `llm_knowledge_ingestion` is explicitly an MVP today. Parser classes for PDF/HTML/Markdown raise `NotImplementedError`. Chunking is whitespace-window only with a hardcoded guard against any other strategy. The dedup module computes hashes but nothing compares them. Sources are local filesystem only. A single bad file aborts the whole run. This milestone closes those gaps.
51+
**Status (as of 2026-06-18): all five epics implemented.** Parsers
52+
(PDF/HTML/Markdown/JSON) are real and wired into the live pipeline; chunking
53+
offers `fixed_tokens` / `sentence_aware` / `heading_aware` with a pluggable
54+
tokenizer; sources are pluggable (local/S3/HTTP, streaming); runs are idempotent
55+
with real content-hash deduplication; and operational hardening adds per-document
56+
error isolation, failure thresholds, an exit-code contract, structured logging,
57+
and an `ingestion_runs.jsonl` observability hook. See `docs/mvp-ingestion.md`.
58+
59+
**Why:** `llm_knowledge_ingestion` was explicitly an MVP. Parser classes for PDF/HTML/Markdown raised `NotImplementedError`. Chunking was whitespace-window only with a hardcoded guard against any other strategy. The dedup module computed hashes but nothing compared them. Sources were local filesystem only. A single bad file aborted the whole run. This milestone closed those gaps.
5260

5361
### Ingestion epics
5462

0 commit comments

Comments
 (0)