Repository navigation
feat(meko/knowledge): detect Markdown headings with goldmark - #530
sushmithamohanraj wants to merge 3 commits into
Conversation
rajmaddy89
left a comment
There was a problem hiding this comment.
Tested at a936402. The heading parity is very good. Headings matched markdown-it (CommonMark) exactly on 30,309 real .md files (358,120 headings), and 64-byte windows gave the same result as 1 MiB windows on every one. Headings changed from main in 236 files, and main was wrong every time. Mismatches with mistune (Cloud) dropped from 240 files to 12, all of them mistune quirks. 4,000 generated documents gave the same headings at every window size from 8 B to 4 KiB, and 5 more minutes of fuzzing (39.7M inputs) found nothing. Run end to end through Chunks(), lone \r, ## inside comments and <pre>, multi-line setext headings and front matter titles all work.
Findings, by priority:
P0 — blocker
- Front matter with a spaced key is a regression from
main(inline onyamlKey). Obsidian-style keys likeDate Created:get the block rejected. The file loses its title, and the YAML block becomes a fake H2 that every chunk then carries as itsSection:.
P1 — high
-
Memory on some 50 MiB shapes is 2.5–3×
main's. That contradicts "112–239 MB on every shape tried" indocs/knowledge.mdand the PR body.50 MiB shape mainthis PR one paragraph of <b>x</b>lines287 MB 755 MB one paragraph of 2. xlines299 MB 782 MB one paragraph of 4-space-indented lines 257 MB 561 MB fence with a 600 KiB info string 390 MB 1,056 MB <div>with a 600 KiB opening line399 MB 1,171 MB Plain prose improves (152 MB against 330 MB). The table's shapes find no restart point, so the window doubles until it holds the whole file. Details are inline on
paragraphOnlyand thewindow/2check. At minimum, the measured claim needs correcting.
P2 — minor
- A lone
</pre>,</script>,</style>or</textarea>line. commonmark.js, markdown-it and mistune (Cloud) all start an HTML block there. goldmark doesn't, so a# Hthat follows is a heading only in Local. This accounts for all 108 mismatches in the generated documents, and 0 of the 30k real files. - A junk chunk. A leading
---thematic break that isn't front matter becomes its own chunk,Document: x\n\n---.
I didn't run golangci-lint locally, because the local build is too old for Go 1.26. CI's run passes.
| titleLine = regexp.MustCompile(`^title:\s*(.+?)\s*$`) | ||
| // yamlKey is a front matter line that starts a key: a bare key without spaces, or a quoted one, | ||
| // then ':' and a space or the line's end. "Steps to follow: install it" isn't one. | ||
| yamlKey = regexp.MustCompile(`^(?:[^\s#:"'-][^\s:]*|"[^"]*"|'[^']*'):(?:[ \t].*)?$`) |
There was a problem hiding this comment.
P0. The key has to be space-free, so this front matter is rejected:
---
title: Notes
Date Created: 2024-01-01
---
# Body
The title is lost, and the block parses as a setext H2 with the text title: Notes Date Created: 2024-01-01. That text becomes the Section: of every chunk in the file. main read this correctly, and keys like this are common in Obsidian vaults.
I understand the goal is to keep prose such as Steps to follow: install it from reading as front matter. Some options: accept spaced keys once the block has already matched a strict key such as title:, or require every line to be key-shaped and nothing else. Or keep the rule, but document the trade-off and add a test pinning the behavior.
There was a problem hiding this comment.
Fixed in ee63f4b: a key with spaces (Date Created:) now counts once the block also has a key without spaces (title:, tags:), so your example keeps its title Notes and makes no heading. A rule over a lone sentence such as Steps to follow: install it still stays Markdown, and a block of only spaced keys does too. TestFrontMatter pins all three.
| if n > 1 || op == "" { | ||
| op = openingLine(kids[n-1], src) | ||
| } | ||
| if op == "" || len(op) > window/2 { |
There was a problem hiding this comment.
P1. If the opening line is over half the window, the window doubles instead of carrying the line. A fence with a 600 KiB info string, or a <div> with a 600 KiB first line, in a 50 MiB file ends up parsed whole: 1,056 MB and 1,171 MB, against about 390–400 MB on main. A synthesized short opener would probably be enough to carry instead: the fence characters alone for a code block, and the tag name for an HTML block, since goldmark only needs the block type to read on.
There was a problem hiding this comment.
Documented rather than changed: a first line over 512 KiB is a made-up shape rather than something people write, and files are capped at 50 MiB, so the worst case is the ~1.1 GB you measured (I measured 1,076–1,173 MB), parsed in about a second. docs/knowledge.md and the PR now state it as the one exception to the memory numbers.
| // space: "#tag"), '*', '-' or '+' (not a list marker or a rule: "*draft*"), '=' or '-' (not only | ||
| // those: "=x"), or a backtick or '~' (fewer than three in a row: "~x"), but not '>', '<' or a tab, | ||
| // nor a list marker or a thematic break. | ||
| func paragraphOnly(line []byte) bool { |
There was a problem hiding this comment.
P1. A paragraph whose lines all start with something that could open another block, such as <b>, 2. or 4 spaces, has no restart line. The window doubles until it holds the whole 50 MiB file: 755 MB, 782 MB and 561 MB, against 257–299 MB on main. One idea: when restarting inside a paragraph, put a synthetic plain-text line (x\n) at the front of the next window. A paragraph can then continue from any line, the same trick openAtEnd already uses with its probe line.
There was a problem hiding this comment.
Fixed in ee63f4b, with your idea: a paragraph still open at a window's end carries a plain x line into the next window, so goldmark continues it from any line. Re-measured at 50 MiB, finding headings: <b>x</b> lines 209 MB, 2. x lines 268 MB, indented lines 270 MB (with 5-byte lines they had reached 2.3 GB before the fix). TestMarkdownWindowStaysBounded checks no parse is given more than a window on these shapes, and the fuzz test found one bug on the way (a paragraph a blank line had closed was carried), now fixed and kept as a seed.
Markdown's heading finder was a line scanner that only knew code fences: headings inside HTML blocks became sections (millions on a large comment or <pre>), fences closed early, multi-line setext headings were missed, a leading rule swallowed prose as front matter, and lone CR line ends failed the file. goldmark now finds the headings, parsing blocks only, a 1 MiB window at a time, restarting where it can and carrying an open code or HTML block's opening line into the next window. A nesting cap and a ctx check per window guard hostile files. Front matter is taken only when it reads as YAML, and lone CRs read as LF for every text type. Claude-Session: https://claude.ai/code/session_01MRGTM3QdVKuqkHG916qdyL
Front matter took a leading rule over any "word word: text" or indented line as YAML; a bare key may no longer hold spaces, and list items and indented lines count only after a key. The nesting cap could turn a long rule inside a quote into list items and lose the heading after it; it now stops where the rest of a line is a rule. The link-definition cases goldmark reads as headings are named in the parser's comment and the docs and pinned in tests, two comments are corrected, and the knowledge doc no longer describes how Meko Cloud makes its chunk ids. Claude-Session: https://claude.ai/code/session_01MRGTM3QdVKuqkHG916qdyL
Front matter with a spaced key ("Date Created:") was rejected, losing
the title and turning the block into a fake heading; a spaced key now
counts once the block has a plain one. A paragraph that fills a window
carries a plain-text line into the next one, so a 50 MiB paragraph of
lines that could each open another block no longer makes the window
grow to the whole file. A section of only thematic breaks makes no
chunk. A lone closing pre, script or style line is documented as text,
and the docs state the measured memory, with the one remaining case:
a code or HTML block whose first line is over 512 KiB.
Claude-Session: https://claude.ai/code/session_01MRGTM3QdVKuqkHG916qdyL
a936402 to
ee63f4b
Compare
|
Thanks for the thorough testing. The two minor findings:
The memory claim is corrected too: 112–270 MB on the 19 shapes measured, with the long-first-line case stated as the exception. |
Overview
Design doc: Markdown heading detection for Meko Local
Why: Meko Cloud recently changed its Markdown chunker to find headings correctly. It no longer
takes a
#line inside a code fence or an HTML block (a comment,<pre>,<div>) as a heading,and it follows CommonMark's rules for where a fence opens and closes. Meko Local still had the old
behavior, so the same file was cut into different sections on each. This change brings Local's
headings in line with that change, but uses a CommonMark parser (goldmark) rather than copying
Cloud's hand-written rules, so there are no heading rules of our own to keep current.
Meko cuts an uploaded Markdown file into sections at its headings, and every chunk carries its
section's
Document:/Section:lines. The heading finder was a line scanner that only knewabout code fences, so it got 11 of 28 CommonMark heading cases wrong:
## xinside an HTML comment,<pre>,<script>or<div>became a section. A 50 MiBcomment gave 4.0 million fake sections, and a 50 MiB
<pre>8.7 million.---rule at the top of a file swallowed the prose under it as front matter.\rline endings (classic Mac OS) failed to index.This change has a CommonMark parser find the headings instead of hand-written rules.
What changes
Headings come from goldmark, parsing blocks only, a 1 MiB window at a time
(
internal/knowledge/markdown.go,findHeadings). Whole-file goldmark holds its syntax tree,1.6 GB for a 50 MiB list, so each window is parsed alone. A window's blocks are trusted up to the
last one a parse can restart at, and the next window starts there. A restart point must hold one
>or list marker for each quote and list item it opens, before its text.When one block fills a window:
goldmark itself says whether the block is open: the window is parsed again with a line of plain
text after it, and an open block takes that line in.
x) into the next window, sogoldmark stays inside it whatever its lines look like (
<b>x</b>,2. x, indented lines). Asetext heading that ends it still takes the real paragraph's text.
Nothing inline is parsed, and tables and paragraph transformers are off. Inline parsing and
tables never move a top-level heading. Leaving out the link-definition transformer has one effect,
accepted and documented: link definitions at the start of a paragraph under a setext underline
are a heading or part of its text.
[a]: /urlthen---is a heading[a]: /url, whereCommonMark reads a definition and a rule. Sections are still cut from the original text, so chunks stay
raw Markdown.
Checks on hostile files. goldmark's time grows with the square of a line's nesting. Unguarded,
512 KiB of
>took 60 s, and 512 KiB of1.took 20 s. Two guards:>or list marker at the start of a line becomesx, byte for byte. Nothing nested that deep is a top-level heading. Rule characters such as- - -are left alone, at the start of a line or after its quotes and list markers.ctxbefore each window, so it stops once theindexer has given up, where before it ran on unseen.
Front matter is taken only when it reads as YAML, closed by
---or...: comments, andkey: valuelines each followed by any list items or indented lines. A key with spaces(
Date Created:, common in Obsidian notes) counts once the block has a key without them(
title:). Anything else stays Markdown, so a---rule over a sentence such asSteps to follow: install itkeeps it. A singleNote: read this first.line under a rule is valid YAMLand is still taken. A section that holds only a rule (a leading
---that isn't front matter)makes no chunk.
A lone
</pre>,</script>or</style>line is text. commonmark.js starts an HTML blockthere, so a heading right after it counts in Meko only. It's documented and pinned by a test; in
5,514 real
.mdfiles no such line was stray, and a reviewer found none in 30,309.Lone
\rline endings read as\nfor every text type, inextract().Where Markdown readers differ, CommonMark wins.
> quotefollowed by## Hmakes H aheading. Meko Cloud keeps H in the quote; the difference is recorded in
docs/knowledge.md.Files indexed before this change keep their old chunks; chunk ids are row ids, so nothing breaks.
Uploading the same bytes again doesn't re-chunk a completed file: the upload finds it by its
SHA-256 and answers "already uploaded". To pick up the new headings, delete the file and upload it
again. A Markdown file that failed on lone
\rline endings is queued again when re-uploaded.goldmark v1.7.13 (MIT, pure Go) is a new direct dependency.
Measured
On 50 MiB generated files, comparing peak memory with
main:main<b>x</b>,2. xor indented lines<pre>>,1.,[,*a_, 50 MiB each)mainon every shapeThe
<pre>file still takes 932 MB end to end (1,775 MB onmain). That rest issplitText'scharacter-by-character fallback, which gets its own PR.
Windowed parsing gives the same headings as parsing a whole file: 5,514 real
.mdfiles and44,161 headings matched exactly at 8, 64 and 4,096-byte windows.
Rejected:
would be ours to keep current.
[, and acrash in its C would take the server down.
Local review
Kode's local review found 2 major and 2 minor issues, all fixed here:
*a_,#x) had no restart point, so thewindow grew to the whole file (1.2–1.5 GB at 50 MiB). Lines that can't start another block now
restart a paragraph (150–167 MB).
failed. The docs and this description now say so.
as a design.
A second round found 6 minor issues, all fixed:
word word: textor indented line; keys now have no spaces,and list items and indented lines count only after a key.
> - - - …) into list items, so a headingafter it was lost; the cap now stops where the rest of the line is a rule.
docs/knowledge.mdno longer describes how Meko Cloud makes itschunk ids.
Review on GitHub
A reviewer tested this at an earlier commit (same headings as markdown-it on 30,309 real files)
and found 1 blocker, 1 high and 2 minor issues:
heading. Spaced keys now count once the block has a plain key.
each open another block made the window grow to the whole file (up to 2.3 GB at 50 MiB). They
now carry a plain line. A code or HTML block whose first line is over 512 KiB still grows to
about 1.1 GB; such lines are made, not written, so the docs state it instead.
</pre>,</script>or</style>line (above).---that isn't front matter no longer makes a chunk.Tests
TestMarkdownHeadings: 39 heading cases, each at a 1 MiB and an 8-byte window. They coverHTML blocks of every kind, fences of every kind, setext headings, the quote case,
#hashtag,7 hashes, an empty
#, link definitions under setext underlines, a rule in a quote, and a lone</pre>.TestParagraphOnly: which lines a quote of one paragraph may restart at.TestMarkdownWindowStaysBounded: paragraphs of<b>x</b>,2. x, indented and symbol-ledlines, a fenced code block and a comment, each 64 KiB with a 1 KiB window: no parse is given more
than a window, and the headings equal a whole-file parse.
TestRuleOnlySection: a leading---makes no chunk.TestMarkdownSections,TestFrontMatter,TestLoneCarriageReturns,TestLimitNesting,TestMarkdownStopsOnContext.TestMarkdownHostileShapesLinear: 13 hostile shapes timed at 64 KiB and at 512 KiB. Thelarger must take under 16 times as long. It fails on 4 shapes with the nesting cap off.
FuzzMarkdownHeadings: compares windowed with whole-file headings at windows of 8 to 263bytes.
go testruns its seeds, including four inputs it found during development.make fuzzfuzzes for 5 minutes; it ran clean for the last 5 minutes after each change, about 40 in all.
TestChunksByTypepasses unchanged.me_kode_review:6 minor addressed;Sagittarius3.0
cloud.mekodata.ai://da2f1657-262b-44d4-8bf0-a6c59db7f80f;adlc/proj/meko/data-preparation
meko_design:design-explora-phase.md;f6c8aa94115d8d9b530ce0df65de767c776acdd31db8ea0184f6eadb2911c3d3
meko_conversation_id:cf74a692a3844ec78414ceedba2e36e0