Skip to content

feat(meko/knowledge): detect Markdown headings with goldmark - #530

Open
sushmithamohanraj wants to merge 3 commits into
mainfrom
ak_sushmithamohanraj/explora-phase
Open

sushmithamohanraj wants to merge 3 commits into
mainfrom
ak_sushmithamohanraj/explora-phase

Conversation

@sushmithamohanraj

@sushmithamohanraj sushmithamohanraj commented Nov 12, 2026 •

Copy link
Copy Markdown

Overview

Design doc: Markdown heading detection for Meko Local

Why: Meko Cloud recently changed its Markdown chunker to find headings correctly. It no longer
takes a # line inside a code fence or an HTML block (a comment, <pre>, <div>) as a heading,
and it follows CommonMark's rules for where a fence opens and closes. Meko Local still had the old
behavior, so the same file was cut into different sections on each. This change brings Local's
headings in line with that change, but uses a CommonMark parser (goldmark) rather than copying
Cloud's hand-written rules, so there are no heading rules of our own to keep current.

Meko cuts an uploaded Markdown file into sections at its headings, and every chunk carries its
section's Document: / Section: lines. The heading finder was a line scanner that only knew
about code fences, so it got 11 of 28 CommonMark heading cases wrong:

  • A ## x inside an HTML comment, <pre>, <script> or <div> became a section. A 50 MiB
    comment gave 4.0 million fake sections, and a 50 MiB <pre> 8.7 million.
  • A short fence closed a longer one, and a fence line with text after it closed an open fence.
  • A multi-line setext heading was missed.
  • A --- rule at the top of a file swallowed the prose under it as front matter.
  • A file with lone \r line endings (classic Mac OS) failed to index.

This change has a CommonMark parser find the headings instead of hand-written rules.

What changes

Headings come from goldmark, parsing blocks only, a 1 MiB window at a time
(internal/knowledge/markdown.go, findHeadings). Whole-file goldmark holds its syntax tree,
1.6 GB for a 50 MiB list, so each window is parsed alone. A window's blocks are trusted up to the
last one a parse can restart at, and the next window starts there. A restart point must hold one
> or list marker for each quote and list item it opens, before its text.

When one block fills a window:

  • A code or HTML block that is still open carries its opening line into the next window.
    goldmark itself says whether the block is open: the window is parsed again with a line of plain
    text after it, and an open block takes that line in.
  • A list or quote restarts at one of its later items.
  • A paragraph that is still open carries a line of plain text (x) into the next window, so
    goldmark stays inside it whatever its lines look like (<b>x</b>, 2. x, indented lines). A
    setext heading that ends it still takes the real paragraph's text.
  • Otherwise the window doubles.

Nothing inline is parsed, and tables and paragraph transformers are off. Inline parsing and
tables never move a top-level heading. Leaving out the link-definition transformer has one effect,
accepted and documented: link definitions at the start of a paragraph under a setext underline
are a heading or part of its text. [a]: /url then --- is a heading [a]: /url, where
CommonMark reads a definition and a rule. Sections are still cut from the original text, so chunks stay
raw Markdown.

Checks on hostile files. goldmark's time grows with the square of a line's nesting. Unguarded,
512 KiB of > took 60 s, and 512 KiB of 1. took 20 s. Two guards:

  • A nesting cap: before parsing, the 33rd > or list marker at the start of a line becomes
    x, byte for byte. Nothing nested that deep is a top-level heading. Rule characters such as
    - - - are left alone, at the start of a line or after its quotes and list markers.
  • The file's deadline: the finder checks ctx before each window, so it stops once the
    indexer has given up, where before it ran on unseen.

Front matter is taken only when it reads as YAML, closed by --- or ...: comments, and
key: value lines each followed by any list items or indented lines. A key with spaces
(Date Created:, common in Obsidian notes) counts once the block has a key without them
(title:). Anything else stays Markdown, so a --- rule over a sentence such as Steps to follow: install it keeps it. A single Note: read this first. line under a rule is valid YAML
and is still taken. A section that holds only a rule (a leading --- that isn't front matter)
makes no chunk.

A lone </pre>, </script> or </style> line is text. commonmark.js starts an HTML block
there, so a heading right after it counts in Meko only. It's documented and pinned by a test; in
5,514 real .md files no such line was stray, and a reviewer found none in 30,309.

Lone \r line endings read as \n for every text type, in extract().

Where Markdown readers differ, CommonMark wins. > quote followed by ## H makes H a
heading. Meko Cloud keeps H in the quote; the difference is recorded in docs/knowledge.md.

Files indexed before this change keep their old chunks; chunk ids are row ids, so nothing breaks.
Uploading the same bytes again doesn't re-chunk a completed file: the upload finds it by its
SHA-256 and answers "already uploaded". To pick up the new headings, delete the file and upload it
again. A Markdown file that failed on lone \r line endings is queued again when re-uploaded.

goldmark v1.7.13 (MIT, pure Go) is a new direct dependency.

Measured

On 50 MiB generated files, comparing peak memory with main:

main This change
Finding headings, 19 shapes 128–1,730 MB 112–270 MB
A 50 MiB paragraph of <b>x</b>, 2. x or indented lines 257–299 MB (reviewer) 209–270 MB
One giant fence, comment or <pre> 306–1,730 MB, with millions of fake sections 122–195 MB, with the right headings
A code or HTML block with a 600 KiB first line about 390–400 MB (reviewer) about 1.1 GB: such a first line isn't carried, so the window grows to hold it (documented)
Hostile files (>, 1. , [, *a_, 50 MiB each) — 141–191 MB, under 0.5 s
End to end, with chunking — at or below main on every shape

The <pre> file still takes 932 MB end to end (1,775 MB on main). That rest is splitText's
character-by-character fallback, which gets its own PR.

Windowed parsing gives the same headings as parsing a whole file: 5,514 real .md files and
44,161 headings matched exactly at 8, 64 and 4,096-byte windows.

Rejected:

  • A line scanner with CommonMark's fence and HTML block rules written by hand: every rule
    would be ours to keep current.
  • md4c through cgo: its inline pass can't be skipped and took 3.7 GB on 50 MiB of [, and a
    crash in its C would take the server down.

Local review

Kode's local review found 2 major and 2 minor issues, all fixed here:

  • A paragraph whose every line started with a symbol (*a_, #x) had no restart point, so the
    window grew to the whole file (1.2–1.5 GB at 50 MiB). Lines that can't start another block now
    restart a paragraph (150–167 MB).
  • The docs said re-uploading a file re-chunks it; same bytes are not re-chunked unless the file
    failed. The docs and this description now say so.
  • The link-definition exception above is now stated in the parser's comment and the docs.
  • The docs described the rejected scanner as the code this change replaced; they now describe it
    as a design.

A second round found 6 minor issues, all fixed:

  • Front matter took a rule over any word word: text or indented line; keys now have no spaces,
    and list items and indented lines count only after a key.
  • The nesting cap could turn a long rule inside a quote (> - - - …) into list items, so a heading
    after it was lost; the cap now stops where the rest of the line is a rule.
  • The link-definition cases are named in full and pinned in tests.
  • Two comments were corrected, and docs/knowledge.md no longer describes how Meko Cloud makes its
    chunk ids.

Review on GitHub

A reviewer tested this at an earlier commit (same headings as markdown-it on 30,309 real files)
and found 1 blocker, 1 high and 2 minor issues:

  • Blocker, fixed: Obsidian front matter with a spaced key lost its title and became a fake
    heading. Spaced keys now count once the block has a plain key.
  • High, fixed for paragraphs, documented for long first lines: paragraphs whose lines could
    each open another block made the window grow to the whole file (up to 2.3 GB at 50 MiB). They
    now carry a plain line. A code or HTML block whose first line is over 512 KiB still grows to
    about 1.1 GB; such lines are made, not written, so the docs state it instead.
  • Minor, documented: a lone </pre>, </script> or </style> line (above).
  • Minor, fixed: a leading --- that isn't front matter no longer makes a chunk.

Tests

  • TestMarkdownHeadings: 39 heading cases, each at a 1 MiB and an 8-byte window. They cover
    HTML blocks of every kind, fences of every kind, setext headings, the quote case, #hashtag,
    7 hashes, an empty #, link definitions under setext underlines, a rule in a quote, and a lone
    </pre>.
  • TestParagraphOnly: which lines a quote of one paragraph may restart at.
  • TestMarkdownWindowStaysBounded: paragraphs of <b>x</b>, 2. x, indented and symbol-led
    lines, a fenced code block and a comment, each 64 KiB with a 1 KiB window: no parse is given more
    than a window, and the headings equal a whole-file parse.
  • TestRuleOnlySection: a leading --- makes no chunk.
  • TestMarkdownSections, TestFrontMatter, TestLoneCarriageReturns, TestLimitNesting,
    TestMarkdownStopsOnContext.
  • TestMarkdownHostileShapesLinear: 13 hostile shapes timed at 64 KiB and at 512 KiB. The
    larger must take under 16 times as long. It fails on 4 shapes with the nesting cap off.
  • FuzzMarkdownHeadings: compares windowed with whole-file headings at windows of 8 to 263
    bytes. go test runs its seeds, including four inputs it found during development. make fuzz
    fuzzes for 5 minutes; it ran clean for the last 5 minutes after each change, about 40 in all.
  • TestChunksByType passes unchanged.

me_kode_review:6 minor addressed;Sagittarius3.0
cloud.mekodata.ai://da2f1657-262b-44d4-8bf0-a6c59db7f80f;adlc/proj/meko/data-preparation
meko_design:design-explora-phase.md;f6c8aa94115d8d9b530ce0df65de767c776acdd31db8ea0184f6eadb2911c3d3
meko_conversation_id:cf74a692a3844ec78414ceedba2e36e0

@sushmithamohanraj
sushmithamohanraj requested a review from a team November 12, 2026 23:28
@CLAassistant

CLAassistant commented Nov 12, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@sushmithamohanraj sushmithamohanraj changed the title fix(meko/knowledge): find Markdown headings with goldmark feat(meko/knowledge): detect Markdown headings with goldmark Nov 13, 2026

@rajmaddy89 rajmaddy89 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested at a936402. The heading parity is very good. Headings matched markdown-it (CommonMark) exactly on 30,309 real .md files (358,120 headings), and 64-byte windows gave the same result as 1 MiB windows on every one. Headings changed from main in 236 files, and main was wrong every time. Mismatches with mistune (Cloud) dropped from 240 files to 12, all of them mistune quirks. 4,000 generated documents gave the same headings at every window size from 8 B to 4 KiB, and 5 more minutes of fuzzing (39.7M inputs) found nothing. Run end to end through Chunks(), lone \r, ## inside comments and <pre>, multi-line setext headings and front matter titles all work.

Findings, by priority:

P0 — blocker

  1. Front matter with a spaced key is a regression from main (inline on yamlKey). Obsidian-style keys like Date Created: get the block rejected. The file loses its title, and the YAML block becomes a fake H2 that every chunk then carries as its Section:.

P1 — high

  1. Memory on some 50 MiB shapes is 2.5–3× main's. That contradicts "112–239 MB on every shape tried" in docs/knowledge.md and the PR body.

    50 MiB shape main this PR
    one paragraph of <b>x</b> lines 287 MB 755 MB
    one paragraph of 2. x lines 299 MB 782 MB
    one paragraph of 4-space-indented lines 257 MB 561 MB
    fence with a 600 KiB info string 390 MB 1,056 MB
    <div> with a 600 KiB opening line 399 MB 1,171 MB

    Plain prose improves (152 MB against 330 MB). The table's shapes find no restart point, so the window doubles until it holds the whole file. Details are inline on paragraphOnly and the window/2 check. At minimum, the measured claim needs correcting.

P2 — minor

  1. A lone </pre>, </script>, </style> or </textarea> line. commonmark.js, markdown-it and mistune (Cloud) all start an HTML block there. goldmark doesn't, so a # H that follows is a heading only in Local. This accounts for all 108 mismatches in the generated documents, and 0 of the 30k real files.
  2. A junk chunk. A leading --- thematic break that isn't front matter becomes its own chunk, Document: x\n\n---.

I didn't run golangci-lint locally, because the local build is too old for Go 1.26. CI's run passes.

titleLine = regexp.MustCompile(`^title:\s*(.+?)\s*$`)
// yamlKey is a front matter line that starts a key: a bare key without spaces, or a quoted one,
// then ':' and a space or the line's end. "Steps to follow: install it" isn't one.
yamlKey = regexp.MustCompile(`^(?:[^\s#:"'-][^\s:]*|"[^"]*"|'[^']*'):(?:[ \t].*)?$`)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0. The key has to be space-free, so this front matter is rejected:

---
title: Notes
Date Created: 2024-01-01
---
# Body

The title is lost, and the block parses as a setext H2 with the text title: Notes Date Created: 2024-01-01. That text becomes the Section: of every chunk in the file. main read this correctly, and keys like this are common in Obsidian vaults.

I understand the goal is to keep prose such as Steps to follow: install it from reading as front matter. Some options: accept spaced keys once the block has already matched a strict key such as title:, or require every line to be key-shaped and nothing else. Or keep the rule, but document the trade-off and add a test pinning the behavior.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ee63f4b: a key with spaces (Date Created:) now counts once the block also has a key without spaces (title:, tags:), so your example keeps its title Notes and makes no heading. A rule over a lone sentence such as Steps to follow: install it still stays Markdown, and a block of only spaced keys does too. TestFrontMatter pins all three.

if n > 1 || op == "" {
op = openingLine(kids[n-1], src)
}
if op == "" || len(op) > window/2 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1. If the opening line is over half the window, the window doubles instead of carrying the line. A fence with a 600 KiB info string, or a <div> with a 600 KiB first line, in a 50 MiB file ends up parsed whole: 1,056 MB and 1,171 MB, against about 390–400 MB on main. A synthesized short opener would probably be enough to carry instead: the fence characters alone for a code block, and the tag name for an HTML block, since goldmark only needs the block type to read on.

@sushmithamohanraj sushmithamohanraj Nov 16, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documented rather than changed: a first line over 512 KiB is a made-up shape rather than something people write, and files are capped at 50 MiB, so the worst case is the ~1.1 GB you measured (I measured 1,076–1,173 MB), parsed in about a second. docs/knowledge.md and the PR now state it as the one exception to the memory numbers.

// space: "#tag"), '*', '-' or '+' (not a list marker or a rule: "*draft*"), '=' or '-' (not only
// those: "=x"), or a backtick or '~' (fewer than three in a row: "~x"), but not '>', '<' or a tab,
// nor a list marker or a thematic break.
func paragraphOnly(line []byte) bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1. A paragraph whose lines all start with something that could open another block, such as <b>, 2. or 4 spaces, has no restart line. The window doubles until it holds the whole 50 MiB file: 755 MB, 782 MB and 561 MB, against 257–299 MB on main. One idea: when restarting inside a paragraph, put a synthetic plain-text line (x\n) at the front of the next window. A paragraph can then continue from any line, the same trick openAtEnd already uses with its probe line.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ee63f4b, with your idea: a paragraph still open at a window's end carries a plain x line into the next window, so goldmark continues it from any line. Re-measured at 50 MiB, finding headings: <b>x</b> lines 209 MB, 2. x lines 268 MB, indented lines 270 MB (with 5-byte lines they had reached 2.3 GB before the fix). TestMarkdownWindowStaysBounded checks no parse is given more than a window on these shapes, and the fuzz test found one bug on the way (a paragraph a blank line had closed was carried), now fixed and kept as a seed.

Markdown's heading finder was a line scanner that only knew code
fences: headings inside HTML blocks became sections (millions on a
large comment or <pre>), fences closed early, multi-line setext
headings were missed, a leading rule swallowed prose as front matter,
and lone CR line ends failed the file.

goldmark now finds the headings, parsing blocks only, a 1 MiB window
at a time, restarting where it can and carrying an open code or HTML
block's opening line into the next window. A nesting cap and a ctx
check per window guard hostile files. Front matter is taken only when
it reads as YAML, and lone CRs read as LF for every text type.

Claude-Session: https://claude.ai/code/session_01MRGTM3QdVKuqkHG916qdyL
Front matter took a leading rule over any "word word: text" or
indented line as YAML; a bare key may no longer hold spaces, and list
items and indented lines count only after a key. The nesting cap
could turn a long rule inside a quote into list items and lose the
heading after it; it now stops where the rest of a line is a rule.

The link-definition cases goldmark reads as headings are named in the
parser's comment and the docs and pinned in tests, two comments are
corrected, and the knowledge doc no longer describes how Meko Cloud
makes its chunk ids.

Claude-Session: https://claude.ai/code/session_01MRGTM3QdVKuqkHG916qdyL
Front matter with a spaced key ("Date Created:") was rejected, losing
the title and turning the block into a fake heading; a spaced key now
counts once the block has a plain one. A paragraph that fills a window
carries a plain-text line into the next one, so a 50 MiB paragraph of
lines that could each open another block no longer makes the window
grow to the whole file. A section of only thematic breaks makes no
chunk. A lone closing pre, script or style line is documented as text,
and the docs state the measured memory, with the one remaining case:
a code or HTML block whose first line is over 512 KiB.

Claude-Session: https://claude.ai/code/session_01MRGTM3QdVKuqkHG916qdyL
@sushmithamohanraj
sushmithamohanraj force-pushed the ak_sushmithamohanraj/explora-phase branch from a936402 to ee63f4b Compare November 16, 2026 20:40
@sushmithamohanraj

Copy link
Copy Markdown
Author

Thanks for the thorough testing. The two minor findings:

  • Lone </pre> / </script> / </style>: confirmed (commonmark.js 0.31.2 and markdown-it 14.1 start an HTML block there; goldmark excludes those tags from type 7). Kept goldmark's reading and documented it: matching would mean rewriting those lines before parsing, which would also break the real end of a <pre> block. In 5,514 real .md files, 12 have such a line and all 12 close a real block, none stray. Pinned in TestMarkdownHeadings.
  • Leading --- chunk: fixed; a section that holds only thematic breaks makes no chunk (TestRuleOnlySection).

The memory claim is corrected too: 112–270 MB on the 19 shapes measured, with the long-first-line case stated as the exception.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants