CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

1.13.0 - 2026-08-19

Added

• Figures carry a caption: Image.caption. Image had url, alt_text, title, width and height and no way to record the text printed beside a figure, so two parsers had independently worked around the gap by writing captions into alt_text. Those are different things — alt text substitutes for an image nobody can see, a caption sits next to one everybody can — and an image with both had to give one up. The field is typed str | None to match Table.caption and sits in the same position on the node. Formats with a native spelling now use it: AsciiDoc's block title, reStructuredText's figure directive (an image directive is promoted when it has a caption), HTML's <figure>/<figcaption>, and MediaWiki's trailing caption field. Markdown has no caption syntax, so it reuses the two-part device built for table captions in #237 — a visible italic line plus a marker comment naming it a caption — with the caption placed below the figure, since that is where a figure's caption is conventionally set. The AsciiDoc and HTML parsers read their own spelling back, so those round-trip too. The PDF parser does not populate the field yet. Its caption detector was, at the time this field landed, a regex over a fixed 50pt band whose fallback accepted any capitalised text under 200 characters — on 20 cached research PDFs only 44 of the 155 strings it returned even began with a figure cue. Since include_image_captions defaults to True, binding that output to a field the renderers now print would have put spurious italic caption lines under most extracted images. The detector has since been rebuilt on the layout model's caption regions (see Fixed); routing its result into this field is the remaining step, and will be graded against the figure-binding oracle rather than assumed. (#338)
• The AST has a home for figures: a Figure block container with children and a caption. A figure is not always one image — multi-panel journal figures embed one raster per panel, LaTeXML wraps every arXiv table in <figure>, and a vector-drawn PDF figure has a caption and no raster at all, which Image.caption alone could not represent (#338). Figure holds block children (possibly none) plus an optional caption: str, mirroring Table.caption. All 16 renderers emit it — natively where the format has a spelling (HTML <figure>/<figcaption>, reST's figure directive, LaTeX's figure float, AsciiDoc block titles, org #+CAPTION:), children plus an italic caption line elsewhere — and Markdown round-trips it through an extent-based variant of the #237 marker device (<!-- all2md:figure --> … <!-- all2md:figure-caption -->/<!-- all2md:figure-end -->). The HTML parser gains an opt-in figures_parsing="figure" mode that reads <figure> back as the container (made the default in a separate change, noted below); the PDF parser does not emit Figure yet — that follow-up is its own deliberate change. NodeVisitor.visit_figure is concrete rather than abstract (the visit_mark precedent), so third-party visitors degrade to the figure's children instead of crashing; the figure:/image: extraction selector now returns a multi-panel figure as one figure rather than N images.
• The PDF parser emits Figure containers. A captioned raster is now wrapped in a Figure with the caption on the container (it used to ride on Image.caption, both unreleased), panels grouped by a layout picture region or an identical detected caption fold into one multi-panel figure, and a picture region holding no raster at all — a vector-drawn chart — becomes a caption-only Figure, because the caption is the only record the figure exists (#338, #340). The picture region also rescues captions the per-image search cannot reach: a stacked panel's below-band finds the next panel, not the caption, while the region's extent ends where the caption starts. Caption body-copy suppression now follows emission: when OCR replaces a page or image_placement_markers is off, no figures are emitted and caption paragraphs are no longer dropped from the text. Measured on the 12-article PMC born-digital sample, figure_binding rose 0.47 → 0.56 (19 of 34 captions bound, both controls held at zero); the newly bound captions leave the prose stream, the same trade every bound caption already makes. The PMC oracle now counts a Figure container once — not once per panel — matching JATS <fig> granularity.
• The PDF parser recovers borderless tables from word-box gutters. Layout-predicted table regions that PyMuPDF's strategies could not grid — 56 of the 63 tables missing from the born-digital corpus, every one shredded by the text strategy and then correctly refused by the split-word guard — now get a third pass that builds the grid from the page's own word boxes: columns from vertical bands no word crosses, whole words assigned to the column holding their center, wrapped cell lines folded into their logical row with hyphenation repair across the join (#386). Cut and space-joined words are impossible by construction. Three measured guards keep prose out: a grid needs three-plus columns (one gutter is what any two-column layout has), a sequential-integer column beside sentence-length cells reads as a numbered bibliography and demotes to prose (gridding one scrambles every citation), and a region of predominantly rotated words is declined in favor of the rotation-aware prose path. Measured on the 12-article PMC sample: tables emitted rose 12 → 30 of 32 expected, table_content_similarity median 0.000 → 0.88, table_structure_similarity median 0.000 → 0.73, with whole-article attainable recall at 0.941 (baseline 0.951 — a table's cell stream breaks a few truth blocks' n-grams that its prose form kept) and both wrong-article controls at zero.
• The PDF parser recovers rotated (landscape) tables in their own frame. The word-gutter pass declined any region whose words were predominantly taller than wide, because gridding a rotated table in page coordinates scrambles its reading order — measured, a 28x4 truth table came back 8x12 with its containment destroyed. Declining was the right call and the wrong ending: on the PMC born-digital corpus, 3 of the 13 still-missing tables were genuine landscape tables behind exactly this guard (#389). Such regions now go through the same gutter sweep with their boxes transposed into the table's own frame. Transposing is a reflection, so one axis always runs backwards for one of the two rotation directions — undecidable from the boxes alone, so both axes are checked against PyMuPDF's own stream order, which holds the words as they read, and mirrored where they disagree; getting that wrong would not mis-shape the grid but reverse its rows or columns, putting every cell in the wrong place. Dispatch demands stronger evidence than the old decline did: a word counts as rotated only when its box is taller than wide by a measured margin (real rotated table words sit at median aspect 2.5–2.7, a mixed-orientation region that must not be transposed at 1.05), because transposing upright text manufactures perfect fake "gutters" out of its line spacing. Ambiguous regions are still declined to the prose path, exactly as before — including the fourth rotated table in the deficit, whose orientation evidence is genuinely mixed.
• The word-gutter table pass admits two-column grids. A single gutter is what any two-column layout has, so the pass shipped refusing it outright -- and that refusal cost the 4 real two-column tables still missing on the PMC born-digital corpus (Questions | Answers, Male patients | 226 (69.8%)) to save junk the downstream guards were already catching (#389). Measured, the corpus's whole two-column population is 12 regions: the 4 real tables, 7 numbered reference lists, and 1 chart whose axis ticks and legend grid perfectly. Six reference lists were already condemned by the bibliography guard; the seventh numbered its entries 2), a spelling the guard's integer pattern now counts. The chart is caught by a new drawing-density gate that runs only at the two-column tier: a chart's labels float over its plot's vector paths (541 in the measured region) while a borderless table has at most its own rules (0-4 in all four real ones). Wider grids carry two aligned boundaries, which chart labels do not produce, so no established path changes.

Changed

• Removed an unreachable second block-processing pipeline from the PDF parser. _process_text_region_to_ast and the five helpers it alone called (_process_text_blocks_to_nodes, _process_blocks_line_text, _process_blocks_line_monospace, _apply_column_detection and _merge_columns_for_reading_order) had no callers anywhere in the package or the test suite. They were a near-duplicate of the live per-block path that had drifted away from it — same responsibilities, different rotated-text, code-block and heading handling — so reading them gave a misleading picture of what the parser actually does, and any fix applied to one copy silently missed the other. No behaviour changes: nothing called them.
• Removed enrich_metadata_with_conversion_info (utils/metadata.py). A repo-wide search found no reference to it outside its own definition — not imported, not exported, not called from any parser, test, or script. ~85 unmaintained lines, including a stream-position trap for file-like input. No behaviour changes: nothing called it.
• HTML figures_parsing now defaults to "figure". A <figure> element parses to the Figure AST container introduced alongside #338 — children plus a caption — instead of degrading to a BlockQuote with the caption folded into its prose. Callers that read the caption as paragraph text should read Figure.caption; the previous behaviour remains one option away (figures_parsing="blockquote"). With the default flipped, the generative figure round-trip gate now covers HTML alongside ast and markdown.
• Default options no longer drop every PDF figure silently. The default attachment_mode="alt_text" returned before extracting anything, so a journal PDF's figures left no trace — a 23-page arXiv paper with 251 embedded rasters produced zero Image nodes (#340). The mode now runs a decode-free geometry pass and emits the figures that carry a detected caption, as URL-less Image nodes whose caption renders through the caption marker device. Uncaptioned images stay suppressed under that mode: with no bytes and no caption, an ![alt]() placeholder is noise, which was the sound half of the old rationale (#338). No pixmap is decoded on this path, so the default mode keeps its performance edge over save/base64. Vector-drawn figures still yield nothing — they emit no raster placement to hang a caption on, and reaching them is #338's caption-bearing container, deliberately not attempted here.
• Changelog entries are now written as changelog.d/ fragments, not as edits to CHANGELOG.md. Every branch appended its entry to the same place in the same file — under ## [Unreleased], at the end of the same ### section — so any sweep landing more than one PR hit a conflict in CHANGELOG.md on every merge after the first, and resolving it by hand next to two thousand lines of prose is exactly the situation in which an entry gets dropped. A PR now adds changelog.d/<slug>.<category>.md holding the bullets it wants published; two branches never write the same lines because they never write the same file. scripts/compile_changelog.py --version X.Y.Z folds the fragments into a new released section at release time, updates the link references at the bottom of the changelog, and deletes what it consumed; --check validates fragments without writing. Entries already sitting under ## [Unreleased] were deliberately left there rather than migrated: the compiler merges hand-written content and fragments into the same sections by design, so the two styles coexist and nothing had to be rewritten to adopt this. towncrier and scriv were both rejected — they rewrite the whole changelog with their own newline and formatting conventions, which would turn a two-line release edit into a whole-file diff and flatten the long-form entries this project writes. Nothing enforces the fragment yet; adding a CI check that a PR touching src/ carries one is a separate decision.
• Re-recorded benchmarks/pmc/reference.json on post-merge main and brought every figure on the fidelity page up to date with it. The tables story inverted: 164 tables emitted against 121 expected (was 92), with the surplus mostly continuation tables the expected count does not yet credit, and table-block text recall fell to 69.1% (was 83.6%) because table text now routes through structured cell extraction instead of flowing out as prose — a trade the page now states instead of netting away. The page also documents the new 110-article held-out corpus and its first validation result.

Fixed

• A nested ordered list numbered from anything but 1 no longer collapses into the paragraph above it. CommonMark lets only a 1. ordered item interrupt a paragraph. The Markdown renderer put a nested list on the line straight after its item's text, so a sublist starting at 10. — or at 0. — was read back as more paragraph text and the entire sublist vanished: two List nodes went in, one came out, and the numbers ended up inside a sentence. This hit tight and loose parents alike, since loose rendering only put blank lines between items and never between a paragraph and the sublist inside one. The renderer now emits a blank line before such a sublist and renders the list containing it loose, which is what makes that blank line legal; the output is a fixed point, so the reparsed (now loose) document renders byte-identically. Renumbering the sublist from 1 was rejected: ordinal markers are real data — a PDF bibliography continues its numbering across list fragments, and rewriting it would silently falsify the citation numbers. Formatting note: a list gains blank lines between its items when one of them holds an offset-numbered sublist after some other block — that is the shape that was broken. A sublist that is an item's very first block sits against the marker, already reads as a nested list, and keeps its tight rendering. Lists with no offset-numbered sublist are untouched — tight stays tight, and the golden snapshot suite is unchanged.

• A PDF's ruling-line table fallback no longer deletes the text of any region it rejects. Text that falls inside a detected table's bounding box is removed from the page's ordinary text blocks before the table is validated, so that it is not emitted twice. The find_tables() path knows this and hands the region's text back as a paragraph from every one of its rejection branches. The ruling-line fallback — _extract_table_from_ruling_rect, which reads a page's stroked lines directly — did not: it returned bare None when extraction was switched off, when fewer than 2x2 ruling lines were found, and from each of its sparsity, uniformity and dot-leader/TOC guards. Every one of those deleted the framed region's prose outright, and the conversion still reported success. On a synthetic page holding a stroked frame with one internal rule and a sentence inside it, table_detection_mode="ruling" produced no output at all — the 2x2 grid was 75% empty, the sparsity guard rejected it, and the sentence went with it. All five paths now return the region's text as a paragraph, the same way the find_tables() path does. table_fallback_extraction_mode="none" is included: it means "detect the region, don't build a table from it", and the region's text has already been excluded by the time it is honoured — but it is not counted as a table rejection, because nothing was rejected.

• A PDF list no longer nests a parent item underneath its own child. _determine_list_level_from_x assigned each newly seen indent the level len(x_levels) — arrival order — and never compared the x-coordinates to each other. The first list item of a run was therefore level 0 whatever its indent, and every new indent after it was one level deeper whether it lay to the right or to the left. A nested list that continues at the top of a column or page begins on a sub-bullet, which is routine in two-column typesetting; the sub-bullet took level 0, the genuine top-level bullet after it took level 1, and since the list builder reads a larger level as deeper, the parent list ended up nested inside its own child. Levels are now assigned by comparing x: within tolerance of an established indent is that indent's level, further right than all of them opens a deeper one, further left than all of them opens a shallower one, and an indent arriving between two known ones lands between their levels. The numbers are no longer 0-based or contiguous — they are an ordering key, and renumbering them would strand the levels the list builder has already recorded on its stack. A list run that starts on a sub-bullet now emits that sub-list in its own right rather than dropping it when the stack unwinds past its bottom. Not addressed: this is still blind to columns, so the first item of a right-hand column reads as deeper than anything in the left one.

• A PDF page whose text cannot be read no longer disappears without a trace. _process_page_to_ast wrapped its page.get_text("dict", ...) call in except (AttributeError, KeyError, Exception): return [] — no log line, no degraded event, no progress event. A page that failed to extract was indistinguishable in the output from a page that was genuinely blank, and the conversion still reported success; worse, if every page tripped it, the document-level OCR safety net saw an empty document and could put a perfectly good text PDF through OCR. The failure is now logged at WARNING with the page number and the underlying exception, and recorded as a page_text_extraction_failed degraded event at error severity so it reaches the confidence report. The tolerance itself is unchanged and deliberate — one unreadable page must not cost the other four hundred, and the parser is driven with mock pages that cannot answer get_text() at all — so the page is still skipped rather than raising. The in-place dehyphenate_blocks() call, which had drifted inside the same try, has been hoisted out of it: it runs on blocks that were already read successfully, so a failure there is a bug in all2md rather than an unreadable page, and it was being laundered into the same silent empty page.

• A framed text box no longer becomes a one-cell PDF "table". _pdf_tables states that its caps "apply to both PyMuPDF's find_tables() output and our ruling-line detector since both can fire on the same false-positive shapes", and the find_tables() path enforces MIN_TABLE_ROWS x MIN_TABLE_COLS accordingly. The ruling-line detector did not: it asked only for two horizontal and two vertical lines, which is a single cell, so a stroked callout box that cleared the sparsity guard came out as one cell of prose wrapped in pipes — the exact shape those constants exist to reject. It now applies the same minimum to the grid the lines actually bound (len(lines) - 1 per axis), and demotes what it rejects to a paragraph like every other guard. The same function had also re-hardcoded two shared thresholds as bare literals — > 0.70 beside the imported MAX_TABLE_EMPTY_RATIO, >= 5 beside MIN_FILLED_FOR_UNIFORMITY_CHECK — which happened to agree with them and would have drifted silently the first time either was retuned; both now read the constant.

• Chunk character spans stay on the basis they claim under --avoid-table-split / --avoid-code-split. Every chunk is stamped char_basis="section_text" — the span indexes the section's rendered Markdown — and that held only while a section was chunked in one piece. Segmenting a section around an atomic table or code block computed each segment's offsets against that segment's own text: the atomic piece got the constant span (0, len(text)) and each prose segment restarted at 0. On a prose/table/prose section every chunk after the first therefore reported a span that overlapped its predecessors and, sliced out of the section text as documented, returned the wrong text — 45 of 119 chunks over this repository's README across five strategies. Each segment is now located in the section's own rendering and its windows shifted by that offset, so the spans are true, ordered and non-overlapping. Searching forward from the previous segment's end keeps two identically-rendered segments (two tables with the same cells) in document order. Rendering a fragment is not guaranteed to reproduce a substring of rendering the whole — footnote definitions, for one, are collected at the end of whatever document they are rendered in — so a segment that cannot be located keeps its segment-relative span and says so with a new char_basis="segment_text", rather than reporting a section offset that is wrong. Check char_basis before slicing.

• An unknown keyword argument passed beside an options object now warns instead of raising TypeError. to_ast, from_ast, to_markdown, convert and roundtrip document a single rule for a keyword argument no options class has a field for: warn and drop it. That rule only ever ran on the branch where the caller passed no options object. Pass one — to_ast(src, parser_options=MarkdownParserOptions(), bogus=1) — and the kwargs went straight to create_updated, which is dataclasses.replace, so the same typo escaped as MarkdownParserOptions.__init__() got an unexpected keyword argument 'bogus' instead. The same branch could not reach a field of a nested options dataclass either: network_timeout lives on options.network, the no-options branch has always folded it in, and the with-options branch raised on it even though it is a perfectly valid option for that format. Both are now handled the same way on both branches, with the existing warning categories and wording. Related: the kwargs that to_markdown/convert pre-split were checked against the detected format's options class rather than the class of the options instance that actually receives them. Options classes inherit (MboxOptions is an EmlOptions) and parsers accept any subclass of what they expect, so a caller could legitimately pass an instance whose class is not the format's own — and a field of that class, such as max_messages on an eml parse, was dropped as "not for the formats in this conversion" and silently applied nothing. The split now uses the receiving instance's class when one is given.

• An Org greater block containing a blank line is no longer torn apart. The Org parser split a heading's body on blank lines and only then looked for #+BEGIN_/#+END_ delimiters, so a block whose contents contained a blank line — an ordinary thing to write in source code, and the only way to write two paragraphs in a #+BEGIN_QUOTE — was cut into fragments before anything could see it was one element. The fragment holding #+BEGIN_SRC had no end delimiter, the fragments after it re-parsed as prose (complete with Org inline markup applied to code), and the fragment holding #+END_SRC printed that delimiter verbatim as body text: #+BEGIN_SRC python / x = 1 / blank / y = 2 / #+END_SRC produced a CodeBlock of just x = 1 followed by a paragraph reading y = 2 #+END_SRC. Body segmentation is now aware of open blocks, the same way the file-property filter above it already was: a blank line inside a #+BEGIN_x region is content, and only a matching #+END_x closes it, so code that itself contains #+-prefixed lines stays intact. Because a greater block is an element in its own right, a delimiter now also bounds the elements around it when no blank line separates them — which additionally recovers text written directly after #+END_SRC, previously swallowed by the block and dropped. Affiliated keywords such as #+CAPTION: still attach to the block beneath them.

• An Org list item's wrapped continuation line is no longer deleted. _parse_list kept only the lines that matched a bullet or number marker and had no branch for anything else, so a line continuing an item's text onto the next line — how any reasonably long item is written — simply vanished. - item one continues / onto a wrapped line / - item two produced a two-item list in which onto a wrapped line appeared nowhere at all, with nothing to indicate text had been dropped. A non-marker line now joins the preceding item's principal text separated by a single space, which is the rule the AsciiDoc parser was given for the same defect in #343; a wrapped item and the same item written on one line now parse to equal documents. Nested items are still flattened to a single level — the loop strips each line before matching it, so the indentation that marks a sub-item is gone before the marker is read — and that is unchanged by this fix.

• A single ; in AsciiDoc prose no longer creates a description list. The lexer's description-list pattern was ^(.+?);(?:\s+(.*))?$, which matches an enormous amount of ordinary writing: Alpha; beta gamma. was lexed as the term Alpha with the description beta gamma., a line merely ending in ; became a bare term with no description, and a semicolon anywhere on a wrapped line broke the paragraph it belonged to in two — the text before the wrap staying a paragraph and the rest becoming a definition list. AsciiDoc has no such marker; its description lists are written ::, :::, :::: or ;;. The pattern now requires the doubled ;;, which parses exactly as :: does, and the class docstring's term:: or term; has been corrected. One existing test asserted the single-semicolon behaviour and has been updated: it encoded the defect rather than the language.

• Words no longer fuse across a formatting change in run-based formats. group_and_format_runs — the shared helper that turns a paragraph's runs into inline nodes for PPTX (and available to any run-based parser) — joined each same-format group and then called .strip() on it. The comment claimed this preserved inter-run whitespace, and it did; what it also removed was inter-group whitespace, which is the only thing separating the words either side of a formatting boundary. Runs ("This is ", plain), ("bold", bold), (" and after.", plain) therefore produced Text("This is"), Strong([Text("bold")]), Text("and after.") and every renderer emitted This is**bold**and after. — a bug that fired on every bolded, italicised or underlined span with a space next to it. Whitespace at a group edge is now collapsed to a single separating space and re-emitted; whitespace inside a group is still preserved verbatim, whitespace at the true edges of the run sequence (the paragraph boundary) is still dropped, and a whitespace-only group between two content groups collapses to one space rather than vanishing. The separating space is always placed outside any formatting wrapper — on an adjacent plain Text node where there is one, otherwise as its own Text(" ") node — because a space inside emphasis markers (**bold **) is not valid markdown. The PPTX text extractor's own compensation for the old behaviour (strip each run, then re-append a trailing space) has been removed now that it is not merely redundant but harmful: it dropped leading run whitespace, which the helper needs in order to see the boundary. The hyperlink-segment path keeps its separate boundary-space handling, since the helper is called once per link segment and cannot see across two of them.

• MarkdownRendererOptions.max_line_width and table_alignment_default now do something. Both fields carried help metadata, so both surfaced as CLI flags and in the generated options documentation, and the Markdown renderer read neither — setting them changed nothing at all. table_alignment_default is now used for columns that state no alignment of their own: the default "left" still writes a bare --- (a column with no alignment is left-aligned anyway, and spelling it :--- would rewrite every table this renderer has ever emitted), while "center" and "right" write :---: and ---:. A column with an explicit alignment is unaffected. max_line_width (still None by default, meaning no wrapping and byte-identical output) now soft-wraps paragraph prose, and only paragraph prose — code blocks, tables, headings, link destinations and reference definitions are never touched. The wrap is deliberately timid: a paragraph containing a code span, a link or image destination, a reference label, an autolink, raw HTML or math is left unwrapped rather than broken at a guess, and a break is never taken in front of a word that would make the continuation line reparse as a new block (-, #, 1998., and friends). Note that soft-wrapped lines come back as soft line breaks when reparsed, which is what a soft wrap means.

• Table and figure captions are escaped before being wrapped in the *...* caption device. Markdown has no caption syntax, so a caption is rendered as a single-emphasis paragraph plus a marker comment naming it a caption. The caption text was interpolated raw, so its own metacharacters closed that emphasis early: Sales *2024* results rendered as *Sales *2024* results* and round-tripped to Sales 2024 results, the literal asterisks silently deleted. Shapes that left the paragraph with more than one child failed the parser's single-Emphasis test outright, which lost the caption and leaked a stray italic paragraph plus the <!-- all2md:table-caption --> comment into the AST. Captions now go through the renderer's normal text escaping (and, like a table cell, have any newline flattened, since one would end the caption paragraph and break the marker triple); the ordinary text unescape on reparse gives the original string back.

• A line break inside a Markdown table cell or heading no longer destroys the block it sits in. visit_line_break emitted a real newline ("\n", or " \n" for a hard break) whatever the surrounding context was, and cell rendering escaped pipes but never newlines. A cell holding a hard break therefore rendered as | line1 <newline>line2 | b |, which splits the pipe row: the table reparsed with zero data rows and the remains became a stray paragraph. The same break in a heading ended the heading early and dropped everything after it into a paragraph of its own. Cells and headings now render their inline content in a single-line context: a hard break becomes <br> (GFM's spelling inside a cell, which our Markdown parser keeps as inline HTML in the cell it belongs to) and a soft break — a source-wrapping artifact — becomes a space, matching what the CSV renderer already did with the same nodes. Newlines arriving from anywhere else (a Text node with an embedded newline, multi-line raw inline HTML) are flattened to a space as a backstop. The ASCII-art table fallback gets the same treatment, since a newline broke the grid it was being measured for. Line breaks in paragraphs are unchanged.

• A Markdown block quote nested in a list item is no longer indented twice. The quote's children were rendered to a string while the list item's marker-width indent was still active, so a child paragraph already carried that indent; the quote then prefixed "{indent}> " onto the very same lines. For a marker four columns wide or more ("10. ", or any bullet at the second nesting level) the quoted text landed four-plus spaces past the >, and the round trip read it back as an indented code block inside the quote rather than a paragraph. Only quotes that were not the item's first child were affected, because the first child renders with the indent state already cleared. The indent state is now suspended while the children render and applied once, by the quote, as it prefixes each line. Material for MkDocs admonitions render through the same helper and now carry the indent on their !!! header too, so a nested admonition no longer splits its header off at column zero.

• Six table renderers no longer drop cells when a span collides. BaseRenderer._layout_table_grid resolves declared colspan/rowspan values — which real documents routinely overstate — onto a grid, truncating a span rather than letting it overlap and widening the table rather than dropping a cell that no longer fits. Only the DOCX and PPTX renderers were migrated onto it; the reStructuredText, Org, LaTeX, ODT, ODP and PDF renderers each kept a copy-pasted fill loop that took the table's width from that layout but then placed cells at their declared spans, ending the row with if col_idx >= num_cols: break. Any cell pushed past the last column by an earlier row's rowspan was discarded with no warning: a two-row table whose second row declares colspan=3 under a rowspan=2 neighbour lost its final cell's content entirely in all six formats. All six now consume the shared grid's placements and emit the effective spans, so a collision costs a merge instead of content. Well-formed tables render byte-for-byte as before.

• An email's attachment section is now built from AST nodes instead of Markdown spliced into the message body. process_email_attachments() returned a Markdown string ("\n\n## Attachments\n\n" plus ![name](url) lines) that was concatenated onto message["content"]. A plain-text body — the common case — is emitted as Text nodes, so the Markdown renderer escaped the whole thing: an email with one PNG attachment rendered \## Attachments and !\[pic.png\], a broken heading and a dead image reference. The section is now a Heading plus a paragraph of Image/Link nodes, the same shape every other parser produces via attachment_result_to_image_node, so it survives into non-Markdown renderers too. Three golden snapshots that had recorded the escaped output were updated. Affected the .eml, .mbox/Maildir and .msg paths; on the latter two it was unconditional, since neither ever re-parsed the body as Markdown.

• MBOX and Outlook bodies converted from HTML or RTF are no longer flattened into escaped plain text. parse_single_message() reports content_is_markdown=True when it converts an HTML or RTF body to Markdown, and the EML parser has honoured that flag; the MBOX and Outlook parsers ignored it and split every body into Paragraph(content=[Text(...)]). The renderer then escaped the Markdown source, so a heading arrived as \## Heading and a link as \[text\](url). RTF bodies are converted unconditionally — convert_html_to_markdown gates only the HTML branch — so Outlook-originated mail reaching the mbox or PST path was mangled under default options. All three parsers now share one parse_email_body() helper, and the PST branch sets the flag when it converts an HTML body. Structure loss also affected non-Markdown renderers, since headings and lists never became AST nodes at all.

• An Outlook .msg file now reaches the Outlook parser instead of the RFC-822 email parser. The eml converter claimed .msg in its extension list at a higher detection priority than the outlook converter, and extension matching runs before content sniffing — so any path or named stream ending in .msg routed to EmlToAstConverter. Neither converter defines a content detector, so nothing corrected the choice afterwards. The RFC-822 parser does not reject OLE/CFBF input: message_from_binary_file accepts the binary happily and yields a header-less message whose body is the compound file's bytes decoded as text, so all2md mail.msg produced mojibake rather than an error, and extract-msg was never invoked. The same bytes without a filename already detected correctly as outlook via magic bytes. .msg has been dropped from the eml extension list; its magic-byte patterns are all text mail headers and cannot match a compound file, so the eml parser loses no reachable input.

• Format detection no longer reads a sentence as a filename. registry.detect_format ran os.path.splitext over any str it was given, so a string holding document content was extension-matched on its tail: to_markdown("Reminder: check results.csv") detected csv and returned the one-cell table | Reminder: check results.csv |, and to_markdown("Payload attached as invoice.pdf") raised FileNotFoundError naming a path the caller never passed — the failure class #233 fixed in the input loader, resurfacing one layer down because detection ran before the loader and derived its own answer. Detection now applies the loader's own looks_like_path_attempt rule, so the two agree: a Path, an openable str, and a path-shaped str (short, single-line, no whitespace, no ://, ending in an extension all2md knows) all keep extension and MIME matching, and everything else is content. Path-ness is deliberately not gated on the file existing, since detect_format is also called on output paths that have not been written yet. Such a string is now also handed to the content detectors, which never saw it before — content stayed None for any str that was not an openable file. Inline string content therefore detects identically to the same content passed as bytes: HTML, JSON, XML and the rest are now recognised, and the structured-text detectors claim the same strings they already claimed on the bytes path ("- a\n- b" is YAML there and is YAML here). Where no detector matches, the fallback to plaintext is unchanged.

• Writing to out.txt no longer silently strips every piece of formatting. When no target_format was given, convert() inferred one from the output path — and registry.detect_format answers "plaintext" for anything it does not recognise, including a .txt extension and any unknown one. That answer is a no-match signal, but it was used as a renderer choice, so convert("in.md", "out.txt") rendered through the plaintext renderer: # Title became Title, **bold** became bold, and [a link](https://example.com) lost its URL entirely. The code had a guard for exactly this, comparing the inferred format against "txt" and substituting markdown, but detect_format has returned "plaintext" — never "txt" — since it was rewritten, so the guard had been dead the whole time and the documented "defaults to markdown" behaviour never fired. All three sites (convert(), and the CLI's merge and single-file conversion paths) now compare against "plaintext". Only inference is remapped: an explicit target_format="plaintext" or --to plaintext still selects the plaintext renderer, and a recognised extension such as .html or .docx is still honoured.

• PPTX run hyperlinks are no longer discarded. _process_paragraph_runs_to_inline built a Link node for every hyperlinked run into a local result list, but both of the function's exit paths returned inline_nodes (the output of group_and_format_runs, which never captures hyperlinks) instead — the hyperlink loop was dead code, so every PPTX hyperlink's URL was silently dropped and only the anchor text survived. Runs are now split into consecutive stretches by hyperlink address; each stretch is formatted independently (so bold/italic inside a link's text still applies) and hyperlinked stretches are wrapped in a Link node, preserving run order relative to surrounding plain text.

• PPTX grouped shapes no longer drop all of their contents. _process_shape_to_ast handled text frames, tables, images, and charts and then fell through to None for everything else. A GroupShape has none of those attributes itself — its content lives entirely in its member shapes — so slides with a grouped diagram, SmartArt-converted shapes, or manually grouped callouts (all extremely common in real decks) lost that content with no warning and no degraded-content signal. The parser now recurses into shape.shapes for any GroupShape, in document order, and handles groups nested inside groups.

• A PDF figure caption is now taken from the layout model's caption region, not guessed from a fixed band of text near the image. The old rule matched a figure cue (Figure 3) against a 50pt band above and below the image and, failing that, accepted any text under 200 characters beginning with a capital letter — which on a journal page is a running head, an author list, or an ordinary sentence. Scored against JATS figure captions over 12 PMC articles, it returned 32 strings of which 19 were captions (59% precision, 50% recall). Reading the caption regions the layout model already predicts and binding the nearest one below the figure — below rather than above, which is the convention for figures and the opposite of a table's — returns 31 strings of which 27 are captions (87% precision, 74% recall). Where pymupdf-layout is not installed the cue match still applies but the catch-all fallback is gone, which trades 6 points of recall for 20 of precision (79%/44%). Requiring a cue on top of a layout region was measured and rejected: 0.9 points of precision for 8.8 of recall, because real captions do not all open with the word "Figure". No output changes yet — the detector's result still reaches nothing, and routing it into Image.caption is the next step. (#338)

• An image next to a blank line is no longer silently dropped from a PDF. The caption detector checked a text band for content before stripping it, so a band holding only whitespace became an empty string and then raised IndexError on text[0]. Image extraction wraps each image in except Exception: continue, so the crash never surfaced as an error — it discarded the whole image, after it had been successfully decoded and encoded. Measured at 1 image in 70 across 20 PMC articles, in save and base64 modes with include_image_captions left at its default of True. (#338)

• A nonexistent Path input no longer raises a misleading "Unsupported input type." LocalPathRetriever.can_handle() returned False for a Path that does not exist, and no other retriever accepts a Path, so the loader fell through to a generic "Unsupported input type: WindowsPath" ValidationError instead of the accurate, path-naming error load() already raised but could never reach. can_handle() now accepts any Path instance, letting load()'s existence and file-vs-directory checks run and surface their message — "Path does not exist: ..." — instead.

• An AsciiDoc hard line break inside a table cell no longer splits the row. AsciiDocRenderer.visit_line_break emitted ' +\n' for every hard break regardless of context, and a table row is written as one source line, so a cell with an embedded hard break carried a literal newline into the middle of it. The project's own AsciiDocParser reads rows line by line: re-parsing 'line1 +\nline2 |b' produced a one-cell row ('line1 +') followed by a spurious second row ('line2', 'b') instead of the original two-cell row, and the ' +' marker leaked into the first cell's text. The renderer now tracks whether it is inside a table cell and, in that context, renders a hard break the same way it already renders a soft one — as a single space — since AsciiDoc has no in-cell line-break idiom this project's parser reads back as anything other than a new row.

• Text after a MediaWiki list or : block quote is no longer silently dropped. mwparserfromhell lumps everything up to the next markup construct into a single Text node, so the paragraphs that follow * item\n* item\nSome text. lived inside the last list item's Text node — the parser truncated that node to its first line for the item and discarded the rest, along with the trailing paragraphs. The list/quote parsers now return the unconsumed remainder so the caller can run it back through normal paragraph processing; a list or quote whose Text node ends exactly at the last marker line still emits no stray paragraphs.

• MediaWiki inline formatting no longer fuses with the words around it. mwparserfromhell splits a paragraph into one Text node per inline-markup boundary, so "This is '''bold''' text." parsed to Text('This is'), Strong([Text('bold')]), Text('text.') — every fragment fully stripped of its separating space — and rendered as This is**bold**text.. Text fragments now collapse internal whitespace runs to a single space instead of stripping them away, keeping the one space that separates a fragment from a neighbouring Strong/Emphasis/Link node; only the true edges of a paragraph or heading (trimmed once, when the fragment buffer is flushed) are stripped, so trailing newlines and leading blank lines still produce clean paragraph/heading text with no stray edge whitespace.

• The PDF image-caption fallback no longer treats a plural sentence opener as a figure cue. The fallback (used when the layout model has no caption region for an image) matches a line against a "Figure 3" / "Fig. 2b"-style opener, but the pattern allowed zero whitespace before its letter alternative and matched the whole thing case-insensitively — so [A-Z] also matched the lowercase trailing "s" of a plural: "Figures in this study", "Images were acquired using a confocal microscope" and "Tables 1 and 2" all satisfied the cue as if the "s" were a figure-letter locator like "Fig B". The locator half of the match (the digits or letter after the keyword) is now checked case-sensitively and, for a letter, requires real whitespace before it, so a plural opener's trailing "s" can no longer stand in for one. Genuine cues ("Figure 3:", "FIGURE 4", "Fig B", "Table 1.") are unaffected.

• header_min_occurrences now actually filters by occurrence. The option is documented as "minimum occurrences of a font size to consider it for headers" (its docstring's stale "default 3" is also corrected to the real default, 5), but the statistic it was checked against accumulated characters, not occurrences: fontsizes[size] += len(text) per span. A font size needed fewer than 5 rendered characters — trivial for almost any span — to be dropped, so the filter was a near no-op regardless of how many times that size actually appeared. A separate line-occurrence count is now tracked and checked against header_min_occurrences instead. Body-text detection ("which size covers most of the page") still runs on the character-count statistic, unaffected by this change, since characters and occurrences answer different questions and a paragraph condensed onto one packed line should still out-rank a heading for body status. The single largest font size on the page is exempt from the occurrence check: by convention it is the document's title, which renders once by design and would otherwise never clear a repetition threshold now that repetition is measured for real. Every other size — subordinate headings, and any one-off oversized span the filter exists to catch — is filtered as documented. This is a real behaviour change for header detection: a font size that previously qualified on character count alone but occurs only a couple of times (and is not the page's largest) will no longer be treated as a heading size.

• WebArchive subresource extraction no longer silently drops files on Windows. _extract_subresources had two defects, both masked by the method's blanket except Exception: logger.warning(...). A text subresource (a plist <string>, not <data>) was written with Path.write_text() and no explicit encoding, so on Windows — where the platform default is cp1252 — any non-Latin content raised UnicodeEncodeError and the resource was dropped; it's now written as UTF-8. Separately, the filename was taken as Path(resource_url).name, which keeps a URL's query string (img.png?v=1); ? is illegal in a Windows filename, so the write raised OSError and the resource was dropped. The filename is now taken from the URL's path only (query string and fragment stripped) and run through the existing attachment-filename sanitizer.

• A detected PDF figure caption now reaches Image.caption instead of being discarded. The caption was routed through fallback_alt_text, a dead path: extraction writes a non-empty placeholder alt text (Image from page N), so the fallback never fired and include_image_captions=True could not affect output in any attachment mode. The caption now rides on the node's caption field — visible page content set beside the figure, not a substitute for it — and the Markdown renderer already round-trips it as an italic line plus a marker comment. The default alt_text mode still extracts nothing; that is the remaining half of the defect and is tracked separately. (#340)

• DokuWiki footnote definitions no longer fuse their paragraphs into one token. A multi-paragraph definition was rendered through the inline path with nothing between the blocks, so first para and second para came out first parasecond para -- a destroyed word boundary, the same corruption class fixed for reST and Org definition lists (#347). Blocks now render separately and join with a space. The paragraph boundary itself is still lost -- DokuWiki's inline footnotes have nowhere to carry it -- and the round-trip fuzzing gate continues to document that; the words survive.

• Org footnote definitions keep their paragraphs across a round trip. Org continues a footnote definition across a single blank line and ends it at two, but neither side spoke that dialect (#347): the renderer joined a definition's paragraphs with a bare newline (one continuation line on re-parse) and followed a definition with a single blank line (which would swallow the next block), while the parser's block splitter discarded the blank counts entirely. The renderer now separates a definition's paragraphs with one blank line and follows a definition with two; the splitter counts the blank lines between blocks, and single-gap paragraph blocks after a [fn:id] join its definition. Also ported the boundary-break hoisting cure (#391) to the Org span delimiters (/, *, +, _): a hard break at a span's edge stranded the delimiter at a line start, where * opens a headline and + a list item.

• reST footnotes survive a round trip with their identifiers and paragraphs. Three defects stacked (#347): the renderer spelled every footnote [a1]_ / .. [a1], which for an alphanumeric label is reST citation syntax, so the footnote stopped being one -- non-numeric identifiers now use the named auto-numbered form ([#a1]_ / .. [#a1]), with escaped whitespace (word\ [#a1]_) where a marker rides a word, since docutils only starts inline markup after a boundary. A multi-paragraph definition rendered as continuation lines and read back as one paragraph -- its blocks now separate with blank lines at the marker's body column, and hard breaks inside a body fall back to raw newlines so | line-block syntax cannot displace it. And the parser preferred docutils' normalized anchors (ids, e.g. footnote-1) over the label as written (names), mangling identifiers even for plain numbered footnotes; it now takes the name on both definitions and resolved references, and resolves docutils' internal \x00 escape markers instead of copying them into text.

• reST and Org definition descriptions no longer fuse or lose words. Both renderers concatenated a description's paragraphs with nothing at all between them, so alpha and beta came back as the single token alphabeta -- a destroyed word boundary, not a lost break (#352). The reST renderer now separates blocks with a blank line at the same indent, which round-trips the paragraph count exactly; the Org renderer joins blocks as indented continuation lines, so the boundary degrades to a line rather than a fused word. Worse than the reported fusion, the Org parser silently deleted every definition line that did not start a new - term :: item -- a wrapped definition lost all but its first line; continuation lines now join the open definition. A term's several descriptions still flatten into the one definition each syntax can hold (docutils: one definition per term; Org: one :: per item) -- words intact, count inherently lost -- and the fuzzing-gate entries now say exactly that.

• PDF: a subsection heading printed directly under its section heading is no longer fused into it. The wrap-merge that reassembles a long title set on two printed lines had no width test, so Methods over Study design became one heading — and both section titles went missing, the largest single class (~30%) of the heading residual on the PMC born-digital corpus (#400). A line only wraps because it filled its measure, so the merge now requires the first line to fill at least HEADING_WRAP_MIN_FILL (0.8) of the two lines' shared width — a threshold read off 307 labeled merges: true wraps fill 0.852–1.0, the separable fused band 0.22–0.84. Pairs whose first line is the wider one (Methods and Design over Study design) remain geometrically inseparable and still merge; that residual is documented on the issue.

• The AsciiDoc renderer/parser pair stops rejecting, corrupting, or leaking its own output. Four defects, one format. A bold or italic span wrapping only a hard break stranded its delimiters at a line start, where * is a level-1 list marker and ** a level-2 one -- the first silently turned emphasis into a list item, the second crashed the parser outright, the round-trip matrix's only open crash (#353); boundary breaks now hoist outside the delimiters, the same cure as markdown's #391. A definition-list description on the line directly below its term:: -- the standard placement, and what this renderer itself emits -- parsed to nothing: the term came back empty, the text as a sibling paragraph, and the list split at every term (#351); unindented lines adjacent to the term now bind to it as one wrapped paragraph. The renderer also fused a description's paragraphs into one token (only+extra -> onlyextra, #352's class); blocks now join as continuation lines. And the parser did not recognise the named inline footnote form footnote:a1[text] -- valid Asciidoctor and the renderer's own spelling -- so the raw markup leaked into the prose as literal text (#346); it now parses, a hard break inside the macro's brackets degrades to a space instead of an unparseable embedded newline, and an id referenced but never defined gets an empty definition rather than an unbalanced round trip.

• The Markdown renderer no longer emits markdown its own parser misreads. Three defects shared that shape. A spanned table was written one pipe cell per AST cell while the delimiter row was sized to the logical width, so a colspan header made the cell counts mismatch and GFM read the whole table as prose; cells are now placed on the resolved grid, spans padded with empty cells -- the merge is lost (pipe tables cannot express one), the table is not, and rows after a rowspan stay in their own columns instead of sliding left (#385). A hard line break was always spelled as two trailing spaces, so a break on a line with no visible text (one break following another) left a whitespace-only line, which is a paragraph boundary in every conformant parser -- consecutive breaks silently split their paragraph in half; such breaks now use the backslash spelling, which puts a visible character on the line (#384). And an emphasis, strong, or strikethrough span wrapping nothing visible -- or ending in a line break -- stranded its delimiter run alone at a line start, where *** is a thematic break and ~~~~ opens a tilde code fence that swallows the rest of the document; nested strikethrough now renders its inner content bare (GFM strikethrough does not nest), spans over nothing visible emit no delimiters, and boundary breaks are hoisted outside the delimiters (#391). The round-trip fuzzer's figure-gate strategies were constrained to avoid the first two defects and its footnote allowlist carried the third under a wrong attribution; the constraints and the stale allowlist entry are removed, so the gates now guard all three classes.

• The PMC born-digital lane runs again. Moving the corpus fetchers' XML parsing to defusedxml left the scheduled PMC Born-Digital Fidelity workflow crashing at import: it syncs a deliberately lean environment (--extra pdf_layout --extra ocr), and defusedxml lived only in format extras the lane does not install, so the 2026-08-15 scheduled run died in 29 seconds before scoring a page. A run that cannot execute also cannot count toward the lane's exit criterion (two consecutive clean scheduled runs before a fidelity baseline is recorded), so this was holding the gate open. What the benchmark lanes import beyond the library now has its own named home — a benchmarks extra — instead of borrowing from a format extra that happened to carry it.

• reST line blocks are no longer silently dropped. A | line block fell through the parser's unknown-node branch, so every line of it vanished from the output -- text loss, not formatting loss. A line block now parses as a paragraph whose lines join with hard breaks; nested (indented) lines flatten into the same paragraph, since the AST does not model their indentation.

Changed

• An HTML <figcaption> no longer overwrites the image's alt text under figures_parsing="image_with_caption". It binds to Image.caption, so a figure that carries both a real alt attribute and a caption keeps both. Previously the caption was only absorbed when the image had no meaningful alt text of its own, and absorbing it destroyed the alt text when it did. A caption with no image to bind to is still emitted as its own paragraph rather than dropped.
• The born-digital lane records why a table region was rejected, and how many were (payload schema 4). The parser distinguishes nine reasons for rejecting a table region and coalesces repeats of (parser, kind, detail, severity) while summing their counts. The lane read back only kind, and then counted coalesced event objects — so an article rejecting twelve regions and one rejecting a single region contributed identically, and table_rejected: 101 was neither a count of regions nor of articles but of event objects, which is the least meaningful of the three. Nothing downstream could tell a corpus-wide detection failure from one pathological document, and nothing could tell an improvement from a regression, because some of those reasons are the parser correctly refusing to grid a page of prose. degraded_events now reports occurrences broken down by reason, with the number of articles each reached beside it — the two answer different questions and diverge sharply here, since a single article contributed twelve of the twenty-nine text_grid_splits_words rejections measured across twelve articles. This is a benchmark payload change only; no conversion behaviour changes, and no published figure moves.

Fixed

• A list item's text may run onto the lines below it (AsciiDoc). AsciiDoc wraps freely: the lines after * item are part of that item until a blank line, another item, or a block. The parser never implemented this. Where it was merely lossy, the run-on line surfaced as a sibling paragraph after the list instead of as the item's own text; where it was worse, the stray line ended the list, so a nested item on the next line had no level-1 parent and the conversion failed outright with ValueError: Cannot nest to level 2 without a parent item at level 1. Three lines of ordinary hand-written AsciiDoc — * a, b, ** c — were enough to hit it. The joining rule, hard breaks and all, already existed for paragraphs and is now shared with list items rather than duplicated, and it merges the text nodes a wrap leaves adjacent, so * a / b and * a b now parse to equal documents. Found by the generative round-trip gates on their first per-PR run. (#343)

• One rejected table region is no longer counted as two (PDF). When the layout model predicts a table region and a grid is found there but then refused, the refusing guard records its own specific reason — text_grid_splits_words, degenerate_grid, mostly_empty. The method that owns this already carried a note forbidding the vaguer layout_region_not_tabular from being added on top, because that counts the same region twice and takes a second bite out of the confidence score. The flag the note relies on did not work for the case it was written for: found_grid was set after the split-word guard, and that guard continues, so the flag was only ever true for grids that survived — never for a grid found and then rejected. The signature is unmistakable on the born-digital corpus. Across 12 articles the two reasons appear in exact 1:1 correspondence in every affected article — 1/1, 4/4, 6/6, 12/12, 29 each in total — which is one region counted twice rather than two regions rejected. So the confidence score was penalised twice for a single refusal, and the benchmark's headline table_rejected count roughly doubles its dominant case, which matters because that number is read as "tables we threw away". The flag is now set when a grid is found, which is what its name and the note both already said it meant. The existing regression test asserted this property but only on the arm that satisfied it: it drives lines_strict with a degenerate grid, reaching the foot of the method with the flag already set. The new one drives the text-alignment arm where the defect lived, and asserts the recorded reasons rather than the counter — a counter cannot see one region rejected under two names.

• The PMC born-digital corpus is back to 66 articles. PMC11000011.1 was withdrawn upstream (#329), and since #330 the lane degraded gracefully and scored 65 of 66. That is the right behaviour for an incident and the wrong steady state: a permanently-false complete_corpus is a permanently-yellow test, so the second withdrawal would be invisible against a run that already looks like that — and it quietly consumes the tolerance budget that exists for real incidents, which aborts the load past a tenth of the selection. The replacement is drawn by the selection process rather than chosen: only the PMC11000000 seed was re-walked, through the same build_manifest entry point with the committed stride, per_seed, candidate cap and filter, because a hand-picked article would bias the corpus in a way nothing downstream could detect. The walk also shows the withdrawal is a rejection rather than an outage — the prefix still lists and only the PDF 404s — so stride alignment is preserved and the seed's two surviving members re-select byte-for-byte identically. Only the third slot moves, to PMC11000033.1. The pin moves 34cc7c50 → 9ba5c0cd, so the next run uses a new digest-keyed cache directory. Re-recorded benchmarks/pmc/reference.json against the new pin on CI, and updated every figure in docs/source/benchmarks.rst in the same change. The headline readings barely move, which is what a representative corpus should do when one article of 66 is swapped: recall of attainable text 95.3%, novel n-grams 1.0%, both unchanged to the published precision. Tables go from 88-against-117 to 92-against-121 (#332).

• The published fidelity figures are now checked against the artifacts they come from. docs/source/benchmarks.rst is the one place the project makes a public quality claim, and the claim rests entirely on each number coming from a committed artifact — but nothing enforced the correspondence. The page and the artifacts were written in the same change and agreed on the day; the first time a pin moved they could silently stop agreeing, and a stale figure on that page reads exactly like a measured one. This lane had already published a comparison between two runs covering different corpora, so the failure mode is demonstrated rather than theoretical. The check is built the strong way round: rather than scraping numbers out of the prose and asking whether each looks plausible — which cannot see a figure that should be there and is not, and needs an exemption for every incidental integer — each published claim is declared as a snippet rendered from the artifact, and the page must contain it verbatim. Matching the whole line rather than the number alone means a stale line fails too. Verified the judge can fail: against the pre-update page it reports exactly the nine figures that moved, while the unchanged OmniDocBench readings still match. Two stronger invariants ride along, because matching rendered text still assumes the artifact itself describes the right corpus. reference.json's corpus_pin must equal the SHA-256 of the committed manifest.json, so a manifest edit landing without a re-record — precisely what #332 had to repair — fails immediately rather than at the next scheduled run; and the published reference must be a complete run covering every article the manifest names, since complete_corpus going false is the right signal for a run and the wrong state for the artifact a public page quotes as its reading.

• Three AST walkers stopped dead at List and Table nodes. AddAttachmentFootnotesTransform._collect_footnote_refs, GenerateTocTransform._collect_headings, and GenerateTocTransform._inject_heading_ids recursed only through hasattr(node, "children") / hasattr(node, "content"), but List stores its items in .items, Table in .header/.rows, TableRow in .cells, and DefinitionList in .items tuples — none of which the check saw, so recursion silently stopped the moment it reached one of those nodes. In practice, the CLI's add-attachment-footnotes transform never emitted a FootnoteDefinition for an empty-URL Image/Link nested in a list or table cell, and GenerateTocTransform (Python-API-only) omitted headings nested the same way from the generated TOC and never injected their ids. All three walkers now recurse via the shared get_node_children() helper (src/all2md/ast/nodes.py) that every other AST walker in this codebase already uses, so they see every node type uniformly. _inject_heading_ids also had to gain a hand-rolled DefinitionList case, since replace_node_children() deliberately refuses to rebuild that node's (term, [descriptions]) tuple structure generically.

• HTML: bare text directly under <body> (or in a body-less fragment) is no longer discarded. The top-level walk only processed Tag children of <body>/root and skipped every NavigableString, so <body>Loose text. <p>Paragraph.</p>Trailing text.</body> kept only the <p>, and a pure fragment like Just plain text with <b>bold</b> inside parsed to a Document containing just Strong('bold') with all the surrounding plain text deleted. The rest of the parser already had a mechanism for this — _process_block_container accumulates adjacent inline content and flushes it into a single Paragraph when a block sibling is reached — so the top-level walk now delegates to it instead of duplicating a narrower, buggy version of the same logic. Whitespace-only text between block elements still produces no paragraph.

• XLSX cell hyperlinks render as real links again, instead of escaped literal text. The parser built a markdown-syntax string ([text](url)) for a hyperlinked cell and handed it to build_table_ast() as plain cell text, which wrapped it in a Text node — so every renderer saw an ordinary string, not a link. The Markdown renderer escapes brackets in Text content, so a hyperlinked cell rendered as \[text\](http://x) instead of a working link, and the HTML renderer emitted the literal bracket syntax instead of an <a> tag. The parser now extracts each cell's (text, hyperlink URL) pair and, for a hyperlinked cell, builds a real Link node directly in the TableCell — the same thing the DOCX, PPTX and HTML parsers already do. build_table_ast() (shared by CSV, ODS and XLSX) gained a second accepted cell shape for this: a cell can still be a plain string, unchanged for every other caller, or a pre-built list[Node] used as-is. ODS has no cell-hyperlink extraction at all yet, so it is unaffected and left as-is.

1.12.0 - 2026-08-12

Added

• A conversion fidelity page (docs/source/benchmarks.rst), with a committed evidence artifact behind it (benchmarks/pmc/reference.json). The benchmark lanes have been producing numbers for several releases and none of them were published anywhere a user would look: the documentation could say all2md converts documents well, but not how anyone would check. Every figure on the page now comes from a committed artifact and is printed beside its control — the same measurement applied where it ought to fail — because on most text metrics the highest-scoring converter is one that dumps the raw text layer with no structure at all, and a fidelity score with nothing to falsify it is not evidence. The born-digital reading, over 65 articles and 699 pages of publisher PDFs scored against publisher JATS: 95.3% of attainable text recovered against a 0.4% wrong-article control, and 1.0% of emitted n-grams novel — containing a word the document does not have anywhere — against a 0.7% control. Both figures are reported with the denominators that make them readable rather than flattering, and the page explains why raw recall (60.1%) and raw unsupported output (6.2%) are the wrong numbers to quote. It is candid where the numbers are weak. Tables are the worst area and say so: 88 emitted against 117 expected, with detection rather than extraction as the bottleneck. It also states what the lanes structurally cannot see — every corpus is English, so a change that deleted all CJK, Cyrillic and Arabic text would score perfectly on all three.
• Content precision beside content recall on the born-digital lane (benchmarks.pmc.article.measure_precision). Recall alone cannot be trusted, and this lane has already been misread once because of it: the highest recall available on this corpus comes from emitting the raw text layer with no structure whatsoever, so recall rising is not by itself good news. The new instrument asks the converse — does the output contain anything the document does not — against the PDF's own text layer rather than JATS, because JATS is not what the page prints and scoring against it would charge the parser for reproducing the document faithfully. Raw precision turned out to need a denominator of its own, for the same reason raw recall needed its attainable ceiling. Most of what a correct conversion emits unsupported is the document's own words in an adjacency the text layer does not have: all2md orders columns and joins blocks, the layer comes out in PyMuPDF's order, and every disagreement mints n-grams at the seam. Measured over five articles that is 4.8% of emitted n-grams, against 0.5% carrying a word the layer never has anywhere — so reporting the raw figure would have made the result nine times worse than the parser deserves. The two are reported separately, and novel_share is the figure to read. Duplication is counted apart from both, because a block emitted twice is an unchanged set and a doubled multiset — no set-based score can see it. Both figures ship with the mismatched-article control the lane already applies to recall.
• A PDF conversion guide (docs/source/pdf.rst). PDF is the format all2md works hardest at and the one with no dedicated page: the deepest hand-written coverage was five bullets in the overview that predated layout analysis entirely, and of the twelve OCR flags, nine appeared only in the generated option reference — listed, never explained. The new page starts from what a PDF actually is (positioned glyphs, so every heading, table and reading order is inferred) and then covers the two per-page routes through a document, layout analysis and why pdf_layout is excluded from all, the table strategies and the whole-word guard on the borderless fallback, column ordering, the heading/list/dehyphenation rules that explain otherwise-arbitrary output, and when auto OCR fires — including that image_area_threshold changed meaning as well as default this release. It records the trade-offs as measured numbers rather than claims, and it is candid about the limits, such as list items the PDF prints with no marker at all, which no marker rule can recover. The overview and format pages now link to it instead of restating a stale summary.
• Every registered format's options class is now importable from both all2md and all2md.options. Twenty were missing from all2md.options and thirty-one from the top level, including whole formats — ArchiveOptions, EnexOptions, MboxOptions, OutlookOptions, WebArchiveOptions, and the JSON, YAML, TOML, INI, Textile, OpenAPI and BBCode option classes — so configuring those converters from the Python API meant importing from a private submodule path. The split did not track anything: ArxivPackagerOptions was reachable from all2md but not from all2md.options, which is backwards from every sibling. The two surfaces are now derived from the converter registry and a test fails if either drifts from it again, so adding a format cannot quietly skip the export (#184).
• OCROptions and MetadataRenderPolicy are importable from all2md and all2md.options too. Both are the declared type of a field on an exported options class — PdfOptions.ocr and BaseRendererOptions.metadata_policy, the latter reaching every renderer — so a caller has to construct one to set that field, and neither was reachable from either public surface. Configuring OCR from the Python API meant importing all2md.options.common, which the PdfOptions docstring told you to do; passing a plain dict instead did not work, and failed deep inside parsing with 'dict' object has no attribute 'enabled' rather than at the call. The check that was supposed to catch this could not: #184 derives its expectation from the converter manifest, which names only format options classes, and a nested configuration group belongs to no format. The new rule is derived rather than listed — any dataclass appearing in a public options class's field annotations must be public itself — and it resolves the annotations before looking, because this package uses from __future__ import annotations and 91% of those fields carry their type as a plain string, so the obvious version of the check inspects 89 of 982 fields and reports success.
• layout_feature_set, choosing which layout classifier reads the page (--pdf-layout-feature-set, requires pymupdf-layout; inert otherwise). The package bundles three: imf+rf (the default, image and text-geometry features), imf (image only) and rf (text geometry only). Which one is right depends on the document, so it is now a searchable option and is registered as an optimize knob rather than decided globally. The difference is not subtle: on a two-column reference page the image-feature models read the dense left column as a table — deleting nine reference entries before the fix elsewhere in this release, and splitting one table into two on three further pages — while rf labels all 41 entries correctly and predicts no table at all. Across 20 born-digital articles rf led on every axis measured (title recall 0.9916 → 0.9972, identical table content recall, three fewer spurious table nodes on the same 16 pages, 29% faster for skipping image inference). The default is deliberately unchanged: that is one corpus of one document kind, and image features plausibly earn their place on scanned pages, of which it contains none. Models are now cached per feature set instead of in a single global — an unkeyed cache returned whichever model loaded first, which would have made every arm of a search over this knob identical and the setting look inert.
• A pinned PMC Open Access corpus for born-digital PDF benchmarking (benchmarks/pmc). The existing external ground-truth lane is 981 rasters, so it measures the OCR path; nothing external covered text-layer extraction, vector table detection or layout-derived reading order, which is most real-world PDF conversion. Articles come from the pmc-oa-opendata bucket, where each versioned prefix holds the publisher PDF beside its JATS XML — publisher-produced ground truth with real sections, paragraphs and table cell markup. The bucket has no corpus-wide revision to pin against, so a committed manifest of SHA-256 digests for both files of all 66 articles takes that role: loading never lists the bucket, revalidates every byte, and keys its cache by the manifest digest. Selection keeps an article when its JATS has at least one <p> and its PDF carries vector drawings; every rejected candidate is recorded by reason. Corpus bytes are not committed, and this ships no oracle or gate — those follow separately. A characterize command measures what the built corpus actually contains, deliberately independent of the filter that selected it: 750 pages, 100% with a text layer, 81.1% with vector drawings, 0% with the single-full-page-image scan shape — the inverse of the raster lane, and the zero was checked against a known scan first to confirm the test can fire at all. An align command measures whether article-level JATS truth can be projected onto PDF pages: 95.7% of blocks place onto a page or an identifiable adjacent pair, with a 1.9% miss rate. Placement is content-only, never using extracted reading order, so it cannot quietly grade the reading-order metric against itself — and it ships with a control that scores every block against a different article's pages, where the false-placement rate is 0.8%.
• A born-digital scoring lane over that corpus (benchmarks.pmc score). Each article is converted once and every page scored against the JATS truth projected onto it, through the same oracle the raster lane uses, so a number here means what a number there means. Page boundaries come from the parser's own per-page separators, which keeps cross-page context intact — splitting the PDF into single-page files would hide exactly the defects this lane exists to find — and a dropped page raises rather than silently shifting every later page's ground truth. Blocks that will not resolve to a page are excluded and reported as an error budget printed with every run. Three controls ship inside the run: each page is scored again against the next page of the same article, against deliberately reversed, scrambled and halved output, and with OCR left enabled in auto mode so "no page needed OCR" stays a measurement that could fail rather than a configuration. Two dimensions are reported but flagged unusable as gates, with the measurement that disqualified them — block_structure_similarity separates own-page from wrong-page output by only ~0.06 and rises when half the content is deleted. Whole-article content recall ships with its attainable ceiling: only 61.1% of JATS blocks are recoverable from the PDF text layer by any parser, because structured citations and bylines record words in an order the page never prints, so raw recall on its own reads as parser loss that is not there.

Changed

• Raw HTML in Markdown is passed through instead of escaped. The Markdown renderer's html_passthrough_mode now defaults to pass-through. Escaping was described as a security posture, but it was not the one doing the work: in the html → markdown direction — the direction where untrusted input actually arrives — the HTML parser maps tags to AST nodes and drops <script>, <iframe>, <form>, <object>, <embed> and <svg onload> outright, so they never reach a renderer to be escaped. The HTMLBlock and HTMLInline nodes the Markdown renderer actually saw came from the Markdown parser: HTML an author wrote in their own Markdown, which Markdown permits by design and which any downstream renderer would have rendered from the original file anyway. Escaping it broke markdown → markdown for no gain — our own README.md lost the contents of all eleven <details> sections and scored 97; it now scores 100 with no deltas, as do all six tracked root documents, and the CI fidelity gate is re-recorded from 97 to 100. Converting untrusted Markdown is the case that wants the old behaviour: pass html_passthrough_mode="escape" (or --markdown-html-passthrough-mode escape). Only the Markdown renderer changes; the HTML, AsciiDoc and Textile renderers still default to escape (#178).
• PyMuPDF is imported under its own name, not the deprecated fitz alias. Some PyMuPDF releases emit a DeprecationWarning on import fitz, which reached anyone running the CLI as noise about a dependency they did not choose and could not act on. pymupdf has been the real module name since 1.24.3 and we require 1.27.2, so nothing is lost: 149 references across 15 modules now use it. No public API changes — the fitz module object is still what PyMuPDF hands back, and _pdf_layout.py still neutralises the layout hook on both module objects, since the alias can be a distinct object in some installs (#284).
• AttachmentOptionsMixin and CloneFrozenMixin are no longer part of the public API. They remain importable from all2md.options and nothing about them has changed; they are simply no longer listed in __all__, because exporting a mixin promises its shape to anyone who subclasses it and that is a larger commitment than an options dataclass. Only from all2md.options import * is affected.
• A dropped keyword argument is now diagnosed by which mistake it was. A name that is an option of no format still reports as Unrecognized keyword arguments were ignored — the typo case #273 was about. A name that is a real all2md option the conversion's formats simply have no field for (pages on a markdown parse, flavor on an HTML render) now says so instead of telling the caller to check for a misspelling they did not make. Both can arrive from one call, as two warnings. to_ast and from_ast previously stayed silent for the second case while to_markdown warned; they now agree, which closes the last gap #273 left open. Callers who pass a real-but-inapplicable option to to_ast will see a warning where they saw none — it is telling them the option did nothing.

Fixed

• A withdrawn PMC article no longer takes the whole born-digital lane down with it. The committed manifest pins each article's SHA-256, which guarantees what the bytes are but says nothing about whether they are still served — and PMC reprocesses articles and drops the superseded version. PMC11000011.1 was fetched successfully one morning and 404'd that same evening, from two networks, with .2 and .3 equally absent (#329). The lane's only response was to abort the load, so a single upstream withdrawal made the corpus unscoreable on any machine without a warm cache, and the monthly scheduled run would have begun failing regardless of anything in this repository. A 404 is now recorded rather than raised: the article lands in CorpusSnapshot.unavailable, the snapshot stops reporting itself as complete, and the ids appear in the payload as corpus.articles_unavailable (payload schema 3) and in the CLI summary beside the article count. That last part is the point — a score over 65 of 66 articles is a different measurement from one over 66, and nothing downstream could previously tell. Tolerance is capped at a tenth of the selection, and losing every article is spelled out separately so a one-article run cannot pass having scored nothing. Transient failures and digest mismatches are still fatal, because those are claims about the bytes rather than about the object's existence.

• The PDF optimization page no longer compares two different corpora. It led with "reduced the corpus benchmark from 21.4 minutes to 6.7 minutes — a 3.2x improvement", taken from the two committed reference runs' corpus-wide totals. The arithmetic is right and the comparison is not: the runs cover 160 and 149 documents with 115 in common, and the arxiv overlap is zero — that source samples the most recent cs.CL submissions and the runs are nine days apart. Documents present in only one run account for 46% of the baseline's wall time and 86% of the optimized run's, so the headline measured a change of corpus as much as a change of code. The same applies to the PDF percentile rows: both runs have 80 PDFs, 30 of them different papers. The page now leads with the 115 shared documents: 11.5 minutes to 55.4 seconds, a 12.4x improvement — a defensible figure, and a larger one than the number it replaces. The govdocs1 rows (p50 30x, mean 12.7x) and the 000887.pdf case study (10.7x) were already sound, because that shard is fixed, and they remain.

• Three places called the corpus sample "deterministic" or "reproducible" when two of its four sources are neither. corpus.toml has marked arxiv and poi reproducible = false all along, and benchmarks/corpus/README.md contradicted itself between its opening line and its own reproducibility section. Roughly 60 of 160 documents can differ between runs, which is exactly the trap the entry above fell into.

• The born-digital benchmark workflow no longer discards the scorer's exit status. It pipes into tee without set -o pipefail, so tee supplied the exit code and a traceback in the summary file went green. The lane is ungated on fidelity by design; that was never a reason to lose a hard crash as well.

• report-fail-under is a measurement again instead of a constant. The GitHub Action advertises it as "fail if any document's conversion confidence falls below this score", and for most formats it could not fail at any threshold. all2md report returns a hardcoded score of 100 banded not_assessed whenever no detector ran for the document's producer — which is every markdown, text, docx, pptx and html input, PDF being the only richly instrumented format. all2md.confidence names this "a vacuous 100 that means 'no detector ran', not 'verified clean'"; the gate read score and discarded band, so an empty file, a valid file and a deliberately broken file all passed --report-fail-under 100 identically. The gate now reads the band. An unassessed document is neither a pass nor a failure: it is reported as not assessed, excluded from the threshold comparison, and never rendered as a bare 100. If a threshold was set and nothing could be assessed, the run is a configuration error — the same class as an empty glob, and exit 2 rather than 1, so it cannot read as "your documents failed". A mixed corpus keeps working; only a wholly unassessable one is refused. This repo's own CI was one of the affected consumers: --report-fail-under 100 over *.md has been dropped from the docs gate, because Markdown can never satisfy it honestly. Fidelity remains gated at 100.

• The Semgrep security scan actually scans again. It had been reporting success without reading a single rule since 2026-07-01. semgrep/semgrep-action@v1 pins semgrep 1.36.0, which cannot parse registry rules carrying severity: MEDIUM, so every run died on ValueError: invalid rule severity value: MEDIUM — and the action wrapper swallowed the non-zero exit. The last genuine scan was 2026-06-30 (Found 0 findings from 1059 rules, 57 seconds); every run after it finished in 11–16 seconds having loaded nothing. Because Security Scan (Semgrep) is both a required check on main and a release gate, v1.11.0 published through a gate that was structurally incapable of failing. SEMGREP_APP_TOKEN was never set either, so the old comment's reasoning about a cloud ruleset described a token-authenticated scan this job had never performed. The CLI now runs directly against the pinned p/python and p/security-audit rulesets, which needs no token. A positive control runs first: semgrep is pointed at a deliberately vulnerable file written at run time, and the step fails if it reports no findings. That is the specific failure that went unnoticed for six weeks — a scanner that finds nothing because it loaded nothing looks exactly like a clean repository. The first working run scanned 1241 files with 346 rules and reported 18 findings, of which 2 are in shipped code: both the deliberate autoescape=False in the Jinja renderer, which emits Markdown rather than HTML. That line already carried a nosemgrep for one rule id; the new ruleset flags it under a second, which nothing could have noticed while the job was crash-passing. The remaining 16 are in tests/, benchmarks/ and stubs/ — none of which the wheel ships — so the blocking scan is scoped to src/ and those are triaged separately rather than blocking a release.

• The two external ground-truth lanes no longer disagree about which dimensions may support a verdict. block_structure_similarity compares block-category sequences without ever inspecting the text underneath, so content-free output scores 1.0 on it (issue #256). The born-digital lane measured that directly — against deliberately damaged output it rises when half the emitted content is deleted, and it separates own-page from wrong-page output by only ~0.06 — and refused to gate on it. The scanned-page ratchet went on comparing it against a tolerance every month. One project cannot hold both positions, and the one backed by measurement is the refusal, so the declaration now lives in one shared benchmarks/omnidocbench/dimensions.py that both lanes read. Not gated is not unchecked: a run whose block_structure_similarity is missing, malformed, out of range, or inconsistent with its own recorded page scores is still red. Only the comparison of its value against the baseline is skipped, because a number that moves the wrong way under damage cannot be evidence of a regression or of an improvement. The gate now prints a NOT GATED line naming the dimension and why on every run, pass or fail — a gate that quietly stops comparing something reads exactly like one that compared it and found it fine.

• python -m all2md.cli runs the CLI instead of refusing to start. python -m all2md worked; the cli package had no __main__.py, so Python answered 'all2md.cli' is a package and cannot be directly executed. That fails before argparse sees the arguments, which makes it worse than an ordinary unknown-command error: every invocation fails identically no matter what follows it, so anything checking whether the CLI accepts a given flag gets the same answer for a real flag and an invented one. Both entry points now reach the same main(), and tests assert they can still reject an unknown flag, since an entry point that cannot start looks exactly like one that accepts everything.

• Seven CLI flags the documentation named do not exist, and now the docs say what does. Most followed one rule the docs had backwards: a boolean option generates only the flag that changes its default, so require_https=True yields --html-network-no-require-https and there is no positive form to type. Documented as --html-network-require-https, and worse, as "Default: Disabled" when HTTPS is in fact required by default — the one entry where believing the docs would have left someone thinking transport security was off when it was on. --html-network-max-remote-asset-bytes was a renamed option still documented under its old name and its old value (the real one is --html-max-asset-size-bytes, 50MB not 20MB). --markdown-page-separator-template was filed under Markdown but belongs to the paginated parsers, which its own document.pdf examples already showed. Also corrected: --pptx-slide-numbers → --pptx-include-slide-numbers, --pdf-detect-columns → --pdf-no-detect-columns, --output-type → --output-format, and --rich-word-wrap, which never existed alongside the --rich-no-word-wrap documented directly beneath it. A test now fails if any prose page names a flag the CLI will not accept.

• Table captions survive a Markdown round trip. Markdown has no caption syntax, so the renderer demoted Table.caption to an italic paragraph — which a reader could see, but which came back as ordinary prose, so the trip lost the caption and gained a Paragraph node that was not in the input. The caption is now written as that same italic paragraph followed by an <!-- all2md:table-caption --> marker, which the Markdown parser folds back into Table.caption along with the paragraph. Readers still see the caption, the AST comes back with the node count it started with, and since the marker carries no copy of the text, editing the visible line edits the caption. The fold is deliberately narrow: a bare marker, a marker with no table after it, or an italic paragraph with no marker are all left alone. comment_mode="ignore" suppresses the marker, and the caption then degrades to the italic paragraph as before. This was the last of the four formats in #237 — asciidoc, rst and org already round-tripped their captions (#237).

• json_to_ast no longer crashes on explicit JSON nulls. A producer outside this library routinely writes null for a field it has nothing to say about, and "children": [null], "content": null or a null metadata object reached the node constructors unchanged. The failure then surfaced much later, inside a renderer, as a TypeError naming nothing that would help. Null list and object fields now read as empty, null child entries are skipped, and a field that is structurally required — a Heading level, a List ordered flag — raises a ValueError naming the node and the field instead of defaulting to something plausible. Thanks to @santhreal (#265).

• "Your PyMuPDF is too old" now says so instead of raising TypeError (PDF). The version guard built its message by joining PyMuPDF's version tuple, which holds ints, so the one branch that exists to tell a user to upgrade crashed from inside itself and reported nothing useful. It also reconciles the minimum version, which had drifted to three different values: the runtime guard demanded 1.27.2 while the converter's declared dependency — the number list-formats prints and dependency errors quote — still said 1.26.4. Both now read one constant.

• Two columns that start level are read left-to-right, not by a hairline (PDF). Blocks were ordered on y alone, so on a page whose gutter is too narrow for column detection to split, whichever column's top edge happened to be a fraction of a point higher was read first — and that decided the order of the entire page. On page 16 of PMC7500012.1 the two columns begin at y=89.708191 and y=89.702942, five thousandths of a point apart, and the references came out running 39–61 and then 19–38: a document that looks correct and is not. Blocks whose tops fall within a twentieth of the page's average line height are now treated as starting on the same row and ordered by x. That is a size-relative measure rather than a constant, since what reads as "level" scales with the type size, and it is deliberately small: it only reaches blocks that begin at effectively the same height, where the y order was noise to begin with. Measured on the PMC born-digital corpus (117 pages), reading-order similarity rises from 0.779 to 0.785 mean and 0.821 to 0.835 median, with block structure and text content up slightly and tables and whole-article recall unchanged — no dimension regressed.

• <del>, <s>, <sup>, <sub> and <mark> survive a Markdown round trip. mistune hands raw inline HTML through untouched, so the Markdown parser produced loose HTMLInline nodes rather than the AST node that already existed for the meaning; the default html_passthrough_mode="escape" then escaped them on the way back out and a <del>x</del> b returned as a <del>x</del> b. They now fold into Strikethrough, Superscript, Subscript and Mark the way <u>/<ins> already folded into Underline, and render as ~~x~~, ^x^, ~x~ and ==x==. The existing constraints are unchanged: a tag carrying attributes stays raw HTML, because the attributes hold information the node cannot, and an unmatched opener or stray closer stays raw rather than being guessed at.

• <mark> no longer disappears when parsing HTML. It was listed as an inline element but had no handler, so it fell through to the generic unwrap and the highlight was dropped outright — <mark>x</mark> became a bare x. Unlike the Markdown side this was silent loss rather than escaping, so no round-trip text comparison would have noticed it; a golden snapshot had captured the damaged output as expected. Same gap <ins> had.

• PyMuPDF's layout advisory no longer lands inside the converted document (PDF). PyMuPDF prints Consider using the pymupdf_layout package for a greatly improved page layout analysis. with a bare print() — not a warning, not a log record — the first time find_tables() runs in a process where pymupdf.layout is not installed. all2md writes converted documents to stdout, so the advisory arrived inside the document: all2md report.pdf > report.md made it line one of the markdown, and any pipeline reading all2md's stdout got it as content. This was the common case rather than an exotic one — pymupdf-layout is deliberately excluded from the all extra over its Polyform Noncommercial license, so the plain and [all] installs both hit it. all2md now opts out through PyMuPDF's own entry point when it opens a PDF; it ships that package as the pdf_layout extra and already reports its absence through its own dependency machinery, which writes to stderr.

• .tar.gz, .tar.bz2 and .tar.xz are accepted on the command line again. Every read-side command collects inputs through one extension filter, and that filter compared Path.suffix — which returns only the last dot-separated component — against the set of registered extensions. Path("bundle.tar.gz").suffix is ".gz", which no converter declares, so the file was dropped before any converter saw it and the CLI reported No valid input files found about a file that was present, correctly named, and readable: to_markdown("bundle.tar.gz") had opened it all along. The split was exact — every single-part spelling (.tar, .tgz, .tbz2, .txz) worked and every two-part one failed. Matching now walks the tails of Path.suffixes longest-first, so .tar.gz wins where it applies while report.2024.pdf still resolves to .pdf. This reached all2md, grep, search, view and watch alike. The same rule now backs the interactive batch type table, so the extension it offers is the one its filter accepts.

• --zip projects its namespaced options like every other path. The CLI carries options fully qualified — pdf.pages, markdown.flavor, and the subcommand sections' own settings such as view.dark — and flattens them against the format that will actually handle each file. Every path that writes to disk does that per input file; --zip handed the whole namespaced dict to convert(), which matched none of it and reported the leftovers as the user's typos: Unrecognized keyword arguments were ignored: ['view.no_wait', 'view.dark']. Nobody typed those — they are the [view] section's settings, and the function that exists to drop them was never called on this path. Projection is now per item, so a mixed batch applies pdf.* to the PDFs and html.* to the HTML rather than to everything, matching what the same files get when converted to disk.

• Packaging no longer warns the user about an attachment_mode it injected itself. create_package_from_conversions forces attachment_mode="base64" so attachments stay in memory, then passed it to convert(); for a source format with no such option (markdown, txt) it was dropped and reported as an unrecognized keyword argument. The user never typed it. An ordinary all2md *.md --package out.zip said "Check the API documentation for valid parameter names" about a command line that was entirely valid. Library-internal call sites can now mark an option as their own, and the three that inject one — the CLI packager, the MCP server's attachment_mode from server config, and convert's flavor shorthand — do. An attachment_mode the caller passed explicitly is still theirs and still warns.

• Text rescued from a rejected table region is dehyphenated like any other prose (PDF). When a detected table's grid turns out to be degenerate the region is emitted as a paragraph instead, and its text is recovered with page.get_textbox() — raw extraction, which returns the glyphs with their printed line breaks intact. Every other route into the AST passes through dehyphenate_blocks() first and this one did not, so a word broken across a line stayed broken: Coroman- and del never became Coromandel, and the word appeared nowhere in the output at all, which no text-recall measure notices because both fragments are still present. This is a growing path rather than an edge case — the layout model over-predicts tables, and each over-prediction routes a whole region's prose through it; on the page this was found on the predicted region covered the entire page body. On that article, 32 line-break fragments become 0 and 8 words reappear. The join now reads identically to ordinary prose, including where the two agree to leave a hyphen alone: a capitalised continuation keeps it (Anglo-Saxon) and a digit continuation is never merged.

• A list marker set in a symbol font now starts a list (PDF). Whether a line opened with a marker was decided by finding the first top-level Text node, which is the wrong question in both directions. A bullet set in a symbol font carries that font's flags, so it arrives wrapped — italic flags make Emphasis(Text("-")) — and the line has no top-level Text at all: it read as empty and never started a list, so one born-digital article's bullets came out as plain paragraphs each beginning with a literal - . In the other direction, reading past the first node to find some Text answers for the middle of the line, so a citation opening with a styled journal name, Nature 12. 45-67, reported a numbered marker and became a list item. Detection now descends into inline wrappers, and the marker is removed by character count so that it comes off even when it sits inside a wrapper or straddles two nodes. Reading the raw spans instead would not work, which is worth recording because it looks like the obvious fix: the parser rewrites four bullet glyphs (U+F0B7, U+00B7, U+2022, U+25CF) to -, and three of those four are not markers in their printed form, so detection has to run after that conversion rather than before it.

  A marker and the space that disambiguates it are also routinely separate spans, which is why Word's second-level Courier o bullets never became list items at all — the rule that an o must be followed by a space (so that "office" is not a bullet) could never fire, because the space was in the next node. Those bullets now nest under the item above them instead of landing as loose paragraphs between the items they belong to.

  That look-ahead is allowed only for a bullet, never for a number, and the restriction is the load-bearing part rather than a tidiness. A numbered marker arrives split exactly the same way — Text("44.") then Text(" Konema, Nigeria …") — and nothing in a PDF distinguishes the 44th bibliography entry from the 44th item of a list. Reading across that boundary turned reference lists into ordered lists, and since nothing carries a start number through, the renderer printed them from 1: reference 44 came out as item 1 and no citation in the body could be matched to its reference. A numbered marker must therefore be complete within one span, as it always was.

  Measured on the born-digital corpus over the twelve list-bearing articles that complete locally, list-item recall rises 0.231 → 0.269 and precision 0.116 → 0.141; only three articles change at all. The items still missed are overwhelmingly ones the PDF prints with no marker of any kind, which no marker rule can reach.

• A bulleted or numbered list is no longer collapsed into a single item (PDF). The vertical gap between two list items is the same gap that separates two paragraphs, so the paragraph-break rule is suspended once a list has started — otherwise every item that wrapped would be split at its own second line. Nothing was put in its place, so an item could only end when its block did, and a whole list arrived as one item: one born-digital article emitted 1 list item for its 16 bullets. A line carrying its own marker now starts the next item, whatever the spacing says. The rule is narrow on purpose — it applies only inside a paragraph already recognised as a list, so it cannot turn prose into one. Measured on the born-digital corpus, list-item recall rises 0.059 → 0.212 and precision 0.040 → 0.113.

• A heading that wraps onto a second printed line is now one heading (PDF). A PDF has no notion of a wrapped heading — it has two lines of type — and each line reached the emitter on its own, so a long section heading became two sibling headings and an article title set on three lines became three #s. None of them matched the real title, so the text was there and the structure was not. A heading line now continues the heading directly above it when the two are at the same level, nothing was emitted between them, and the second sits within HEADING_WRAP_GAP_RATIO line heights of the first — the gap is a ratio rather than a point count so it means the same thing for a 24pt title and a 9pt subheading. Two headings that genuinely follow one another are separated by the space above a new section, which is what puts them beyond it, and a second line opening with its own numbering starts a new heading however tightly it is set. Measured on the born-digital corpus, section-title recall rises 0.729 → 0.766 and precision 0.507 → 0.623 — 231 headings emitted for 188 real titles, down from 270, so the join removes 39 duplicate headings as well as recovering 7 real ones.

• A lone math glyph is no longer promoted to a heading (PDF). Heading classification validated a candidate by font size, bold/all-caps requirement, maximum length and an internal-sentence-boundary check, and never asked whether the text contained a letter. Large delimiters are set in a symbol font well above body size, so they cleared the size gate and passed every remaining one — short, non-empty, no sentence boundary — and were emitted as headings from inside displayed equations. One chemistry paper in the born-digital corpus produced 179 headings for its 9 sections, 122 of them a single Private Use Area glyph (∑, ∫, large parentheses and brackets); it now emits 55, with its real headings untouched and its text content byte-identical once heading markers are stripped. The gate is str.isalnum, not an ASCII character class, precisely so that headings in CJK, Cyrillic, Arabic, Devanagari, Greek and Hangul still qualify — an ASCII test would have deleted every one of them while fixing this.

• Section headings inside a multi-line block are no longer flattened to prose (PDF, requires pymupdf-layout). Layout labels were assigned per block, by IoU against the model's predicted region. That is a fair test only when a block is one semantic unit, and PyMuPDF returns a whole journal column as a single block — so a two-line heading inside it scored an IoU near 0.03 against its own block and its section-header label was discarded before anything could use it. Measured on the born-digital corpus, 54% of section-header predictions never reached a block, the median miss being a block 38x the prediction's area; the same plumbing lost 52% of list-item and 51% of page-header labels. Labels are now also stamped per line, by containment rather than IoU — a line inside a correct region has a low IoU with it by construction — and the heading path consults the line's own label before the block's. Against JATS section titles on 12 articles, the share emitted as headings rose from 0.420 to 0.729 (79 → 137 of 188) for five additional headings not backed by a title. Two articles went from 0/32 and 3/33 to 24/32 and 26/33. Nothing else on the corpus moved: text recall is identical to the digit in both arms (95.6% of attainable overall; title 94.2%, text_block 96.3%, table 96.7%), reading order is unchanged at 0.779, and block_structure_similarity rises 0.548 → 0.567 — a change in structure and nothing else, which is what was intended. This is the same class of defect as the partial table region fixed above: a block is the wrong unit to judge, and the fix is to judge the line.

• The structural half of heading detection is now measured. The corpus lane scored title as text recall, which cannot distinguish ## Materials and Methods from a bare paragraph carrying the same words — a parser that flattened every heading in the corpus would have scored 1.00 on it. Note the ground truth for this is section titles specifically: the oracle's title kind also maps article-title, which appears inside every bibliography <ref>, so scoring headings against it rewards exactly the wrong behaviour on a 60-reference article.

• A table region covering part of a text block no longer deletes the rest of that block. Blocks a table region covers are withheld from the text stream, because the region is emitted in its own right — as a table, or as a paragraph when the grid is rejected — and emitting both would duplicate it. That decision was made per block on a majority-area test, so a region covering more than half of a block removed the whole block. But "more than half" is not "all of it": PyMuPDF returns a full-height journal column as a single block, so a region predicted over its lower half cleared the bar for the entire column and the upper half was carried out with it — deleted outright, since the re-emitted region text does not include it. On page 16 of one benchmark article this silently removed nine reference entries, a region over y=380–733 of a column spanning y=90–733 taking 54.8% of it and everything above with it. Coverage is now judged per line: lines a region covers are still withheld, and whatever lies outside every region survives with its bounding box tightened to what remains, so reading order still sorts correctly against the table it precedes. A line straddling a region boundary is assigned by the same majority rule rather than duplicated or dropped by both sides. Lines that are blank or whitespace are not rescued: those border a table region routinely, and emitting them would replace a dropped block with an empty paragraph. Measured on 20 articles of the PMC born-digital corpus, title recall of attainable rose from 97.6% to 99.2%, and the four full-height column drops the corpus contained fell to two.

• Words no longer run together where a bold or italic run wraps onto the next line. Lines of a paragraph are joined with a separator space unless the text already ends with whitespace there, and the check that decides this walks back to the last text leaf, which may sit inside a Strong/Emphasis/Link. It conflated two different answers from a wrapper — "its text does not end in whitespace" and "it holds no text at all" — and treated both as keep looking. A line ending in a styled run therefore fell through to whatever preceded that run, and if that ended with a space the separator was suppressed and the two halves were concatenated: negotiating Roang on + Lamotrek Atoll came back as Roang onLamotrek. The two runs then looked adjacent to the inline consolidator, which merged them into a single node, so the space could not be recovered downstream either. The walk now distinguishes "no text here" from a definite verdict about the join point. This affects any wrapped styled run, not one document kind, but it surfaced through bibliography entries on the PMC born-digital corpus: a wrapped reference title is short enough that one bad join costs it more n-grams than a recall threshold allows, while a long paragraph absorbs the same damage unnoticed. Over a 20-article subset it recovers 16 of the 33 unrecoverable titles with none newly lost, taking title recall of attainable from 95.4% to 97.6%; one review article with a large reference list goes from 22 lost to 6. The remaining losses are a separate defect — text genuinely absent from the output rather than reflowed — and are still open.

• Auto-mode OCR no longer discards a good text layer because the page has figures on it. In mode="auto" the decision to OCR a page was made by summing the area of every image on it and comparing that to image_area_threshold (then 0.5) — regardless of how much text the page already had. Since preserve_existing_text defaults to False, a page that tripped this had its publisher-supplied text thrown away and re-read from a picture. On the PMC born-digital corpus, characterized as 0.0% scan-shaped, this fired on 20 pages across 11 of 66 articles; every one of those pages had real extracted text (median 536 characters, up to 1998). Two separate faults. Summing image areas does not measure whether a page is a scan: one affected page carried six figure panels covering a tenth of the page each, which summed past the threshold while nothing on the page was remotely page-sized. And the threshold sat far below where scans actually live. The trigger now measures the largest single image, and the default is 0.8. That boundary is calibrated rather than picked: across 101 scanned pages the largest image covers exactly 100% of every one, while across 851 born-digital pages it never passes 64% (median 13% of pages carrying any image at all). Note the threshold's meaning changed with its default, so an explicit image_area_threshold is now read against the largest image rather than the summed area. The narrowing was checked in both directions: a real scan raster carrying a thin text layer — a running header, which clears text_threshold so only this branch can reach it — still triggers OCR, and reverting the measure makes the born-digital pages fire again. Measured over the 11 affected articles: text_content_similarity median 0.515 → 0.599, block_structure_similarity mean 0.530 → 0.563, whole-article recall of attainable 94.6% → 95.1%, and 9 of the 11 articles stop OCR'ing entirely. Table scores and title recall are unchanged, and the 55 unaffected articles are untouched by construction — exactly 16 of the corpus's 750 page decisions change, all of them from OCR to no-OCR.

• Borderless tables in layout-predicted regions are recovered instead of being emitted as prose. With layout analysis on (pdf_layout), a region the layout model predicts to be a table was searched with PyMuPDF's default find_tables() strategy, which requires ruling lines on both axes. Journal tables are typically booktabs-style — horizontal rules only, or none — so the search found nothing and the region was demoted to a paragraph. On the PMC born-digital corpus that was 0 of 31 such regions recovered. The parser's own layout_region_not_tabular telemetry made this look like guards rejecting tables they had found; instrumenting the branch showed no grid was ever found to reject, and that the event also fired a second time on regions whose specific rejection reason had already been recorded, double-charging the confidence score. Text-alignment detection is now tried as a fallback, and the duplicate event is gone. Because that fallback has no ruling lines corroborating it and the layout model over-fires, it is held to one extra test the line strategies are not: its columns must not cut through words. Without it, a mis-predicted region rendered a page of abstract prose as a seven-column table of half-words (study was condu | cted to explore) — every table metric improved while whole-article recall fell from 92.6% to 83.8%. Grid shape, reading-order preservation and region corroboration (ruling lines, a Table N caption) were each measured as guards and each failed to separate real tables from gridded prose; whole-word integrity, measured against the page's own unclipped word segmentation, separates them cleanly. Net effect on a 12-article subset: tables emitted 4 → 12 of 32 expected, table_content_similarity mean 0.075 → 0.241, table_structure_similarity 0.091 → 0.235, with whole-article recall unchanged at baseline.

• Scanned PDFs now keep their block structure instead of collapsing to one paragraph. Every OCR'd page previously projected as exactly one Paragraph — no Heading, no Table, no list structure — because the OCR result was returned as a flat string and wrapped in a single synthetic block spanning the whole page, with one line, one span and a hardcoded font size. Everything downstream of OCR segments on block, line and span geometry, so column detection, header/footer trimming, table-region filtering and block-to-node conversion all received one page-sized rectangle and had nothing left to work with. Measured across all nine OmniDocBench data sources: 33 semantic blocks over 36 pages against 704 annotated regions. It was specific to the OCR path — under the same parser policy, born-digital PDFs emit 7–20 blocks per page with real headings and tables. Tesseract already assigns block, paragraph and line numbers and per-word boxes; image_to_string discards them and image_to_data reports them, so OCR now returns paragraphs mapped back into PDF points and the parser emits one block each. Engines that cannot report layout, and any failure recovering it, fall back to the previous behaviour rather than losing the page.

• Scanned multi-column pages no longer interleave their columns. Sorting blocks by vertical position assumes a column is one top-to-bottom run. OCR paragraph boxes are tight to their glyphs rather than the wide regular blocks column detection looks for, so a two-column scan was read as one column and the sort then alternated left and right fragments — turning a reference list into "Norlund / 1997. Occupational / NRC / Sluiter". The OCR engine already emits blocks in reading order across columns, so that order is now preserved rather than rebuilt from geometry. Both fixes are scoped to OCR-derived blocks; PDFs with a text layer are unaffected, and tests pin that.

• Scanned pages now carry usable positions. Because paragraphs are real blocks, each one reaches the AST with its own SourceLocation.metadata['bbox']. A scanned page went from a single box covering the entire page to 54 distinct ones, so a citation into an OCR'd document can resolve to a region rather than to "somewhere on this page".

1.11.0 - 2026-08-04

Added

• Tests: generative round-trip fuzzing across the format matrix. tests/document_strategies.py builds arbitrary Document trees with Hypothesis and feeds them to roundtrip_report for all 24 round-trippable formats, behind four gates: every format must be classified (so a newly registered one cannot silently skip the fuzzer), ast must score exactly 100 as the control, no format may raise outside a known-crashes allowlist, and shapes drawn from previously fixed defects must survive the trip. Both allowlists are xfail(strict=True), so they can only shrink — fixing an entry turns it into an XPASS and fails the build until it is removed. It found six crash classes and a set of invariant gaps on existing formats, now filed as issues. Thanks @santhreal (#204).
• Tests: external PDF fidelity against pinned OmniDocBench ground truth. benchmarks/omnidocbench downloads the immutable 981-page v1.0 corpus, calls all2md.to_ast once for each page, and compares AST text, tables, formulae, and reading order directly with annotation fields. Dimensions the parser cannot yet express are reported as unsupported instead of scored, so absent capabilities cannot earn credit. A fail-closed ratchet records score denominators, variance, and exact page scores, rejects corpus, oracle, parser-policy, parser-runtime, or conversion drift, and requires review for both regressions and improvements. The full lane runs only on a monthly schedule or manual dispatch. Pull request and release CI keep using synthetic, network-free tests. The first baseline is recorded: text content 0.5058, reading order 0.6034, block structure 0.1176 over all 981 pages. Exact reproduction of the annotation scores 1.0, and no degradation of it scores higher: checked across eleven degraded variants and all six dimensions that then existed, on all 981 pages, plus the variants that sweep did not cover — every one of its variants deleted blocks, so output that kept the block structure and destroyed the content, or kept both and only re-chunked, went untested. Both went on to matter. Blanking every block scored a perfect reading order, so a block now has to be located before it votes on the ordering; and pairing emitted blocks against annotated ones one-to-one scored segmentation rather than order, so that text reproduced exactly but emitted as a single block scored 0.0. The first full-corpus run put 894 of the 981 pages at exactly zero, 128 of them scoring 0.9 or better on text content. Blocks are now located inside the concatenated output, which is blind to how it was chunked, and the block-category sequence is reported on its own as block_structure_similarity — segmentation is a real question and a different one from ordering. Read that dimension as a granularity ratio rather than a quality score: it is bounded by the ratio of the two block counts, it never inspects the text under a block, and it is independent of the other two (+0.03 and +0.05, where those two correlate +0.84). Thanks @santhreal.
• The OmniDocBench lane records what its corpus actually contains. provenance.corpus_characterization counts how many pages carry a text layer, vector drawings, or the full-page-image shape of a scan, and how many documents the parser ran OCR on. The lane was built, gated and baselined before anyone asked that question, and the answer changes what the scores mean: every page in the pinned corpus is a raster, so they grade OCR rather than the PDF text and table paths. Counted from the PDF directly rather than from a projection, so a parser change cannot alter what the corpus is reported to contain, and excluded from the gate's identity fields so evidence like this can be added without invalidating a recorded baseline. Every page of a document is measured, not just its first — a title page is not a sample of the article behind it — and the scan shape is decided by image area: a page counts as scanned when one image covers at least 80% of it. Calibrated against 49 pages of scanned journal back-catalogue, where the largest image covered exactly 100% of every page, and 101 pages of modern born-digital articles, where the largest figure reached 61%. The earlier rule of "exactly one image and no vector drawings" was wrong in both directions on that sample: it fired on born-digital pages carrying a single figure, and missed scans that ship a second small raster beside the page image.
• An erased benchmark dimension no longer reads as a verdict on the parser. unsupported_dimensions said "all2md emitted no Table nodes on N converted page(s)", which names one side of a two-sided fact. On this corpus it is the wrong side: every page is a full-page raster, so the PDF table path never runs and there is nothing for it to have missed. The message now states the parser's output, how many pages carry ground truth for the dimension, and how many of the characterized pages are full-page images, then points at provenance.corpus_characterization and leaves the cause to the reader (#257).

Changed

• A str that names a missing file now raises instead of becoming the document. Every text parser accepts a str that may be either a file path or the document content, and nothing in the signature distinguishes them — so each parser guessed, and the library guessed two different ways. Fifteen parsers fell through to "it must be content", which meant a typo in a filename produced a one-line document containing the filename: MarkdownToAstConverter().parse("does_not_exist.md") returned a Paragraph reading does_not_exist.md, which downstream is a successful conversion of the wrong thing. The other three (csv, fb2, rtf) read every str as a path, so raw content was unusable. Both halves now go through one classifier in all2md.utils.inputs.resolve_str_input, so all 18 agree: raw content parses, and a string that looks like a path but does not resolve raises FileNotFoundError naming the escape hatches. "Looks like a path" is deliberately narrow — no whitespace, no ://, and ends with an extension all2md knows — so prose such as "read config.json" or "Visit https://example.com" is still content. Callers who pass content that fits that shape should pass bytes or io.StringIO; Path(...) still always means a path. ini, json, toml and yaml also stop re-wrapping typed all2md failures as ParsingError, so the more specific exception survives (#233).
• Type inference no longer reads 1 and 0 as booleans. With type_inference=True, the JSON, YAML and TOML renderers treated the table cells 1 and 0 as true/false, so an age column of 1 serialized as a boolean. They are now integers; true/yes/on and false/no/off still infer as booleans, matching YAML 1.1. Output changes for anyone who relied on 1/0 inferring as booleans. Thanks @santhreal (#202).

Fixed

• Bold and italic no longer swallow the images, links and code they wrap. The inline consolidator asked _has_nested_formatting whether a Strong/Emphasis held anything worth recursing into, and that check looked only for nested Strong/Emphasis. A node wrapping an Image, Link or Code therefore looked like a plain-text node, and the fallthrough path rebuilt it from its concatenated text alone. An Image contributes no text, so **![alt](img.png)** rebuilt to an empty Strong and rendered as the empty string — the image and the paragraph around it both vanished, with no error anywhere. Any non-Text child now counts, so the node is recursed into instead of rebuilt. Strikethrough, Mark, Underline, Superscript and Subscript are consolidated recursively too, rather than passed through untouched. Thanks @santhreal (#264).

• HTML table cells keep their colspan, rowspan and alignment. The parser read a cell's alignment only to build the table's column-alignment list and dropped the span attributes on the floor, so <td colspan="2"> parsed as an ordinary cell and the merge no longer existed by the time anything rendered. All three now land on the TableCell itself, and a span that is missing, non-numeric or below 1 falls back to 1 rather than raising. Spans now survive HTML→HTML, DOCX and AsciiDoc round trips; Markdown has no span syntax, and its renderer still does not pad for a spanned cell, so a colspan="2" there continues to put the following cell in the next column. Thanks @santhreal (#266).

• A nested EPUB table of contents is no longer flattened to its top level. ebooklib returns a TOC as a tree, where a section with children is a (Section, [children]) tuple, and the builder iterated it one level deep — keeping entries that had a .title and silently discarding every tuple. A book that nested its chapters under parts emitted the parts and nothing else. The TOC is now walked recursively, one heading level per level of depth, clamped at 6. Thanks @santhreal (#267).

• A Word or ODT heading deeper than level 6 no longer crashes the conversion. Heading validates its level as 1–6 in __post_init__, but the DOCX parser passed through whatever number it matched in a Heading N style name and the ODT parser whatever outlinelevel said. Word ships built-in styles through Heading 9, so an ordinary document raised ValueError: Heading level must be 1-6, got 7 and converted not at all. Both parsers now clamp into range. Thanks @santhreal (#262, #263).

• A null path item or components section no longer crashes an OpenAPI spec. YAML lets a key be written with nothing under it — /pets: or components: on a line by itself — and that parses as None, which is legal in a spec that is still being filled in. Both were then used as dictionaries (path_item.items(), components.get("schemas")) and raised AttributeError. Non-dict path items are skipped, and a null components now falls back to Swagger 2.0's definitions the same way a missing one always did. Thanks @santhreal (#260, #269).

• An escaped pipe in a PDF table cell no longer splits the cell or doubles its backslash. The markdown-table fallback in the PDF parser split rows on every |, so a cell containing \| fractured into two cells. Escaped pipes are now masked before the split and restored as a bare | — the AST holds unescaped text and the Markdown renderer adds the backslash back on the way out, so restoring the backslash here as well would have escaped it twice and emitted \\|. Thanks @santhreal (#268).

• Table captions survive a round trip in AsciiDoc, RST and Org. All three have a caption syntax and none of them used it in both directions. AsciiDoc already wrote the caption as a .My caption block title, and its parser ignored the line, so the caption made the trip out and not the trip in. RST emitted a bare grid with nowhere to put one; it now uses the .. table:: directive, which takes the caption as its argument and the table as its indented body, and reads it back. Org emitted nothing; it now writes the #+CAPTION: affiliated keyword. Doing that also narrowed the filter that keeps #+TITLE: and friends out of the body text, which previously dropped every #+KEYWORD: line and would have eaten the caption with them — it now works from a list of genuinely document-level keywords. One visible consequence: an RST table caption that used to vanish on conversion now reaches the output, so RST→Markdown gains an italic line above the table. Markdown has no caption syntax and its arm of this is still open (#237).

• AsciiDoc and Org: an ordered list keeps the number it starts at. Both formats renumber a list themselves, so 1. through n. is all either one preserved and a List(start=5) came back as start=1. The two lost it at opposite ends. AsciiDoc emitted no marker at all; it now writes the [start=N] block attribute, and the parser reads it (and consumes it, so it cannot attach to a later list). Org did emit the number — a literal 5. — but Org renumbers from 1 unless the first item carries an explicit counter set, so the information was in the document and thrown away by Org and by our own parser alike. The renderer now adds [@5] to the first item, and the parser reads both [@N] and, failing that, the list's own first number, which is what the Markdown parser already did. Hand-written 5. First in Org therefore starts at 5 now (#239).

• RST: all six heading levels are now distinguishable. Two defects collapsed them, and both are visible on any document with more than a couple of heading levels. The renderer had five underline characters for six levels, so levels 5 and 6 shared * — and RST derives a heading's level from the order in which each underline character first appears, which makes two levels sharing a character the same level. The default gains a sixth character ("), and a repeated character is now rejected by RstRendererOptions rather than silently merging two levels. Separately, on the parsing side, docutils was promoting a lone top-level section's title to the document title and a lone subsection's to the subtitle. That takes both out of the section tree, so the depth count restarted underneath them: a document reading Title then Section came back with both at level 1, and six properly nested headings came back [1, 2, 1, 2, 3, 4]. Sections stay sections now (doctitle_xform off, as Sphinx does), so the level is the nesting depth. Bibliographic fields written under a title are still read as metadata — docutils only lifts them into a docinfo node when it promotes the title, so they are now read from the field list directly (#238).

• AsciiDoc: heading levels no longer shift, and level 6 is no longer dropped. The renderer added one = to every level, reserving a bare = for a document title it never actually wrote. Nothing read it back that way — the parser counts = and returns that number — so every heading came back one level deeper than it went in. Level 6 was worse than shifted: it rendered as seven =, which is not a heading at any level, so the node was dropped and six headings became five. AsciiDoc has exactly six markers for six levels, so all six only fit if level 1 is a single = — which is what the renderer's own docstring already claimed it did. Output changes: a level-1 heading is now = Title rather than == Title (#236).

• AsciiDoc: a table no longer gains a phantom column. Rows were written with a trailing delimiter — |A |B | — and AsciiDoc reads whatever follows the final | as one more cell, so every N-column table parsed back as N+1. It was silent and it did not depend on the contents: a one-column table came back with two. The extra column also persisted into anything converted onward from that AsciiDoc. Each cell is now introduced by its own delimiter and the row ends with the last cell's content. Fixing the renderer exposed the matching parser gap: a span spec belongs to the cell after it, so canonical AsciiDoc writes |A 2+|B, and the parser only recognized the spec when it stood alone in its own segment — the form the old renderer happened to emit. Hand-written |A 2+|B silently lost the span. Both forms parse now. Output changes for anyone diffing rendered AsciiDoc tables (#235).

• Org: a source block above the first heading is no longer deleted. Text before the first * heading went through a filter meant to keep file properties such as #+TITLE: out of the body, and it dropped every line starting with #+. Org spells its block delimiters with the same prefix, so #+BEGIN_SRC/#+END_SRC were removed and the code between them re-flowed as a paragraph — an org file that opens with a code block silently lost it, while the identical block one line below a heading parsed correctly. Only #+KEYWORD: lines are filtered now, and never inside a block, so a #+TITLE: written in Org source stays source. #+BEGIN_QUOTE and #+BEGIN_EXAMPLE are also recognized for the first time — in both positions, since neither was ever handled: they used to reach the output as literal text, mangled on the way, because +…+ is Org's strikethrough syntax and #+BEGIN_QUOTE therefore rendered as #~~BEGIN_QUOTE. A quote block becomes a block quote, an example block a fence with no language, and any block kind still unrecognized contributes its contents without its delimiters. Also: a bullet with nothing after it is now a list item. The item pattern required content, so - \n- b parsed as a one-item list (#240).

• AsciiDoc: nested lists survive a round trip. The renderer emitted a + list continuation before a nested list, which detaches it into a block of its own and leaves the ** marker with no * parent at the level below it — output the AsciiDoc parser rejected with Cannot nest to level 2 without a parent item at level 1. Every document with a nested list was affected, not only the task-list shape the fuzzer first reported. Nesting is now carried by the marker alone; + is still emitted for code blocks, tables, block quotes and additional paragraphs, which do need it. Closing the round trip also required the parser to learn list continuations at all: it treated a lone + as the end of the list, so the item after an attached block started a fresh list with no parent to nest under, and valid hand-written AsciiDoc such as * a\n** b\n+\nc\n** d was rejected. Continuation blocks now attach to the open item and the list stays open (#206).

• Rendering failures now always raise All2MdError, whatever the underlying library raised. from_ast documents that failures surface as All2MdError subclasses, but renderers delegate to third-party libraries that raise their own types, and only some renderers translated them — from_ast(doc, "pptx") raised a bare ValueError and from_ast(doc, "rtf") a bare KeyError, so a caller writing the documented except All2MdError crashed instead. Probing the whole matrix showed 15 of the 24 formats leaked, not just the two the fuzzer happened to expose, so the translation now happens at the single point every render passes through; a renderer that already raises All2MdError keeps its own message. The PPTX renderer, which guarded only its save step, and the RTF renderer, whose render_to_string was unguarded while render was not, are both brought in line with the DOCX renderer (#212).

• HTML parser: nested tables no longer duplicate their cells into the outer row. Cells were collected with a recursive find_all, so a table inside a <td> had its own td/th pulled into the enclosing row as well — a one-cell inner table turned | inner | into | inner | inner |. Only direct children are collected now, matching how the row walk already worked. Thanks @santhreal (#199).

• DokuWiki parser: a whole-line <del>, <sub> or <sup> keeps its formatting. Those tags were taken as a plugin or HTML block when they were the entire line, so with plugin parsing off the content was dropped outright. They now fall through to inline parsing and produce strikethrough/subscript/superscript. Thanks @santhreal (#200).

• Org parser: a line that is only +strikethrough+ is no longer an empty list. The list check matched -, + or * at the start of a line without requiring the space that follows a real bullet, so +gone+ on its own line parsed as an empty list and lost the text. A space is now required after the marker. Thanks @santhreal (#201).

• AsciiDoc parser: //// opens a block comment instead of commenting out one line. The lexer checked // before block delimiters, so //// was read as a line comment and the block's body was parsed as ordinary content — text meant to be hidden appeared in the output as paragraphs, and the closing //// was consumed as another comment. //// is now a delimiter pair whose body becomes a single Comment node, dropped entirely under strip_comments. Thanks @santhreal (#203).

• Textile renderer: an empty table no longer crashes the render. A Table with no header and no rows appends nothing, so the trailing-newline cleanup indexed an empty buffer and raised IndexError whenever such a table was the first thing rendered. The buffer is checked before indexing. Thanks @santhreal (#205).

• ODT/ODP renderers: a nested link no longer fails the whole conversion. ODF forbids <text:a> inside <text:a> and odfpy enforces it, so a Link inside a Link raised IllegalChild and aborted the render. This was reachable from ordinary input rather than only from a hand-built AST: browsers accept nested <a> and the HTML parser preserves the nesting, so all2md page.html --out page.odt failed outright on any page containing one. The inner link is now unwrapped — its text is kept inside the enclosing hyperlink and only its own target is dropped. Unwrapping rather than splitting the outer link into siblings, which is what a browser does, keeps the surrounding inline flow intact and stays correct at any depth, including a link buried under a Strong that could not be split out without discarding the emphasis (#211).

• Table cells whose spans overflow or collide no longer crash the render or vanish from it. A colspan/rowspan wider than the row it sits in is ordinary in real-world HTML, but every renderer sized the grid by summing each row's colspans, which ignores the columns an earlier rowspan already consumes. Cells displaced past that width were dropped, and a span reaching into ground another cell held produced an impossible merge: DOCX raised no `tc` element at grid_offset, PPTX range contains one or more merged cells. The fuzzer reported the two crashes, but probing the whole matrix found the same geometry in eight renderers — the other six silently lost a cell instead, which is worse for being quiet. Grid layout now happens once, in BaseRenderer, under two rules: a span truncates rather than overlapping, and the grid widens rather than dropping a cell. Losing a merge is recoverable, losing content is not. DOCX, PPTX, LaTeX, ODT, ODP, Org, RST and PDF all take the shared pass (#207, #208).

• RTF: nested lists no longer crash the round trip, and stop losing their deepest items. The RTF renderer builds a pyth document, and pyth writes one \ilvl level prefix per paragraph inside a list entry — so an entry whose content was a nested list got the outer level's prefix and the inner level's prefix on the same output paragraph. Reading that back, pyth charges both controls to the preceding paragraph and sees a level decrease it never saw a matching increase for, at which point it pops the document off its own stack (IndexError, #209). Carrying a task status took a different path: List subclasses Paragraph in pyth, so the marker run was inserted into the sub-list's content, where only entries belong, and the writer then dispatched on the bare string it found there (KeyError, #210). Nested lists are now flattened before being handed to pyth. That gives up nesting depth, which the RTF parser never reconstructed anyway — a plain two-item list already read back as two paragraphs — and in exchange it fixes a data-loss bug neither issue had noticed: a list three levels deep silently dropped its deepest item, because pyth pushed a list for the level increase and never re-attached it (#209, #210).

• to_ast() and from_ast() now warn about keyword arguments they don't recognise. to_markdown() has always warned; the other two dropped unknown kwargs at a logger.debug call nobody sees. The result looked fine, which is the bad combination: to_ast(content, filename="x.md") is a natural thing to write, filename is not a parameter of anything (the real one is source_format), so the hint was discarded, detection fell through to the plaintext parser, and the caller got back a Document of bare Paragraphs — headings, lists and tables all gone, with no error. The same applied to any misspelled option name. All three entry points now route through one helper, so they cannot drift apart again. Two details worth knowing: the warning fires only for names that are not options of any format, because a real option that this particular format ignores is usually the library's own doing — the CLI packager forces attachment_mode, the MCP server sets it from server config, and convert injects its flavor shorthand — and blaming the caller for those would be wrong. And the existing warning pointed one frame past the caller, landing on <sys>:0, which made it useless to act on and collapsed every such warning into a single dedup entry; it now names the calling line (#273).

• An HTML comment stays a comment when rendering Markdown. comment_mode defaulted to "blockquote", so <!-- note --> — valid Markdown, and invisible when rendered — came out as a visible > note, and an inline comment was glued into the middle of a sentence as [Comment by Reviewer: ...]. That is a content-fidelity bug rather than a formatting preference: the round trip added text a reader can see, and it was not even reversible, since the blockquote reads back as a BlockQuote node rather than a Comment. "html" is the only mode that is a fixed point, and it is what every other renderer already does — each defaults to its own native comment syntax, and textile, which offers the identical three choices, already defaulted to "html". Markdown was the lone outlier. Found by our own quality gate: it is what blocked adding an mcp-name registry marker to README.md, which dropped the README's round-trip fidelity from 97 to 96 and failed the build (#271, #272).

  One visible consequence to know about: converting a DOCX with reviewer comments to Markdown now emits <!-- Comment ... --> rather than a bracketed [Comment by ...] glued to the end of the annotated sentence, so the annotations no longer show up in the rendered prose. They are still there, and in fact carry more than before — the HTML form keeps the author and the timestamp, where the visible form dropped the timestamp. Pass comment_mode="blockquote" (or --markdown-comment-mode blockquote) to get the old, reader-visible output back; that is the case the previous default was chosen for, and it is now opt-in rather than the default for every Markdown document.

1.10.1 - 2026-07-27

A maintenance release. Almost all of it is infrastructure — the harnesses this project already had are now wired to CI as blocking gates — but wiring them up surfaced one real conversion bug, which is the reason to take the release.

Added

• GitHub Action: a conversion-quality gate. action.yml at the repository root ships all2md as a reusable action that scores every matched document and fails the build when fidelity degrades — the same ratchet this project runs on itself, pointed outward. It wraps roundtrip --fail-under and report --fail-under, and gates on the worst score across the matched set.

  - uses: thomas-villani/all2md@v1.10.1
    with:
      paths: docs/**/*.md
      roundtrip-fail-under: 97

  It refuses to pass quietly, which is the whole point: a paths glob that matches nothing, a run with no threshold set, and a document that cannot be converted at all are each a failure rather than a silent green. Config errors exit 2, quality failures exit 1. It also warns when a threshold sits ten or more points below your documents' real scores — documents that convert well score 99-100, so a threshold that sounds strict has enough dead headroom to never fire.

  The action lives in this repository rather than a separate one so @v1.10.1 installs all2md 1.10.1: the gate's verdict is the library's score, so the two versions must not drift. See docs/source/github_action.rst.

Fixed

• Prose containing a | is no longer swallowed into a table. mistune's _process_thead never checks that the second line of a candidate table is a delimiter row — it matches each cell against the three alignment patterns and falls back to "no alignment" for anything else — so any line with the same number of pipe-separated cells as the line above it was accepted as the delimiter and then consumed. Two ordinary sentences that each happened to contain a pipe became a two-column table, and the second line was deleted outright. So did two fully-piped rows written without a delimiter, and two lines whose pipes sat inside code spans (whose backticks were mangled on the way through). A delimiter row is now required, per GFM, which is where the loss stops. Found by the new quality-gate action, which scored a table-free CHANGELOG.md as containing a table — the metric was right and the code was wrong (#177).
• CI: the lint gate could not fail. fix = true in [tool.ruff] applies to plain ruff check, not just ruff check --fix, so CI rewrote every auto-fixable violation inside the runner and exited 0 — green build, repair discarded with the runner, violation still on main. Only rules with no auto-fix could ever turn it red. The setting is gone and CI passes --no-fix; a test pins both (#149).
• Benchmarks: a failed corpus download is no longer cached as an empty result. An empty _index.json reads back as a valid cache hit, so one transient network failure zeroed a source out permanently on any machine that keeps the cache. Found when a DNS blip reduced the corpus to 50 documents and the run still exited 0.
• Benchmarks: the two large corpus archives (Enron, ~423 MB; govdocs1, ~250 MB) now get a bounded retry with backoff. Exhausting it still fails loudly — the retry buys tolerance for flakiness, never silence about failure — and a 404 is not retried at all (#182).
• Benchmarks: a corpus source that yields none of a format its corpus.toml entry requests now says so, instead of quietly covering a narrower format mix than the manifest advertises (#181).
• Tests: pytest -m unit works again. tests/performance/conftest.py overwrote markexpr in pytest_configure, silently discarding any -m — the documented fast path collected 7997 tests instead of 3857. CI never noticed because it passes no -m.
• Tests: TestSplitCLIE2E no longer races under parallel workers. It used a fixed directory under the project root, so concurrent xdist workers collided in it and a crashed run left the directory behind. Thanks @rkfshakti (#185, #187).

Changed

• Faster cold start: import all2md no longer loads every options module. The all2md.options package re-exports lazily (PEP 562) instead of importing all 30-odd submodules at package-init time. Because from all2md.options.base import ... executes that package init first, any options import used to cost the whole set — which is how a bare import all2md ended up loading 31 options modules to reach the two classes all2md/api.py actually needs. Now 4. Total eagerly-imported all2md modules drops from 64 to 37. No API change: from all2md.options import PdfOptions works exactly as before, and dir() still lists every export.

  Measured on CI (median of six runner VMs, nine samples each): import all2md 230 ms → 162 ms (−29%) and all2md --version 246 ms → 178 ms (−28%). --help and a small conversion are roughly unchanged, as expected — both build the full parser or run the pipeline, so they load the options either way. The win lands on the per-invocation path that CLI and agent workflows actually pay.

• CI: the Markdown roundtrip fidelity benchmark (benchmarks/roundtrip) now runs as a blocking gate on every push and PR, and is required before a release. A Markdown -> AST -> Markdown regression that either oracle can see now fails the build instead of waiting to be noticed. Knowingly-accepted failures are declared in EXPECTED_FAILURES (currently one: raw HTML, which the html_passthrough_mode="escape" policy makes lossy by design); the gate also fails if such an entry starts passing or goes stale, so the list can't decay into a permanent excuse.

• CI: cold start is now guarded on every push and PR, and is required before a release. Two complementary gates, because the cost is milliseconds but the cause is an import graph: tests/unit/test_eager_imports.py pins the exact set of modules a bare import all2md pulls in (deterministic, cannot flake), and a Cold Start Gate job compares wall-clock timings against benchmarks/startup_baseline.json with a 20% tolerance. An unrecorded speedup also fails, so the baseline can't drift into describing code that no longer exists. The timing gate judges each scenario twice — raw milliseconds and milliseconds relative to the same run's bare interpreter — and goes red only when both agree, which is what separates a real regression from a runner that landed on a faster or slower CPU class.

• CI: a weekly Corpus Fidelity Gate (also on dispatch) converts the corpus benchmark's reproducible half and compares the failure set — which documents fail, by name — against benchmarks/corpus/corpus_baseline.json. Only the two sources whose sample is fixed across cold runs are gated; arxiv and POI resolve against upstream state that moves, so a baseline would report document-mix churn as a regression. The recorded baseline converts 100/100 gated documents, so its accepted-failure list is empty. A short corpus is also a failure, which is what caught the download bug above.

• CI: a Format Benchmarks job converts the small per-format fixtures across the dozen formats the corpus gate cannot reach (it covers PDF and email only). It gates on conversion, not on time: these fixtures run in 5–885 ms, and gating a 5 ms measurement needs a variance study on these runners first. Timings are recorded to an artifact, which is what such a study would be built from.

• tests/performance is kept but disarmed. Its seven absolute ceilings had 106×–1170× headroom on CI and its other seventeen assertions were mean_time > 0, true by construction — so nothing there could ever have failed, and nothing had ever run it, because a bare pytest tests/ deselects benchmark. What is checked now is that every format still converts to something non-empty: deterministic, and unable to flake.

• Docs: performance.rst no longer reports throughput figures that came from no harness. The old table gave pages/sec, a unit the benchmarks have never produced, beside one row in MB/sec — measured and invented numbers side by side, which launders the invented ones. They are replaced by figures that name their source, and by the advice to measure your own corpus. It also documented PubMed Central as a live corpus source with runnable commands, though [sources.pmc] is commented out, and described the Apache POI sample as stable when ref = trunk is a moving branch. Adds the startup-cost section the page never had (#180).

• CI: bumped actions/setup-python 6 → 7 and actions/download-artifact 7 → 8 (#188, #189). Created the dependencies, github-actions and python labels that dependabot.yml had always referenced but that never existed on the repository, so Dependabot could not apply them.

1.10.0 - 2026-07-24

Added

• insert_mode Markdown renderer option (--markdown-insert-mode): how to render insertions — markdown (^^text^^, the default, which round-trips), html (<ins>), or ignore (#113).

Fixed

• OCR'd PDF pages with links no longer crash conversion. OCR-synthesized text spans omitted the bbox key that every real PyMuPDF span carries. On a page that was OCR'd and held a link annotation (common in signed documents), link resolution read span["bbox"] and aborted the whole file with KeyError('bbox'). OCR spans now carry the page-rect bbox, so link resolution runs normally (and a page-sized span never spuriously matches a small link).

• Markdown tables with ragged rows no longer vanish. mistune rejects any body row whose cell count differs from the header's, and then discards the entire table and re-emits it as a literal paragraph — so one row with a missing or extra | silently cost the whole table on the way to DOCX, PDF, ODT and every other format. The table rule is now GFM-compliant: short rows are padded with empty cells and cells past the header width are dropped, matching how GitHub renders the same source. Applies to tables nested in list items and blockquotes too. A header/delimiter width mismatch still correctly means "not a table."

• Binary renderers: soft line breaks render as spaces, not hard breaks. The DOCX, PDF, PPTX, ODT, ODP, RTF and CSV renderers rendered soft breaks (plain newlines inside a Markdown paragraph) as hard line breaks, so any hard-wrapped Markdown source produced output with a visible line break at every wrap point — 214 stray breaks in one real-world memo. Soft breaks now render as a space, matching the text renderers (HTML, LaTeX, RST, …) and CommonMark semantics; hard breaks are unchanged.

• INI renderer: DEFAULT section no longer fails to render. configparser reserves DEFAULT, so add_section("DEFAULT") raises ValueError — which meant any document whose keys landed in the default section (orphan key/value lists with no heading, or a heading literally named DEFAULT) aborted the whole conversion with a RenderingError. Keys are now set on the parser directly for that section. Thanks @santhreal (#116).

• Org renderer: table header separators now end with a newline. The header rule and the first body row were emitted glued together (|---|---|| 1 | 2 |), which is not a valid Org table and loses every body cell when re-parsed. Thanks @santhreal (#117).

• CSV renderer: line breaks inside table cells survive export. LineBreak nodes and <br> HTML inlines were dropped when flattening a cell to text, silently joining the two lines. They now emit a newline, which the CSV writer quotes. Thanks @santhreal (#118).

• DokuWiki renderer: empty list items keep their bullet. An empty ListItem produced no output at all, so - a / - / - c rendered as two bullets instead of three and shifted the list. Matches the Markdown/MediaWiki/AsciiDoc/RST renderers, which already preserve the blank bullet. Thanks @santhreal (#119).

• RTF renderer: lists nested in block quotes and definitions no longer crash. pyth's List subclasses Paragraph, so prefixing a paragraph rewrapped a nested list as Paragraph(content=[ListEntry…]) and raised TypeError — any Markdown with a list inside a block quote failed to convert. Thanks @santhreal (#122).

• AsciiDoc parser: escaped braces in an attribute-ref shape no longer hang. Input as small as {\{} spun forever: escape preprocessing left a {…} shape, the combined inline pattern matched at the cursor, every handler declined, and the fallback advanced by zero. An unclaimed match is now consumed as literal text. Thanks @santhreal (#123).

• AsciiDoc parser: cross-reference targets restore their escapes. <<id>> and <<id,text>> skipped escape postprocessing that the sibling link: branch already did, leaking internal placeholders into the URL and link text. Thanks @santhreal (#124).

• AsciiDoc parser: column spans survive a space after the pipe. The span regex ran against the stripped cell but sliced the unstripped one, so | 2+| spans two left a stray + cell and split the row wrong. Thanks @santhreal (#130).

• AsciiDoc renderer: merged table cells keep their spans. colspan/rowspan were dropped entirely, so an HTML table with merged cells lost its structure on conversion even though the AsciiDoc parser already understood 2+|, .3+|, and 2.3+|. A leading cell's spec replaces the row-opening |, since a spec binds to the cell after the pipe it precedes. Thanks @santhreal (#131).

• MediaWiki parser: literal pipes in table cells survive. Cell-attribute stripping matched any cell whose text merely contained a |, so pipe: a|b became b and [[Target|label]] lost its target. A leading segment is now treated as attributes only when it contains =. Thanks @santhreal (#125).

• MediaWiki parser: table captions are read from |+ lines. |+ was skipped outright, so Table.caption was always None and real wikitable captions were lost on parse and round-trip even though the renderer already emitted them. Thanks @santhreal (#132).

• MediaWiki renderer: table captions stay on one line. A caption containing a newline (common from HTML <caption>) split across lines, which is invalid wikitable markup and dropped the remainder of the caption. Thanks @santhreal (#129).

• DokuWiki renderer: table column spans are emitted. TableCell.colspan was ignored, so merged columns were lost; DokuWiki expresses a span with empty continuation cells (| Wide || Tall |). Thanks @santhreal (#127).

• Textile renderer: block children of a list item are separated properly. Children were joined with a single space, so a table after a list item's lead paragraph became * intro |_.H| — ordinary text to Textile, and the table vanished on round-trip. Thanks @santhreal (#126).

• AST serialization: Comment, CommentInline, and Mark nodes are supported. The dispatch tables had no entries for them, so serializing any document containing a Markdown HTML comment, ==mark==, an AsciiDoc // comment, or an RST .. comment raised ValueError: Unknown node type for serialization. Thanks @santhreal (#128).

• HTML parser: <ins> is no longer silently dropped. It was listed as an inline element but had no handler, so it fell through to the generic unwrapping and lost its markup entirely — the counterpart <del> has always mapped to Strikethrough. It now parses to an Underline node with semantic="insert". This also recovers Textile's +inserted+, which python-textile renders as <ins> and which therefore used to arrive as unmarked plain text (#113).

• Markdown parser: <u> and <ins> survive a Markdown round trip. They came back as raw HTMLInline, which the default html_passthrough_mode="escape" then turned into <u> on the next render. Both tags are now read back into Underline nodes, so they round-trip losslessly. Tags carrying attributes, and unmatched or stray tags, are still passed through untouched rather than guessed at. The remaining inline tags (<del>, <sup>, <sub>, <mark>) still self-escape (#113).

• JSON and YAML renderers: duplicate table column names keep their values. Header text was mapped straight to dict keys, so a table with two tag columns kept only the last cell of each row. Repeats are now suffixed (tag, tag_2) in both renderers. Thanks @santhreal (#137).

• LaTeX parser: section titles are read from the right argument. pylatexenc gives sectioning macros the argspec *[{, and the parser took slot 0 — the * marker — as the title. \section*{Introduction} produced # *, and because slot 0 is empty for unstarred sections too, every heading fell through to a regex fallback that lost inline markup (\section{Hello \textbf{World}} gave # Hello \textbf\{World). Thanks @santhreal (#138).

• LaTeX parser: \hline no longer leaks into table cell text. Rule macros inside tabular were converted as ordinary content, so a cell rendered as \hline c. Thanks @santhreal (#143).

• LaTeX parser: itemize/enumerate items keep their text. Only braced arguments on \item were read, but a LaTeX item body is the sibling nodes up to the next \item — so every bullet was empty and lists vanished from the output entirely. Item bodies are now collected from siblings, nested lists stay nested instead of being flattened away, and the source newline/indent after \item is trimmed without disturbing spacing around inline markup. Thanks @santhreal (#142).

• LaTeX parser: \chapter maps to a heading. Only section through subparagraph were routed to the sectioning handler, so with the default parse_custom_commands=False every chapter of a book- or report-class document was dropped. Thanks @santhreal (#144).

• LaTeX parser: \paragraph{...} titles are parsed as titles. Default pylatexenc has no macro spec for \paragraph, so the title stayed a sibling group and emitted an empty level-4 heading followed by loose text (#### \n\nTitle) — which broke round-trips, since the LaTeX renderer emits \paragraph{...} for level-4 headings. Inline markup in those titles now survives too. Thanks @santhreal (#145).

• HTML parser: <tr> elements with no cells are skipped. An empty row became a zero-column header, rendering as invalid GFM (| | over ||) and corrupting any table that followed it. A row with an empty <td> is still preserved. Thanks @santhreal (#146).

• Org parser: marker-only headlines are no longer dropped. orgparse reports * TODO as an empty heading with todo='TODO', and the parser discarded any headline whose text was empty — losing the headline and reparenting its body. The marker stays in metadata rather than the title, so * TODO round-trips as itself instead of * TODO TODO. A priority-only headline (* [#A]) is kept the same way. Thanks @santhreal (#147).

• Org parser: headlines with more than six stars no longer crash. Org allows any star depth, but Heading accepts levels 1-6, so ******* Deep raised on parse. The level is now clamped at 6, matching the HTML and CHM parsers. Thanks @santhreal.

• CSV renderer: inline code, inline math and image alt text survive export. Code, MathInline and Image leaves were skipped when flattening a cell to text, so a cell holding `code` or an image came out as an empty field. Thanks @santhreal.

• MediaWiki parser: empty list items keep their bullet. A bare * or # line produced no item at all, so the list came back short and every following item shifted up a position. Matches the DokuWiki renderer fix above. Thanks @santhreal.

• MediaWiki parser: attributes are stripped from |+ table captions. Cell attributes were already handled for | cells, but |+ style="..." | Caption kept the whole attribute segment in the caption text. Thanks @santhreal.

• BBCode parser: empty [*] items keep their bullet. Same shape as the MediaWiki fix — an empty item was skipped, shortening the list and shifting the items after it. The segment before the first [*] is still correctly not an item. Thanks @santhreal.

• BBCode parser: [color], [size] and [font] no longer flatten their contents. These tags have no Markdown equivalent, so they are stripped — but the strip discarded any markup nested inside them, and [color=red]see [b]this[/b][/color] lost its bold. The inner content is now parsed and spliced in place of the tag. Thanks @santhreal.

• FB2 parser: notes-body section titles are kept. The first heading of a notes body was dropped unconditionally to avoid duplicating the "Notes" heading, so a notes body with no body-level <title> — or with an empty <title> placeholder — silently lost its first footnote heading. The leading heading is now dropped only when a body-level <title> actually produced one. Thanks @santhreal.

• FB2 parser: <cite> is converted as a block quote. It was flattened into a plain paragraph while its structural twin <epigraph> became a quote, so a citation's block children collapsed into inline text. Both now take the epigraph path. Thanks @santhreal.

Changed

• ^^text^^ and underline are now distinct in the AST. ^^ is pymdownx's insert extension, not underline, but both parsed to the same Underline node, so an insertion could not render as <ins> and a genuine underline emitted insert syntax. Underline gained a semantic: Literal["underline", "insert"] discriminator (default "underline", omitted from serialized output when default, so existing JSON is unaffected). The HTML renderer emits <ins> for insert and <u> for underline (#113).
• underline_mode now defaults to "html" (<u>) instead of "markdown" (^^text^^). Markdown has no underline syntax of its own, and ^^ means insert; <u> now round-trips losslessly thanks to the parser fix above. Set underline_mode="markdown" to keep the pre-1.10 spelling (#113).
• CI: bumped actions/setup-python 6 → 7 and actions/setup-node 6 → 7 (#120, #121).

Security

• Relocked torch 2.12.1 → 2.13.0 (GHSA-rrmf-rvhw-rf47 / CVE-2025-3000, memory corruption in torch.jit.script), pulling torchvision 0.27.1 → 0.28.0. torch reaches us only through the ocr-easyocr and search extras and all2md never calls torch.jit.script, so practical exposure was nil — but the lock pinned a vulnerable range for anyone installing those extras.
• Relocked setuptools 81.0.0 → 83.0.0 (GHSA-h35f-9h28-mq5c / CVE-2026-59890, MANIFEST.in exclusions bypassable via a Unicode normalization collision on macOS APFS/HFS+). Transitive via torch only; all2md builds with hatchling.

1.9.0 - 2026-07-15

Added

• Conversion optimizer (all2md optimize). Converts a document many times under different converter settings and reports the ones that recover the most well-formed structure — emitted both as a runnable command and as a .all2md.toml snippet. Built for the documents that need it most (the gnarly PDF with no known-good output to diff against), so the objective is reference-free — and it is deliberately neither of the two existing scores. The confidence score is a saturating breakage detector: on anything not visibly broken it pins to 100 regardless of settings, so it has no gradient to search (measured: 16 option combinations on a two-column PDF produced one distinct confidence score while the parsed AST produced four distinct outcomes). The round-trip score measures the renderer, not the parser — a garbled table round-trips through Markdown perfectly. So all2md.optimize scores the parsed AST directly.

  Body text gates the score rather than contributing to it. Losing a paragraph is data loss; leaving a running header in is an annoyance, and the two are not interchangeable at any exchange rate — so a candidate's body-text retention multiplies its fitness (cubed), and no amount of tidiness buys back deleted content. The weighted dimensions are the ones that are genuinely tradeable: tables (scored as quality-weighted recall — filled cells discounted by shape regularity, so a hallucinated table earns almost nothing while a missed real one still costs its cells), structure, and cleanliness (how much repeated furniture the setting left behind, where furniture is content that repeats across a substantial fraction of the document's pages).

  The search is cheap by construction — the named presets first, then coordinate descent, which costs sum(len(values)) conversions instead of a full grid's prod(len(values)). It is still tens of full conversions, though, and a PDF page costs about a second to parse, so --sample-pages tunes against a slice of a long document and --cache makes repeat runs nearly free (18.5s → 0.3s warm, on a 31-candidate run); the command warns up front when it is about to tune a whole document. Available from Python as all2md.optimize_options(source, ...) (with optimizable_formats()), and from the command line as all2md optimize <file> (--rounds, --sample-pages, --no-presets, --top, --out, --json). Tunable formats: pdf, html, docx. The reported fitness ranks candidates against each other and is not an absolute quality score — all2md report and all2md roundtrip remain the scores for that.

• Round-trip fidelity scoring (all2md roundtrip). Converts a document to another format, parses it straight back, and scores the structure that survived. Unlike the confidence report this comparison has a ground truth — the source document itself — so a lossless round trip scores exactly 100 and anything less is a concrete, itemized loss. Five dimensions are scored against independent alignments and combined: structure (0.40 — heading levels, list nesting, table placement), text (0.30 — the document-wide word stream), inline (0.15), tables (0.10) and references (0.05); dimensions the source does not exercise are dropped and the rest renormalized, so a document with no tables is neither rewarded nor punished for the tables it lacks. Alongside the score the report lists concrete StructuralDelta incidents ("heading(h1) -> paragraph", "table 1: 4x3 -> 4x2", "3 of 229 words") so a low score is actionable. Tight/loose list items and nested-paragraph artifacts are normalized away, so format-legal spelling differences do not read as loss. Available from Python as all2md.roundtrip_report(source, via=...) (with roundtrippable_formats() listing the 24 valid via formats) and from the command line as all2md roundtrip <file> (aligned card by default, --via to pick the intermediate format, --json, --fail-under SCORE as a CI gate, --max-deltas N, --format to override source detection for stdin). The score responds to converter options, which is what makes it usable as the fitness function for the planned all2md optimize.

• Conversion confidence report ("quality card"). Every conversion now attaches a reference-free ConfidenceReport to Document.metadata['confidence'] — a 0-100 score, a high/medium/low band, the signals behind it, and the discrete degraded-content incidents the converter recorded — surfacing sanity signals converters previously computed and threw away. PDF reports meaningful-text density (chars_per_page), OCR reliance (ocr_page_fraction), detected/rejected table counts (each rejected non-tabular region is recorded with its reason), and running-heading demotions; DOCX reports table/image counts and flags silently-dropped embedded objects, charts, and SmartArt. The single score doubles as an optimizer fitness function (no ground-truth needed). Read it programmatically with all2md.confidence_report(source) or from the command line with the new all2md report <file> verb (aligned pretty card by default, --json for machine use, --fail-under SCORE as a CI gate). Any parser can contribute incidents via BaseParser._record_degraded; container formats (zip, archives) already flag members that could not be parsed.

• Opt-in conversion cache (--cache). grep, search, chunk, view, report, roundtrip and optimize all take a --cache flag (and --cache-dir DIR) that stashes parsed documents on disk so repeated runs over unchanged files skip the expensive parse step. The cache is keyed by a fingerprint over the source file (path + size + mtime), the resolved format and parser options, and the all2md version + AST schema — so a changed file, changed options, or a version bump all miss cleanly rather than serving a stale AST. Off by default; also enable globally with ALL2MD_CACHE=1, and point it anywhere with ALL2MD_CACHE_DIR (defaults to the per-OS user cache directory via platformdirs). Exposed programmatically as all2md.conversion_cache.use_conversion_cache(...), which transparently caches every to_ast() call made inside the context.

• DOCX run-level character styles round-trip. Named character styles on runs ("Intense Emphasis", "Quote Char", a custom style, …) are now captured on the AST inline node's metadata['source_style'] and re-applied when rendering back to DOCX with a template — the run-level analog of the existing paragraph source_style handling. This preserves run styling across a DOCX → AST → DOCX round-trip (and combines with direct bold/italic). Character styles have no Markdown representation, so the name rides only the AST and is dropped on Markdown serialization; without a template that defines the style, application falls through silently, so default output is unchanged.

Fixed

• Markdown: footnotes round-trip as Markdown under the default flavor. A footnote reference and its definition rendered to raw HTML on the default flavor, which the default html_passthrough policy then escaped on the next pass — so a footnote did not survive a markdown → markdown round trip. Footnotes now render in Markdown syntax by default and round-trip intact.

• Markdown: inline marks, superscript and subscript are flavor-aware and round-trip by default. Highlight (==text==), superscript (^text^) and subscript (~text~) defaulted to HTML tags that the default passthrough policy escaped on reparse. They now default to the roundtrip-safe Markdown spelling (flavors that support the syntax natively emit it directly); set the corresponding *_mode option to html for wider display support.

• Markdown: underline (^^text^^) and non-GFM strikethrough round-trip instead of self-escaping. Underline rendered <u>…</u> by default and the <del> fallback for flavors without ~~ did the same, both of which the default passthrough policy escaped to <u>… on the next pass. Underline now defaults to the pymdownx ^^text^^ insert spelling (the old "markdown" mode emitted __…__, which every flavor parses as bold, silently losing the underline); strikethrough on a flavor without ~~ now emits ~~ by default. Explicit html still opts into the tags.

• Markdown: inline $$…$$ display math is kept, not dropped. An inline $$…$$ span was silently discarded instead of being preserved as display math.

• Markdown: a list survives an admonition that degrades to a labelled quote. When an admonition inside a list item degraded to a labelled block quote, the surrounding list was broken apart; it now stays intact.

• PDF: prose from every rejected table is preserved, not just degenerate grids. Text inside a detected table's bbox is stripped from the ordinary text stream before the table is validated, so a rejection path that returned None deleted that text. A prior fix covered only degenerate (1×N / N×1) grids; the oversized-grid, mostly-empty, uniform-cell and dot-leader-TOC rejections still dropped a sparse-but-real table (a financial statement, a form) or a table of contents. All four now demote the region to a paragraph.

• Round-trip scoring counts code, math and raw-HTML block content. CodeBlock, MathBlock, HTMLBlock and their inline siblings keep their payload in a plain string with no Text children, so the text dimension never compared it — a round trip that dropped or mangled an entire code block scored a false 100. Their content (and image alt text) is now part of the comparison.

• Confidence: conversions with no quality instrumentation report not_assessed, not a false high. Formats that emit no scored signals and no degraded events (docx, pptx, html) scored a vacuous 100/HIGH, so a mangled .docx read as verified clean. Such a report is now banded not_assessed; the numeric score is unchanged.

• DOCX: title-promotion inversion clamps heading levels at 6. Demoting the headings after a leading title used an unbounded level += 1, pushing an H6 to an out-of-spec level 7 that serialization and the round-trip scorer saw. It is now clamped, mirroring the forward transform's bottom clamp.

• all2md optimize searches only valid figures_parsing / details_parsing values. The HTML search space listed values (figure, image, details, content) that no parser accepts; they were silently no-ops and could be written into a recommended .all2md.toml. Replaced with valid choices, guarded by a test.

• Markdown: multi-paragraph and multi-line list items round-trip without collapsing. Three problems in the same surface conspired to flatten lists on a Markdown round trip:

  Loose lists were read as tight. mistune 3.x carries the loose/tight flag on the list token itself, not in attrs; reading it only from attrs marked every list tight, so the renderer dropped the blank lines that separate a loose item's paragraphs and they merged into one on reparse.

  Continuation lines were emitted at column zero. A list item whose content wrapped across several lines (soft-wrapped source, or a multi-paragraph item) indented only its first line; every continuation went to the margin, where it reparses as a lazy continuation that collapses the wrapped lines together — or, when a nested block landed there, breaks the item apart. Code blocks and block quotes inside a list item had the same flaw and could escape to column zero as siblings of the list. Continuation lines, and nested code/quote blocks, now carry the item's indentation.

  A nested list as an item's first child double-indented. The first-child render path cleared the indent stacks but left the in-list flag set, so a nested list added its own indent level on top of the marker — rendering 1. - x as 1. - x. It now renders flush and is shifted to the marker's content column like any other continuation.

  A task checkbox ([ ] ) is treated as first-line content rather than marker width, so continuations align to the list marker and don't over-indent into an accidental code block. Net effect: a document like a nested ordered/task list with wrapped prose now survives markdown → markdown unchanged (idempotent), where before it flattened onto single lines.

• DOCX: inline code and block quotes survive the Markdown round trip. Two independent renderer/parser asymmetries on the md → docx → md path, both filed as #71:

  Inline code was dropped. The renderer emitted a `code` run with a monospace font but no named character style, so the parser — which recovers inline styling from run styles, not fonts — had nothing to key on, and `inline code` came back as plain inline code. The renderer now tags inline-code runs with a Verbatim Char character style (matching pandoc's name; created on demand and only when use_styles is on), and the parser maps that style back to a Code node. Recognized style names are configurable via the new code_char_style_names DOCX parser option.

  A block quote came back as a bullet list. The renderer wrote the quoted paragraph as a Normal paragraph with a left indent, and the parser read that indent as list nesting — so > a quoted line silently became * a quoted line, which looks intentional and is arguably worse than dropping it. The renderer now applies Word's built-in Quote paragraph style (which also makes the generated document look right in Word) and, for a single level, no longer sets a bare indent that the parser would misread; the parser maps Quote / Intense Quote back to a BlockQuote, coalescing adjacent quote paragraphs into one quote. Recognized style names are configurable via the new quote_style_names DOCX parser option.

  Together these take all2md roundtrip … --via docx on a document with inline code and a quote from structure: 33 / inline: 0 to 100 / 100. Both recoveries require styles (use_styles=True, the default); with styles disabled the render still falls back to font-and-indent as before.

• DOCX: a document title survives the Markdown round trip. Rendering to DOCX applies TitlePromotionTransform — a leading # H1 becomes Word's Title style and every following heading is promoted one level (H2 → "Heading 1") so the document reads correctly in Word. The parser had no inverse: it mapped the Title style to a plain paragraph and left the promoted headings where they were, so # Title / ## Section came back as body text plus # Section — the title silently demoted to prose and every heading shifted up a level (all2md roundtrip … --via docx scored structure: 67). The parser now maps Word's Title back to Heading(level=1, is_title=True) and, when that title leads the document, demotes the following headings one level to undo the promotion — making the transform exactly invertible (structure: 100) while keeping the nice-looking Word output. Word's Title is semantically the document title, so this also gives natively-authored Word documents a sensible outline (Title → #, its Heading 1 → ##).

• HTML: loose list items no longer grow a paragraph-inside-a-paragraph. A loose item — one whose <li> already holds a block, <li><p>x</p></li> — was parsed to ListItem > Paragraph > Paragraph > Text, because _process_list_item_to_ast wrapped every item's content in a freshly synthesized Paragraph whether or not that content was already a block. No format represents a paragraph nested directly in a paragraph, so the inner node was pure artifact: a consumer walking the AST saw a different ListItem shape depending on how the source HTML happened to be written, and — because our own HTML renderer emits <li><p>…</p></li> — an html → html round trip accreted one extra Paragraph per item on every pass. The parser now adopts a <li>'s block children directly and only synthesizes a wrapping Paragraph for loose inline runs, so <li><p>x</p></li> and <li>x</li> produce the identical AST and the round trip is stable.

• String page ranges select the pages you asked for. validate_page_range() converted 1-based page numbers to 0-based twice on the string path: parse_page_ranges() already returns 0-based indices, and the result was then decremented again. So every string range was wrong — one including page 1 (pages="1-3") raised Invalid page number: 0, and every other one silently returned the wrong pages, shifted by one: PdfOptions(pages="2") gave you page 1. The list form (pages=[2]) was correct, and the tests only ever exercised lists, so nothing caught it. The string path now returns the already-0-based parse directly.

  Two adjacent failures went with it. The CLI could not express the ranges the option documents: --pdf-pages 1-3 was rejected with "Expected comma-separated integers", because the builder resolved pages: list[int] | str | None to just list[int] — it takes the first non-None member of a union — and ignored the field's own "type": str metadata, which only int and float were honored. An explicit metadata type now overrides inference, so the page spec reaches the converter verbatim. (--pdf-pages is the only option in the library whose declared metadata type differs from its annotation, so nothing else changes. --save-config now records the spec as written — "pdf.pages": "1-3" — and configs carrying the older list form still load.) And a range that selected nothing converted the whole document: pages="99" on a 10-page PDF parsed to an empty selection, which pdf.py read as "no selection, use every page". It now raises.

• HTML: tables no longer vanish inside <figure> or inline layout wrappers. On a real arXiv paper (LaTeXML output), 13 tables / 150 rows / 794 cells parsed to 3 empty tables and not a single row. The captions survived, so the output still looked plausible. Two independent defects, either of which was enough to lose every table in the document:

  <figure> was special-cased to images — _process_figure_to_ast looked for a <figcaption> and an <img> and built its result from those two alone, so any other child (a <table>, a <pre>, a <video>) was never visited. With a caption present it returned a BlockQuote holding only the caption; with neither image nor caption it returned None and dropped the figure whole. A figure is a container that carries a caption — HTML5 recommends it for captioning tables and code listings too — so its content now goes through the normal block dispatch, whatever that content happens to be.

  Separately, a block element inside an inline element was discarded outright. An inline context has nowhere to put a block, so _process_children_to_inline skipped it behind a logger.debug whose message said this "should not happen with proper block/inline separation". It happens constantly: LaTeXML scales an oversized table by wrapping it in <span class="ltx_transformed_inner" style="transform:scale(0.7)">, and a <table> inside a <span> was lost — figure or no figure. An inline element wrapping block content is a layout wrapper, not inline content, and is now processed as a block container. Where a block genuinely cannot be kept (<a><div>…</div></a>, since a link cannot hold a block), the drop is now recorded as a degraded-content incident instead of being silent, so all2md report can see it.

  The same paper now parses to 13 tables, 150 rows, 794 cells — every row and cell in the source. Four of the six documented figures_parsing modes were also unimplemented and fell through to the blockquote branch: skip did not skip, paragraph returned a blockquote, and caption_only kept the image. All six now behave as documented, and image_with_caption no longer drops a caption it cannot fold into the image's alt text.

• all2md optimize no longer recommends breaking words in half. The objective's text signal was a count of whitespace-separated tokens, which rewards a parse that fragments words — chop one word into two and the document appears to contain more text. When merge_hyphenated_words began working on native-text PDFs (it was previously a silent no-op), the optimizer immediately found this: repairing hyphen- + ation into hyphenation joins two tokens into one and so read as losing a word, and it recommended disabling the repair on 17 of 17 real papers. Judged against the publisher's HTML rendering, that advice made every one of them measurably worse (mean −0.03, worst −0.066). Word counts now go through content_tokens(), which rejoins hyphen-broken tokens and drops hyphens, so a fragmented parse and a clean one produce an identical count and the metric cannot have a preference. This is the third time this exploit surfaced (previously via consolidate_inline_formatting, where "hello" was counted as "hel" + "lo"); fixing the metric rather than the setting closes the whole class.

  Relatedly, KNOBS is now guarded by FORBIDDEN_KNOBS, because two categories of setting look tunable and are not. Correctness settings (merge_hyphenated_words, consolidate_inline_formatting) have one right value — there is no document for which the broken word is the better answer, so there is nothing to search. Content-inclusion preferences (include_comments) change what the user asked for rather than how well it was extracted; the objective rewards recovering more content and comments are words, so it would have recommended include_comments=True on essentially every DOCX — advice that leaks reviewer comments the author never meant to publish.

• auto_trim_headers_footers now removes running headers and footers. It largely did not. Three defects compounded, and each was hidden by the optional pdf_layout extra, which labels headers and footers directly — so on a development machine with the extra installed the feature looked fine, while a stock install got almost nothing. (1) Candidates were keyed on their exact text, so Page 1 of 12 and Page 2 of 12 looked like two unrelated blocks, neither ever repeated, and a footer carrying a page number — very nearly every running footer there is — could never be detected at all. Digit runs are now collapsed when keying, so a running footer is recognized as one. (2) Detection refused to run on documents with fewer than three pages, making the option a silent no-op on every two-page document; two pages are enough to show repetition. (3) The zone filter dropped any block that began inside the header zone, rather than one that lies entirely within it — so a body paragraph starting a few points below the running head was deleted in full, taking the rest of the page with it. On a real FCC filing whose body opened 4pt under the header, the opening paragraph of every page was destroyed. Furniture is always fully contained in the zone (the zone is derived from furniture's own far edge); body text merely pokes into it.

  Because collapsing digits makes Section 1 and Section 2 key alike, a candidate must now also hold still: real furniture is anchored to the page, whereas a heading that merely recurs is anchored to the text flow and lands somewhere different on each page. Verified against arXiv's HTML rendering of 29 papers as an external ground truth — recall did not fall on a single one — and the feature now has tests, which it did not before.

• PDF table detection no longer invents tables out of prose — or deletes the prose when it declines to. find_tables() fires on plenty of things that are not tables, and a grid with only one dimension is never one: a single column is prose wrapped in pipes, and a single row is a line of text chopped at its word boundaries (on one arXiv paper the sentence "What is the capital of this country?" was rendered as an eight-column table). Those detections are now rejected — but rejecting them was not simply a matter of dropping them. Text inside a detected table's bbox is removed from the ordinary text stream before the table is validated, so a rejection path that returned nothing did not demote the region to prose, it deleted it: doing that silently cost 256 words of real body text across the corpus. Rejected regions now come back as paragraphs. The same applied to regions the layout model predicted as tables and which turned out not to be — a common misfire on academic PDFs, where suppressing the fake tables also removed 530 words of body text with them. Measured across the PDF corpus: junk tables 21 → 2, real tables 37 → 37 (none lost), and body text strictly improves — the junk grids had been shredding words into per-cell fragments (Gender → G + ender), so ~500 real words come back.

• merge_hyphenated_words now actually works on text PDFs. The option is on by default, but for any PDF that did not go through OCR it silently did nothing: the parser delegated the merge to PyMuPDF's TEXT_DEHYPHENATE extraction flag, and that flag is inert — on PyMuPDF 1.28 / MuPDF 1.29 it does not change get_text() output in any extraction mode. A word split at a line break came back as "hyphen- ation" instead of "hyphenation" in every ordinary text PDF. The merge is now performed directly on the extracted text blocks (dehyphenate_blocks()), moving the continuation word up into the preceding line so the joined word survives the line-to-paragraph join that callers perform. The existing capitalization rule is unchanged and now applies to native text too: an uppercase continuation keeps the hyphen ("Anglo-\nSaxon" → "Anglo-Saxon"), a lowercase one drops it ("be-\nwusst" → "bewusst"), and hyphens not between two letters ("10-\n20") are left alone. This is the other half of the fix for #51, which addressed only the OCR path — on the explicit assumption that the flag already covered native extraction.

• The options reference regenerates reproducibly. Two MarkdownRendererOptions fields default to an UNSET sentinel, and the generator rendered it with repr() — emitting <object object at 0x...>, a memory address that changed on every run. docs/source/options.rst therefore showed a spurious diff after any documentation build, burying real option changes in noise. Those fields now render as unset, matching the wording all2md --help already used. Running scripts/generate_options_doc.py standalone also works now: its --output and --narrative defaults resolved against scripts/ rather than the docs/source/ tree they named, so a hand-run always failed on a missing narrative file.

• DOCX no longer opens with a stray blank line. A Word document whose first paragraph is empty (a common template artifact) produced a leading blank line in the Markdown output — including when that empty paragraph carried a list style, which slipped past the empty-paragraph filter as a blank bullet. Empty paragraphs are now dropped uniformly across regular, list-item, and post-list paths, and the Markdown renderer strips any leading blank line as a final safeguard (so no converter can emit one).

• Capitalization-aware dehyphenation. When merging words split across a line break by a hyphen (OCR text, merge_hyphenated_words), an uppercase continuation letter now keeps the hyphen — "Anglo-\nSaxon" becomes "Anglo-Saxon" rather than "AngloSaxon" — so legitimately hyphenated compounds and names survive instead of being fused.

• Persistent search index no longer serves stale results. A keyword index saved with --search-index-dir (MCP search_documents) was reused whenever the directory existed, with no record of the corpus it was built from — so pointing it at a changed corpus, or a different paths set, could silently return stale hits. The index now records a fingerprint of the documents and index-relevant options at save time and is rebuilt when they no longer match.

• HTML: an ordered list keeps its start attribute instead of renumbering from 1. The HTML parser built the List node without ever reading <ol start="N">, so <ol start="3"> came back numbered from 1 on every HTML conversion — the Markdown renderer already honored List.start, the parser just never populated it. Ordered lists now carry their start value through (a non-numeric start falls back to 1), so a list that does not begin at 1 survives conversion and the markdown → html → markdown round trip.

• Markdown: multi-paragraph definition lists round-trip without collapsing. Two defects in the definition-list renderer flattened a list on a Markdown round trip: a description's paragraphs were joined with a single newline (so a second paragraph merged into the first as a lazy continuation on reparse), and consecutive term/description groups were separated by a single newline (so the next term merged into the previous description). Description blocks and term groups are now separated by a blank line and continuation lines indented four spaces, so multi-paragraph descriptions and multiple terms survive intact.

• Markdown: footnote definitions survive parsing under mistune 3.x. The parser recognized only the legacy footnote_def token and read identifiers from attrs['label']; mistune 3.x instead groups definitions in a footnotes container of footnote_item tokens (the label living in attrs['key'] / the reference's raw field, with attrs holding only a numeric index), so every footnote definition was dropped from the AST and every reference lost its identifier. The parser now handles the footnotes container and reads key/raw with a label fallback; a multi-paragraph footnote is additionally rendered with a blank line between its blocks and four-space continuation indent, so it no longer collapses into one paragraph on reparse.

• Markdown: a table nested in a list item stays in the list. A table inside a list item was emitted at column zero instead of under the item's content margin, so on a Markdown round trip it re-parsed as a top-level sibling and broke the list apart. Idempotency did not catch it — the broken output was stable — but the round-trip benchmark's HTML-equivalence oracle did. Table rendering now shifts every line to the current indent, mirroring code-block handling, so the table stays inside its item; at the top level the indent is empty and output is unchanged.

• A long single line is no longer mistaken for a file path and leaked as OSError. During source resolution LocalPathRetriever.can_handle called Path(value).exists() on raw input before the parse-error wrapper, so a one-line string whose path component exceeds the OS name limit raised OSError(ENAMETOOLONG) and escaped convert() instead of surfacing as an All2MdError. The stat calls are now guarded (mirroring the existing guard in the parser base), so oversized inline content is handled as content.

Performance

• Faster CLI cold start. Building the dynamic CLI parser imported every format's options module and introspected each field, adding ~1.7s to startup even when the invocation never needed it. --version/-V and --about/-A are now short-circuited before the parser is built (dropping --version from ~2.0s to ~0.8s, the bare import cost), AttachmentOptionsMixin is imported from all2md.options.common rather than the eager all2md.options package, and get_type_hints() is memoized per options class — cutting create_parser() from ~1.7s to ~0.24s after a warm import, which also roughly halves the small-file conversion path. A cold-start benchmark (benchmarks/startup.py) guards these wins against regression.
• Cheaper repeated and batch conversions. Four conversion-hot-path wins help many-small-file and repeated-conversion workloads: the flattened, priority-sorted converter list is memoized — and invalidated on register/unregister — instead of being rebuilt on every detect_format; check_version_requirement is cached per (package, spec) so the dependency-guard decorator stops re-reading installed versions and re-parsing specifiers on every parse/render; resolved option type hints are shared across option construction; and DOCX conversion with attachment_mode="skip" short-circuits before reading the image blob (output is byte-identical).

1.8.2 - 2026-07-09

Fixed

• Bullet lists no longer disappear when rendering DOCX with a custom template. When a template lacked the List Bullet / List Number styles, the generated numbering part interleaved w:abstractNum and w:num elements. CT_Numbering requires every w:abstractNum to precede every w:num; Word does not reject the malformed part but silently mis-associates the stray definition, so bulleted lists rendered as plain paragraphs. Numbering definitions are now spliced in ahead of any existing w:num, which also fixes templates that already ship a numbering part (--docx-renderer-template-path, DocxRendererOptions.template_path).
• Generated bullets use the correct glyph. The bullet level specified U+00B7 (MIDDLE DOT) while pinning the run font to Symbol, yielding the wrong character. It now uses U+F0B7, the Symbol font's bullet, matching Word's own output.
• Generated list styles keep their names. List Bullet / List Number were created as custom styles, colliding with Word's latent built-ins and getting renamed to List Bullet1 / List Number1 — so any styling a template applied to List Bullet never took effect. They are now created as built-in styles.

Changed

• CI now runs on pushes to and pull requests against release/** branches, so patch releases cut from a release tag get a full lint/type/test run before merge.

1.8.1 - 2026-07-06

Added

• --remote-input-no-require-head-success. Remote document fetching (all2md https://…) previously always required a successful HEAD request before downloading, with no way to opt out — servers that reject or mishandle HEAD could not be read at all. RemoteInputOptions gains require_head_success (default True) with a matching CLI flag and ALL2MD_REMOTE_INPUT_REQUIRE_HEAD_SUCCESS environment variable.

Fixed

• Legacy <center> no longer swallows page content. <center> was not in the HTML parser's block-element set, so pages that wrap their main content in it — notably Hacker News item pages — converted to empty output. It is now treated as a block container and its children (paragraphs, tables, …) are preserved.
• Options docs now list only flags that actually exist. The auto-generated options reference invented --network-* flags with no per-format prefix and showed positive forms of boolean flags the CLI only exposes negated (e.g. --html-network-no-require-https). The generator now mirrors the CLI builder's real naming rules (per-format --<format>-network-* / --<format>-renderer-network-* prefixes, negated defaults, skipped internal fields), and every emitted flag is cross-checked against the live parser.

Security

• Redirect limits are now actually enforced. The max_redirects check ran in an httpx response event hook, which fires before httpx assigns response.history — so the redirect count it inspected was always empty and the limit never triggered. Enforcement now uses httpx's native max_redirects, surfacing violations as NetworkSecurityError.
• Four NetworkFetchOptions fields were accepted but silently ignored when fetching attachments/images: max_redirects, allowed_content_types, max_requests_per_second, and max_concurrent_requests. They are now wired through a single shared fetch helper used by the HTML parser and the DOCX/EPUB/ODP/ODT/PDF/PPTX renderers (rate limiting is applied per converter instance), with a guard test asserting every field of the dataclass is forwarded so new fields can't silently drop out again.

1.8.0 - 2026-07-01

Added

• all2md help cheatsheet. A bundled, grouped quick reference of the most common commands (convert, view/serve/edit, extract/navigate, grep/search, chunk, diff/lint, generate, transforms, stdin pipes, utilities), printable offline from the terminal (--rich renders it as Markdown). The cheatsheet ships in the wheel as a single source of truth and is mirrored into the docs (:doc:cheatsheet); the quick-help footer now points at it.

• all2md chunk: provenance-aware document chunking for RAG/LLM pipelines. Splits any supported document into chunks and emits them as JSONL (one object per line) — or --format json/pretty. Unlike flat-text chunkers, every chunk carries AST-derived provenance: its section heading/level, and the source page span where the parser tracks it (PDF and friends). Eleven strategies: semantic (default; section-bounded real-token windows), heading, section, auto (coarse, one chunk per boundary), and token, sentence, paragraph, word, line, char, code (fine). --max-tokens/--overlap/--min-tokens bound size; --max-heading-level, --include-preamble/--heading-merge toggles control structure; --token-counter {auto,tiktoken,whitespace} selects the tokenizer. Real BPE token counting uses tiktoken (new optional extra: pip install all2md[chunk]); count-only strategies fall back to a whitespace approximation when it is absent. Element handling: --avoid-table-split and --avoid-code-split keep each table or fenced code block whole (one atomic chunk rather than fragmenting it), --drop-elements image,table,… strips noisy node types before chunking, --elide-data-uris (on by default) replaces long base64 data: URIs with a short placeholder so embedded images never inflate token counts or shred into noise, and --attachment-mode {skip,alt_text,save,base64} (plus any [pdf]/[html]/ top-level converter keys in a config file) controls how the underlying conversion handles images — so base64 blobs need never reach a chunk. Exposed from Python as a one-call all2md.chunk(source, …) (mirrors to_markdown: converts and chunks in a single step, deriving document_id/path from the source and forwarding converter kwargs), with all2md.chunking.chunk_ast(doc, …) for an AST you already hold; both return ProvenanceChunk records. The fine-grained chunkers are vendored from the localvectordb sister project.

• Mermaid diagrams, syntax highlighting, and custom themes for view/serve. The browser preview (all2md view) and local server (all2md serve) now render mermaid fences as diagrams (via mermaid.js) and syntax-highlight fenced code and raw source files (via highlight.js). Both are on by default with graceful offline degradation, toggle off with --no-mermaid / --no-syntax-highlight, and pick dark variants under --dark. Mermaid rendering is also exposed on the HTML renderer via the new HtmlRendererOptions.render_mermaid (off by default; view/serve enable it). serve's directory listing is rewritten as an aligned table (Name/Size/Modified/Created) plus a card view with a localStorage-remembered toggle and HTML-escaped names. --theme now also accepts a plain .css file (wrapped in a minimal shell) and a theme name registered in a new [themes] config table. New "Document Viewer & Server" guide (:doc:viewer).

Fixed

• merge_hyphenated_words now applies to OCR text. PyMuPDF's TEXT_DEHYPHENATE flag only affects native text extraction, so words hyphenated across a line break (be-\nwusst) survived unmerged whenever a PDF page went through OCR (--pdf-ocr-enabled), even with merge_hyphenated_words = true. OCR output is now dehyphenated the same way, joining the split halves (bewusst). Numeric ranges (10-\n20) and hyphens not sitting between two letters are left untouched. (#51)
• Config-file discovery is now bounded at the home directory. find_config_in_parents() walked from the working directory all the way to the filesystem root, so an .all2md.* sitting in a shared parent (a drive root, /) would silently apply to every project underneath it. The upward walk now stops at Path.home() (inclusive). Real behavior is unchanged — ~/.all2md.* is still found, and the home fallback still covers a working directory outside the home subtree.

1.7.1 - 2026-06-25

Added

• Lint profiles: all2md lint --profile NAME. Curated, named rule bundles built entirely from the existing 47 rules — prose (typographic polish for long-form writing, ideal for a converted DOCX), accessibility (alt text, link/table semantics, heading hierarchy at error severity), and technical-docs (structure and links enforced, prose typography relaxed). --list-profiles prints them with descriptions. Profiles are a base layer: config files and CLI flags layer on top in precedence profile < config file < CLI flags. Exposed from Python via all2md.linter.get_profile_config / available_profiles. New "Linting & Enforcing a Style Guide" how-to guide in the docs walks the full convert → lint → fix → profile workflow.
• --extract is now repeatable and understands tables and figures. In addition to sections (by name/pattern or #: index) and line: ranges, --extract now selects tables (table:2, table:1-3, table:*) and figures/images (figure:1, image:*). Pass --extract multiple times to pull several pieces at once; results are emitted in the order the flags appear, separated by ---. A single line: range still cannot be mixed with other selectors.
• --extract … ::N word limit. Append ::N to a selector to cap its output at roughly N words, cut at node boundaries so the result stays valid (e.g. --extract "Introduction::500").
• --slice X/Y paging. Return the Xth of Y semantic slices of a document to stdout/file without writing split files. The document is divided into exactly Y balanced slices at section boundaries, and the chosen slice is emitted with a footer hint pointing at the next slice. Mutually exclusive with --extract/--outline/--split-by/--collate.
• --head [N], --tail [N], and --lines START:END. Simple windows over the rendered Markdown output (1-based, inclusive), mirroring head/tail and the existing --extract line: range. --head/--tail default to 10 lines and honor --line-numbers.

Fixed

• GFM tables nested in list items and blockquotes are now parsed. Pipe tables indented inside a list item or > blockquote were previously left as plain text; they are now recognized and parsed into table nodes.

1.7.0 - 2026-06-24

Changed

• --pager no longer refuses to page Rich output on Windows/WSL. Paging is left to the environment via PAGER/MANPAGER and the platform default. When --pager --rich is used on Windows without a configured PAGER (where the default more mangles ANSI color codes), all2md now prints a one-line hint pointing at an ANSI-capable pager such as less -R instead of silently dropping paging.
• EML: HTML and RTF bodies keep their formatting. Email bodies converted from HTML (with convert_html_to_markdown) or RTF are now re-parsed into rich AST nodes, so headings, bold/italic, links, and lists survive into the output instead of being flattened to escaped plain text. Genuine plain-text bodies are still treated as plain text, and raw HTML is never passed through (the Markdown renderer escapes it by default), preserving the parser's sanitization stance.

Added

• [rich] config table for theming --rich terminal output. A new [rich] table in the config file customizes the colors Rich uses for Markdown elements (headings, links, block quotes, list bullets, inline code, ...) in --rich output. Bare element names auto-prefix to markdown.*; dotted keys pass through verbatim; invalid or non-string entries are skipped with a warning. Previously only code-block syntax themes were configurable. all2md config generate emits a commented [rich] example.
• all2md help markdown (and help md). Added as aliases for the verbose help common-markdown-formatting topic, matching the help <format> pattern used by every other format.
• view/serve honor converter options from the config file. A single config file now drives all2md, view, and serve identically -- e.g. [pdf] detect_columns = true or a top-level attachment_mode applies when viewing or serving, not just when converting. (serve still forces base64 attachments so images render in-browser.)
• Shorthand flags for view and serve. view gains -d/--dark, -w/--window, -t/--theme, -x/--extract, -N/--no-wait. serve gains -p/--port, -H/--host, -B/--browse, -C/--config, and a new -a/--address HOST:PORT that sets host and port together (-a 0.0.0.0:9000, -a :9000, -a host:). Host uses -H because -h is reserved for --help.
• EML: RTF message bodies are converted to Markdown. Emails whose body is an application/rtf / text/rtf part (e.g. Outlook messages exported via libpst/readpst) previously yielded empty content; the RTF body is now routed through the existing RTF parser as a fallback after plain-text and HTML, and rendered to Markdown. Controlled by the new include_rtf_parts option (--no-include-rtf-parts). (GitHub #39)

1.6.0 - 2026-06-18

Added

• list_workspace_files MCP tool. A new read-only tool (enabled by default) that lets an agent discover the files it is allowed to read before reading or editing them. Returns each file's absolute path and size, supports a glob pattern and a workspace-relative subdirectory scope, recurses by default, and flags truncated when the listing is capped. Toggle with --enable-list-files / --no-list-files or ALL2MD_MCP_ENABLE_LIST_FILES.
• Additional read-only folders for the MCP server. A new --additional-read-dirs flag and ALL2MD_MCP_ADDITIONAL_READ_DIRS environment variable append folders to the read allowlist only (never the write allowlist), and are surfaced in the MCPB manifest.
• Batch, in-place edit_document. edit_document now accepts an ordered edits batch applied to a single parse; the batch is atomic (any failure writes nothing). When a batch contains a mutating action, the document is written back to disk in its original format (disk_written / output_path in the response). In-place write-back supports md/html/docx/pptx/rst/epub; other formats and read-only targets fail with a clear message. Responses echo only the edited region, not the whole document.

Changed

• edit_document auto-detects the source format instead of assuming Markdown, so a .docx (or html/rst/epub/…) is parsed correctly rather than yielding zero sections and cryptic index errors. Mutating edits now require the target to be within the write allowlist (it was read-only before). DOCX write-back uses the original file as a template to preserve styles where possible.
• MCP path handling. Relative paths and bare filenames are resolved against the workspace (the read/write allowlist acts as the working directory) across the read, edit, outline, diff, and save tools. A source that is unmistakably a file path but cannot be found now fails loudly — listing the folders searched — instead of being silently treated as inline document text.

Fixed

• MCP stdio protocol corruption on PDFs. PyMuPDF prints an advisory to stdout when processing PDFs, which corrupted the JSON-RPC channel and crashed the connection for any PDF. The server now redirects fd 1 → stderr around each tool's conversion work and sets PYMUPDF_MESSAGE=fd:2 as an import-time backstop.

1.5.0 - 2026-06-15

Added

• MCP query tools. The MCP server gained three read-only tools so an agent can query a document corpus, not just convert single files: search_documents (grep plus keyword/BM25 search across a corpus, returning ranked snippets), diff_documents (compare two documents of any format with unified or JSON output), and get_document_outline (list a document's heading structure, with indices aligned to edit_document's #N notation). All three are enabled by default and read-only; each has its own --no-<tool> flag and ALL2MD_MCP_ENABLE_<TOOL> environment switch, and path inputs are enforced against the read allowlist. search_documents rebuilds a fresh in-memory index per call by default; opt into a persistent keyword index with --search-index-dir / ALL2MD_MCP_SEARCH_INDEX_DIR (validated against the write allowlist). Vector/hybrid search modes are rejected with a clear error.
• Interactive all2md batch wizard. A guided workflow that walks through file selection (with a file-type preview), output layout, attachment handling, per-format options, and advanced parameters, then prints the equivalent command and offers to run it. Uses Rich when available, with a plain-input fallback.
• Near-source batch attachments. With --preserve-structure and --attachment-mode save (and no explicit --attachment-output-dir / --attachment-base-url), saved attachments are now co-located in a shared .attachments folder beside each output file and linked with relative paths. Explicit overrides and the legacy single-folder behavior are preserved.
• Batch help and docs. The multi-file flags are now grouped under a "Batch options" group so all2md help batch works, all2md help attachments resolves to the global attachment topic, and a new batch page documents the batch-conversion CLI.
• Material for MkDocs markdown syntax. The markdown parser now understands several niche flavor constructs common on MkDocs sites: admonitions (!!! note "Title") and their collapsible ??? / ???+ variants, and the pymdownx inline mark family — highlight (==text==, a new Mark AST node), insert/underline (^^text^^), superscript (^text^) and subscript (~text~). Admonitions round-trip to native !!! / ??? blocks on the markdown_plus flavor and degrade to labelled block quotes elsewhere; marks round-trip on flavors that support them and otherwise fall back to HTML. Controlled by the new parse_marks / parse_admonitions options (--no-parse-marks, --no-parse-admonitions).
• Dark mode for all2md edit. The in-browser editor now has a 🌙/☀️ toggle in its header, a --dark flag, and an [edit] config dark = true setting. The toggle choice is remembered across launches via the browser's localStorage.
• Standalone-window mode for all2md view and all2md edit. A new --window flag (and matching [view]/[edit] config setting) opens the preview/editor in a native OS window with no address bar or browser chrome. It uses the new optional pywebview dependency (pip install all2md[window]); without it, all2md prints a hint and falls back to a normal browser tab.

Changed

• The raw-Markdown pane in all2md edit now uses a monospace font, matching the expectation for editing source text (the rendered preview pane is unchanged).

Fixed

• Definition lists are now parsed. The parse_definition_lists option and its AST handling existed, but the underlying mistune plugin was never enabled, so Term / : definition syntax was silently dropped. It is now wired up (and the handler updated for the current mistune def_list_item token).
• MCPB bundle now ships rank-bm25. The search_documents MCP tool defaults to keyword (BM25) mode, but the Claude Desktop bundle didn't install rank-bm25, so corpus search failed out of the box with an install hint. The bundle now depends on rank-bm25 directly (not the full search extra, whose faiss-cpu / sentence-transformers back the vector/hybrid modes that the MCP server rejects).

1.4.0 - 2026-06-11

Added

• EasyOCR engine for PDF OCR. A new binary-free OCR backend, selectable via OCROptions(engine="easyocr") or --pdf-ocr-engine easyocr. Unlike the default Tesseract engine it needs no system binary (pip install all2md[ocr-easyocr]); it pulls in PyTorch and downloads recognition models on first use. Added an OCROptions.gpu flag (EasyOCR only). Tesseract remains the default with unchanged behavior.

Fixed

• Corrected stale OCR CLI flags in the README (--ocr-* → --pdf-ocr-*).
• rcat opened a transient console window that closed instantly on Windows instead of rendering in the terminal (regression in 1.3.0). When the Windows context-menu integration added a [project.gui-scripts] table, the rcat entry point was inadvertently absorbed into it, so its launcher used the GUI subsystem and detached from the console. Moved rcat back to [project.scripts]; it renders in the terminal again.

1.3.0 - 2026-06-11

Added

• all2md llm-minify — a token-lean conversion command for feeding documents to LLMs. The default preset keeps Markdown structure (headings, lists, code, tables) while dropping comments, frontmatter, and raw HTML, replacing embedded base64 image data with an alt-text-only reference (so a single inlined screenshot no longer costs tens of thousands of tokens), and collapsing redundant blank lines and interior whitespace. --aggressive (alias --text) strips all formatting down to bare text, and --strip-links/--strip-images/--strip-formatting layer additional pruning on top of either preset.
• Windows right-click context-menu integration via all2md context-menu (per-user, no administrator rights). It installs a View entry on files (browser preview), an Edit entry on files (in-browser editor), and a Serve entry on folders (local server). install registers View by default; add --edit, --serve, or --all for the others. status reports which entries are installed and uninstall removes them all. The file entries honor --extensions/--all-text for which file types they appear on; the folder Serve entry is unaffected.
• generate-site gained MkDocs, Zola, and Eleventy generators, joining the existing static-site backends.

Changed

• JSON, YAML, TOML, and INI inputs now convert to a fenced, syntax-highlighted code block by default instead of a table/definition-list document (comments are preserved for the formats that have them). This is easier to read and round-trips cleanly. Pass --<fmt>-no-literal-block (e.g. --json-no-literal-block) to restore the previous structured-document output.
• In all2md view and all2md serve, external links now open in a new browser tab (target="_blank" rel="noopener noreferrer") so clicking an off-site link no longer navigates away from the document; internal and relative links are unchanged. Plain --to html output is unaffected.
• Restructured the bundled agent skills into a single all2md skill following Anthropic's progressive-disclosure pattern: a lean SKILL.md overview that routes to per-task guides under references/ (read, convert, generate, grep, search, diff), replacing the previous six top-level all2md-* skills. install-skills installs the one skill tree; llm-help <topic> maps to the reference files (topics unchanged, plus overview).
• Faster CLI startup: a generated converter manifest lets the CLI resolve formats without importing every converter module at launch.

Fixed

• auto OCR mode now recovers scanned PDFs that previously came back empty. The per-page heuristic counts meaningful (alphanumeric) characters instead of raw string length, so pages whose extracted "text" is only whitespace or invisible glyphs now trigger OCR; a document-level safety net additionally re-runs OCR when the entire document renders near-empty under auto. When OCR is disabled, a hint now suggests --pdf-ocr-mode force.
• Corrected stale/renamed CLI flags throughout the bundled skills and docs (GitHub issue #16). Notably --html-standalone (HTML is standalone by default; use --html-renderer-no-standalone for a fragment), --docx-template → --docx-renderer-template-path, --pdf-page-size → --pdf-renderer-page-size, --jinja-template* → --jinja-renderer-template*, --pdf-detect-tables → --pdf-table-detection-mode, search --semantic/--mode bm25 → --vector/--keyword, and several others. Added a regression test that fails if any removed flag reappears in bundled skill content.

Documentation

• Comprehensive documentation audit: split the overview into a user-facing guide and a separate architecture-internals page, reconciled configuration-precedence docs, removed overlapping/duplicated guidance, and fixed a range of accuracy and correctness errors across the guides. The supported-format matrix is now auto-generated from the converter registry during the Sphinx build, so it can no longer drift from the code.

1.2.0 - 2026-05-29

Added

• Config-file support for the view, serve, diff, edit, arxiv, and generate-site subcommands. Each command reads its own same-named section — [view], [serve], [diff], [edit], [arxiv], [generate-site] — from .all2md.toml/.yaml/.json (or the equivalent [tool.all2md.<command>] block in pyproject.toml), so flags like view --no-wait or serve --port can be set once instead of typed every time. Precedence is built-in default < config section < explicit CLI flag, and every one of these commands now also accepts --config <path> and --no-config, mirroring the main converter. Keys are the option name (hyphens or underscores both work); only the matching section is read, so subcommand config never affects a normal conversion and vice versa. A config value can also satisfy an otherwise-required option (e.g. [arxiv] with output = "paper.tar.gz" lets all2md arxiv paper.tex run without -o).
• all2md config generate now emits a template section for each of those subcommands alongside the format sections, so generating a config is the quickest way to discover every available subcommand key and its default. See the new "Subcommand Options" section in docs/source/configuration.rst.

Fixed

• The main converter no longer mishandles non-format config sections ([view], [serve], [diff], etc.) when the input format can't be determined (stdin or failed detection). On that fallback path, any format-qualified option was previously applied blindly, so a subcommand section's keys (e.g. port, no_wait) could be injected as parser keyword arguments and crash the conversion. The fallback is now restricted to recognized parser/renderer format prefixes; unrecognized sections are dropped.

1.1.3 - 2026-05-21

Added

• rcat — a standalone "rich cat" command equivalent to all2md --rich. Renders any supported document with rich terminal formatting (syntax highlighting, colors) and automatically falls back to plain Markdown when output is piped or redirected, so rcat doc.pdf pretty-prints while rcat doc.pdf | grep ... stays parseable.
• all2md serve now accepts glob patterns (e.g. all2md serve "docs/*.docx"). The pattern's anchor directory is served as a listing filtered to matching files; a ** segment enables recursive matching. The background live-rescan continues to honor the filter, and a hand-authored index.html/README.md no longer overrides the filtered listing.
• --include-hidden flag for both conversion and all2md serve. Dot-files and dot-folders are now skipped by default when scanning directories or expanding globs; pass --include-hidden to include them. Explicitly named files (even hidden ones) are always converted.
• -f as a short alias for --force-rich.
• install-skills, edit, lint, and arxiv subcommands now appear in the all2md --help listing (previously hidden).

Fixed

• --force-rich now actually emits ANSI styling when stdout is not a TTY, so piping forced rich output to a pager works (e.g. rcat file --force-rich | less -R). Previously the forced-rich path still produced plain text because the Rich console was not placed into terminal mode.

1.1.2 - 2026-05-20

Added

• all2md serve now auto-renders an index.html, index.htm, index.md, or README.md (case-insensitive, priority order) from the served directory through the active theme instead of the generated file listing. Applies to every directory the server can reach, including subdirectories in --recursive mode. New --force-auto-index flag opts back into the generated listing.
• all2md serve directory mode now picks up newly added, removed, and modified files automatically via a background polling thread. New --poll-interval SECONDS flag (default 2.0, set 0 to disable) controls the rescan cadence; on detected change the cached index page is invalidated and stale file-cache entries for vanished files are dropped.
• Line-number navigation for the CLI. --line-numbers/-ln annotates Markdown output with line numbers: --outline --line-numbers labels each heading with the line it occupies in the full conversion, a normal conversion numbers every line (cat -n style), and --extract keeps the returned lines' original numbers. Line numbers reference the Markdown rendering and are ignored for other targets.
• --extract line:X-Y selects content by output line range (line:42, line:42-87, line:42-, or line:1-10,42-87; 1-based, inclusive). The selection is taken on the Markdown rendering and re-parsed so it can still render to any --to target. Paired with --outline --line-numbers, this lets a reader (or an LLM/agent) map a document then pull back just the range it needs.

Changed

• all2md serve now handles requests on per-connection threads (ThreadingHTTPServer), so a slow conversion no longer blocks other visitors.

Fixed

• all2md serve Ctrl+C shutdown was previously delayed until the next inbound request arrived to unblock select() on Windows. The server now runs serve_forever() in a background daemon thread and the main thread reacts to SIGINT immediately, calling httpd.shutdown() for a prompt clean exit.
• --to/--output-format was silently ignored when converting to stdout (e.g. all2md doc.md --to html printed Markdown). The option is now tracked as explicitly provided, so it is honored for stdout and takes precedence over output-path extension inference; ALL2MD_OUTPUT_FORMAT also works as a default.
• Short UTF-8 files could be mojibaked when chardet misdetected rare multi-byte characters (en-dash, em-dash, smart quotes) as Windows-1252 (e.g. turning "–" into "â€""). A strict UTF-8 decode is now attempted first; since invalid UTF-8 byte sequences raise rather than mis-decode, a successful decode is definitively correct.

1.1.1 - 2026-05-15

Added

• New PDF parsing options for handling brittle real-world layouts: min_image_dimension (filter decorative artifacts under a pixel threshold), filter_header_footer_images (drop images sitting inside detected page-header/footer bands), collapse_excess_whitespace (collapse long whitespace runs that PDF spans use as layout padding), dedup_running_headings (merge split numbering-prefix headings like "I." + "Background" into "I. Background"), and annotate_rotated_text (opt-in *[rotated 90° counter-clockwise]* marker; default off).
• DOCX round-trip formatting preservation. to_ast/from_ast/from_markdown/convert accept a new preserve_formatting kwarg, and all2md edit gains a --preserve-formatting flag (on by default for .docx → .docx; pass --no-preserve-formatting to opt out). Round-tripping a .docx through Markdown now keeps page setup, theme, headers/footers, and named paragraph styles instead of collapsing them to defaults. The parser stashes paragraph.style.name on AST nodes via metadata['source_style'], and the renderer re-applies it when the template defines the style — so custom paragraph styles like "Chapter Title" survive instead of degrading to "Heading 1". to_ast auto-stashes Document.metadata['source_path'] for file-path inputs so the original document can be reused as a rendering template. Out of scope: run-level character styles still collapse on round-trip (tracked separately).
• DocxRendererOptions.clear_template_body (default False) — gates whether a loaded template_path keeps its body content (letterhead use case) or has it stripped before the AST is rendered (round-trip use case). Section properties, headers/footers, and style definitions are always preserved.
• Corpus benchmark harness under benchmarks/corpus/ — pulls deterministic samples from arxiv, PubMed Central, govdocs1, Apache POI, and Enron, times conversion, and emits a stratified Markdown report. Companion inspect command saves converted Markdown next to the source for manual quality review on the slowest, largest, and random subsets. See benchmarks/corpus/README.md and the new "Corpus Benchmark Harness" section in docs/source/performance.rst.
• Manual-dispatch GitHub Actions workflow (.github/workflows/benchmark.yml) that runs the corpus harness on a clean ubuntu-latest VM, caches the ~1 GB corpus between runs, and uploads results as a 90-day workflow artifact. Use for reproducible perf numbers when the local dev box is too noisy.
• Benchmark CLI ergonomics: purge subcommand to delete the ~1 GB corpus cache, --purge-after flag for post-run cleanup (CI / ephemeral disks), and --use-layout-model to opt back into the optional pymupdf-layout ONNX classifier — off by default in the benchmark for reproducibility across machines.
• Reference benchmark snapshots under benchmarks/reference/ — committed before/after .md + .json reports (b0e4224-baseline, 3516bc9-optimized) that anchor the performance numbers cited in the docs.
• New documentation page docs/source/optimizations.rst walking through the v1.1.1 PDF performance work: methodology (corpus benchmark + cProfile + inspect), headline numbers, the 000887.pdf case study (5.6 min → 11.65 s), per-commit attribution, and a "what's still slow" section.

Changed

• PDF table detection in the default mode now skips PyMuPDF's find_tables() on pages with no ruling-line drawings or large closed rectangles. Avoids ~1s/page of wasted work on prose-only pages where find_tables() would either return nothing useful or fire on decorative frames that downstream guards already reject. Net impact on the 149-doc corpus benchmark: 21.4 min → 6.7 min total (3.2x faster); PDF p50 8.5s → 728ms (12x); the slowest single file 5.6 min → 11.65 s (28x). The new page_has_table_signals() helper is conservative on error (returns True / runs find_tables) so PyMuPDF quirks can't silently lose real tables. table_detection_mode="pymupdf" is unchanged — explicit opt-in to always-run behavior. See docs/source/optimizations.rst for the full writeup.
• image_placement_markers no longer applies when attachment_mode="alt_text" (the default). Markers had no URL to target in that mode, so ![Image from page N]() placeholders were just noise. The option now only takes effect in save and base64 modes. As a side effect, image-heavy PDFs in the default mode also skip pixmap decoding entirely (≈160 decodes avoided on a typical 32-page workshop PDF).
• DOCX rendering re-applies parser-stashed source_style paragraph styles when the template defines them, rather than always falling back to built-in heading mapping.
• DocxRendererOptions field order: network moved to the end so the auto-generated options docs read in a more natural order. All fields remain keyword-friendly with defaults.

Fixed

• PDF heading detection misclassified the body=11pt / header=12pt convention as body text (the 1.2 size-ratio default produced an empty header_id), silently ignored bold-only header styles, and classified mixed-style lines by spans[0] only. Spans are now aggregated per line and style requirements are enforced.
• PDF rotated text flooded output with one *[rotated 90° counter-clockwise]* marker per line (~280 markers on the "Attention Is All You Need" figure-axis labels). Consecutive rotated spans are now grouped within blocks and merged across blocks via metadata, and the annotation is opt-in via the new annotate_rotated_text option.
• PDF table detection fired on TOC dot-leader regions, decorative frames, and oversized empty grids in both PyMuPDF's find_tables() and the ruling-line fallback. Shared size, sparsity, uniformity, and dot-leader-ratio guards now reject pathological detections in both paths rather than emitting them as garbage tables.
• PDF attachment_mode="alt_text" emitted 100+ empty ![Image from page N]() placeholders on image-heavy documents. extract_page_images() now returns early in alt_text mode (suppresses the placeholders and avoids decoding every pixmap only to throw the bytes away).
• Tiny decorative PDF images (logo strokes, signature artifacts) and images sitting inside detected page-header/footer regions are no longer emitted as ghost markers — see the new min_image_dimension and filter_header_footer_images options.

1.1.0 - 2026-05-01

Added

• all2md edit FILE command — launches a local web-based editor (Toast UI Editor v3.2.2 with Markdown and WYSIWYG modes) pre-loaded with any supported document converted to Markdown. Saves back to disk in any installed target format, with automatic .bak creation when overwriting. For .md sources the default save target is the original file (overwrite enabled); for any other format the default target is a sibling .md file (overwrite disabled). Toast UI assets are vendored under themes/assets/ and served from /assets/ with a strict allow-list.
• Linter v2: 27 new rules across three new categories and four expanded ones, bringing the total to 47 built-in rules. New categories: LST (lists), TBL (tables), IMG (images). Expanded categories: STR (short-section, empty-document, excessive-nesting), HDG (heading-as-sentence, heading-url), LNK (insecure-link, link-text-is-url), TYP (ellipsis-character, space-before-punctuation, consecutive-punctuation).
• Auto-fix framework: all2md lint --fix applies safe auto-fixes in place. Seven rules ship with safe fixes attached: TYP001 (trailing-spaces), TYP002 (multiple-spaces), TYP003 (straight-quotes), TYP004 (double-hyphens), TYP006 (ellipsis-character), TYP007 (space-before-punctuation), and STR004 (empty-heading).
• --dry-run flag for lint --fix: report what would be changed without writing the file.
• Public API: all2md.linter.lint_and_fix_document(), lint_and_fix_file(), LintFixResult, LintFix, FixSafety, FixContext, apply_fixes.
• Reporters now surface auto-fix results: the text reporter prints per-file applied N fix(es) plus deferred-conflict counts; the JSON reporter adds applied_fixes, skipped_fixes, pre_fix_violations, and rewritten keys per result entry.

Changed

• Violation.fixable is now a derived @property (fix is not None) rather than a stored field. Code that constructs Violation(..., fixable=True) will need to pass a LintFix instead.
• LintRule.build_violation() accepts an optional fix= keyword to attach a LintFix to a violation.

1.0.6 - 2026-04-10

Added

• Per-subdirectory index pages with breadcrumb navigation when serving directories recursively (all2md serve --recursive)
• Batch conversion examples added to CLI help output for discoverability

Fixed

• PDF conversion crash when PyMuPDF detects empty tables (tables with no cells)
• view --no-wait deleting the temp file before the browser could load it

Changed

• Default document author is now set to "all2md" when not otherwise specified by the source document
• Auto-release CI workflow: tag pushes now run CI checks then publish to PyPI automatically
• Bumped codecov/codecov-action from 5 to 6 and actions/setup-python from 5 to 6

1.0.5 - 2026-04-08

Added

• --no-wait flag for the view command for non-interactive use

Fixed

• Create missing list styles when rendering DOCX with custom templates

1.0.4 - 2026-03-25

Added

• ArXiv submission package generator (all2md arxiv) — converts any supported document format into a complete ArXiv-ready LaTeX submission archive (.tar.gz or directory) with extracted figures and optional .bib bibliography
• Pre-built Agent Skills — 6 focused skill files (all2md-read, all2md-convert, all2md-generate, all2md-grep, all2md-search, all2md-diff) that teach AI coding assistants (Claude Code, Cursor, Windsurf) how to use all2md. Install with all2md install-skills
• Optional pymupdf-layout integration for GNN-based PDF layout analysis — classifies text blocks by semantic role (title, section-header, caption, footnote, etc.) for improved reading order and structure detection. Install with pip install "all2md[pdf_layout]"

Fixed

• CLI renderer options (e.g. --docx-renderer-template-path) were silently dropped during format filtering, causing renderer-specific flags to have no effect

1.0.3 - 2026-03-16

Added

• Flow layout engine for Markdown-to-PPTX rendering with template placeholder reuse and inherited built-in styles
• H1-to-Title promotion for Markdown-to-DOCX rendering

Fixed

• HTML renderer anchor links now use GitHub-style heading IDs (id="introduction" instead of id="introduction-1"), so #ref links resolve correctly
• PPTX flow layout no longer overlaps template placeholders; HTML comments route to speaker notes
• --collate --out now writes the target format (e.g. DOCX) instead of raw Markdown
• Sphinx documentation build warning from malformed .. deprecated:: directive
• Stale mypy type: ignore comments across pptx renderer, title promotion transform, and archive parser
• Flaky test_detect_latin1 marked as xfail (chardet Latin-1 detection unreliable across platforms)

Changed

• Upgraded to Black 26.x and pinned version (~=26.1) to prevent CI/local formatting drift
• Pre-commit format-sync hooks now use uv run for Windows compatibility
• options= accepted as deprecated alias for parser_options in to_markdown(); unmatched kwargs now warn

1.0.2 - 2026-02-27

Fixed

• PPTX flow layout overlapping template placeholders
• HTML comments in PPTX now route to speaker notes

1.0.1 - 2025-12-18

Added

• Softbreak parsing and DOCX CodeBlock styling support
• Dependency-aware file filtering for shell completions

Fixed

• Diff CLI args renamed to original/modified for clarity
• CLI processor refactoring and PDF parsing internals split out
• Broken test from CLI help text change
• mypy type issue from merging lost branch

Changed

• Refactored CLI processors and split PDF parsing internals

1.0.0 - 2025-10-29

Core Features

• Universal document conversion library supporting bidirectional transformation between various formats and Markdown
• AST-based (Abstract Syntax Tree) pipeline for consistent document manipulation across all formats
• Smart dependency management with format-specific optional dependencies
• Security-conscious design with SSRF protection and archive validation

Supported Input Formats (Parse to AST/Markdown)

• Office Documents: PDF, DOCX, PPTX, RTF, ODT, ODP, ODS, XLSX
• Web & Markup: HTML, MHTML, Markdown, reStructuredText, AsciiDoc, Org-Mode, MediaWiki, Textile, BBCode, DokuWiki
• Email: EML, MBOX, MSG (Outlook), PST/OST (Outlook archives)
• E-books: EPUB, FB2, CHM
• Data & Code: CSV/TSV, Jupyter Notebooks (.ipynb), OpenAPI/Swagger, 200+ source code languages
• Archives: ZIP, TAR, 7Z, RAR and other archive formats
• Other: LaTeX, plain text

Supported Output Formats (Render from AST/Markdown)

• Markdown: Multiple flavors (GFM, CommonMark, etc.)
• Office: DOCX, PPTX, PDF, ODT, ODP
• Web: HTML, RTF
• Markup: reStructuredText, AsciiDoc, Org-Mode, MediaWiki, Textile, DokuWiki, LaTeX
• Data: CSV, Jupyter Notebooks (.ipynb), AST JSON
• Templates: Custom Jinja2 templates for any text-based format
• Plain text

MCP Server Integration

• Built-in Model Context Protocol (MCP) server for AI assistant integration
• Smart auto-detection of input sources (file paths, data URIs, base64, plain text)
• Section extraction by heading name for targeted reading
• Security features including file allowlists and network controls
• Support for vision-enabled models with base64 image embedding

PDF Features

• Advanced table detection and extraction
• Multi-column layout analysis
• Intelligent header/footer removal
• OCR support for scanned documents (via Tesseract)
• Page range selection
• Configurable text extraction powered by PyMuPDF

Transform System

• Built-in transforms:
  • remove-images: Strip images from documents
  • remove-nodes: Remove specific node types
  • heading-offset: Adjust heading levels
  • link-rewriter: Rewrite URLs with patterns
  • text-replacer: Find and replace text content
  • add-heading-ids: Generate heading IDs for anchors
  • remove-boilerplate: Strip common boilerplate content
  • add-timestamp: Add conversion timestamp metadata
  • word-count: Add word count metadata
  • add-attachment-footnotes: Add footnotes for attachments
• Extensible plugin system for custom transforms via entry points

CLI Features

• Multi-file and directory processing with recursive mode
• Parallel execution for batch conversions
• Directory watching for automatic conversion
• stdin/stdout piping support
• Format-specific options exposed as CLI flags
• Progress bars and rich terminal output
• Preset configurations for common workflows
• Transform application from command line

Python API

• Simple to_markdown() function for quick conversions
• convert() function for format-to-format conversion
• to_ast() and from_ast() for AST manipulation
• Type-safe configuration with dataclass-based options
• Programmatic transform pipeline application
• Direct AST node manipulation for advanced use cases

Template System

• Jinja2 template renderer for custom output formats
• Example templates included:
  • DocBook XML
  • YAML metadata
  • ANSI terminal output
  • Custom outlines

Developer Features

• Comprehensive test suite with pytest markers (unit, integration, e2e, format-specific)
• Property-based testing with Hypothesis
• Golden/snapshot testing with Syrupy
• Type checking with mypy and custom type stubs
• Code quality enforcement with Ruff
• Pre-commit hooks
• Extensive documentation with Sphinx
• Entry point system for third-party plugins

Documentation

• Comprehensive README with examples
• API documentation with Sphinx
• Format-specific guides
• Security and threat model documentation
• Plugin development guide
• MCP server configuration guide
• Transform system documentation
• Contributing guidelines

Security

• SSRF protection for remote resource fetching
• ZIP bomb detection and prevention
• Path traversal protection in archives
• Network security controls with allowlists/blocklists
• HTML sanitization with configurable policies
• URL validation and sanitization

Technical Details

• Python 3.10+ required
• Hatchling build backend
• MIT License
• Comprehensive type hints throughout codebase
• NumPy-style docstrings
• Modular architecture with clear separation of concerns