How to stringify a parsed Markdown AST as-is (preserving the original source) while changing only some nodes

[ole]

Intro

I'd like to use remark (or the underlying mdast-util-from-markdown and mdast-util-to-markdown) to selectively edit thousands of Markdown files while keeping the bulk of the original Markdown sources unmodified. In other words, my code should only modify the parts of the Markdown that I actually want to change (e.g. the contents of link nodes) while keeping the rest of the text as-is.

In my experiments (see below for an example), I found that mdast-util-to-markdown "normalizes" the Markdown it outputs, e.g. it will use a consistent style for bullets, emphasis, strong, thematic breaks, etc. I know I can configure which of these styles to use via options, but that still wouldn't do what I want. I'd like to keep the bullet/emphasis/strong etc. styles in the original Markdown sources, even if that means I have to live with inconsistencies etc. I understand that normalizing the output is often a good idea, but in my case I feel that it actually makes my life harder, if only because it creates noisy diffs that make it hard to distinguish cosmetic changes from the material edits I do want to make (and which I want to check by looking through the diffs) Again, we're talking thousands of files, so no noise really matters.

Example

In the following example (written in TypeScript for Deno), my input is a Markdown source text with several incongruities and inconsistencies (multiple strong and emphasis styles, a somewhat unusual format for thematic breaks etc.):

Heading
=======

[Link 1](http://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [link 2](http://example.com)

I want my code to selectively edit the "link" nodes (by changing "http" to "https") and then write the modified tree back as Markdown. (My actual problem uses different AST transformations, e.g. adding a field to frontmatter. But the argument is the same.)

// rewrite-markdown.ts
//
// This is a script written for Deno. You should be able to run it with:
//
//     deno run rewrite-markdown.ts

import type * as Mdast from "npm:@types/mdast"
import * as From from "npm:mdast-util-from-markdown"
import * as To from "npm:mdast-util-to-markdown"
import * as V from "npm:unist-util-visit"

const input = `Heading
=======

[Link 1](http://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [link 2](http://example.com)
`

// Parse Markdown to AST
const tree: Mdast.Root = From.fromMarkdown(input)

// Walk AST and edit link nodes
V.visit(tree, "link", (node: Mdast.Link, _index, _parent): V.VisitorResult => {
  if (node.url.startsWith("http://")) {
    node.url = node.url.replace("http://", "https://")
  }
  return V.SKIP
})

// Write modified AST back into Markdown text
const output = To.toMarkdown(tree)
console.log(output)

Actual vs. desired result

deno run --check rewrite-markdown.ts

This prints (actual result):

# Heading

[Link 1](https://example.com)

***

* **Lorem** **ipsum**
* *dolor* *sit* [link 2](https://example.com)

I would like it to print (desired result):

Heading
=======

[Link 1](https://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [link 2](https://example.com)

I.e. preserving all idiosyncracies in the original source except for the link nodes I edited; the only change compared to the input is "http" to "https". As you can see, the actual output is normalized Markdown, e.g. it uses a consistent style for bullets/strong/emphasis etc. Which is not what I want.

Possible solutions

One idea I had was to install custom toMarkdown handlers for all nodes that I don't want to change. These handlers would then take the node's position information to copy the text directly from the input text into the output. For example, if we want to preserve the original text of "thematicBreak" nodes in the output, we can change the code:

  // rewrite-markdown.ts
  …
  // Write modified AST back into Markdown text
- const output = To.toMarkdown(tree)
+ const toMarkdownOptions: To.Options = {
+   handlers: {
+     "thematicBreak": (node: Mdast.ThematicBreak, parent, state, info): string => {
+       // Slice the relevant segment directly out of the input text
+       return input.slice(node.position!.start.offset, node.position!.end.offset)
+     }
+   }
+ }
+ const output = To.toMarkdown(tree, toMarkdownOptions)
  console.log(output)

Now, the original format of the thematic break is preserved in the rendered output. Great!

This approach seems relatively trivial (if a little tedious to write) for leaf nodes of the tree. But I don't know how to achieve the same for nodes that can have children. E.g. in my example, the "listItem" nodes should generally render all their children (and themselves) as-is, unless one of their children is a "link" node.

I feel this approach could potentially work, but I don't understand the mdast APIs well enough to make it work. Any ideas?

Or are there other solutions you can think of?

Thanks for reading, and thank you to everybody who works on these packages. I love the unified/remark/micromark ecosystem!

[ChristianMurphy]

Duplicate of https://github.com/orgs/remarkjs/discussions/1067, https://github.com/orgs/remarkjs/discussions/1194, and micromark/micromark#59

remark is an AST library

AST is short for Abstract Syntax Tree, the Abstract part focuses on simplifying working on structure, intentionally glossing over or normalizing stylistic parts of the language, like spaces or which list marker is used which don't change how the document will be structured/displayed (https://en.wikipedia.org/wiki/Abstract_syntax_tree)

What you are describing is a Concrete Syntax Tree (https://en.wikipedia.org/wiki/Parse_tree), which could be built on top of micromark (syntax-tree/mdast#36 (comment)), but would have a completely different structure and way of working from remark or rehype.

[ole]

Fair enough. Thanks for the quick reply.

[wooorm]

You could do things with the positional info. It contains positional info about where in the original file things happened. You can then change only things at particular positions, yourself.

[ole]

Thanks, this is a good suggestion. I came up with the following approach, which comes fairly close to what I want:

• When making edits to the AST, record the position info of the modified node(s) in an array.
• For stringifying the modified AST, copy everything outside the modified ranges verbatim from the input text. Only re-render the nodes that have been modified and splice their output into the text.
• This isn’t perfect because re-rendering a node also affects all of its child nodes and will therefore potentially reformat these. But if the nodes you want to modify are primarily leaf nodes (as is the case for me), this looks like a promising workaround.

Here's a full code sample:

#!/usr/bin/env deno run --check

import type * as Mdast from "npm:@types/mdast"
import * as From from "npm:mdast-util-from-markdown"
import * as To from "npm:mdast-util-to-markdown"
import * as V from "npm:unist-util-visit"

const input = `Heading
=======

[Link 1](http://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [_link_ 2](http://example.com)
`

// Parse Markdown to AST
const tree: Mdast.Root = From.fromMarkdown(input)

interface ModifiedRange {
  node: Mdast.RootContent
  start: number
  end: number
}

// Track modified text ranges in the original Markdown source.
const modifiedRanges: ModifiedRange[] = []

// Walk AST and edit link nodes.
V.visit(tree, "link", (node: Mdast.Link, _index, _parent): V.VisitorResult => {
  if (node.url.startsWith("http://")) {
    node.url = node.url.replace("http://", "https://")
    
    // Track this node as modified.
    // Note: error handling (missing position info) intentionally left out.
    modifiedRanges.push({
      node: node,
      start: node.position!.start.offset!,
      end: node.position!.end.offset!,
    })
  }

  // Skip this node's children because links cannot be nested.
  return V.SKIP
})

// Build output by alternating between original and re-rendered segments.
let output = ""
let currentPos = 0
modifiedRanges.sort((a, b) => a.start - b.start)
for (const range of modifiedRanges) {
  // Add original content before this modification.
  if (currentPos < range.start) {
    output += input.slice(currentPos, range.start)
  }
  
  // Re-render the modified node.
  // Note: This *will* potentially modify children of this node
  // whose contents I'd rather preserve.
  output += To.toMarkdown({
    type: "root",
    children: [range.node]
  })
  .trimEnd() // Trim trailing newline.
  
  currentPos = range.end
}

// Add remaining original content after the last modification.
if (currentPos < input.length) {
  output += input.slice(currentPos)
}

console.log(output)

This will produce the following output:

Heading
=======

[Link 1](https://example.com)

- - -

- __Lorem__ **ipsum**
- _dolor_ *sit* [*link* 2](https://example.com)

Which preserves all syntax of the original source except inside the 2 link nodes that have been edited. Note that the emphasis style in "link 2" has changed:

- - _dolor_ *sit* [_link_ 2](http://example.com)
+ - _dolor_ *sit* [*link* 2](https://example.com)

[wooorm]

You may like https://www.npmjs.com/package/magic-string!