Support direct HTML generation mode

[oliviertassinari]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and discussions and couldn’t find anything (or linked relevant results below)

Problem

TL;DR: The way MDX works fundamentally leads to much slower build and runtime compared to a pure Markdown -> HTML generation.

Current solutions

From how I understand MDX, it works like this:

1. On the build side, it takes Markdown and it converts it to React JavaScript (React.createElement), as described in https://mdxjs.com/packages/mdx/#use.
2. On the runtime side:
   a. The React.createElement is inlined in the Next.js RSC payload
   b. React takes the RSC payload and hydrates it (yes Next.js needs to hydrate RSC too, but it's not as costly as client components, see NEXTJS 13: self.__next_f.push vercel/next.js#42170).

This is opposition to tools like marked:

1. On the build side, it takes Markdown and converts it to HTML directly.
2. On the runtime:
   a. The string representation is inlined in the Next.js RSC payload
   b. React takes the RSC payload and hydrates it with innerHTML (dangerouslySetInnerHTML).

Now, the problem is that each step seems much slower on the MDX side:

• 1 The step seems x2 slower if we can trust a benchmark like https://github.com/mui/material-ui/tree/v5.x/benchmark/server#muidocs
• 2.a This step adds more data to the .html page because of how heavy the RSC payload is, see NEXTJS 13: self.__next_f.push vercel/next.js#42170 for some context on this. We could benchmark it to +35%: [docs-infra] Add CodeHighlighter component for loading demos and codeblocks mui/mui-public#379 (comment)
• 2.b This step is x2.5 slower as we could benchmark it in [docs-infra] Add CodeHighlighter component for loading demos and codeblocks mui/mui-public#379 (comment).

Proposed solutions

So I wonder if MDX would be interested in supporting a hybrid rendering mode. For example, allowing a header to be either an HTML string output https://marked.js.org/using_pro#renderer, or a React element. My hope is that this could lead to much faster build time (e.g. no need anymore for Rust https://nextjs.org/docs/app/guides/mdx#using-the-rust-based-mdx-compiler-experimental) and much faster render time.

[ChristianMurphy]

Welcome @oliviertassinari 👋
And thanks for sharing your idea.

TL;DR: The way MDX works fundamentally leads to much slower build and runtime compared to a pure Markdown -> HTML generation.

On the build side, it takes Markdown and it converts it to JSX, as described in https://mdxjs.com/packages/mdx/#use.

It converts it to plain JavaScript, as JSX compiled in auto mode would

On the runtime side:
a. The JSX is inlined in the Next.js RSC payload
b. React takes the RSC payload and hydrates it.

That is one of many possible flows.
MDX is not tied to React, RCS, or Next.
It also works with other renders (vue, svelte, etc), full client side and full server side rendering, and many frameworks (astro, nuxt, etc).

On the build side, it takes Markdown and converts it to HTML directly.

Unified (the parent project of mdx) offers three layers of abstraction to choose from.

1. https://github.com/micromark/micromark a low level markdown to HTML library, small, fast, not particularly customizable. (It sounds like this is what you want)
2. https://github.com/remarkjs/remark a mid level, markdown to (html, jsx, markdown) library, with plugin support for customizability.
3. https://github.com/mdx-js/mdx which is a layer and language on top of markdown and remark, adding native JSX support

The step seems x2 slower if we can trust a benchmark like https://github.com/mui/material-ui/tree/v5.x/benchmark/server#muidocs

I'm not entirely sure what I'm looking at.
A benchmark of how long it takes the doc site to render?
This doesn't appear to directly benchmark any part of the markdown renderer, so no I don't particularly trust it.

mui/mui-public#379 (comment)

Your comment here appears to be about the slowness of one of the syntax highlighters plugged in?
There are other syntax highlights that could be used.

So I wonder if MDX would be interested in supporting a hybrid rendering mode. For example, allowing a header to be either an HTML string output https://marked.js.org/using_pro#renderer, or a React element.

How would that work?
MDX has renderers in the sense of the Marked example you share.
The hybrid HTML/JSX is:

1. something handled at the build tool level, in your case Next
2. Already supported by Next Add support for rendering MDX as a server component vercel/next.js#44651

[ChristianMurphy]

That isn't to say MDX couldn't be sped up. It can!
Feel free to take some time and profile the code and optimize hot paths.

I don't think direct HTML generation fits this project.
It is by definition interactive JSX, which needs a framework (react, vue, svelte, and friends).
If you don't want/need interactive JSX, use micromark or remark, they are faster and do produce HTML.

[oliviertassinari]

I'm not entirely sure what I'm looking at. A benchmark of how long it takes the doc site to render?

@ChristianMurphy I was trying to show a sense of the time the MDX loader spends in webpack to generate the .js asset vs. the time it would take to output a raw JavaScript string (which seems a bit simpler, so faster).

our comment here appears to be about the slowness of one of the syntax highlighters plugged in?
There are other syntax highlights that could be used.

There is a lot of context in this comment that is unrelated to this thread. But the main point was to compare the time it takes to render RSC output with React vs. a plain string HTML output with React. It's so much faster with a plain string HTML output. It's quite counterintuitive to see RSC so slow vs. baseline HTML, but from vercel/next.js#42170, it actually makes sense.

I don't think direct HTML generation fits this project.
It is by definition interactive JSX, which needs a framework (react, vue, svelte, and friends).
If you don't want/need interactive JSX, use micromark or remark, they are faster and do product HTML.

I agree, I'm not pushing for this. For example, I could move to https://markdoc.dev/ if I wanted this.

What I'm exploring is the ability to have a hybrid model. For example, we do this: https://github.com/mui/base-ui/blob/0f2b7a1e4220a61f496257c7497e926e2b69a03a/docs/src/mdx-components.tsx#L54, which feels wasteful: 1. markdown is converted to React.createElement custom h4, 2. This is inlined into an RSC payload. 3. This is hydrated by React.

Instead, for h4, it could support a straight HTML output generation, for fast paces.

  h4: (properties) => `<h4 class="mt-8 mb-1.5 scroll-mt-6 font-medium text-balance">${properties.children}</h4>`,
  h5: (props) => <h5 className="mt-8 mb-1.5 scroll-mt-6 font-medium text-balance" {...props} />,
  h6: (props) => <h6 className="mt-8 mb-1.5 scroll-mt-6 font-medium text-balance" {...props} />,
  p: (props) => <p className="mb-4" {...props} />, 

which would allow faster build and runtime, this applies to React, Vue, Svelte, etc.

How would that work?

The tricky part here is that html element's children can only be html elements. But to give a simple example from https://mdxjs.com/packages/mdx/#use.

export function Thing() {
  return <>Boy!</>
}

#### Hello

<Thing />

would convert to something like this (pseudo-code):

import {Fragment as _Fragment, jsx as _jsx, jsxs as _jsxs} from 'react/jsx-runtime'

export function Thing() {
  return _jsx(_Fragment, {children: 'Boy!'})
}

export function spanHtml(props) {
  return _jsx('span', { dangerouslySetInnerHTML: {{ _html: props.content } })
}

function _createMdxContent(props) {
  const _components = {h1: 'h1', ...props.components}
  return _jsxs(_Fragment, {children: [_jsx(spanHtml, { content: '<h4 class="mt-8 mb-1.5 scroll-mt-6 font-medium text-balance">Hello</h4>' }), _jsx(Thing, {})]})
}

export default function MDXContent(props = {}) {
  const {wrapper: MDXLayout} = props.components || {}
  return MDXLayout
    ? _jsx(MDXLayout, {...props, children: _jsx(_createMdxContent, {...props})})
    : _createMdxContent(props)
}

The larger the HTML string is, the faster the generation speed-up you gain, and the fewer annoying wrapping spans in the DOM output. Effectively, what I envision is to turn all the typography into plain HTML, and use React for rich parts, e.g. Image component, Demo component, etc.

[remcohaszing]

Hi @oliviertassinari

I think you are onto something. Yes, MDX could be faster. Some build-time transforms could be made to make the MDX code faster at runtime. However, MDX is a programming language that compiles to another programming language, JavaScript. Markdown on the other hand is a markup language that can be compiled to another markup language, HTML. So going from MDX to HTML directly is not an option.

What I think you’re really after, and what I see in your last comment, is that you want to preprocess some values avoid overhead of the virtual DOM in React. Some steps have already been made into this direction.

• @babel/plugin-transform-react-constant-elements can memoize calls to jsx() for static content.
• I think React compiler can do the same.
• rehype-mdx-constant-elements can remove the ability to replace elements via components. This means the optimizations by the tools above can kick in.
• It’s already possible to write a recma plugin to transform static JSX to a static HTML string as you suggest.
• There have been some talks a while ago about an alternative more performant JSX transform. I don’t have the details available, but I think the Preact team was working on it.

[ChristianMurphy]

There have been some talks a while ago about an alternative more performant JSX transform. I don’t have the details available, but I think the Preact team was working on it.

Are you thinking of https://marvinh.dev/blog/speeding-up-javascript-ecosystem-part-9/ ?

[remcohaszing]

Yep, that’s the one!

[oliviertassinari]

you want to preprocess some values avoid overhead of the virtual DOM in React.

@remcohaszing Yes, on point. I opened this issue because it feels like the vast majority of the users of MDX are impacted by this (most of their content is composed of static typography React components).

It’s already possible to write a recma plugin to transform static JSX to a static HTML string as you suggest.

This is the idea that was raised internally for us: we could write a custom plugin to transform the output to avoid the virtual DOM, where possible.

[oliviertassinari]

Ah, this seems very close: https://x.com/bluwyoo/status/1791063109056270515. My issue feels like it's about pushing this optimization further, to allow to rely on the userland-provided static components.

[wooorm]

Direct HTML, from the OP, is possible by calling evaluate instead of compile and then your framework. Or, micromark.

The design indeed is fundamentally different from markdown->html. But that’s the design. That’s why this exists. markdown->html also already exists.

Much of your questions relate to React. MDX is not exclusive to React. But React is working on this too, with RSC. And that problem, is not exclusive to MDX either. You can type the same stuff in many giant blog posts as JSX, and have the same problems. MDX is just an easier way to write JSX.

Tho, as all the comments above show, there is much that can be done, and the plugin architecture allows that! Plugins aren’t just for custom MUI needs, plugins (and components) are the fundamental building block that everything is built on! Plugins are also the way to try things out and see if they make sense for more folks, later, here in core!

Closing, as I think the OP is answered and does not lead to actionable things closed by a PR. But, there can be more issues and discussions!