Experimental markdown output #1303

jrturton · 2025-09-26T13:35:50Z

Summary

This PR addresses the changes proposed in #1301. It involves a new translator / semantic visitor / markup walker to create the new outputs from DocumentationNode. This is done for each node, after the render JSON is generated.

Dependencies

None

Testing

There are extensive unit tests in MarkdownOutputTests.swift. Testers can also run the convert command against any catalog with the --enable-experimental-markdown-output and --enable-experimental-markdown-output-manifest flags set.

Checklist

Make sure you check off the following items. If they cannot be completed, provide a reason.

Added tests
Ran the ./bin/test script and it succeeded
Updated documentation if necessary

…ert feature flags

…wn output

d-ronnqvist

There's a lot to unpack here and I've only had the time to skim thought the code but these are a few things that stand out to me:

If this is intended as experimental feature, then it can't add actual public API because consumers of SwiftDocC would have no way of knowing what API is stable and what is experimental and could still have breaking changes or be removed.

If a goal of this API is that clients can use it while it's still experimental then it needs to be marked as SPI (using @_spi(SomeName) so that those clients make a deliberate decision to use it and acknowledge that it may have breaking changes.
The added tests follow a pattern not seen elsewhere in the repo. I left some more specific comments about that.

Sources/SwiftDocC/SwiftDocCMarkdownOutput/MarkdownOutputManifest.swift

Sources/SwiftDocC/Model/MarkdownOutput/Translation/MarkdownOutputSemanticVisitor.swift

Tests/SwiftDocCTests/Rendering/Markdown/MarkdownOutputTests.swift

Tests/SwiftDocCTests/Test Bundles/MarkdownOutput.docc/MarkdownOutput.symbols.json

Sources/SwiftDocCUtilities/Action/Actions/Convert/JSONEncodingRenderNodeWriter.swift

Sources/SwiftDocC/Model/Rendering/RenderNode/RenderMetadata.swift

Sources/SwiftDocC/Model/MarkdownOutput/Translation/MarkdownOutputMarkdownWalker.swift

d-ronnqvist · 2025-12-15T17:29:43Z

Sources/SwiftDocC/Model/MarkdownOutput/Translation/MarkdownOutputMarkdownWalker.swift

+import Markdown
+
+/// Performs any markup processing necessary to build the final output markdown
+internal struct MarkdownOutputMarkupWalker: MarkupWalker {


FYI (not blocking): it's redundant to specify internal here because it's the default access level.

d-ronnqvist · 2025-12-15T18:32:39Z

Sources/DocCCommandLine/Action/Actions/Convert/ConvertFileWritingConsumer.swift

+    }
+
+    func consume(markdownManifest: MarkdownOutputManifest) throws {
+        let url = targetFolder.appendingPathComponent("\(markdownManifest.title)-markdown-manifest.json", isDirectory: false)


Question: what's the reason for including the title in the manifest file?

Have I misunderstood that archives would only have a single manifest file?

One archive will only have a single manifest file, but if you are processing the output of multiple archives this allows you to distinguish them

Wouldn't the caller already know what archive the manifest belongs to based on what archive the manifest file is inside?

Sources/SwiftDocCUtilities/Action/Actions/Convert/JSONEncodingRenderNodeWriter.swift

Tests/SwiftDocCTests/Rendering/Markdown/MarkdownOutputTests.swift

d-ronnqvist · 2025-12-15T18:38:45Z

Tests/SwiftDocCTests/Rendering/Markdown/MarkdownOutputTests.swift

+                
+                ## Overview
+                
+                Nothing to see here


nit: This is just noise (and just slightly malformed because the rendered page would have two "Overview" sections)

d-ronnqvist

There continues to be lot of API with a higher access level than what's needed and a handful of unused API.

… level

d-ronnqvist · 2025-12-18T15:03:44Z

@swift-ci please test

d-ronnqvist · 2025-12-18T15:14:55Z

I started a cross-repo test in swiftlang/swift#84510 (comment) to double check that these changes don't impact the rest of the toolchain.

d-ronnqvist · 2025-12-19T09:00:53Z

@swift-ci please test

* Add experimental markdown output flag and pass it through to the convert feature flags * Initial export of Markdown from Article * Initial processing of a type-level symbol * Adds symbol declarations and article reference links * Output tutorials to markdown * Be smarter about removing indentation from within block directives * Baseline for adding new tests for markdown output * Basic test infrastructure for markdown output * Adds symbol link tests to markdown output * Tutoorial code rendering markdown tests * Adding metadata to markdown output * Include package source for markdown output test catalog * Output metadata updates * Adds default availability for modules to markdown export * Move availability out of symbol and in to general metadata for markdown output * Refactor markdown output so the final node type is standalone * Add generated markdown flag to render node metadata * Only include unavailable in markdown header if it is true * Initial setup of manifest output, no references * Output of manifest / relationships * Manifest output format updates * Remove member symbol relationship * More compact availability, deal with metadata availability for symbols * Update tests so all inputs are locally defined * Remove print statements from unused visitors * Added snippet handling * Remove or _spi-hide new public API * Remove or _spi-hide new public API (part 2) * Prevent recursion when the abstract of a link contains a link and we are rendering links-with-abstracts * Test and temporary fix for colspan issue swiftlang/swift-markdown#238 * Remove separate SwiftDocCMarkdownOutput target * Bump swift-markdown, remove temporary bug fix * Updates to handle removed bundle parameter * Extract common writer code into a helper function * Clarify use of article.md in test catalog * Deal with internal links with anchor tags * Deal with unresolved symbol links * Updates to improve public API / clarity of MarkdownOutputNode * Improve public API of manifest, sort contents before writing to prevent unneccessary diffs on re-run * Don't use unstructured `Data` for WritableMarkdownOutputNode * Reduce public SPI surface around markdown writing * Make relationship subtype a specific type rather than a string * Make it clear that path component is a fallback title, harden linked list rendering * Add todos and missing tests * Move markdown output types from public SPI to package * don't include scheme or host in documentation links * Reformat multi-condition if statements * remove whitespace-only change * Remove unused declaration * Remove redundant return types * Remove MarkdownOutputNodeTranslator * Add TODO for alternate declarations * Add TODO for structural review * Add stacks todo, set writable type back to package * uri -> identifier * Use RelationshipsGroup.Kind instead of creating a mirror type * Lower access level of manifest, remove children method * Restrict package level declarations to those actually used at package level * Merging main --------- Co-authored-by: Richard Turton <[email protected]>

Richard Turton added 23 commits September 2, 2025 16:25

Add experimental markdown output flag and pass it through to the conv…

cf7ebc6

…ert feature flags

Initial export of Markdown from Article

4b1b94a

Initial processing of a type-level symbol

55e7836

Adds symbol declarations and article reference links

53a6196

Output tutorials to markdown

3513a73

Be smarter about removing indentation from within block directives

81d2d5a

Baseline for adding new tests for markdown output

f3fa5ab

Basic test infrastructure for markdown output

5013a09

Adds symbol link tests to markdown output

6798162

Tutoorial code rendering markdown tests

132cd6c

Adding metadata to markdown output

1753cca

Include package source for markdown output test catalog

301d7da

Output metadata updates

9607e3b

Adds default availability for modules to markdown export

5244f0f

Move availability out of symbol and in to general metadata for markdo…

a6a740e

…wn output

Refactor markdown output so the final node type is standalone

d0dbf44

Add generated markdown flag to render node metadata

595831a

Only include unavailable in markdown header if it is true

104f9eb

Initial setup of manifest output, no references

a2aa8f1

Output of manifest / relationships

b5ed559

Manifest output format updates

74b6023

Remove member symbol relationship

198d8e8

More compact availability, deal with metadata availability for symbols

53ba222

jrturton requested review from d-ronnqvist, franklinsch and heckj September 26, 2025 13:35

Merge branch 'main' into 159600318/experimental-markdown-ouput

af890be

jrturton linked an issue Sep 26, 2025 that may be closed by this pull request

LLM-accessible output #1301

Closed

d-ronnqvist requested changes Sep 26, 2025

View reviewed changes

Update tests so all inputs are locally defined

4784380

Richard Turton added 3 commits December 15, 2025 10:08

Merge branch 'main' into 159600318/experimental-markdown-ouput

737be98

don't include scheme or host in documentation links

33fa249

Reformat multi-condition if statements

9481ccf

d-ronnqvist reviewed Dec 15, 2025

View reviewed changes

Richard Turton and others added 10 commits December 16, 2025 10:08

remove whitespace-only change

0df68dd

Remove unused declaration

172674c

Remove redundant return types

8a91913

Remove MarkdownOutputNodeTranslator

f9331f5

Add TODO for alternate declarations

c9f9131

Add TODO for structural review

5a4cedf

Add stacks todo, set writable type back to package

ae69449

uri -> identifier

0a4c251

Use RelationshipsGroup.Kind instead of creating a mirror type

86e40fa

Merge branch 'main' into 159600318/experimental-markdown-ouput

fcb3734

jrturton requested a review from d-ronnqvist December 17, 2025 12:11

d-ronnqvist reviewed Dec 17, 2025

View reviewed changes

Richard Turton added 2 commits December 17, 2025 16:58

Lower access level of manifest, remove children method

78361eb

Restrict package level declarations to those actually used at package…

71916e7

… level

d-ronnqvist approved these changes Dec 18, 2025

View reviewed changes

Richard Turton added 2 commits December 18, 2025 14:48

Merging main

562fcc4

Merge branch 'main' into 159600318/experimental-markdown-ouput

53def00

d-ronnqvist mentioned this pull request Dec 18, 2025

[DNM] Test toolchain build swiftlang/swift#84510

Draft

Merge branch 'main' into 159600318/experimental-markdown-ouput

8cfa398

d-ronnqvist merged commit 8a086b6 into swiftlang:main Dec 19, 2025
2 checks passed

jrturton mentioned this pull request Jan 5, 2026

[6.3] Experimental markdown output #1415

Merged

Experimental markdown output #1303

Experimental markdown output #1303

Uh oh!

Conversation

jrturton commented Sep 26, 2025

Summary

Dependencies

Testing

Checklist

Uh oh!

d-ronnqvist left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

d-ronnqvist Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

d-ronnqvist Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

jrturton Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

d-ronnqvist Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

d-ronnqvist Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

d-ronnqvist left a comment

Choose a reason for hiding this comment

Uh oh!

d-ronnqvist commented Dec 18, 2025

Uh oh!

d-ronnqvist commented Dec 18, 2025

Uh oh!

d-ronnqvist commented Dec 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

d-ronnqvist left a comment •

edited

Loading