Content negotiation for AI agents

[stichbury]

What's the problem this feature will solve?

This article describes how to reduce bloat when AI agents request documentation. It's basically looking at the Accept header and serving markdown when it's an option.

Lots of projects are now building markdown equivalents of their pages (we are in Vizro) and we'd be interested in whether this is something you'll implement as part of hosting.

Describe the solution you'd like

Not sure if it's a solution, but maybe worth considering negotiation where the docs confirm via a setting that they have a suitable markdown alternative published alongside their HTML?

Alternative solutions

Have you tried to workaround the problem? Or a different approach to solving this issue?

Additional context

Add any other context, links, etc. relevant to the feature request.

[humitos]

Hi, I've thought a little about this and found a few issues that we will hit here.

One of them is something we have struggled in the past already: "it's hard to match the URL to the corresponding source file (.md, .rst, etc)". This was easier to do when we only supported Sphinx since we knew how to convert a URL to the source file based on the extensions of the file and some Sphinx settings (dirhtml builder, etc). However, now that we support any documentation tool that outputs HTML, this is harder to do without explicitly asking the user for this pattern.

On the other hand, returning the source file without rendering won't work on all the scenarios since those source files may have .. include: Sphinx directive (to include other files in the current one) and that content won't be returned to the AI bot.

Finally, doesn't this feature request cover the same purpose than llms.txt file?

[stichbury]

Thanks @humitos!

doesn't this feature request cover the same purpose than llms.txt file?

To answer your last question first, not really. It's possible to build reverse-build markdown from built HTML and deliver up simple markdown for LLMs. We're doing this for Vizro using a plugin for MkDocs, which enables us to offer a page to download or view, that's simpler markdown than the source it was built from. See this page as an example of how we present it, and the reconstructed markdown here which we are hosting on our RTD project alongside every HTML page.

AIUI, the llms.txt file is just a kind of guide to where to find useful pages, and the LLM will then go get them if it respects llms.txt of course. If it doesn't start there, and starts crawling the site, but can use content negotiation, it's going to get a better set of content than HTML builds (in terms of size, and also possibly in readability -- we've noticed that our tabbed code examples cause some issues).

Does that sound reasonable? I don't know how you'd manage with sites that aren't offering post-built markdown publishing, but maybe you don't, and only negotiate to deliver the markdown if it's there, as it is in our site, for example?