DTD/VmService - add optional descriptions and schemas to service extensions

[jakemac53]

See the Model Context Protocol and specifically what it looks like to register a "tool" as a server.

These are very similar to DTD service extensions and VM service extensions, and we very likely want to expose those from our own MCP server as "tools". The main difference is that the MCP "tools" require a schema and a description. These are used by the AI to try and figure out when a user is trying to invoke a given tool, as well as how to invoke it properly.

Wait, what is this?

A quick tldr; here is that MCP is an open protocol for AI models to interact with other tools. We want to write an MCP server for Dart programmers, which can connect the AI models to their apps - exposing all the same functionality that devtools etc get today.

Use cases

Some example use cases are taking a screenshot to provide as context to the model, providing runtime and static errors as context, providing info about the selected widget as context, or even calling arbitrary service extensions registered from inside or outside the app.

For known extensions, we can provide our own description and schemas, so we can start there. But eventually it would be useful for the extensions to actually provide descriptions and schemas, so that we can support invoking arbitrary extensions in the future.

[jakemac53]

cc @bkonyi

[jakemac53]

We may also want this to be opt-in, so service extensions authors can choose whether they think their extension should be made available to the MCP server or not, via some sort of flag.

[bkonyi]

Is this also something we'd want for the core VM service / DTD APIs?

We may also want this to be opt-in, so service extensions authors can choose whether they think their extension should be made available to the MCP server or not, via some sort of flag.

Is there any reason they wouldn't want these extensions available? Would the absence of a schema and description be enough to indicate that an extension shouldn't be used?

[jakemac53]

Is this also something we'd want for the core VM service / DTD APIs?

I do think we should yes - even going so far as updating registerExtension.

It makes exposing these by default to AI models trivial, and it will work for any tool registering extensions without us needing to build specific support in for each one.

I also believe it would be a general improvement across the board - it makes anybody trying to invoke extensions have a much better understanding about how they are intended to be used, not just AIs.

Is there any reason they wouldn't want these extensions available? Would the absence of a schema and description be enough to indicate that an extension shouldn't be used?

I think these fields would be generally useful even if you don't want your extension exposed to AI models. So, and explicit flag might be useful. But, I am also not sure why you wouldn't want something to be exposed.

Maybe it should just be a "private" flag - indicating its only intended for use by the owners of the extension.

[bkonyi]

Is this also something we'd want for the core VM service / DTD APIs?

I do think we should yes - even going so far as updating registerExtension.

At least for the VM service APIs, this is going to be a large change since we'd need to provide an endpoint to list the APIs available along with their schemas and descriptions. The majority of the VM service endpoints are implemented in C++, so we'd probably have to use some macro magic to do this in a scalable way. The Dart implementations of the VM service, DWDS, and DDS would also need to be modified to support schemas for the endpoints defined in Dart code.

We'd also need to handle service extensions registered from the target Dart program, which can be individually registered on multiple isolates. These extensions all require an isolateId to parameter to be provided, which AI models will need to know how to retrieve.

Service extensions registered by VM service clients would probably be easier to handle since we don't need to worry about multiple instances being registered as they're associated with a single client. DTD services basically work in the same way.

Is there any reason they wouldn't want these extensions available? Would the absence of a schema and description be enough to indicate that an extension shouldn't be used?

I think these fields would be generally useful even if you don't want your extension exposed to AI models. So, and explicit flag might be useful. But, I am also not sure why you wouldn't want something to be exposed.

Maybe it should just be a "private" flag - indicating its only intended for use by the owners of the extension.

By convention, extensions with names starting with an underscore are considered private. I'd be hesitant to add a flag that would effectively convey the same information.

[jakemac53]

At least for the VM service APIs, this is going to be a large change since we'd need to provide an endpoint to list the APIs available along with their schemas and descriptions.

We don't need to do this - the existing notifications for when a service gets registered work just fine. MCP supports updating the list of available tools on the fly.

The Dart implementations of the VM service, DWDS, and DDS would also need to be modified to support schemas for the endpoints defined in Dart code.

This is why these fields should be optional, so we can add the data as desired.

We'd also need to handle service extensions registered from the target Dart program, which can be individually registered on multiple isolates. These extensions all require an isolateId to parameter to be provided, which AI models will need to know how to retrieve.

I did a recent experiment to prove they are actually already capable of exactly this type of interaction. I could put together a more specific demo if you like, but they are more than capable of handling multi-step workflows and asking for user input at various steps along the way.

It does require careful engineering of the descriptions to help it. "You can get an Isolate ID by calling getVM() and then selecting one of those isolates, you cannot just make up the ID" etc.

By convention, extensions with names starting with an underscore are considered private. I'd be hesitant to add a flag that would effectively convey the same information.

The problem with that is there are definitely things that follow this convention but we do want to expose (taking screenshots, for example).

[jakemac53]

We could potentially have a separate "AI description" field, but in general the idea is you just talk to it like a human. But you do have to tell them not to make stuff up for sure, lol. So that is a bit awkward.