Extend Agents Support

[mohdali]

We need to extend agents support to include more model providers (e.g., Anthropic, Deepseek, local llm providers) and also enable tooling support by adding MCP support to agents work.

Refernece for MCP intergation into SK: https://devblogs.microsoft.com/semantic-kernel/integrating-model-context-protocol-tools-with-semantic-kernel-a-step-by-step-guide/

[pugafran]

Is there a viable way to access the Kernel object? I can’t overriding ChatCompletion, and I’ve tried overriding KernelFactory, but I haven’t had any success.

There’s actually a NuGet package that simplifies the whole connection process and the tedious MCP setup for Semantic Kernel, but it’s only accessible through the Kernel object itself.

Here’s the repository:
https://github.com/StefH/McpDotNet.Extensions.SemanticKernel

I urgently need to attach tools to the agents. If you see a way to achieve this, I’d really appreciate your help. And if you need anything from me, just let me know.

This can technically be done without the NuGet package at a lower level, but it’s a real pain, you have to dig deep into the internals.

[pugafran]

I'm going to try to implement, if everything goes well, could I open a pull request?

[sfmskywalker]

Depending on how and where you need access to the Kernel, you could add some extensibility points to Elsa.Agents.Core and submit a PR. This way, you can access the Kernel from your own app and modules.

[pugafran]

Could you take a look at the commit? The idea of what I want to do is there, but I'm lost when it comes to harmonize it with the rest of the project, and what you had already designed for plugins/tools: e4c1220

By the way, OpenAIChatCompletionProvider should allow specifying a custom endpoint (I'm using my local provider, but it could also be something like Grok or OpenRouter). Something like this:

I've created the NuGet packages using dotnet pack --configuration Release. I tried building with the system you use, called nuke, but it was throwing error, specifically, the project itself was locking a .NET 8-9 DLL while it was running.

When I run the solution with my changes in the nugget, it throws errors that I already commented (//) in the commit, but they still happen, so I’m probably missing something.

warn: Elsa.Agents.KernelFactory[0]
      Plugin ManoloMCP not found
warn: Elsa.Agents.KernelFactory[0]
      Plugin Echo (ManoloMCP) not found
warn: Elsa.Workflows.Middleware.Activities.ExceptionHandlingMiddleware[0]
      An exception was caught from a downstream middleware component
      System.ArgumentException: The System.Object instance doesn't implement any [KernelFunction]-attributed methods.
         at Microsoft.SemanticKernel.KernelPluginFactory.CreateFromObjectInternal[T](T target, String pluginName, JsonSerializerOptions jsonSerializerOptions, ILoggerFactory loggerFactory)
         at Microsoft.SemanticKernel.KernelPluginFactory.CreateFromObject(Object target, String pluginName, ILoggerFactory loggerFactory)
         at Microsoft.SemanticKernel.KernelExtensions.<>c__DisplayClass27_0.<AddFromObject>b__0(IServiceProvider serviceProvider)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
         at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
         at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier)
         at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope)
         at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
         at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetServices[T](IServiceProvider provider)
         at Microsoft.SemanticKernel.Kernel..ctor(IServiceProvider services, KernelPluginCollection plugins)
         at Microsoft.SemanticKernel.KernelExtensions.Build(IKernelBuilder builder)
         at Elsa.Agents.KernelFactory.CreateKernel(KernelConfig kernelConfig, AgentConfig agentConfig)
         at Elsa.Agents.KernelFactory.CreateKernel(KernelConfig kernelConfig, String agentName)
         at Elsa.Agents.AgentInvoker.InvokeAgentAsync(String agentName, IDictionary`2 input, CancellationToken cancellationToken)
         at Elsa.Agents.Activities.AgentActivity.ExecuteAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Activity.Elsa.Workflows.IActivity.ExecuteAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Middleware.Activities.DefaultActivityInvokerMiddleware.ExecuteActivityAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Runtime.Middleware.Activities.BackgroundActivityInvokerMiddleware.ExecuteActivityAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Middleware.Activities.DefaultActivityInvokerMiddleware.InvokeAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Runtime.Middleware.EvaluateLogPersistenceModesMiddleware.InvokeAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Middleware.Activities.NotificationPublishingMiddleware.InvokeAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Middleware.Activities.ExecutionLogMiddleware.InvokeAsync(ActivityExecutionContext context)
         at Elsa.Workflows.Middleware.Activities.ExceptionHandlingMiddleware.InvokeAsync(ActivityExecutionContext context)

That’s why I’d appreciate it if you could lend me a hand.

The AddMcpFunctionsFromSseServerAsync function belongs to a nugget that lets you easily connect an MCP server with Semantic Kernel through the Kernel itself. I’ve used it many times very straightforward and it works.

However, I’m having trouble integrating it into Elsa in general, because there’s a different philosophy/code structure around the plugin concept, and they seem to be clashing.

I’m also not really sure how to get it to harmonize with the way tools are selected in Elsa Studio.

I managed to make them appear and be selectable by overriding the GetPlugin method, but I don’t know how you’re linking that later to the Kernel.

I mean, I got it to receive the list via MCP and show the options, and you can select them, but I get lost in how to connect that selection to the Kernel in the way I want.

Sorry if I’m not explaining myself clearly—happy to discuss it further if needed.

[pugafran]

Ok, the error I mentioned earlier was due to the fact that, for some reason, the plugins for the agent remained stored in the database even after I removed them. I’ve solved that part, and I’m still iterating on it, but in any case, your input would be really helpful.

[pugafran]

Also, in the InjectingMCP function I'm calling the tools without using the SK nugget at first, and then I do use it, so I'm duplicating the plugin, which causes the error. I'm fixing that now.

First, I'll test with the nugget, and then I'll try the version I developed myself using pure Model Context Protocol to see if we can avoid depending on yet another nugget.