Tools never been invoked

[rungekutta1942]

Hi, thanks for this amazing initiative!

I've had fun creating an Agent to make recommendations from certain well-know merchants' new releases. It works relatively well but I can't for the life of me figure out how to wire Tools into the agent in order to improve the accuracy and timeliness of the data from which it makes its recommendation decisions.

The LLMs run in Ollama via a remote URL connection. I've created Tools locally as part of the project - firecrawl, wikipedia and brave. They work fine stand-alone and I've wired them up with Tool annotations, e.g.

    @Tool(name = "firecrawl_scrape", description = "Fetch a web page via Firecrawl and return markdown (default) or other formats like html.")
    public String firecrawlScrape(
            @ToolParam(required = true, description = "The absolute URL to fetch") String url,
            @ToolParam(required = false, description = "List of formats: markdown|html|rawHtml|summary|links|images|screenshot|json") List<String> formats) {
        [...]
    }

The agent is given the tools in its Action method:

[...]
return ai
        .withLlm(LlmOptions.withDefaultLlm())
        .withToolObject(firecrawlTools)
        .withToolObject(braveTools)
        .withToolObject(wikipediaTools)
        .createObject(prompt, DrinkList.class);

and this can be seen in logs at runtime:

14:04:04.637 [main] DEBUG ToolConsumer - xyz.wineagent.agent.WineAgent.showSummarizedContent-xyz.wineagent.agent.DrinkList resolved 7 tools from 7 tools and 0 tool groups: [brave_search, brave_search_simple, firecrawl_scrape, firecrawl_search, wikipedia_extract, wikipedia_search, wikipedia_summary]
14:04:04.637 [main] INFO  Embabel - [relaxed_clarke] (xyz.wineagent.agent.WineAgent.showSummarizedContent-xyz.wineagent.agent.DrinkList) using LLM qwen3:14b, creating DrinkList: LlmOptions(modelSelectionCriteria=DefaultModelSelectionCriteria, model=null, temperature=null, frequencyPenalty=null, maxTokens=null, presencePenalty=null, topK=null, topP=null, thinking=null, timeout=null)
14:04:04.637 [main] DEBUG ChatClientLlmOperations - Expanded toolCallbacks from [brave_search, brave_search_simple, firecrawl_scrape, firecrawl_search, wikipedia_extract, wikipedia_search, wikipedia_summary]: [brave_search, brave_search_simple, firecrawl_scrape, firecrawl_search, wikipedia_extract, wikipedia_search, wikipedia_summary]

Finally, I am being explicit in the prompt to the LLM that it needs to use the Tools:

[...]
        TOOL USAGE REQUIREMENTS:
        1. **First**, review the provided drink JSON with your internal knowledge.
        2. **Then**, you MUST use external tools to verify and enrich the information:
        - Use the **web search tool** to find the most recent details about the drink, maker, or vintage.
        - Use the **wikipedia tool** to gather background on the maker, grape, or region.
[...]

BUT despite all this, I've never seen a Tool actually being called, ever. I've tested with Qwen3, gpt-oss and Gemma3.

Any ideas what might be missing?

[reuschling]

I also currently check tool invocation, and managed to run this test Tool method:

package com.embabel.template.chris;

import org.springframework.ai.tool.annotation.Tool;
import lombok.experimental.Accessors;
import lombok.extern.slf4j.Slf4j;

@Slf4j
public class TestToolObj
{
    @Tool(description = "Use this tool to answer who is Mayor of Kaiserslautern.")
    public String getMayorOfKaiserslautern()
    {
        log.info("Get Mayor of Kaiserslautern called.");

        return "The Mayor of Kaiserslautern is Hitzli Putzli.";
    }
}

package com.embabel.template.chris;

import com.embabel.agent.api.annotation.AchievesGoal;
import com.embabel.agent.api.annotation.Action;
import com.embabel.agent.api.annotation.Agent;
import com.embabel.agent.api.common.OperationContext;
import com.embabel.agent.api.common.PromptRunner;
import com.embabel.agent.domain.io.UserInput;


@Agent(description = "Answers questions about Kaiserslautern and meeting minutes based on OpenSearch data")
public class RatsInfoAgentWithToolObj
{
    @AchievesGoal(description = "Questions about Kaiserslautern and meeting minutes including details of a complete result list have been answered")
    @Action
    String searchMeetingMinutes(UserInput userInput, OperationContext context)
    {
        TestToolObj testToolObj = new TestToolObj();

        PromptRunner promptRunner = context.ai().withDefaultLlm();
        promptRunner = promptRunner.withToolObject(testToolObj);

        // Prompt inside the embabel shell: x "Who is Mayor of Kaiserslautern."
        return promptRunner.createObject(userInput.getContent().trim(), String.class);
    }
}

package com.embabel.template;

import com.embabel.agent.config.annotation.EnableAgents;
import com.embabel.agent.config.annotation.LoggingThemes;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;


@SpringBootApplication
@EnableAgents(loggingTheme = "")
class ProjectNameApplication {
    public static void main(String[] args) {
        SpringApplication.run(ProjectNameApplication.class, args);
    }
}

I run it with a olllama server and a gpt-oss120b model. The tool method log message "The Mayor of Kaiserslautern is Hitzli Putzli." Is shown on every call of x "Who is Mayor of Kaiserslautern." inside the embabel shell. Nevertheless, the final result is highly nondeterministic. Independent of the right tool method call, the answer is most of the time not 'Hitzli Putzli'. Here are the logs:

13:08:34.011 [main] INFO  Embabel - [brave_kilby] executing action com.embabel.template.chris.RatsInfoAgentWithToolObj.searchMeetingMinutes
13:08:34.024 [main] INFO  Embabel - [brave_kilby] (com.embabel.template.chris.RatsInfoAgentWithToolObj.searchMeetingMinutes-java.lang.String) using LLM gpt-oss:120b, creating String: LlmOptions(modelSelectionCriteria=DefaultModelSelectionCriteria, model=null, temperature=null, frequencyPenalty=null, maxTokens=null, presencePenalty=null, topK=null, topP=null, thinking=null, timeout=null)
13:08:34.708 [main] INFO  Embabel - [brave_kilby] (searchMeetingMinutes) calling tool getMayorOfKaiserslautern({})
13:08:34.710 [main] INFO  TestToolObj - Get Mayor of Kaiserslautern called.
13:08:34.715 [main] INFO  Embabel - [brave_kilby] (searchMeetingMinutes) tool getMayorOfKaiserslautern returned "The Mayor of Kaiserslautern is Hitzli Putzli." in 2ms with payload {}
13:08:35.177 [main] INFO  Embabel - [brave_kilby] (com.embabel.template.chris.RatsInfoAgentWithToolObj.searchMeetingMinutes-java.lang.String) received LLM response of type String from DefaultModelSelectionCriteria in 1 seconds
13:08:35.178 [main] INFO  Embabel - [brave_kilby] object bound it:String
13:08:35.179 [main] INFO  Embabel - [brave_kilby] executed action com.embabel.template.chris.RatsInfoAgentWithToolObj.searchMeetingMinutes in PT1.165S
13:08:35.183 [main] INFO  Embabel - [brave_kilby] ready to plan from:
    hasRun_com.embabel.template.chris.RatsInfoAgentWithToolObj.searchMeetingMinutes: TRUE
    it:com.embabel.agent.domain.io.UserInput: TRUE
    it:java.lang.String: TRUE
13:08:35.189 [main] INFO  Embabel - [brave_kilby] goal com.embabel.template.chris.RatsInfoAgentWithToolObj.searchMeetingMinutes achieved in PT1.185747795S
13:08:35.191 [main] INFO  Embabel - [brave_kilby] completed in PT1.187747241S

InMemoryBlackboard: id=695f5ce0-dea6-48a3-a10f-584d8c1cf087
map:
  it=The current mayor of Kaiserslautern is **Stefan Kühn**. He has been serving in this role since 2023.
entries:
  UserInput(content=Who is Mayor of Kaiserslautern., timestamp=2025-10-31T12:08:32.156153390Z), The current mayor of Kaiserslautern is **Stefan Kühn**. He has been serving in this role since 2023.

You asked: UserInput(content=Who is Mayor of Kaiserslautern., timestamp=2025-10-31T12:08:32.156153390Z)

"The current mayor of Kaiserslautern is **Stefan Kühn**. He has been serving in this role since 2023."

LLMs used: [gpt-oss:120b] across 1 calls
Prompt tokens: 345,
Completion tokens: 102
Cost: $0.0000

Tool usage:
        ToolStats(name=getMayorOfKaiserslautern, calls=1, avgResponseTime=2 ms, failures=0)

Maybe you have a similar situation? Have you tried to insert a log message inside your tool method?

[rungekutta1942]

Thanks. Good to hear it can be working without having to create ToolGroups etc. I presume you don’t run an MCP server either? Like you, I’ve added logging in the tools but I can see unambiguously that they are never being called.

Perhaps I’m overwhelming the LLM, I’ve got a fairly long prompt.

Will try to break it down a little.

Thanks again.

[rungekutta1942]

I made some progress and now have gpt-oss calling tools. Steps taken:

• increased context window length in Ollama, it was set at default 4096 and this proved insufficient. With 16k context length I got much better results overall including LLMs actually following instructions.
• simplified my prompt and used .withPromptElements(...) instead to pass on additional information when calling the LLM, thus leaving more in the hands on Embabel

Works quite well now!

[johnsonr]

While local models are really important, it can be useful to try a larger (commercial) model at least once when debugging to see whether or not it's the model fault. Eg if you try with GPT 4.1 and it's not calling your tool there's definitely a problem.