Feature Request: Tool Calling Pattern Analysis and Optimization

[cwangg897]

Feature Request: Tool Calling Pattern Analysis and Optimization

Expected Behavior

I would like Spring AI to provide enhanced observability for tool calling patterns, including:

1. Tool Call Sequence Tracking: Track which tools are called in sequence during AI conversations
2. Pattern Analysis: Identify common tool calling patterns and combinations
3. Redundant Call Detection: Detect when the same tool is called multiple times with similar parameters
4. Performance Insights: Provide metrics on tool calling efficiency and optimization opportunities

Example API:

@Component
public class ToolCallPatternAnalyzer {
    
    // Get tool usage statistics
    public ToolUsageStatistics getToolUsageStats(TimeRange timeRange);
    
    // Get tool call sequences (which tools are called together)
    public List<ToolCallSequence> getToolSequences(String conversationId);
    
    // Get most common tool combinations
    public List<ToolCombination> getCommonCombinations();
    
    // Detect redundant calls
    public List<RedundantCall> detectRedundantCalls(String conversationId);
}

// Data structures
public class ToolUsageStatistics {
    private Map<String, Integer> toolCallCounts;        // tool name -> call count
    private Map<String, Double> toolSuccessRates;       // tool name -> success rate
    private Map<String, Duration> avgExecutionTimes;    // tool name -> avg time
    private List<ToolPair> frequentCombinations;        // tools used together
}

public class ToolCallSequence {
    private String conversationId;
    private List<ToolCall> sequence;
    private Duration totalTime;
}

public class ToolCombination {
    private List<String> toolNames;
    private int frequency;
    private double coOccurrenceRate;
}

// Usage example
@RestController
public class ToolAnalyticsController {
    
    @Autowired
    private ToolCallPatternAnalyzer analyzer;
    
    @GetMapping("/tool-usage-stats")
    public ToolUsageStatistics getUsageStats() {
        return analyzer.getToolUsageStats(TimeRange.LAST_7_DAYS);
    }
    
    @GetMapping("/tool-combinations")
    public List<ToolCombination> getCommonCombinations() {
        return analyzer.getCommonCombinations();
    }
}

Current Behavior

Spring AI currently provides basic observability through:

• spring.ai.tool observations for completion time and tracing
• spring.ai.chat.client observations for ChatClient calls
• Basic Micrometer metrics integration

However, there's no way to:

• Analyze tool calling patterns across conversations
• Identify optimization opportunities in tool usage
• Detect redundant or inefficient tool calls
• Understand how tools are used together in real scenarios

Context

How has this issue affected you?
When building AI applications with multiple tools, it's difficult to understand:

• Which tools are actually being used effectively
• Whether the AI is making redundant calls
• How to optimize tool configurations
• What tool combinations work best together

What are you trying to accomplish?

• Optimize AI application performance by understanding tool usage patterns
• Identify and eliminate redundant tool calls
• Improve tool design based on actual usage data
• Provide developers with insights into AI behavior for debugging and optimization

What other alternatives have you considered?

• Custom logging and analysis (requires significant manual work)
• External monitoring tools (lack Spring AI integration)
• Basic metrics collection (doesn't provide pattern insights)

Are you aware of any workarounds?
Currently, developers must implement custom tracking by:

• Adding manual logging to each tool
• Parsing logs to identify patterns
• Building custom analytics dashboards
• Manually correlating tool calls across conversations

This feature would provide a standardized, framework-integrated solution that all Spring AI users could benefit from.

[cwangg897]

@ilayaperumalg Hi, I’d like to work on this enhancement. Is it okay if I take this?

[ilayaperumalg]

@cwangg897 Thank you for your interest in contributing. Please allow us some time to review your proposal and we can engage some discussion before you start implementing it.