[Feature Request] Add separate leaderboard tracks for "Thinking" and "Non-Thinking" evaluations

[Alice39s]

🥰 Feature Description

This request builds upon the potential bug regarding model aliases (Issue #156). While the powerful 'thinking' or 'reasoning' features (e.g. from OpenAI and Anthropic) provide significant performance boosts, they make direct comparisons difficult.

🧐 Proposed Solution

Therefore, I propose creating two distinct tracks on the leaderboard.

1. Base Capability Track (Non-Thinking):

   • All models run in their standard API mode without any reasoning enhancements.
   • Focuses on fundamental performance, measuring critical metrics like TTFT (Time to First Token) and TPS (Tokens per Second).

2. Peak Performance Track (Thinking-Enabled):

   • Explicitly enables reasoning features for all supported models (e.g., gpt-5-thinking, Claude's extended reasoning).
   • Showcases the upper limit of each model's capability in solving complex, multi-step problems.
   • Could report metrics like average reasoning tokens used.

[zjwu0522]

Thanks for the suggestion (and sorry again for the delayed response 🙏).

This feature request has already been implemented in PR #178 , In this update we:

• Switched to LiteLLM as the backend, which gives us a more flexible and unified way to handle different model providers.
• Introduced our own mcpmark-agent, a minimal but general-purpose agent we designed to standardize evaluation framework.

Most of the background discussion is in Issue #156 , so I’ll just refer to it here without repeating the details.