Merge pull request #64 from smith6jt-cop/claude/gifted-ptolemy-3GIL6

feat(mcp): migrate server to FastMCP with structured tool output

Author: smith6jt-cop

.github/workflows/mcp-eval.yml

@@ -0,0 +1,102 @@
+name: MCP Eval
+
+# Evaluates the KINTSUGI MCP tool surface.
+#   * Structural pass (token cost, schema quality, tool coverage): runs on every
+#     PR as a fast, deterministic gate (no API key, --strict fails on regressions
+#     such as a documented-but-unregistered tool).
+#   * Agentic tool-selection pass: runs on manual dispatch and weekly schedule
+#     only, and only when an ANTHROPIC_API_KEY secret is configured (it spends
+#     API tokens and is nondeterministic, so it never runs on PRs).
+on:
+  pull_request:
+    branches: [main, develop]
+    paths:
+      - "src/kintsugi/mcp/**"
+      - "evals/mcp/**"
+      - ".github/workflows/mcp-eval.yml"
+  workflow_dispatch:
+    inputs:
+      model:
+        description: "Model for the agentic tool-selection eval"
+        default: "claude-sonnet-4-6"
+  schedule:
+    - cron: "0 6 * * 1" # Mondays 06:00 UTC
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  mcp-eval:
+    name: MCP tool-surface eval
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libvips-dev
+
+      - name: Install Python dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]" anthropic
+
+      - name: Run MCP evaluation
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          EVAL_MODEL: ${{ github.event.inputs.model || 'claude-sonnet-4-6' }}
+        run: |
+          if [ -n "$ANTHROPIC_API_KEY" ] && [ "${{ github.event_name }}" != "pull_request" ]; then
+            python evals/mcp/harness.py --agentic --model "$EVAL_MODEL" --strict --json mcp_eval_report.json
+          else
+            if [ "${{ github.event_name }}" != "pull_request" ]; then
+              echo "::notice::ANTHROPIC_API_KEY secret not set — running the structural eval only."
+            fi
+            python evals/mcp/harness.py --strict --json mcp_eval_report.json
+          fi
+
+      - name: Write job summary
+        if: always()
+        run: |
+          python - <<'PY' >> "$GITHUB_STEP_SUMMARY"
+          import json, pathlib
+          p = pathlib.Path("mcp_eval_report.json")
+          if not p.exists():
+              print("No eval report produced.")
+              raise SystemExit(0)
+          r = json.loads(p.read_text())
+          s = r["structural"]
+          print("## MCP tool-surface eval\n")
+          print(f"- Tools: **{s['tool_count']}** (output schemas: "
+                f"{s['tools_with_output_schema']}/{s['tool_count']})")
+          print(f"- Tool-definition tokens (est.): **~{s['total_definition_tokens_est']:,}**")
+          print(f"- Params missing description: {s['params_missing_description_count']}")
+          gaps = s["task_coverage_gaps"]
+          print(f"- Task coverage gaps: {gaps if gaps else 'none'}")
+          a = r.get("agentic")
+          if a:
+              print(f"\n### Agentic tool-selection: **{a['passed']}/{a['total']} = "
+                    f"{a['accuracy']:.0%}** (model `{a['model']}`)\n")
+              fails = [x for x in a["results"] if not x["pass"]]
+              if fails:
+                  print("| task | chose | expected |")
+                  print("|------|-------|----------|")
+                  for x in fails:
+                      print(f"| {x['task']} | {x['chosen']} | {', '.join(x['expected'])} |")
+          else:
+              print("\n_Agentic pass skipped (PR event or no ANTHROPIC_API_KEY secret)._")
+          PY
+
+      - name: Upload report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: mcp-eval-report
+          path: mcp_eval_report.json

docs/skills/mcp-fastmcp-structured-output/.claude-plugin/plugin.json

@@ -0,0 +1,10 @@
+{
+  "name": "mcp-fastmcp-structured-output",
+  "version": "1.0.0",
+  "description": "Migrate a Python MCP server to FastMCP with structured tool output, preserve curated schemas, and evaluate the tool surface. Trigger when: (1) migrating kintsugi.mcp or any low-level mcp.server.Server to FastMCP, (2) adding structured output / outputSchema to MCP tools, (3) an MCP tool result fails output validation with 'None is not of type ...', (4) preserving per-parameter descriptions or enums under FastMCP's auto schemas, (5) building an MCP tool-selection evaluation harness or wiring it into CI.",
+  "author": {
+    "name": "KINTSUGI Team"
+  },
+  "skills": "./skills",
+  "repository": "https://github.com/smith6jt-cop/Skills_Registry"
+}

docs/skills/mcp-fastmcp-structured-output/PUBLISHING.md

@@ -0,0 +1,30 @@
+# Publishing this skill to the Skills Registry
+
+This is a drafted `/retrospective` skill, preserved in the KINTSUGI repo because
+the `Skills_Registry` is a **separate** repository
+(`github.com/smith6jt-cop/Skills_Registry`). It already passed the registry's
+validator locally (`OK mcp-fastmcp-structured-output`).
+
+To publish it:
+
+```bash
+# from the KINTSUGI repo root
+git submodule update --init Skills_Registry
+cp -r docs/skills/mcp-fastmcp-structured-output Skills_Registry/plugins/kintsugi/
+
+cd Skills_Registry
+git checkout main && git pull            # leave the detached submodule HEAD
+python scripts/generate_marketplace.py   # adds it to marketplace.json
+python scripts/validate_plugins.py       # expect: OK mcp-fastmcp-structured-output
+git add plugins/kintsugi/mcp-fastmcp-structured-output marketplace.json
+git commit -m "feat(kintsugi): add mcp-fastmcp-structured-output skill"
+git push
+```
+
+Optionally, bump the submodule pointer in KINTSUGI afterward:
+
+```bash
+cd ..                # back to KINTSUGI root
+git add Skills_Registry
+git commit -m "chore: bump Skills_Registry (add mcp-fastmcp-structured-output)"
+```

docs/skills/mcp-fastmcp-structured-output/skills/mcp-fastmcp-structured-output/SKILL.md

@@ -0,0 +1,87 @@
+---
+name: mcp-fastmcp-structured-output
+description: "Migrate a low-level MCP server to FastMCP with structured output, preserve curated input schemas, and evaluate the tool surface. Trigger: FastMCP migration, adding outputSchema/structuredContent, 'None is not of type' output-validation errors, building MCP tool-selection evals."
+author: KINTSUGI Team
+date: 2026-06-22
+---
+
+# MCP Server: FastMCP Migration + Structured Output + Evaluation
+
+## Experiment Overview
+| Item | Details |
+|------|---------|
+| **Date** | 2026-06-22 |
+| **Goal** | Migrate `kintsugi.mcp` from low-level `mcp.server.Server` to FastMCP, deliver structured tool output, and stand up an evaluation/improvement workflow |
+| **Environment** | `mcp==1.27.2` (floor `>=1.10`), Python 3.11/3.12, FastMCP (`mcp.server.fastmcp`) |
+| **Status** | Success |
+
+## Context
+The MCP server hand-wrote a `Tool(...)` list + an `elif` dispatch in `call_tool`,
+returned only `TextContent` (a JSON string), and exposed no output schemas. Goal:
+move to FastMCP so each tool returns machine-readable `structuredContent`, on a
+foundation that makes later resources/prompts/elicitation cheap.
+
+## What Worked (verified approach)
+
+1. **Floor `mcp>=1.10.0,<2`.** Structured output (`Tool.outputSchema`,
+   `CallToolResult.structuredContent`, FastMCP `structured_output`) and
+   elicitation **do not exist below 1.10** (the SDK release after spec 2025-06-18).
+
+2. **Register existing handlers; keep them FastMCP-agnostic.** A declarative
+   registry (`tool_specs.py`: name → module/attr + description + input schema)
+   plus, in `server.py`:
+   ```python
+   server = FastMCP("kintsugi")
+   handler = getattr(module, spec["attr"])           # plain async dict-returning fn
+   server.add_tool(handler, name=spec["name"],
+                   description=spec["description"], structured_output=True)
+   ```
+   Handlers stay plain `async` functions returning `dict[str, Any]`, so the unit
+   tests keep calling them directly.
+
+3. **Returns: `dict[str, Any]` + `structured_output=True`.** FastMCP then emits
+   the dict as `structuredContent` **and** a JSON text block, for both the
+   success and the in-band `{"error": ...}` branches — no per-tool models needed.
+
+4. **Preserve curated input schemas (descriptions + enums) by overriding** the
+   advertised schema after registration; validation still runs from the handler
+   signature:
+   ```python
+   # FastMCP's signature-derived schema drops per-param descriptions + enums.
+   server._tool_manager.get_tool(name).parameters = curated_input_schema  # wrap in try/except
+   ```
+
+5. **Evaluation harness** (`evals/mcp/`): a structural pass (tool-definition token
+   cost, schema-quality checks, confusable-group flags, task coverage) + an
+   agentic pass (`tool_choice="auto"`, score the model's first `tool_use`).
+   Tool **selection** needs only the schemas — **no image data or handler
+   execution** — so it runs with just an API key. Wire agentic into CI behind an
+   `ANTHROPIC_API_KEY` secret (manual + scheduled); gate PRs with the structural
+   `--strict` pass. The harness immediately found `analyze_weighted_subtraction`
+   documented + implemented but **never registered**.
+
+## What Failed (do not repeat)
+
+| Attempt | Result |
+|---------|--------|
+| Strict `TypedDict` returns (`total=False`) to get rich per-field outputSchema | **FAILED.** FastMCP materializes absent keys as `None`, then validates against the non-nullable field type → `Output validation error: None is not of type 'string'`, `structuredContent=None`, `isError=True` on **both** success and error returns |
+| `Optional`-field `TypedDict`s (to dodge the above) | Validates, but **pollutes every result** with `null` keys for absent fields (noisy/misleading) → use plain `dict[str, Any]` instead |
+| Rely on docstrings for per-parameter descriptions under FastMCP | **FAILED.** FastMCP 1.27 does **not** parse docstring args; descriptions are dropped. Use `Annotated[T, Field(description=...)]` or override `.parameters` |
+| Raise the `mcp` floor only in `pyproject.toml` extras | **Incomplete.** `src/kintsugi/deps.py` `OPTIONAL_GROUPS` is a parallel source of truth used by `kintsugi install`/`check` — update both (`claude` *and* `dev`) |
+
+## Verification
+- `pytest tests/test_mcp_tools.py tests/test_mcp_path_safety.py` — direct-call
+  tests stay green (handlers unchanged); add an in-memory client test
+  (`mcp.shared.memory.create_connected_server_and_client_session`) asserting
+  `outputSchema` present + `structuredContent` populated + text block + error
+  branch clean.
+- `python evals/mcp/harness.py --strict` — structural gate (fails on coverage
+  gaps / missing descriptions).
+- Preserve the `create_server()` `ImportError` contract by simulating
+  `MCP_AVAILABLE = False` (don't swallow the import in a bare `except`).
+
+## Key Files (reference implementation)
+- `src/kintsugi/mcp/server.py` — FastMCP wiring, `_advertise_schema` override
+- `src/kintsugi/mcp/tool_specs.py` — declarative registry (mcp-free, also feeds `kintsugi mcp tools`)
+- `evals/mcp/{tasks.py,harness.py,README.md}` — evaluation harness
+- `.github/workflows/mcp-eval.yml` — keyed agentic eval + structural PR gate

evals/mcp/README.md

@@ -0,0 +1,68 @@
+# KINTSUGI MCP — Evaluation Harness
+
+Measures the quality of the KINTSUGI MCP **tool surface** with realistic tasks
+instead of intuition, following the `mcp-builder` philosophy. Use it to drive and
+verify MCP improvements (tool consolidation, description quality, resources/prompts).
+
+## Why
+
+The server exposes 27 tools to Claude. Two questions decide whether that surface
+is *good*:
+
+1. **Is it cheap and clear?** (structural) — how many tokens do the tool
+   definitions cost, and are any tools/params under-described or confusable?
+2. **Does the model use it correctly?** (agentic) — given a realistic request,
+   does Claude pick the right tool?
+
+## Run it
+
+```bash
+# Structural only — no network, no API key. Runs anywhere mcp is installed.
+python evals/mcp/harness.py
+
+# Agentic tool-selection eval — needs an API key + the anthropic SDK.
+pip install anthropic
+ANTHROPIC_API_KEY=sk-... python evals/mcp/harness.py --agentic --model claude-sonnet-4-6
+
+# Save the full report (structural + agentic) as JSON.
+python evals/mcp/harness.py --agentic --json mcp_eval_report.json
+```
+
+The agentic mode measures **tool selection only** — it records the model's first
+`tool_use` per task and never executes the handler — so it needs **no image data
+or project**, just the live tool schemas.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `tasks.py` | 15 realistic natural-language tasks with `expected` acceptable tool(s). Includes deliberate confusable pairs (`denoise` vs `denoise_advanced`; the three background-removal tools; `assess_quality` vs `compute_snr`). |
+| `harness.py` | Builds the live FastMCP server, runs the structural metrics, and (opt-in) the agentic tool-selection loop. |
+
+Add a task by appending an `EvalTask` to `TASKS`; the structural pass will verify
+its `expected` tools exist on the server.
+
+## Findings (baseline, structural pass)
+
+Run against the post-migration server:
+
+- **27 tools**, **27/27 carry an output schema** (structured output is live).
+- **~3.9k tokens** of tool definitions injected on every call (~144/tool) — modest;
+  no urgent token pressure (compare: GitHub's MCP server ≈ 55k for 43 tools).
+- **0 params missing a description**, **0 tools with weak descriptions** — the
+  curated input schemas held up.
+- **First eval-driven fix:** the harness caught `analyze_weighted_subtraction` —
+  documented in `CLAUDE.md` and implemented, but **never registered**. Now wired up.
+
+### Open items the eval points at (future PRs)
+
+- **Confusable groups** worth either consolidating or giving sharper
+  "Use this when …" descriptions, to be confirmed by the agentic pass:
+  - background removal: `subtract_blank` / `gaussian_subtract` / `estimate_background`
+  - denoising: `denoise` / `denoise_advanced`
+  - parameter suggestion: `suggest_parameters` / `suggest_with_learning` / `get_learned_parameters`
+- **Resources/prompts** (deferred P1 items) — once added, extend the harness with
+  resource-read and prompt tasks.
+
+Run the agentic pass with a key to turn the "confusable groups" hypotheses into
+measured tool-selection accuracy, then act on the tools that get mis-selected.