sempkg_agent: roadmap to an effective human-facing knowledge assistant over curated sembundles

[willem445]

Make sempkg_agent an effective human-facing knowledge assistant over curated sembundles

Goal

We want to curate a knowledge base of sembundles and put sempkg_agent in front of it so humans can ask questions and get useful, cited summaries. The current prototype (src/sempkg_agent/) already has the right bones — LangGraph ReAct loop over the sempkg MCP tools, a structured AgentAnswer contract, deterministic citation verification, a recursion-limit synthesis fallback, one warm MCP session, and three transports (A2A / REST+chat UI / MCP). This issue is the plan to take it from prototype to a genuinely useful front end.

The work is organized in phases; each phase is independently shippable. File references are to src/sempkg_agent/sempkg_agent/ unless noted.

---

Phase 1 — Retrieval effectiveness (biggest lever)

The agent's answer quality is bounded by retrieval, and today retrieval is throttled by a self-imposed constraint: the prompt hard-limits the agent to two query calls total because query can take ~100s (prompts.py, "Efficiency & stopping rules"). That trades answer quality for cost on every request. Fix the cost structure instead of rationing:

• Expose the cheap retrieval tiers. The sempkg MCP server also exposes search_symbols, search_code, search_docs, get_context, and docs_metadata, but MCPSettings.allowed_tools (config.py) excludes them. Add them, and rewrite the retrieval policy in both prompts as tiered: start with cheap lexical/symbol search (search_*), escalate to the expensive reranked query only when lexical search doesn't resolve the question, then drill in with read_*. This should let most questions answer well in seconds without ever paying for a reranked query, and lets hard questions use query more than twice.
• Inject the corpus into the system prompt. The agent currently burns a tool round on list_packages and the LLM often guesses package names. KnowledgeAgent.list_installed() (agent.py) already caches the package list for the UI — render it (names + versions + one-line descriptions) into the system prompt at graph build time so the agent always knows what's installed. Invalidate the cache (see Phase 4 refresh work).
• Corpus cards for curated bundles. For a curated KB, each bundle should carry (or the deployment should configure) a short description: what the package is, what kinds of questions it answers. Feed these into the system prompt. This is the lightweight version of the plan doc's "package router" (docs/plan-knowledge-agent-server.md §4) and materially improves routing for multi-bundle corpora.
• Hard version filtering. Version scope is currently a "soft hint" in the user message (prompts.py:build_user_message); nothing stops retrieval from mixing versions when several are installed. Add a real version parameter to the core sempkg query/read tools (Rust side) and thread it through — this is the cross-cutting item already flagged in deploy/README.md and the agent README.
• Encourage parallel tool calls. LangGraph ReAct executes parallel tool calls if the model emits them; the prompt should tell the model to batch independent read_* calls in one turn to cut wall-clock latency.

Phase 2 — Answer quality & trust

• Verify line ranges, not just snippets. verify.py checks that ≥60% of a snippet's significant lines appear in the retrieved evidence — good, but a finding can have a verbatim snippet with wrong file/start_line/end_line and still show ✓. Add a stricter (config-gated) pass that re-reads each cited range via read_code and confirms the snippet matches those exact lines; downgrade the badge if not.
• Unverified-finding policy. Today verified=False findings are just badged ⚠ in the UI. Add a configurable policy: flag (current), drop, or retry (one corrective round-trip that tells the model which citations failed and asks it to fix or remove them). Default retry for the human persona — a human-facing product shouldn't ship visibly broken citations.
• Answer↔findings coverage check. In human mode the prose answer is the product, but only findings are verified. Add a cheap heuristic check that the prose's inline references (package@version path:lines) all correspond to actual findings, and flag prose references that cite nothing.
• Stream the final answer tokens. astream() (agent.py) streams reasoning/tool events but the final answer arrives as one block after a potentially minutes-long run. Stream the synthesis tokens (LangChain on_llm_new_token for the final structured call, or emit the answer field progressively) so the human sees the answer being written — this is the single biggest perceived-latency win in the chat UI.

Phase 3 — Human UX for a curated knowledge base

• "What do I know?" surface. A landing state in the chat UI (static/index.html) that shows the curated corpus: installed bundles, versions, descriptions, and 2–3 starter questions per bundle (configurable, or generated once at startup and cached). First-time users currently face an empty prompt box with no idea what's askable.
• Deep links from citations. Findings carry package/version/file/start_line/end_line. If the bundle manifest carries a source URL (repo + tag), render source cards as links to the exact lines (e.g. GitHub blob/<tag>/<path>#L10-L42). Requires plumbing a source_url through sembundle metadata → MCP list_packages/finding enrichment.
• Feedback capture. Add 👍/👎 + optional comment per answer (POST /v1/feedback), persisted (SQLite alongside SEMPKG_AGENT_STATE_DB) with session id, question, answer, findings, model, latency, and token usage. This is the raw material for the Phase 5 eval set and for judging the curated corpus itself (what do people ask that we can't answer?).
• Session history in the UI. State persistence exists (SEMPKG_AGENT_STATE_DB + SQLite checkpointer) but the UI has no conversation list/resume. Add minimal history (list past sessions, reopen one). Make the state DB on-by-default in the Docker/compose deployment.
• Answer-time metadata. Show which model answered, which packages/versions were searched, elapsed time, and token cost in the UI footer of each answer — builds trust and helps users calibrate model choice in the dropdown.

Phase 4 — Operations, scale, cost

• MCP session resilience. SempkgToolProvider (mcp_tools.py) holds exactly one warm stdio session; if the sempkg mcp subprocess dies, every request fails until the server restarts, and concurrent requests serialize on one stdio pipe. Add: (a) liveness check + automatic session restart with backoff, (b) an optional small session pool (N warm subprocesses) for concurrency, (c) surface MCP health in /healthz.
• Corpus refresh without restart. Bundles update (the deploy pipeline publishes a rolling latest), but the agent process loads tools once and _installed_text is cached forever. Add an authenticated POST /v1/admin/refresh that re-runs sempkg sync, restarts the MCP session, and invalidates the installed-packages/prompt caches.
• Caching. The corpus is static between refreshes, so repeated questions are highly cacheable. Add (a) a tool-result cache for query keyed by (normalized args, corpus digest), and (b) an optional whole-answer cache keyed by (normalized question, package, version, corpus digest). Invalidate on refresh.
• Token usage + budgets. Report per-request token usage/cost in AskResponse and stream events; add a per-session budget guard (config) so a runaway conversation can't burn unbounded OpenRouter spend. max_iterations guards rounds but not tokens.
• Auth parity across transports. Bearer auth exists only on REST (rest.py); A2A and the MCP mount are open. Apply the same shared-secret check to all three (even if production still adds a gateway).
• Model catalog hygiene. Validate DEFAULT_CATALOG/SEMPKG_AGENT_MODEL_CATALOG slugs against OpenRouter /models at startup and warn on unknown ids; refresh the default SEMPKG_AGENT_MODEL (currently anthropic/claude-3.5-sonnet) to a current-generation default.

Phase 5 — Evaluation harness

For a curated KB the operator needs to know the agent actually answers well over their corpus:

• Golden question set. A YAML/JSON eval format: question → expected package(s), expected files/symbols, optionally key facts. Seed it with questions per curated bundle; grow it from Phase 3 feedback.
• Eval runner. sempkg-agent eval <dataset> that runs the set against a live workspace and scores: retrieval hit rate (expected file/symbol appears in findings), citation verification rate, "I don't know" correctness (questions deliberately outside the corpus must be refused, not hallucinated), latency, and cost per question. Emit a markdown report.
• CI hook. Run the eval on a small fixture corpus in CI (mock LLM for the offline path, opt-in real-LLM job) so retrieval/prompt regressions are caught before deploy.

---

Quick fixes (do anytime)

• list_installed() caches "" permanently on a transient failure (agent.py) — retry on next call instead.
• _make_config reads self._settings.agent.max_iterations but default_query_limit (config.py) is defined and never used — wire it into the query tool default or remove it.
• The A2A/MCP transports should also surface the rendered markdown (render.py) consistently — audit parity of the three transports' payloads.

Out of scope (tracked in docs/plan-knowledge-agent-server.md)

Hot/cold bundle tiering, the hosted multi-tenant service, embedding-service extraction, and full package routing at thousand-package scale remain in the hosted-service plan (§4–§8). This issue is about making the local/self-hosted agent excellent over a curated, moderate-size corpus.