[WIP] Clean up PaperKit agent system to eliminate duplicates

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Original prompt

I tried to open that GitHub folder view, but GitHub served the “Uh oh! There was an error while loading” page in the web tool, so I couldn’t reliably see the directory listing from the browser side. 

The good news is: a Copilot “developer agent” can do a better job than any remote viewer anyway, because it can inspect the repo locally and follow references through code. Below is a detailed mission brief you can paste straight into Copilot Chat/Agent mode.

⸻

Copilot Agent Mission Brief: PaperKit .paperkit/ cleanup and de-duplication

Goal

Clean up PaperKit’s agent system so there is one canonical agent definition system that:
• matches the agent-schema.json rules
• is what the runtime actually uses to load/run agents
• has no duplicated “second system” lingering in parallel (or, if backward-compat is required, duplicates are replaced with a deliberate compatibility layer)

Non-negotiables
• Do not break existing PaperKit workflows/CLI behaviour.
• Do not change the schema unless you can prove the schema is wrong (it looks good).
• Preserve academic integrity constraints in agent instructions (don’t accidentally weaken them during migration).
• Every change must be backed by repo evidence (grep + file inspection), not assumptions.

⸻

Phase 0 — Safety + Baseline
1. Create a branch: chore/agent-system-unification.
2. Record a baseline:
• Run existing validation command(s) and save output to docs/dev/agent-audit/baseline.txt.
• Run whatever tests/lints exist (or make test / pytest / etc.), save output too.
3. Make a “do no harm” inventory snapshot:
• Save tree -a .paperkit > docs/dev/agent-audit/paperkit-tree.txt (or equivalent on Windows).

⸻

Phase 1 — Forensic Discovery (find the two systems and who uses what)

1.1 Locate the schema + validator
• Find the schema file in repo:
• rg -n "agent-schema.json|PaperKit Agent Definition Schema" -S .paperkit
• Find the validator script referenced by the manifest comment:
• rg -n "validate-agent-schema|agent-schema" -S .paperkit open-agent-system tools

1.2 Enumerate “agent definition” locations

Collect counts and lists:
• .paperkit/core/agents/*
• .paperkit/specialist/agents/*
• any _cfg agents: .paperkit/_cfg/agents/* (or .paperkit/_cfg/...)
Commands:
• find .paperkit -maxdepth 4 -type f | rg "/agents/"
• find .paperkit -type f -name ".yaml" | rg "agents"
• find .paperkit -type f -name ".md" | rg "agents"

Write a short report to docs/dev/agent-audit/inventory.md listing:
• all agent definition files (path + extension)
• which look canonical (match schema path pattern vs not)
• any “manifest” index files (like your YAML manifest)

1.3 Determine runtime loading path (the key question)

Search code for references to:
• .paperkit/_cfg/agents
• .paperkit/core/agents
• .paperkit/specialist/agents
• “manifest”, “agent registry”, “agent loader”, “open-agent-system”
Commands:
• rg -n "_cfg/agents|core/agents|specialist/agents|agent manifest|agent loader|load.*agent" -S .

Identify:
• The code path(s) that load agents at runtime
• The expected file format (MD? YAML? front matter?)
• Whether the runtime uses the schema validator or bypasses it

Document findings in docs/dev/agent-audit/runtime-paths.md with:
• file + function names
• what directories are scanned
• how agent metadata is parsed

⸻

Phase 2 — Decide Canonical System (based on evidence)

Decision rule

Prefer the system that:
• matches agent-schema.json without hacks
• is already aligned with the repo structure (.paperkit/core and .paperkit/specialist)
• is easiest to keep stable long-term

Likely outcome (verify in repo):
• Canonical agent definitions = .paperkit/{core|specialist}/agents/.md
• _cfg/agents/.yaml is legacy OR internal config that got mistaken for canonical

But do not assume — prove by runtime loader.

Write the decision and rationale to docs/dev/agent-audit/decision.md.

⸻

Phase 3 — Unification Plan (choose one of these and implement)

Option A (preferred): Make MD agents canonical, retire YAML agents

Do this if runtime already supports MD agents or can easily be made to.

Steps:
1. Ensure every active agent in the manifest exists as a .paperkit/{core|specialist}/agents/.md
2. If YAML-only agents exist, migrate them to .md:
• Convert YAML fields → schema fields:
• name, displayName, title, icon, module, path
• identity.role, identity.description, identity.communicationStyle
• capabilities, constraints, principles, examplePrompts
• Put the operational prompt/instructions into the Markdown body.
3. Update the runtime loader to scan only:
• .paperkit/core/agents/.md
• .paperkit/specialist/agents/.md
4. Add a compatibility shim ONLY if needed:
• If parts of PaperKit still expect _cfg/agents/*.yaml, replace them with either:
• symlinks (if supported), or
• a generated file step, or
• a tiny adapter that reads YAML and emits in-memory schema objects
• BUT: the adapter should be explicitly marked “legacy” and have a depreca...

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.