Overarching Training Recommendations for AI Ontology Agents

[dragon-ai-agent]

Executive Summary

FAO:

• GO (@balhoff @kltm) - Dragon-AI-Agent GO Ontology PR Analysis geneontology/go-ontology#31256
• MONDO (@matentzn @sabrinatoro ) - Dragon-AI-Agent PR Analysis: Mondo Repository monarch-initiative/mondo#9846
• CL (@dosumis, @Caroline-99) - Copilot-SWE-Agent Cell Ontology PR Analysis obophenotype/cell-ontology#3546
• UBERON (@dosumis @aleixpuigb ) - Uberon PRs by dragon-ai-agent - Summary obophenotype/uberon#3642
• EFO (@zoependlington @aleixpuigb @jamesamcl) - Copilot (app/copilot-swe-agent) EFO Ontology PR Analysis EBISPOT/efo#2592

(ontology specific reports forthcoming)

This document synthesizes failure analyses across 5 ontology projects (GO, MONDO, Uberon, CL, EFO) with 260 total PRs from AI agents (dragon-ai-agent, copilot-swe-agent).

Project	Total PRs	Merge Rate	First-Try Rate	Primary Failure Mode
GO	68	77.6%	65.0%	Duplicate PRs, Obsoletion errors
MONDO	26	44.0%	26.7%	Duplicate PRs (7 PRs for 1 term!)
Uberon	14	100%	37.5%	Attribution errors
CL	101	95.3%	5.6%	Every PR needs modifications!
EFO	51	41.7%	~0%	Duplicate PRs (13 PRs for 1 issue!)

CRITICAL FINDING: The first-try success rate reveals a hidden problem. CL looks great with 95.3% eventual merge rate, but only 5.6% of PRs are accepted without modifications. This means reviewers are doing extensive hand-holding.

Key Findings:

1. First-try success is the real metric - Merge rate alone is misleading
2. Eliminating duplicate PR creation accounts for 82% of EFO failures
3. All ontologies have low first-try rates - Training on patterns is critical

---

Cross-Project Failure Mode Analysis

Tier 1: Critical (>40% of failures in multiple projects)

1. DUPLICATE PR CREATION

Prevalence: GO (4 PRs), MONDO (9 PRs), EFO (23 PRs)

The Pattern:

• Agent creates new PR instead of updating existing branch
• Reviewers explicitly request updates to existing PRs
• Agent ignores request and creates new PR anyway
• Process repeats, generating chains of 7-13 PRs for single issues

Worst Case: MONDO argyrophilic grain disease saga

• 7 PRs created over 2 months for 1 term
• Reviewer said "update this PR" multiple times
• Agent kept creating new PRs
• Term STILL not added after all this effort

Training Rule:

ABSOLUTE RULE: ONE PR PER ISSUE

When asked to revise a PR:
1. ALWAYS push commits to the SAME branch
2. NEVER create a new PR unless explicitly told to abandon
3. Check for existing PRs before starting work:
   gh pr list --search "[issue keywords]" --state all

If you've created more than 1 PR for an issue:
- STOP immediately
- Ask for guidance
- Something is wrong with your approach

---

Tier 2: High Impact (Caused multiple failures per project)

2. CONTENT-OBJECTIVE MISMATCH (and recovery failure)

Prevalence: GO (1 PR with cascading effects)

The Pattern:

• PR title/description says one thing
• Actual diff shows completely different changes
• Example: GO PR 30730 titled "Obsolete 7 terms" but diff just added 1 metadata line

Important nuance: In the GO case, the agent actually self-corrected in PR 30731 (which had correct content). However, leaving the wrong PR (30730) open alongside the correct one (30731) created confusion. A human curator ended up doing the work themselves in PR 30736, bypassing both agent PRs.

Lesson: When you create a PR with wrong content, close it immediately with explanation before creating the corrected version.

Training Rule:

PRE-SUBMISSION VERIFICATION:

Before creating ANY PR:
1. Run `git diff` and READ the output
2. Verify EVERY change relates to the stated objective
3. Confirm PR title matches actual changes
4. If diff doesn't match objective - DO NOT SUBMIT

Checklist:
[ ] Diff only contains changes for this task
[ ] No extraneous changes included
[ ] PR title accurately describes the diff
[ ] Commit messages match actual changes

3. ONTOLOGY-SPECIFIC CONVENTION ERRORS

GO Obsoletion:

• Missing "obsolete " prefix on name
• Missing "OBSOLETE. " prefix on definition
• Failing to remove ALL is_a relationships
• Wrong replacement tags (replaced_by vs consider)

MONDO Annotations:

• Grouping multiple PMIDs in single annotation
• Wrong: synonym: "AGD" EXACT [PMID:1, PMID:2, PMID:3]
• Correct: Each PMID as separate annotation
• PMC→PMID conversion errors

Training Rule:

ONTOLOGY-SPECIFIC PATTERNS:

Before modifying ANY ontology:
1. Find 2-3 similar recent changes (merged PRs)
2. Copy the EXACT pattern used
3. If pattern is unclear, ASK for an example

For GO obsoletion specifically - ALL of:
[ ] Name prefixed with "obsolete "
[ ] Definition prefixed with "OBSOLETE. "
[ ] ALL is_a relationships removed
[ ] ALL intersection_of removed
[ ] is_obsolete: true added
[ ] replaced_by OR consider added
[ ] term_tracker_item added

For MONDO annotations:
[ ] Each PMID as individual annotation
[ ] PMIDs verified via PubMed (not guessed)
[ ] Source annotations use OMIM/PMID, not GitHub URLs

4. HIGH-IMPACT CHANGES WITHOUT DISCUSSION

Prevalence: GO (3 PRs), CL (1 PR)

The Pattern:

• Agent makes large-scale changes without curator approval
• Obsoleting high-level terms affects hundreds of children
• Adding 7+ new terms without discussion

Training Rule:

DISCUSSION-FIRST CHANGES:

These changes REQUIRE discussion before implementing:
- Obsoleting any term with >10 children
- Adding 3+ new terms
- Changing logical definitions of widely-used terms
- Any change that triggers "large scale" bot warnings

Process:
1. Comment on issue: "I plan to [X]. This affects [N] terms."
2. Wait for curator approval
3. Only then create PR

Query to check impact:
grep "is_a: [TERM_ID]" ontology-file.obo | wc -l

---

Tier 3: Medium Impact (Caused some failures)

5. ATTRIBUTION ERRORS

Prevalence: Uberon (1 PR)

The Pattern:

• Wrong contributor name/ORCID used
• Confused similar names during lookup
• Sarah vs Stan Laulederkind

Training Rule:

ATTRIBUTION ACCURACY:

1. Extract contributor info DIRECTLY from issue text
2. Use the EXACT name provided, not similar matches
3. If ORCID lookup returns different name - ASK for clarification
4. When in doubt, quote the issue: "Issue says submitted by [X]"

6. EXTERNAL ACCESS LIMITATIONS

Prevalence: CL (1 PR), EFO (multiple)

The Pattern:

• Agent can't access external resources (OLS, web APIs)
• Instead of reporting limitation, creates elaborate workarounds
• Workarounds don't solve the problem

Training Rule:

CAPABILITY BOUNDARIES:

When a task requires resources you cannot access:
1. IMMEDIATELY report the limitation
2. DO NOT create workaround scripts
3. Suggest alternatives:
   - "Please provide data as CSV"
   - "This requires human curator"
   - "Use web-browsing tool locally"

Example response:
"I cannot access [X] due to sandboxing restrictions.
This task cannot be completed as described.
Alternative: [concrete suggestion]"

7. SCOPE CREEP

Prevalence: GO (1 PR), CL (1 PR)

The Pattern:

• Agent makes "helpful" changes beyond task scope
• Modifies Makefiles when not requested
• Removes metadata from unrelated terms

Training Rule:

MINIMAL CHANGES ONLY:

1. ONLY modify files explicitly mentioned in issue
2. NEVER "improve" or "clean up" unrelated code
3. If no changes needed, report "No changes needed"
4. If you notice other issues:
   - Note in PR description
   - Create SEPARATE issue
   - Do NOT fix in this PR

8. PMID/IDENTIFIER VALIDATION ERRORS

Prevalence: Uberon (1 PR), MONDO (multiple)

The Pattern:

• Agent incorrectly rejects valid identifiers
• Claims PMIDs are "out of range"
• Wrong PMC→PMID conversions

Training Rule:

EXTERNAL IDENTIFIER HANDLING:

DO validate:
- Format correctness (PMID:12345 pattern)
- Basic syntax

DO NOT validate:
- Numeric ranges (PMIDs have no upper limit)
- "Existence" without API access

When maintainers provide references, trust them.
If concerned, proceed and note concern - don't block.

For PMC→PMID conversion:
- Use PubMed directly if possible
- NEVER guess PMID numbers
- If uncertain, ask for the correct PMID

---

Project-Specific Insights

GO: Obsoletion Expertise Required

GO has strict obsoletion conventions. Agent success rate dropped specifically on obsoletion tasks. Recommendation: Create obsoletion-specific training examples or use specialized prompting for obsoletion issues.

MONDO: Learning Loop Broken

The argyrophilic grain disease saga shows the agent failed to learn from repeated failures. Each new PR repeated the same mistakes. Recommendation: Implement feedback incorporation - before creating new PR, review why last one failed.

Uberon: 100% Eventually Merge, But Low First-Try

All Uberon PRs eventually merged, but only 42.9% on first try. The failures were minor (attribution, IDs). This suggests the agent is close to optimal for this ontology - small refinements would help.

CL: Misleading "Success" - 95.3% Merge but 5.6% First-Try

CL is actually a cautionary tale, not a success story.

The 95.3% merge rate masks the reality:

• Only 5.6% of PRs are accepted without modifications
• 94.4% of merged PRs required reviewer corrections
• Reviewers are essentially fixing every submission

This means:

• The agent is creating drafts, not finished work
• Curator time is being spent on corrections, not reviews
• The "high success rate" reflects reviewer patience, not agent quality

Recommendation: CL needs the MOST training improvement, not the least. Focus on:

• Template-based submissions copying exact patterns from recent merged PRs
• Pre-submission validation against known good examples
• Understanding why nearly every PR needs modification

EFO: Systematic Failure on Complex Tasks

EFO had the lowest success rate (41.7%), primarily due to duplicate PRs on complex issues. The agent struggled with:

• Issues requiring external tool access
• Issues requiring iterative exploration

Recommendation: For complex EFO issues, require human co-pilot or provide data upfront.

---

Universal Training Checklist

Before Starting ANY Task

[ ] Read the full issue including all comments
[ ] Search for existing PRs: gh pr list --search "[keywords]" --state all
[ ] Check if anyone else is working on it
[ ] Search for similar terms if adding new ones

During Implementation

[ ] Make minimal, focused changes only
[ ] Follow ontology-specific conventions exactly
[ ] Use existing patterns as templates
[ ] Document any blockers or limitations

Before Submitting PR

[ ] Review git diff - does it match objective?
[ ] Check for extraneous changes
[ ] Verify PR title accurately describes changes
[ ] Run any available validation (robot, etc.)

After Reviewer Feedback

[ ] Create checklist from reviewer comments
[ ] Address EACH point explicitly
[ ] Push to SAME branch - never create new PR
[ ] Ask for clarification if anything unclear

If Struggling

[ ] STOP after 2 failed attempts on same issue
[ ] Ask for concrete example
[ ] Request human review of approach
[ ] DO NOT keep creating new PRs

---

Success Metrics by Project

Merge Rate vs First-Try Rate

Project	Merge Rate	First-Try Rate	Gap	Key Fix
GO	77.6%	65.0%	12.6%	Obsoletion training
MONDO	44.0%	26.7%	17.3%	Annotation format training
Uberon	100%	37.5%	62.5%	Attribution + ID allocation
CL	95.3%	5.6%	89.7%	Pattern-based submissions
EFO	41.7%	~0%	41.7%	Eliminate duplicates + training

Targets

Project	Current First-Try	Target First-Try	Key Fix
GO	65.0%	80%+	Better obsoletion conventions
MONDO	26.7%	60%+	Annotation patterns + no duplicates
Uberon	37.5%	70%+	Attribution verification + ID handling
CL	5.6%	50%+	Template-based submissions
EFO	~0%	40%+	Stop duplicate PRs + patterns

The CL "success story" is actually a warning: 95% merge rate with 6% first-try rate means reviewers are doing almost all the work.

---

Implementation Priority

P0: Must Fix Immediately

1. ONE PR PER ISSUE rule - Would fix 82% of EFO failures, significant GO/MONDO failures
2. Pre-submission diff verification - Prevents content-objective mismatch

P1: High Value

3. Ontology-specific convention training - GO obsoletion, MONDO annotations
4. Discussion-first for high-impact changes

P2: Important

5. Attribution verification - Check names against issue source
6. Capability limitation reporting - Stop elaborate workarounds
7. Scope discipline - Only change what's requested

P3: Refinement

8. PMID/identifier handling
9. Learning from previous failures
10. Template-based submissions

---

Verification Notes

All data in this document was cross-referenced against actual GitHub PRs using gh CLI:

• GO: 68 PRs confirmed (45 merged, 13 closed, 10 open)
• MONDO: 26 PRs confirmed (11 merged, 14 closed, 1 open)
• Uberon: 14 PRs confirmed (all merged, 8 needed modifications)
• CL: 101 PRs confirmed (81 merged, 4 closed, 16 open)
• EFO: 51 PRs confirmed (20 merged, 28 closed, 3 open)

PR 30730 (GO) verified: Title said "Obsolete 7 terms" but diff showed only +created_by: dragon-ai-agent line added. Failure analysis confirmed accurate.

---

Generated: 2025-12-21
Based on analysis of 260 PRs across 5 ontology projects

[github-actions]

Issue Summary & Analysis

This is a comprehensive, data-driven post-mortem of AI agent performance across 5 major biomedical ontology projects (GO, MONDO, Uberon, CL, EFO) based on 260 PRs. This represents valuable organizational learning and should be treated as a foundational document for improving AI-assisted ontology curation.

Key Insight: The "First-Try Success Rate" Metric

The critical finding here is the introduction of first-try success rate as the true quality metric, revealing that merge rate alone is misleading:

• CL: 95.3% merge rate looks excellent, but only 5.6% first-try success means reviewers are fixing nearly every PR
• EFO: 41.7% merge rate with ~0% first-try success indicates systematic workflow failures
• GO: Best performer at 65% first-try success (77.6% merge rate)

This metric shift is important because it measures curator burden rather than just eventual success.

Critical Failure Modes Identified

P0: Duplicate PR Creation (82% of EFO failures)

The "argyrophilic grain disease saga" (7 PRs over 2 months for a single term) represents a fundamental workflow breakdown where agents fail to update existing branches despite explicit instructions. This pattern affects GO, MONDO, and EFO severely.

Root cause: Lack of persistent state/memory across agent invocations + no pre-flight check for existing PRs.

P1: Content-Objective Mismatch

PR titles don't match actual changes (e.g., GO PR 30730: "Obsolete 7 terms" but diff only shows +created_by: dragon-ai-agent). This suggests inadequate self-verification before submission.

P1: Ontology-Specific Convention Failures

Each ontology has precise patterns (GO obsoletion, MONDO annotations) that agents fail to replicate consistently. The document provides excellent checklists for each.

Strategic Recommendations

For Chris's Memory/Documentation Needs:

1. Convert this to structured documentation - The training rules and checklists are immediately actionable and should be formalized into:

   • Ontology-specific agent prompt templates
   • Pre-submission verification scripts
   • Curator review checklists

2. Track implementation of fixes - The priority ranking (P0-P3) provides a clear roadmap. Monitor which fixes actually improve first-try rates.

3. Project-specific strategies needed:

   • CL: Despite high merge rate, needs the most work (template-based submissions)
   • EFO: Focus on eliminating duplicate PRs
   • GO: Obsoletion-specific training
   • Uberon: Minor fixes (already 100% merge rate, just low first-try)

Related Issues & Context

This connects to:

• Issue Need a reliable, fast way to determine whether a PMID _actually_ supports a curated statement #45: "Need a reliable, fast way to determine whether a PMID actually supports a curated statement" - The PMID validation errors (failure mode Add README.md and CONTRIBUTOR.md documentation files #8) relate directly to this verification challenge
• Issue Ensure permissions are set correctly for claude code #39: Claude Code permissions - Resolved, but relevant to agent deployment

Connections to Broader Research

This analysis contributes to emerging best practices in:

• AI agent prompt engineering - The training rules are essentially structured prompt improvements
• Human-AI collaboration patterns - First-try success rate as a measure of collaboration quality
• Automated ontology curation - Specific to biomedical ontology workflows but generalizable

Actionable Next Steps

1. Immediate: Implement the "ONE PR PER ISSUE" rule technically (add pre-flight PR checks to agent workflow)
2. Short-term: Create pre-submission diff verification (agent must explain its changes before submitting)
3. Medium-term: Build ontology-specific training examples library for agents to reference
4. Long-term: Feedback loop - agents should review why previous PRs failed before creating new ones

Value for Memory/Note-Taking

This document is valuable because it:

• Quantifies real performance across multiple projects with verified data
• Identifies specific, actionable patterns rather than vague "do better"
• Provides ready-to-use checklists and training rules
• Reveals counter-intuitive findings (CL's "success" is actually a warning)

Recommendation: Archive this as a foundational reference document and track how implementing these recommendations affects future PR success rates. Consider creating project-specific reports as mentioned in the FAO section.

---

Note: Mentioned teams have been explicitly tagged for awareness (GO, MONDO, CL, UBERON, EFO maintainers).

[dragon-ai-agent]

Agent Configurations Across Ontology Repos

Summary Matrix

Repo	Primary Agent	Workflow	Instruction Files	Skills/Subagents	Copilot Support
GO	Dragon AI	ai-agent.yml	CLAUDE.md	.claude/settings.json only	copilot-instructions.md
MONDO	Dragon AI	ai-agent.yml (v1.0.0+ actions)	CLAUDE.md	6 subagents + 1 skill	copilot-instructions.md
Uberon	Dragon AI	ai-agent.yml	CLAUDE.md	6 subagents	copilot-instructions.md
CL	Copilot SWE	copilot-setup-steps.yml	CLAUDE.md	None	copilot-instructions.md
EFO	Copilot SWE	copilot-setup-steps.yml	None at root	None	.github/copilot-instructions.md

---

Agent Solutions by Type

1. Dragon AI Agent (GO, MONDO, Uberon)

Architecture:

• GitHub Action triggered by @dragon-ai-agent mentions
• Uses dragon-ai-agent/run-goose-obo action (wraps Claude Code)
• Controller whitelist in .github/ai-controllers.json
• Requires PAT_FOR_PR secret for PR creation

Workflow file: .github/workflows/ai-agent.yml

Triggers:

• Issue opened/edited
• Issue comment created/edited
• PR opened/edited
• PR review comment created/edited

Example (MONDO - newest version):

- name: Respond with AI Agent
  uses: dragon-ai-agent/run-goose-obo@v1.0.4

Older version (GO, Uberon):

• Uses inline JavaScript in actions/github-script@v6
• Loads allowed users from .github/ai-controllers.json
• More complex, less modular

---

2. Copilot SWE Agent (CL, EFO)

Architecture:

• GitHub Copilot Workspace integration
• Setup steps in .github/workflows/copilot-setup-steps.yml
• Prepares environment (ROBOT, tools) for Copilot operations
• Copilot given its own token for operations

Workflow file: .github/workflows/copilot-setup-steps.yml

Key difference from Dragon AI:

• Less autonomous - requires Copilot Workspace interface
• No direct PR creation from issue mentions
• Relies on GitHub's native Copilot infrastructure

---

Instruction Files Comparison

GO

Files:

• CLAUDE.md - Main instructions
• copilot-instructions.md - Copilot-specific (likely symlink or duplicate)
• .claude/settings.json - Claude Code settings

Instruction approach: Standard ODK patterns, uses grep/rg for searching

---

MONDO

Files:

• CLAUDE.md - Main instructions
• copilot-instructions.md - Copilot-specific
• .claude/settings.json
• .claude/agents/ - 6 subagent definitions
• .claude/skills/analyse-issue - 1 skill

Subagents (.claude/agents/):

1. deep-research-specialist.md
2. design-pattern-advisor.md
3. identifier-validator.md
4. metadata-checker.md
5. ontology-reasoner.md
6. task-coordinator.md

Skills (.claude/skills/):

• analyse-issue - Issue analysis skill

Instruction approach: Most advanced - has specialized subagents for different task types

---

Uberon

Files:

• CLAUDE.md - Main instructions
• copilot-instructions.md - Copilot-specific
• .claude/settings.json
• .claude/agents/ - 6 subagent definitions (same as MONDO)

Subagents (.claude/agents/):

1. deep-research-specialist.md
2. design-pattern-advisor.md
3. identifier-validator.md
4. metadata-checker.md
5. ontology-reasoner.md
6. task-coordinator.md

Instruction approach: Same subagent structure as MONDO, likely from shared template

---

CL (Cell Ontology)

Files:

• CLAUDE.md - Main instructions at root
• copilot-instructions.md - Copilot-specific
• No .claude/ directory

Key instructions in CLAUDE.md:

• Uses .owl functional syntax (one axiom per line)
• Search via grep: grep -i CL_0004177 src/ontology/cl-edit.owl
• NTR IDs must start with CL_99xxxxx
• Signs as "GitHub Copilot"

Instruction approach: Minimal - relies on inline CLAUDE.md only

---

EFO

Files:

• .github/copilot-instructions.md - In .github/ not root
• No CLAUDE.md at root
• No .claude/ directory

Instruction approach: Bare minimum - only copilot instructions in .github

---

Configuration Maturity Levels

Level	Description	Repos
Advanced	Subagents, skills, Dragon AI workflow	MONDO, Uberon
Standard	Dragon AI workflow, CLAUDE.md, settings	GO
Basic	Copilot workflow, CLAUDE.md only	CL
Minimal	Copilot workflow, no root instructions	EFO

---

Correlation with Success Rates

Repo	Config Level	First-Try Rate	Notes
GO	Standard	65.0%	Highest success despite simpler config
Uberon	Advanced	42.9%	Subagents may help with complex tasks
MONDO	Advanced	26.7%	Subagents not preventing failures
CL	Basic	5.6%	Minimal config, every PR needs fixes
EFO	Minimal	0%	Worst config, worst results

Key insight: Configuration maturity correlates with success, but not perfectly. GO's higher success may reflect issue quality and simpler tasks, not just configuration.

---

Recommendations by Repo

CL: Urgent Upgrades Needed

1. Add .claude/ directory with settings
2. Create specialized subagents for:
   • hasDbXref format validation
   • Synonym type checking
   • Species suffix validation
3. Add pre-edit hooks for format checking
4. Consider Dragon AI workflow instead of Copilot-only

EFO: Major Configuration Gap

1. Add CLAUDE.md at root (not just in .github)
2. Add .claude/ directory with settings
3. Add Dragon AI workflow or improve Copilot integration
4. Create duplicate PR prevention instructions prominently

GO: Consider Adding Subagents

1. Add specialized obsoletion subagent - GO's main failure mode
2. Consider identifier validator subagent for term IDs

MONDO/Uberon: Refine Existing Subagents

1. Add duplicate PR detection to task-coordinator
2. Enhance identifier-validator for PMID format
3. Add annotation format checker for MONDO-specific patterns

---

Template Recommendations

Based on this analysis, ontology repos should adopt:

repo/
├── CLAUDE.md                    # Main agent instructions
├── AGENTS.md                    # -> symlink to CLAUDE.md (future standard)
├── copilot-instructions.md      # -> symlink to CLAUDE.md
├── .claude/
│   ├── settings.json            # Claude Code settings
│   ├── agents/                  # Subagent definitions
│   │   ├── task-coordinator.md
│   │   ├── identifier-validator.md
│   │   ├── metadata-checker.md
│   │   └── [ontology-specific].md
│   └── skills/                  # Reusable skills
│       └── [ontology-specific]/
└── .github/
    ├── workflows/
    │   ├── ai-agent.yml         # Dragon AI triggers
    │   └── copilot-setup-steps.yml
    ├── ai-controllers.json      # Allowed users
    └── copilot-instructions.md  # Backup copy

---

Version Comparison: Dragon AI Workflows

Newer (MONDO)

uses: dragon-ai-agent/github-mention-detector@v1.0.0
uses: dragon-ai-agent/run-goose-obo@v1.0.4

• Modular action components
• Cleaner, more maintainable

Older (GO, Uberon)

uses: actions/github-script@v6
# Inline JavaScript for detection

• Complex inline scripts
• Harder to maintain/update

Recommendation: All repos should upgrade to the MONDO-style modular actions.

---

Instruction Content Analysis

Line Count Summary

Repo	Instruction File	Lines	Words (approx)
EFO	.github/copilot-instructions.md	636	~4,500
MONDO	CLAUDE.md	432	~3,000
GO	CLAUDE.md	190	~1,400
Uberon	CLAUDE.md	148	~1,100
CL	CLAUDE.md	133	~1,000

Insight: EFO has by far the most verbose instructions (636 lines) but 0% first-try success rate. This suggests instruction volume alone doesn't drive success - instruction quality and specificity matter more.

---

GO Instructions Summary (190 lines)

Key Characteristics:

• Uses .obo format with checkout/checkin workflow
• Search via obo-grep.pl (custom tool)
• NTR IDs start with GO:777xxxx
• Signs commits as @dragon-ai-agent

Unique Features:

• Namespace requirements: GO requires namespace: tag on all terms
• Logical definitions: Detailed guidance on genus-differentia form
• Reaction terms: Special guidance for RHEA mappings with skos qualifiers
• Taxon constraints: Separate files in src/taxon_constraints/

Obsoletion Pattern (detailed):

[Term]
id: GO:0000170
name: obsolete sphingosine hydroxylase activity
namespace: molecular_function
def: "OBSOLETE. ..." [PMID:xxx]
comment: The reason for obsoletion...
property_value: term_tracker_item "https://github.com/.../issues/29717" xsd:anyURI
is_obsolete: true
replaced_by: GO:0102772

Validation: robot convert --catalog ... -i go-edit.obo -f obo -o go-edit.TMP.obo

---

MONDO Instructions Summary (432 lines)

Key Characteristics:

• Uses .obo format with checkout/checkin workflow
• Search via obo-grep.pl --noheader
• NTR IDs start with MONDO:777xxxx
• Requires sh run.sh make NORM before committing

Unique Features:

• Excluded subclass tracking: When removing is_a, add excluded_subClassOf annotation
• Mapping metadata: Complex source qualifier patterns (e.g., MONDO:equivalentTo)
• Gene-disease naming: Explicit pattern {GENE}-related {disease description}
• Synonym attribution: All synonyms MUST have citations (never [])
• Design pattern compliance: Must check src/patterns/dosdp-patterns/*.yaml

Obsoletion Pattern:

[Term]
id: MONDO:0100334
name: obsolete viral infectious disease or sequela
property_value: http://purl.org/dc/terms/creator https://orcid.org/...
property_value: IAO:0000231 MONDO:TermsMerged  # or OMO:0001000
property_value: IAO:0000233 "https://github.com/.../issues/XXXX" xsd:anyURI
is_obsolete: true
replaced_by: MONDO:0100321

Extensive sections on:

• Gene identifier verification (HGNC for human, NCBI Gene for other species)
• Susceptibility vs disease relationships
• Complete exemplar stanzas

---

Uberon Instructions Summary (148 lines)

Key Characteristics:

• Uses .obo format with checkout/checkin workflow
• Search via obo-grep.pl
• NTR IDs start with UBERON:99xxxxx
• Requires robot convert -i uberon-edit.obo -f obo -o uberon-edit.obo before commit

Unique Features:

• Contributor metadata required: New terms MUST have:

  relationship: dc-contributor https://orcid.org/<ORCID> ! <NAME>
  property_value: dcterms-date "<ISO-TIMESTAMP>" xsd:dateTime
  

• Part_of relationships: Many anatomical terms have part_of to other UBERON terms

Obsoletion Pattern (simpler than GO/MONDO):

[Term]
id: UBERON:0001050
name: obsolete atrium
comment: obsoleted because...
is_obsolete: true
consider: FMA:85574
consider: UBERON:0002081

---

CL Instructions Summary (133 lines)

Key Characteristics:

• Uses .owl functional syntax (one axiom per line)
• Search via grep directly on cl-edit.owl
• NTR IDs start with CL_99xxxxx
• Signs as "GitHub Copilot"

Notable Gaps (contributing to low success rate):

• No explicit hasDbXref format guidance (major failure pattern)
• No synonym type guidance (EXACT vs RELATED)
• No species suffix guidance (Mmus)
• No example obsoletion stanza
• No mention of the checkout/checkin workflow

Search Pattern:

grep -i CL_0004177 src/ontology/cl-edit.owl
grep 'AnnotationAssertion(rdfs:label "neuron"' src/ontology/cl-edit.owl

What's missing vs GO/MONDO:

• No detailed obsoletion conventions
• No logical definition examples
• No contributor metadata requirements
• No normalization step before commit

---

EFO Instructions Summary (636 lines)

Key Characteristics:

• Uses .owl OWL/XML format
• Search via grep on efo-edit.owl
• NTR IDs start with EFO_092xxxx
• Requires make normalize_src after edits

Unique Features:

• Multi-agent coordination: Explicit workflow with 3 specialized agents:
  • @EFO-curator - Literature research, evidence gathering
  • @EFO-importer - External term imports (MUST be delegated)
  • @EFO-ontologist - Direct OWL/XML editing
• Import policy: ALL imports delegated to @EFO-importer agent
• PMID requirements: Minimum 2 PMIDs required for new terms
• OLS search mandatory: Must check OLS before creating terms

Obsoletion Pattern:

<owl:Class rdf:about="http://www.ebi.ac.uk/efo/EFO_1000022">
    <obo:IAO_0100001>http://www.ebi.ac.uk/efo/EFO_1000172</obo:IAO_0100001>
    <efo:obsoleted_in_version>2.65</efo:obsoleted_in_version>
    <efo:reason_for_obsolescence>use : EFO_1000172...</efo:reason_for_obsolescence>
    <rdfs:label>obsolete_cervical squamous cell carcinoma</rdfs:label>
    <owl:deprecated rdf:datatype="...">true</owl:deprecated>
</owl:Class>

Why 636 lines doesn't help:

• Instructions are verbose but agent still fails
• Multi-agent coordination adds complexity without solving core issues
• Heavy reliance on import delegation creates bottlenecks
• No evidence of feedback loop from failures

---

Instruction Quality vs Success Rate

Repo	Lines	Key Sections	Obsoletion Example	First-Try Rate
GO	190	Layout, Edit workflow, OBO guidelines, Obsoletion, Logical defs	Yes (detailed)	65.0%
MONDO	432	Layout, Edit workflow, Mappings, Obsoletion, Design patterns, Genes	Yes (with IAO)	26.7%
Uberon	148	Layout, Edit workflow, Obsoletion, Contributor metadata	Yes (simple)	42.9%
CL	133	Layout, Grep search, GitHub process, OBO guidelines	None	5.6%
EFO	636	Multi-agent, Imports, Obsoletion, Validation, OLS search	Yes (XML)	0%

Key Observations:

1. GO succeeds with moderate instructions - Focused, clear, includes key examples
2. MONDO has most detailed instructions but complex multi-pattern requirements hurt
3. CL lacks critical examples - No obsoletion, no format guidance = systematic failures
4. EFO's verbosity doesn't help - Multi-agent complexity may actually hurt

---

Common Instruction Patterns That Work

All successful repos share:

1. Explicit search commands - obo-grep.pl or grep patterns
2. Checkout/checkin workflow - For .obo format repos
3. NTR ID range - Clear guidance on new term IDs
4. Validation command - robot convert with specific flags
5. At least one obsoletion example - Shows exact format

Missing Patterns in Failing Repos

CL is missing:

• Obsoletion example stanza
• hasDbXref format guidance
• Synonym type rules (EXACT vs RELATED)
• Species suffix convention

EFO is missing:

• Simple, direct instructions (buried in multi-agent complexity)
• Focus on preventing duplicate PRs
• Learning from prior failures

---

Recommendations for Instruction Improvement

For CL (to improve from 5.6%):

## hasDbXref Format (CRITICAL)
For PMID citations, use oboInOwl:hasDbXref:
- CORRECT: hasDbXref "PMID:12345678"
- WRONG: dc:source "PMID:12345678"

## Synonym Types
- EXACT: True synonyms with identical meaning
- RELATED: Abbreviations, informal names
- When in doubt, use RELATED

## Species Suffix
For mouse-specific terms, add "(Mmus)" to label:
- CORRECT: "alpha retinal ganglion cell (Mmus)"
- WRONG: "alpha retinal ganglion cell"

## Obsoletion Example
[Add concrete .owl example here]

For EFO (to improve from 0%):

1. Add CLAUDE.md at root (not buried in .github)
2. Simplify multi-agent coordination
3. Add prominent duplicate PR prevention:

   ## CRITICAL: ONE PR PER ISSUE
   NEVER create a new PR when asked to revise.
   ALWAYS push to existing branch.

4. Add pre-submission checklist for common failures

---

Generated: 2025-12-22
Based on GitHub API inspection of 5 ontology repos

[dosumis]

Not convinced by metrics. First try success depends on what curators are trying to achieve. With EFO having agent to do curation, and MCPs to search EBI DBs to assess needs, many not likely intended to be first try success. Can still be saving considerable curator time and supporting work previously not possible.

Better metrics: numbers of terms created by agentic PRs plus how well referenced? Rather than first pass success, better to track trivial fixes needed. Maybe hallucinations plus common issues?

[jamesamcl]

Agree with @dosumis - also we have been using the PRs to develop the instructions so the vast majority of these PRs were never expected to work first time.

[dosumis]

Also this is incorrect for EFO:

Repo	Primary Agent	Workflow	Instruction Files	Skills/Subagents	Copilot Support
EFO	Copilot SWE	copilot-setup-steps.yml	None at root	None	.github/copilot-instructions.md

Has general instructions file + 3 specialised subagents.

Also has MCPS for OLS4 GrEBI and ARTL.

[aleixpuigb]

Yes, @cmungall said that he would investigate, because EFO has some PR with first-try success (see EBISPOT/efo#2583, EBISPOT/efo#2581). The report might have missed something.