Skip to content

qcai0427/Sibyl-Codex

BranchesTags

Repository files navigation

Sibyl Research System

Sibyl Research System

Fully Autonomous AI Research System with Self-Evolution

License: MIT

Inspired by the pioneering work of The AI Scientist, FARS, and AutoResearch, Sibyl now targets a Codex CLI native workflow built around AGENTS.md, repo-local sibyl commands, MCP servers, and isolated codex exec child runs.

中文文档

I am not the oringinal contributor of this repo. What you see right now is the codex Cli version of Sibyl project, which been used by me personally. Sibyl is a fully automated scientific discovery system that autonomously drives ML research from literature survey to paper submission. It operates as an autonomous research organization: 20+ specialized AI agents debate ideas, design and run GPU experiments, write papers, and critically review their own work — all without human intervention.

What truly sets Sibyl apart is its dual-loop architecture:

  • Inner Loop — Research Iteration: Each project automatically iterates across every dimension — refining hypotheses based on experiment results, re-planning experiments, rewriting papers, pivoting to alternative ideas when needed — until quality meets publication standards.
  • Outer Loop — System Self-Evolution: Sibyl learns from the research process itself. After every iteration, it classifies issues across 8 categories, accumulates reusable lessons, and automatically updates its own agent prompts, scheduling strategies, and architectural patterns. The system that runs your research is itself getting better at running research.

What Makes Sibyl Different?

  • Autonomous Multi-Dimensional Iteration — Not just "run experiments and write a paper." Every aspect of the research improves automatically across iterations: ideas sharpen through multi-agent debate, experiments expand with better baselines and ablations, writing tightens under 6-agent cross-review, and resource utilization optimizes through GPU scheduling feedback. The quality gate decides when to stop or pivot — no human in the loop.
  • Self-Evolving System — Most AI research tools are static — they run the same way every time. Sibyl evolves. It extracts lessons from every research iteration (issues, success patterns, efficiency metrics), keeps them time-weighted and context-filtered, and injects the relevant ones back into agent prompts. Across projects, the system accumulates institutional knowledge — each project makes every future project better.
  • Codex CLI Native — Not a wrapper around API calls. Built around Codex CLI, AGENTS.md, MCP tools, repo-local orchestration commands, and optional codex exec child runs for isolated roles.

Get Started

Recommended: Run It With Codex CLI

Clone the repo, create the local environment, and start Codex from either the repo root or a workspace root:

git clone https://github.com/Sibyl-Research-Team/sibyl-research-system.git
cd sibyl-research-system
chmod +x setup.sh && ./setup.sh
tmux new -s sibyl                                           # recommended: persistent session
codex

Then either use the shell entrypoints directly:

sibyl start spec.md
sibyl continue .

or ask Codex to read AGENTS.md plus .codex/loop-prompt.txt in the target workspace and continue the Sibyl control loop.

Manual Setup

Click to expand manual setup instructions

Prerequisites

  • Python 3.12+, Node.js 18+
  • Codex CLI
  • GPU server with SSH access
  • OPENAI_API_KEY when using Codex MCP or non-ChatGPT auth flows
  • tmux (strongly recommended) — enables persistent sessions and automatic recovery via Sentinel watchdog. Install: brew install tmux (macOS) / apt install tmux (Linux)

1. Install

git clone https://github.com/Sibyl-Research-Team/sibyl-research-system.git
cd sibyl-research-system
chmod +x setup.sh && ./setup.sh    # Interactive: creates venv, installs deps, configures MCP

setup.sh also adds or updates export SIBYL_ROOT="..." in your shell rc file so workspace-root Codex sessions and helper scripts can resolve the repo root.

2. Configure MCP Servers

Two MCP servers are required. setup.sh configures them interactively, but for manual setup the preferred path is codex mcp add ... so the configuration stays Codex-scoped:

codex mcp add ssh-mcp-server -- npx -y @fangjunjie/ssh-mcp-server \
  --host YOUR_GPU_IP --port 22 --username YOUR_USER --privateKey ~/.ssh/id_ed25519

codex mcp add arxiv-mcp-server -- /ABSOLUTE/PATH/TO/sibyl-research-system/.venv/bin/python3 -m arxiv_mcp_server

If you already manage Claude Code MCP servers through JSON, update the existing MCP config instead of creating a second source of truth:

{
  "mcpServers": {
    "ssh-mcp-server": {
      "command": "npx",
      "args": ["-y", "@fangjunjie/ssh-mcp-server",
               "--host", "YOUR_GPU_IP", "--port", "22",
               "--username", "YOUR_USER",
               "--privateKey", "~/.ssh/id_ed25519"]
    },
    "arxiv-mcp-server": {
      "command": "/ABSOLUTE/PATH/TO/sibyl-research-system/.venv/bin/python3",
      "args": ["-m", "arxiv_mcp_server"]
    }
  }
}

Server names must be exact: "ssh-mcp-server" and "arxiv-mcp-server".

3. Configure GPU Server

Create config.yaml at project root (git-ignored):

ssh_server: "default"
remote_base: "/home/user/sibyl_system"
max_gpus: 4
language: zh
codex_enabled: false

Use ssh_server: "default" when ssh-mcp-server was registered with explicit --host/--username arguments. If your MCP setup resolves a named SSH host alias instead, use that alias.

4. Run

# `setup.sh` normally writes this for you; set it manually only if you skipped setup.sh
export SIBYL_ROOT=/path/to/sibyl-system

# Repo root: setup, init, status, migrate, evolve
cd "$SIBYL_ROOT"
tmux new -s sibyl-admin
codex

# Workspace root: actual project execution (recommended)
cd "$SIBYL_ROOT/workspaces/my-project"
tmux new -s sibyl-my-project
codex

# Shell entrypoints:
sibyl start spec.md               # New project from this workspace's spec
sibyl continue .                  # Resume the current workspace

Why tmux? Sibyl experiments can run for hours. Running inside tmux ensures the session persists through terminal disconnections. The Sentinel watchdog runs in a sibling tmux pane and automatically restarts Codex if it crashes or goes idle.

Which directory should Codex start in? Use the repo root only for setup and global maintenance. For an actual research run, start Codex from the target workspace root (workspaces/<project>/), not from the repo root and not from workspaces/<project>/current. This makes Codex load the workspace-specific AGENTS.md, .codex/loop-prompt.txt, and project memory directly.

Parallel projects: run one Codex session/pane per workspace root.

Docs: Full Setup Guide · Configuration (35+ options) · MCP Servers · SSH & GPU · All 12 Commands

System Overview

Sibyl orchestrates 20+ AI agents through a 19-stage state-machine pipeline, automatically completing literature survey, idea generation, experiment design & execution, result analysis, paper writing, and peer review. The system supports multi-round iterative optimization with built-in cross-project learning that continuously improves research quality.

Sibyl Research System Architecture

Core Features

  • 19-Stage Research Pipeline: End-to-end automation from literature search to camera-ready paper
  • Multi-Agent Collaboration: 6-agent debate for idea generation, 6-agent result analysis, 6-agent parallel writing
  • GPU-Parallel Scheduling: Topological sort + dynamic dispatch, maximizing GPU utilization with automatic task dependency management
  • Autonomous Iterative Optimization: Quality gate auto-decides whether to continue iterating, pivot to new ideas, or terminate — every dimension of research improves across iterations
  • Self-Evolving System: Automatically extracts lessons across 8 categories, tracks effectiveness, prunes what doesn't work, and updates agent prompts — the system improves itself with every project
  • Self-Healing System: Background agent continuously monitors for runtime errors, auto-fixes them using skill pipelines, adds regression tests, and commits fixes — all without human intervention
  • Multi-Model Collaboration: Claude Opus/Sonnet + GPT-5.4 (Codex) independent cross-review

Pipeline

+== Research Iteration =============+  +== Paper Writing ====================+
|                                    |  |                                     |
|  Literature Search (arXiv + Web)   |  |  Outline                            |
|       |                            |  |       |                             |
|       v                            |  |       v                             |
|  Idea Debate (6 Agents)            |  |  Section Writing (seq/para/Codex)   |
|       |                            |  |       |                             |
|       v                            |  |       v                             |
|  Experiment Planning               |  |  Cross Review (6 Agents)            |
|       |                            |  |       |                             |
|       v                            |  |       v                             |
|  Pilot Experiments                 |  |  Integration & Editing              |
|       |                            |  |       |                             |
|       v                            |  |       v                             |
|  Full Experiments (GPU parallel)   |  |  Final Review (NeurIPS level)       |
|       |                            |  |       | fail --> back to edit (x2)  |
|       v                            |  |       v                             |
|  Result Debate (6 Agents)          |  |  LaTeX --> compile PDF              |
|       |                            |  |       |                             |
|       v                            |  +-------|---------+-------------------+
|  Decision                          |          |
|       | PIVOT --> back to Idea     |          |
|       | PROCEED                    |          v
+-------|-----------+----------------+  +== Review & Reflection ==============+
        |                               |                                     |
        +----------> Outline            |  Review (Critic+Supervisor+Codex)    |
                                        |       |                             |
                                        |       v                             |
                                        |  Reflection (lessons learned)       |
                                        |       |                             |
                                        |       v                             |
                                        |  Lark Sync (cloud docs)             |
                                        |       |                             |
                                        |       v                             |
                                        |  Quality Gate                       |
                                        |       | >= 8.0 & >= 2 iter --> DONE |
                                        |       | else --> next iteration     |
                                        |                                     |
                                        +-------------------------------------+

Stage Details

Stage Description Agent Mode
literature_search Dual-source survey via arXiv + Web Single Agent
idea_debate 6-perspective idea debate (Innovator / Pragmatist / Theorist / Contrarian / Interdisciplinary / Empiricist) 6-Agent Team
planning Design experiments, generate task_plan.json with dependencies Single Agent
pilot_experiments Small-scale feasibility validation Single Agent
experiment_cycle GPU-parallel full experiments, topologically sorted batch scheduling Single Agent + GPU Scheduler
result_debate 6-perspective result analysis (Optimist / Skeptic / Strategist / Methodologist / Comparativist / Revisionist) 6-Agent Team
experiment_decision Supervisor decision: PIVOT (change direction) or PROCEED Single Agent
writing_outline Generate paper outline Single Agent
writing_sections Write by section (sequential / parallel / Codex modes) Configurable
writing_critique 6-agent cross-review of each section 6-Agent Parallel
writing_integrate Editor integrates into complete paper Single Agent
writing_final_review NeurIPS/ICML-level final review (can loop for revision) Single Agent
writing_latex Convert to NeurIPS LaTeX format and compile PDF Single Agent
review Critic + Supervisor + Codex parallel review Parallel Skills
reflection Classify issues, generate improvement plan, record lessons Single Agent
lark_sync Sync research data to Feishu/Lark cloud docs Single Agent
quality_gate Evaluate completion (≥8.0 score and ≥2 iterations) Automatic

Agent Roles

Idea Generation Team

Agent Perspective Responsibility
Innovator Cross-domain innovation Bold methodology transfer and novel combinations
Pragmatist Engineering feasibility Ensure ideas are implementable
Theorist Mathematical foundations Focus on theoretical guarantees and proofs
Contrarian Challenge assumptions Find counter-evidence and blind spots
Interdisciplinary Analogical inspiration Import methods from cognitive science, physics, biology
Empiricist Experiment-first Focus on reproducibility and data quality

Result Analysis Team

Agent Perspective Responsibility
Optimist Positive findings Discover positive results and extension directions
Skeptic Statistical rigor Question statistical significance and confounders
Strategist Next steps Suggest resource allocation and research direction
Methodologist Method review Evaluate internal and external validity
Comparativist SOTA benchmarking Compare and position against existing best methods
Revisionist Hypothesis revision Reflect on and adjust hypotheses based on results

Model Tiers

Tier Model Usage
Heavy Opus 4.6 Synthesis, supervision, editing, criticism, reflection
Standard Opus 4.6 Literature survey, planning, experiments, writing
Light Sonnet 4.6 Result debate, cross-review, section critique
Codex GPT-5.4 High Independent third-party review, optional writing mode

Self-Evolution System

Sibyl doesn't just run research — it learns how to run research better. After every iteration, the system analyzes what worked, what failed, and what was inefficient, then automatically updates itself:

Research Iteration completes
       |
       v
  Reflection Agent ──> Analyze outcomes across 8 dimensions
       |                    ├── Experiment design quality
       |                    ├── Writing clarity & structure
       |                    ├── Resource efficiency (GPU utilization, scheduling)
       |                    ├── Idea novelty & contribution
       |                    └── System reliability, analysis depth, planning, pipeline
       v
  Evolution Engine ──> Track & rank lessons
       |                    ├── Time-weighted frequency analysis (30-day half-life)
       |                    ├── Schema-normalized issue clustering (stable issue keys)
       |                    └── Success pattern extraction (what to keep doing)
       v
  Auto-Update ──> Inject relevant improvements into agent prompts
       |              ├── Context-aware overlay filtering per agent/stage
       |              ├── Stale overlays removed automatically
       |              └── Efficiency insights: scheduling & resource optimization
       v
  Self-Check ──> Detect systemic anomalies
                    ├── Declining quality trend across iterations
                    ├── Recurring errors that lessons haven't fixed
                    └── Ineffective lesson accumulation

Why Self-Evolution Actually Works

Most AI systems that claim to "learn" are stateful processes — they accumulate context within a single session, but lose everything when the process restarts. Sibyl takes a fundamentally different approach: stateless architecture with persistent artifacts.

  • Every prompt is loaded from disk at call time. There is no in-memory cache, no long-running daemon. Each agent reads its prompt file (sibyl/prompts/*.md) fresh every time it is invoked. If the evolution engine rewrites a prompt, the very next agent call picks up the change — zero restart, zero redeployment.
  • Every agent runs as an independent subprocess. Skills execute via python3 -c "..." in a fresh process, so Python modules are re-imported every time. Code changes in sibyl/*.py take effect immediately on the next stage.
  • Config is re-parsed per orchestrator call. cli_next() instantiates a new Orchestrator each time, re-reading config.yaml from disk. Parameter tuning by the evolution engine is picked up on the next tick.
  • Lesson overlays are plain files. Experience extracted from past projects is written to .sibyl/evolution/lessons/{agent}.md (or the runtime path overridden by SIBYL_STATE_DIR / SIBYL_EVOLUTION_DIR). The load_prompt() function appends the overlay content on every call — new lessons are injected into the next agent invocation automatically.

This means evolution is not a "batch update" that requires a maintenance window. It is a continuous, incremental process: the system that runs iteration N+1 is already different from the one that ran iteration N, because the reflection after iteration N has already modified prompts, overlays, and potentially code on disk. The entire system is designed so that every file is the source of truth, and every file is read fresh — making self-evolution a natural consequence of the architecture rather than a bolted-on feature.

Safety: All system file modifications are gated by mandatory tests (.venv/bin/python3 -m pytest tests/) and tracked via git commits, ensuring every evolution step is reversible and auditable.

8 Issue Categories: SYSTEM, EXPERIMENT, WRITING, ANALYSIS, PLANNING, PIPELINE, IDEATION, EFFICIENCY — each automatically routed to the relevant agents. The planner learns to design better experiments, the experimenter learns to use GPUs more efficiently, the writer learns to avoid recurring style issues — all without manual intervention.

Self-Healing System

While the self-evolution system learns from completed iterations, the self-healing system operates in real time — continuously monitoring for runtime errors and fixing them autonomously as the research pipeline runs.

Runtime Error Occurs
       |
       v
  Error Collector ──> Structured capture to logs/errors.jsonl
       |                    ├── Exception type, traceback, file, line
       |                    ├── Pipeline stage & project context
       |                    └── Automatic categorization (7 types)
       v
  Error Router ──> Intelligent triage
       |                    ├── Deduplication (hash-based)
       |                    ├── Priority sorting (import > build > type > test > ...)
       |                    ├── Skill routing (error type → repair skill pipeline)
       |                    └── Circuit breaker (3 failures → escalate to human)
       v
  Self-Healer Agent ──> Autonomous repair
       |                    ├── Invoke mapped skills (systematic-debugging, tdd-workflow, ...)
       |                    ├── Apply fix with scope limits (max 5 files, protected file rules)
       |                    ├── Generate regression test to prevent recurrence
       |                    └── Verify: full test suite must pass
       v
  Git Commit ──> fix(self-heal): <description> [auto]
                    └── All fixes tracked on dev branch, periodically synced to main

How It Works

The self-healing system is a three-layer architecture:

  1. Error Collector (sibyl/error_collector.py) — Captures runtime exceptions with full context (traceback, stage, project) into structured JSONL records. A @wrap_cli decorator automatically catches errors from all orchestrator CLI functions. Errors are categorized into 7 types: import, test, type, state, config, build, prompt.

  2. Error Router (sibyl/self_heal.py) — Deduplicates errors by content hash, sorts by priority (import errors before config errors), and maps each error category to a repair skill pipeline via the skill route table. A circuit breaker prevents infinite fix loops: after 3 failed attempts on the same error, it is logged and skipped, then retried in the next iteration.

  3. Self-Healer Agent (sibyl-self-healer skill) — A fork skill running on the standard tier (Opus) that receives repair tasks and autonomously:

    • Invokes the appropriate skills (e.g., systematic-debugging → tdd-workflow)
    • Applies the fix within scope limits (max 5 files per fix, surgical changes to protected files)
    • Writes a regression test covering the exact failure condition
    • Runs the full test suite to verify the fix
    • Commits with fix(self-heal): ... [auto] format for full traceability

Safety Mechanisms

Mechanism Purpose
Circuit breaker Same error failing 3 times → stops and flags for human review
File scope limit Max 5 files modified per fix — prevents over-reaching changes
Protected files Core files like orchestrate.py only allow minimal, surgical edits
Test gate Full test suite must pass before any fix is committed
Git tracking Every fix is a separate commit on dev — fully reversible

Configuration

self_heal_enabled: true        # Enable self-healing (default: true)
self_heal_interval_sec: 300    # Background scan interval (default: 5 min)
self_heal_max_attempts: 3      # Circuit breaker threshold (default: 3)

Project Structure

sibyl-system/
├── sibyl/                      # Core Python modules
│   ├── orchestrate.py          # State-machine orchestrator (19-stage pipeline)
│   ├── config.py               # Configuration (models/GPU/modes)
│   ├── workspace.py            # Workspace file & Git management
│   ├── gpu_scheduler.py        # GPU topological sort & parallel scheduling
│   ├── evolution.py            # Cross-project evolution engine
│   ├── reflection.py           # Iteration logging
│   ├── error_collector.py      # Structured error capture for self-healing
│   ├── self_heal.py            # Error routing, circuit breaker, repair orchestration
│   └── prompts/                # 33 agent prompt templates
├── .claude/
│   ├── agents/                 # Agent tier definitions (heavy/standard/light)
│   └── skills/sibyl-*/         # 30+ Fork Skills (isolated context execution)
├── plugin/commands/            # Claude Code plugin commands
├── tools/                      # Repo-level helper tools (not research projects)
├── workspaces/                 # Sibyl research project workspaces only
├── tests/                      # Unit tests (~320 tests)
└── requirements.txt            # Dependencies (PyYAML, rich)

Workspace Structure

Each research project has an independent filesystem under workspaces/<project>/:

workspaces/<project>/
├── CLAUDE.md                   # Effective runtime prompt (system + project memory)
├── status.json                 # Orchestrator state (stage/iteration/score)
├── config.yaml                 # Project-level config overrides
├── topic.txt / spec.md         # Research topic & requirements spec
├── .sibyl/project/             # Project memory and prompt overlays
├── context/literature.md       # Literature review
├── idea/                       # Proposals, alternatives, debate records
├── plan/                       # Experiment plan, task_plan.json
├── exp/                        # Code, results, logs, GPU progress
├── writing/                    # Outline, sections, reviews, full paper, LaTeX
├── logs/                       # Iteration archives, research diary
└── lark_sync/                  # Feishu/Lark sync registry

Repo-level helper utilities belong under tools/, not workspaces/. If a directory under workspaces/ does not have Sibyl project state such as status.json, it should be moved out.

Documentation

Document Description
Setup Guide Claude-readable setup checklist (recommended)
Getting Started Full installation and first-run guide
Configuration All 35+ config options reference
MCP Servers Third-party MCP dependencies & setup
SSH & GPU Setup GPU server configuration
Plugin Commands All 12 plugin commands reference
Codex Integration GPT-5.4 cross-review setup
Feishu/Lark Setup Cloud document sync
Architecture System internals for contributors

Third-Party Dependencies

MCP Servers

Server Required Purpose Source
SSH MCP Yes Remote GPU execution npx @fangjunjie/ssh-mcp-server
arXiv MCP Yes Paper search pip install arxiv-mcp-server
Google Scholar MCP Recommended Citation search GitHub clone
Codex MCP Optional GPT-5.4 review npm install -g @openai/codex
Lark MCP Optional Feishu Bitable/IM npm install -g @larksuiteoapi/lark-mcp
Feishu MCP Optional Feishu documents npm install -g feishu-mcp
bioRxiv MCP Optional Biology preprints pip install biorxiv-mcp-server
Playwright MCP Optional Web browsing npm install -g @playwright/mcp

See MCP Servers Guide for installation and MCP registration details.

Python Dependencies

  • PyYAML >= 6.0 — Config file parsing
  • rich >= 13.0 — Terminal formatted output

Optional Tools

  • OpenAI Codex CLI — Independent cross-review (opt in with codex_enabled: true)
  • Ralph Loop — Autonomous iteration loop (Claude Code plugin)
  • AI Research Skills — 85 expert skills covering fine-tuning, inference, evaluation, paper writing, and more. When installed, Sibyl agents automatically discover relevant skills and invoke them on demand for best-practice guidance. See setup guide for installation.

Key Mechanisms

GPU Parallel Scheduling

The experiment stage reads task_plan.json, topologically sorts tasks by dependencies, then greedily assigns parallel execution based on available GPUs:

{
  "tasks": [
    {"id": "train_baseline", "depends_on": [], "gpu_count": 2, "estimated_minutes": 60},
    {"id": "train_model_a", "depends_on": ["train_baseline"], "gpu_count": 1, "estimated_minutes": 90},
    {"id": "train_model_b", "depends_on": ["train_baseline"], "gpu_count": 1, "estimated_minutes": 90},
    {"id": "ablation", "depends_on": ["train_model_a", "train_model_b"], "gpu_count": 1, "estimated_minutes": 30}
  ]
}

Cross-Project Self-Evolution

Lessons learned in one project automatically improve all future projects:

  1. Record: Classify issues (8 categories) and success patterns after each iteration
  2. Analyze: Aggregate with time-decay weighting (30-day half-life) — recent lessons matter more
  3. Normalize: Canonicalize action plans (category/severity/status/trajectory) and assign stable issue_keys
  4. Apply: Generate per-agent prompt overlays — each agent receives only the lessons relevant to its role and current context
  5. Prune: Remove stale overlays automatically when no longer supported by current insights
  6. Self-Check: Detect quality decline and recurring unresolved errors

PIVOT Mechanism

When experiment results are unsatisfactory, the supervisor decision agent can trigger PIVOT:

  • Analyze whether results support the original hypothesis
  • Evaluate whether continued investment is worthwhile
  • If PIVOT: roll back to idea debate stage with alternative proposals
  • Maximum 6 PIVOT cycles (configurable)

Comparison

Feature Sibyl Research System AI Scientist AutoResearch
Architecture Claude Code native (skills, teams, MCP) API wrapper Single-file script
Agent count 20+ specialized agents Single LLM Single agent
Idea generation 6-agent multi-perspective debate LLM brainstorming N/A
Experiment execution GPU-parallel with topo-sort scheduling Template-based Single-GPU loop
Paper writing Multi-agent write + review + revise LLM generation N/A
Self-evolution Cross-project lesson learning None None
Self-healing Auto-detect & fix runtime errors None None
Quality control Multi-round review + quality gate Automated review Metric-based
Human intervention Fully autonomous Minimal Minimal

License

MIT License

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages