Skip to content

ARTICLE_v2.md

Latest commit

 

History

History
399 lines (261 loc) · 21.5 KB

ARTICLE_v2.md

File metadata and controls

399 lines (261 loc) · 21.5 KB
Raw

Teaching AI Agents to Think Like Domain Experts (v2): A Quant Research “Idea Discovery” Loop

Repo: alexmihalache/cursor_domain_agent

How to read this (pick your path)

  • Curious builder / generalist: read the Main article only.
  • Quant / research workflow: main + Appendix A.
  • Cursor / agent configuration: main + Appendix B.
  • Tooling / architecture: main + Appendix C.

TL;DR (what this is)

I’m learning systematic trading. The bottleneck isn’t “I can’t write code” — it’s idea discovery and not fooling myself. So I tried to turn a coding agent into something closer to a junior researcher:

  • It proposes candidate mechanisms (hypotheses).
  • It tests them quickly in “exploration mode”.
  • If anything looks promising, it has to pass stricter “validation mode” gates.
  • Everything gets written down as artifacts so I can audit it later.

The trick is: a coding agent is useful for domain work because it can write deterministic tools (scripts, gates, extractors) and run them — and with the right scaffolding it’s forced to use them — not because it’s “pretending to be a quant”.

1) Why I built this

Two hobby directions I keep circling with AI and Machine Learning are football data and algo trading.

The practical problem with football is that the good stuff (especially tracking data) is expensive and hard to access. Trading data can also be expensive, but platforms like cTrader give you years of tick data for free. It’s not full exchange microstructure (Level 2), but it’s a decent proxy and it’s a huge volume of real time-series data to learn from. Also: it overlaps with my day job enough that it’s easier to justify spending time on.

I started “prompt-first” (ChatGPT + copy/paste) and quickly hit the same wall everyone hits: context decays, you forget what you tried, and you end up re-deriving decisions you already made.

The shift for me wasn’t “use a better prompt.” It was: treat research like a workflow, not a conversation.

2) The real problem: agents can code, but research is a workflow

Coding agents are very good at implementation. But domain work (quant research, ops, legal research, whatever) is mostly:

  • experimental design (what would falsify this?)
  • data validity and leakage paranoia
  • baselines and comparisons
  • record keeping and reproducibility
  • decision-making under uncertainty

So instead of trying to make a coding agent “magically know quant”, I used the coding agent as the platform to build a research workflow on top of:

idea → look things up → get data → sanity-check → run → compute metrics → decide → choose next step

Cursor’s deep filesystem + terminal integration matters here because it lets the agent actually run things and leave behind durable artifacts.

3) The system in one mental model (3 parts)

Everything I built reduces to three parts:

  1. Lab notebook (experiments as folders)
    Each run becomes a folder under research/experiments/ with a plan, sanity checks, results, and a decision.

  2. Safety rails (hard gates)
    A small set of gates that stop you from “discovering” nonsense (data validity, leakage, baselines, OOS discipline, etc.).

  3. Memory that survives the session (methods registry)
    A repo-wide registry of methods + append-only usage log + a scorecard so you can see what’s actually helping over time.

If you only take one idea from this article, it’s this: if the state lives in files, the agent and the human can both recover and audit.

4) What it looks like in practice: one end-to-end discovery loop

This is a real example from the repo. In-repo it’s logged as EXP-20260119-004; the important structure is: plan → sanity → results → decision (tracks/task envelopes are explained in Appendix B).

  • Experiment folder: research/experiments/EXP-20260119-004/

Brief (what we were trying to do)

We wanted a way to screen candidate features for NAS100_SB on M5 bars, without falling into p-hacking.

Instead of “build a whole strategy and pray”, the idea was:

  • generate a moderately sized set of plausible OHLCV-derived features (30–50)
  • score them with an information-style metric (null-calibrated MI proxy)
  • check stability across time buckets
  • only then try “minimal monetization” tests to see if any feature can stand on its own - meaning run actual backtests through the cTrader engine to ensure they'd execute under real conditions.

In plain English: it’s a score for “does this feature predict anything beyond chance?”, calibrated against shuffled data so we don’t fool ourselves.

In plan.md, the agent also produced a quick fanout of “what to try first” (targets, feature families, and scoring approaches), then we chose the simplest falsifiable route:

  • Targets: forward return sign/quantile, drawdown proxy, chop/cost regime proxy
  • Feature families: returns/momentum, candle shape (wicks/CLV), volatility, volume, gaps, trend/levels, time-of-day
  • Scoring options: (A) null-calibrated MI proxy, (B) out-of-sample log-loss improvement, (C) MDL-style compression (“bits saved”)

That’s deliberately humble: it’s an “idea discovery / triage” loop, not “found alpha”.

Hypothesis (mechanism + falsifier)

From EXP-20260119-004/plan.md:

  • Hypothesis: a small set of simple, non-leaky OHLCV features reduce uncertainty about a trade-relevant target in a way that is stable across time buckets and worth monetizing.
  • Falsifier: after multiple-testing controls, the top-ranked features are unstable and show no net-of-cost improvement when implemented as minimal cBots on a clean OOS split.

Step 1 — Scaffold the experiment (the lab notebook)

We created the usual experiment files:

  • plan.md (what we’re doing and what would falsify it)
  • sanity.md (data window validity + alignment)
  • results.md (what happened, including evidence pointers)
  • decision.md (what we will do next, or why we stop)
  • RUN_METADATA.json (repro metadata + artifact pointers)

This is boring on purpose: it’s the part that makes the work auditable later.

Step 2 — Run the minimum sanity gates (so exploration isn’t garbage)

From EXP-20260119-004/sanity.md, the “Data Window Validity Gate” captured:

  • bar counts per split (IS / OOS1 / OOS2)
  • timestamp coverage
  • gap counts ((\Delta t > 5) minutes)
  • basic alignment rules (features at (t), labels at (t+1..t+H))
  • holdout exhaustion rules (what we’re allowed to do with OOS1 vs OOS2)

This is the kind of thing that’s easy to skip when you’re iterating fast — and exactly the kind of thing that creates fake edges.

One concrete example from earlier iterations (the kind of thing that happens when you let an agent explore fast): I had a “session start” style feature with the timezone wrong, which effectively produced 0 bars in most windows (everything looked like zeros and the loop happily proceeded). A data-window gate would have stopped that immediately — and it’s the kind of bug that can waste days if you don’t have a hard fail-fast check.

Step 3 — Exploration run: screen features and don’t overthink it

From EXP-20260119-004/results.md:

  • The screening produced rankings for 43 features, written to:
    • research/experiments/EXP-20260119-004/artifacts/feature_screen_rankings.csv
  • It also ran leakage “trap” tests and wrote:
    • research/experiments/EXP-20260119-004/artifacts/leakage_trap_tests.json

The interesting part (for me) is the shape of the outcome:

  • A handful of features looked “OK” on OOS1 (as screening signals).
  • But when we forced the “minimal monetization” test (standalone policies), the results were strongly negative net-of-cost across IS/OOS1/OOS2.

That’s the system working: it didn’t let “screening looks promising” turn into “ship it”.

Step 4 — Minimal monetization (a deliberately harsh gate)

In the results, the top-3 features were implemented as minimal cBots and backtested net-of-cost.

The outcome was unambiguous: the standalone versions were not viable (negative net profit and high drawdown).

That doesn’t mean the features are useless — it suggests they might be better as filters/sizers on a base strategy — but it does fail the “minimal monetization” gate as written.

Step 5 — Decision

From EXP-20260119-004/decision.md:

  • Decision type: Do not promote (yet)
  • Next actions if continuing:
    • test features as conditional filters/sizers on a chosen base strategy (with retention-matched baselines)
    • tighten multiple-testing discipline further

This is the “research OS” idea in practice: you can do a lot of exploration quickly, but the output of exploration is not “profit”, it’s a clear decision and a next step.

What I got out of this loop (even without “an edge”)

  • A ranked list of candidate signals to test as conditional features, not vibes.
  • A record of what failed, and why it failed.
  • A reusable screening harness that can be rerun on other instruments/timeframes.
  • A concrete example of why gates are necessary: without them, you would rationalize your way into calling this “promising” and stop there.

5) What worked / what didn’t (trade-offs)

This is the honest part. The system helps, but it’s not magic.

What worked:

  • Artifacts beat chat for anything longer than a single sitting.
  • Hard gates prevent self-deception, especially around data windows, leakage, and OOS discipline.
  • The agent can build tools for itself (scripts, validators, extractors), which is a real advantage over “prompt-only” workflows.
  • Tracks help both the agent and the human stay oriented (and call out when you’re drifting).

What didn’t (or needs tuning):

  • Silent failure / stalled runs are still a real pain (commands hang, background runs fail, and you only notice later).
  • Guardrail paradox: too strict and the agent becomes bureaucratic; too loose and you get chaos. The balance depends heavily on the model and the task.
  • Workflow gate friction: early versions blocked too much and made it annoying to kick off longer runs (or run things overnight). Tuning “enforce vs warn” is part of making this usable.
  • Output volume is a trap: you can generate a lot quickly, but productivity without value is pointless. The track system is partly there to protect me from that.
  • Reproducibility is hard: RUN_METADATA.json helps, but you still have drift (deps, datasets, platform changes).
  • Methods registry underuse: unless the workflow forces it (slash commands + gates), the agent doesn’t reliably consult/update the registry. That’s fixable, but it’s not “automatic compounding” by default.
  • Bad priors are sticky: I tried “persona prompting” to give the agent a quant vibe; it often made output more formulaic and less creative. I also accidentally introduced a rogue “xEV” framing that the agent started applying everywhere. The safer pattern is: keep exploration flexible, but ground it in logged methods + gates + evidence.

6) What you can steal for your own domain (a checklist)

If you ignore the trading part, the transferable pattern is:

  1. Define your “experiment object” (the thing you want to be able to audit later).
  2. Define a small set of gates that stop catastrophic mistakes - this might take a bit of trial and error to see what is needed for your workflow.
  3. Make files the default output (not chat).
  4. Add enforcement where it matters (hooks, CI, or whatever your platform supports).
  5. Build a methods registry + usage log so “what works” compounds over time.
  6. Add context isolation (tracks / work streams / workspaces) so you can run multiple threads without mixing them.

7) Limits + when not to use this

This is not a replacement for domain expertise. It’s a way to compress time while keeping the loop honest.

  • Don’t use this for high-stakes decisions without independent validation.
  • Hooks and rules are workflow guardrails, not security boundaries.
  • If your domain can’t define validation gates, you’ll struggle to make autonomy safe.

8) Next steps + links

  • I’m going to keep refining this on the quant workflow that started the project.
  • If you try a similar setup (in trading or elsewhere), I’d love to hear what works and what breaks.
  • If you’re reading this in-repo, the “what exists today” inventory is roughly: strategies under research/shared/strategies/ (indexed in STRATEGIES.yml), a growing methods catalog in research/methods/, and a durable experiments log under research/experiments/ (with older, legacy runs preserved under research/workspaces/).

Appendix A — Quant research protocol (exploration vs validation)

This appendix is the “protocol” part: what the agent should do, and what it must not skip when results start looking good.

Modes

Exploration (fast, permissive)

  • Goal: generate and triage ideas quickly.
  • Non-negotiable: data presence/coverage sanity before trusting any metric.
  • Output: hypotheses + rough evidence + next test.

Validation (slow, strict)

  • Goal: decide if an idea is real enough to invest in.
  • Hard gates become mandatory (see below).
  • Output: decision + reproducibility metadata + explicit evidence.

Hard gates (the ones that matter)

These are “stop yourself from lying to yourself” gates:

  • Data Window Validity Gate: counts/coverage/gaps; ensure windows exist and aren’t degenerate.
  • Leakage Gate: features strictly prior to labels; no same-bar dependence disguised as “feature”.
  • Baseline discipline: beat do-nothing and a naive baseline; define what “baseline” means.
  • OOS / walk-forward: time-split discipline; no repeated peeking and retuning.
  • Cost realism: spread/slippage/commissions sensitivity, at least conservatively.
  • Robustness: regime splits, parameter perturbations, stability checks.
  • Tail risk / concentration: don’t hide drawdowns in averages.
  • Confounds: time-of-day, volatility, calendar effects (if those explain it, you didn’t find a new edge).

Experiments as folders (system-of-record)

The repo uses research/experiments/EXP-YYYYMMDD-NNN/ folders. A typical folder includes:

  • plan.md
  • sanity.md
  • results.md
  • decision.md
  • config.json
  • RUN_METADATA.json
  • code_ref.txt
  • artifacts/ (small durable outputs; large logs live in tmp/agent/<TRK>/...)

See research/experiments/README.md for the canonical structure and intent.

Methods registry (what worked, over time)

Repo-wide registry lives in research/methods/:

  • METHODS.yml: method cards (signals, how-to, failure modes)
  • USAGE_LOG.ndjson: append-only usage records (why, outcome, evidence pointers)
  • METHOD_SCORECARD.csv: generated rollup from usage logs

This is what I mean by “memory that survives the session”, append-only evidence you can audit.

Appendix B — Cursor implementation (rules / hooks / skills / commands)

This appendix is for “how is this wired up in Cursor?”

If you want the full, current technical doc, start here:

  • documentation/cursor_agent_setup.md
  • documentation/quant_research_loop_workflow.md
  • docs/RUNBOOK.md
  • docs/ARTIFACTS_INVENTORY.md

Tiny glossary

  • Rules: persistent instructions applied by scope (file patterns) written in natural language.
  • Hooks: scripts that run on lifecycle events (before shell, after edit, on stop) written in Python.
  • Skills: reusable procedures/macros (e.g., track router, checkpointing) written in natural language.
  • Commands: slash-invoked playbooks (repeatable workflows) written in natural language - they're like pre-canned prompt snippets that you can add into a prompt without having to write them out every time.

As a concrete example of each

Rule (quant loop gates) — from .cursor/rules/quant_research_loop.mdc:

### Exploration (fast)
Required minimum:
- hypothesis (mechanism + falsifier)
- data scope (instrument/timeframe/window)
- **Data Window Validity Gate output** in `sanity.md`

Hook (workflow gate: soft vs strict) — from .cursor/hooks/gate_shell.py:

def _gate_strict_enabled():
    # Default SOFT when gate is enabled: warn/allow.
    # Strict mode (deny shell until docs updated) can be enabled by either:
    # - setting environment variable CURSOR_WORKFLOW_GATE_STRICT=1
    # - creating file .cursor/workflow_gate.strict.enabled
    ...

In practice: soft mode nudges (and leaves an audit trail) but still lets you keep moving; strict mode blocks further shell execution until you update the track/task/checkpoint artifacts for the current milestone.

Skill (track router) — from .cursor/skills/track-router/SKILL.md:

## Selection (deterministic)
1) If the user explicitly names a track, TRK = that value (uppercased + underscores).
2) Else if a referenced task file exists and contains `TRK: <TRK>`, use it.
3) Else read `docs/context/INDEX.md` and use `LastTouchedTrack`.

Tracks (context isolation)

The “work stream” unit is a track under:

  • docs/context/tracks/<TRK>/CURRENT.md (living state)
  • docs/context/tracks/<TRK>/checkpoints/ (snapshots)
  • docs/results/tracks/<TRK>/latest.md (short summaries, link-heavy)

Execution conventions (uv-first)

From docs/RUNBOOK.md:

  • Prefer uv run ... for Python execution from repo root.
  • Long outputs go to tmp/agent/<TRK>/... and are linked from track results.

Why this matters

This is the “workflow” layer behind the “artifacts beat chat” idea above: it prevents drift and makes the work resumable.

Appendix C — Meta + tooling patterns (repo-as-tools, ad hoc tooling, AI-building-AI)

ChatGPT vs Cursor agent (how I used them)

  • ChatGPT was useful for research/synthesis/planning.
  • Cursor agent was useful for implementation and iteration because it could read/write files and run commands.

Ad hoc tooling (workspace-scoped) → promotion into core

One pattern I kept using: let the agent write disposable scripts inside a workspace to unblock an analysis, then only promote the proven ones into core.

That keeps the core stable while still letting you move fast.

In this repo, you can see that pattern clearly in research/workspaces/research_1/ where I let the agent build small “local instrumentation” tools, for example:

  • Deterministic metrics extraction / reporting (small JSON/CSV artifacts instead of chat paste), e.g. research/workspaces/research_1/research_log/tools/report_metrics.py
  • Simple portfolio aggregation from multiple runs, e.g. research/workspaces/research_1/research_log/tools/portfolio_curve.py

This matters because it:

  • keeps velocity high without destabilizing the core mid-research
  • reduces token/attention cost by generating small, computed artifacts (JSON/CSV) instead of copying huge logs into chat
  • increases confidence: “this metric changed” can point to a script parsing real outputs, not model narrative

Over time, the useful ones get promoted into core once they prove reusable (for example, sweep monitoring that started as workspace scripts becoming first-class CLI capabilities).

“Repo as lightweight MCP” (what I mean, and what I don’t mean)

What you get from repo-as-tools:

  • deterministic scripts (tools)
  • versioned artifacts (memory)
  • discoverable context (files)
  • local execution (fast iteration)

What you don’t get:

  • strong security boundaries
  • clean standardized remote tool APIs
  • permissioned multi-client separation

So it’s not an MCP replacement. It’s an architectural shortcut that works surprisingly well for personal and small-team workflows — especially when your “tools” are already code in-repo and your “memory” is versioned artifacts. Once you need true remote services, strict permissions, or standardized interfaces, MCPs (or similar) start to make more sense.

If you need remote execution, shared multi-user tool access, or policy enforcement, repo-as-tools stops being enough.

References & related materials