Trio Retina

The state layer of the world-model stack — bring any perception model on top, get one standard, model-agnostic WorldState, build any dynamics underneath. Swap the model or the dynamics; Retina is the constant in the middle.

docs · quickstart · the world-model stack · benchmark · examples · notebooks

A lightweight, model-agnostic computer-vision pipeline for object detection & tracking that emits structured events — zone intrusion, line-crossing, dwell, people-counting — from YOLO, VLM, or Grounding DINO detectors over video, files, or RTSP. Runs on CPU at the edge; feeds digital twins, dynamics models, and LLMs.

Just want camera events (zone intrusion, line-crossing) pushed to a webhook? → jump to the 5-line quickstart, or copy examples/rtsp_to_webhook.py.

👋 hello

Trio Retina (Retina for short) turns raw signals — video, sensor — into a queryable world-state: readable events (zone.enter, dwell, line.cross) plus a standardized latent vec channel on the same records, on one small model-agnostic standard. The latent channel is a real, serializable interface (attach your own embedding — see examples/latent_vec.py), and the automatic producers now ship: DinoV2Embedder fills per-object entity.vec and VJepa2Embedder fills the scene latent ws.scene. Bring any model (YOLO, V-JEPA, DINO, a VLM, or none); Retina assembles its output into state a dynamics model, rule engine, or LLM can consume — and a small example dynamics model imagines the future off that state.

Think OpenTelemetry for perception — it doesn't build the sensors, it normalizes any of them into one state. In world-model terms it's the encoder (s = Enc(x)), and only the encoder; dynamics and policy build on top. Retina isn't trying to win a vertical — driving, games, and robotics each already have their own stack — it's the neutral state standard those structured, multi-sensor world models can share. → see DESIGN.md.

💻 install

pip install trio-retina            # core: numpy only
pip install 'trio-retina[yolo]'    # + Ultralytics YOLO adapter
pip install 'trio-retina[video]'   # + OpenCV frame source (files / RTSP / webcam)
pip install 'trio-retina[all]'     # everything

Then try the CLI — retina demo runs a synthetic dock scene and prints the retina.event stream (numpy only, no model/GPU/video):

pip install trio-retina && retina demo        # instant event stream, no model
retina validate events.jsonl                  # check a JSONL stream against the spec
retina --version                              # also: retina run / retina bench

🔥 quickstart

Runs on a bare pip install trio-retina (numpy only) — no model, no GPU, no video file. A stand-in detector walks one "person" across a dock zone; Retina emits the real retina.event stream:

import numpy as np

from retina import CountRule, IoUTracker, Retina, Zone, ZoneRule
from retina.detect import Detection


class ScriptedDetector:
    """A stand-in model: one 'person' walking across a dock zone."""

    def __init__(self):
        self._xs = list(range(0, 102, 6))

    def __call__(self, frame):
        x = self._xs.pop(0) if self._xs else 100
        return [Detection(label="person", bbox=(x - 10, 40, x + 10, 60), confidence=0.9)]


dock = Zone("dock", [(40, 0), (60, 0), (60, 100), (40, 100)])

cam = Retina(
    source_id="cam_01",
    detector=ScriptedDetector(),
    tracker=IoUTracker(min_hits=2),
    rules=[
        ZoneRule(dock, classes={"person"}, dwell_s=2.0),
        CountRule(1, classes={"person"}),
    ],
)

frames = [(np.zeros((100, 100, 3), dtype=np.uint8), float(i)) for i in range(18)]
for event in cam.run(frames):
    print(event.to_json())
    # {"type":"count.threshold","t":1.0,"src":"cam_01","n":1,"frame":1,...}
    # {"type":"zone.enter","t":7.0,"src":"cam_01","id":1,"label":"person",...}
    # {"type":"zone.dwell","t":7.0,...,"zone":"dock","dur":2.0,...}
    # {"type":"zone.exit","t":7.0,...,"zone":"dock","dur":3.0,...}

▶ with a real model + video — pip install 'trio-retina[yolo]' (add [video] for the frame source), then point it at your clip:

from retina import Retina, Zone, ZoneRule, YoloDetector
from retina.sources import video_frames

dock = Zone("dock", [(0.3, 0.2), (0.7, 0.2), (0.7, 0.9), (0.3, 0.9)], normalized=True)

cam = Retina(
    source_id="cam_01",
    detector=YoloDetector("yolo11n.pt", classes={"person"}),
    rules=[ZoneRule(dock, classes={"person"}, dwell_s=30)],
)
for event in cam.run(video_frames("your.mp4")):
    print(event.to_json())
    # {"type":"zone.dwell","t":1718254799.8,"src":"cam_01","id":42,
    #  "label":"person","zone":"dock","dur":31.0,"conf":0.91}

More no-model examples ship with the source (not the wheel) — git clone the repo and run python examples/quickstart.py (the forecast / video demos need [video] + a clip).

▶️ Or run it in your browser — no install:

notebook	what it shows
	quickstart — detector → zone / line / count / dwell events + validate()
	camera → webhook — a restricted-zone alert pushed to your endpoint
	from Supervision — pipe your existing sv.Detections straight in

compose models with |

Wire models like n8n / LangChain, no GUI. Add a cheap gate and a VLM enricher anywhere in the chain:

from retina import MotionGate, GateNode, YoloDetector, IoUTracker, EnricherNode, ZoneRule, JsonlSink

pipe = (
    GateNode(MotionGate())                 # skip static frames (cut model calls)
    | YoloDetector("yolo11n.pt", classes={"person", "forklift"})
    | IoUTracker()
    | EnricherNode(my_vlm_describe)        # attach a VLM read to frame.user
    | ZoneRule(dock, dwell_s=30)
    | JsonlSink("events.jsonl")
)

Two more ways to wire it (explicit list · declarative JSON) + the node catalog

# explicit node list
from retina import Pipeline, DetectorNode, TrackerNode, RuleNode
pipe = Pipeline([DetectorNode(yolo), TrackerNode(), RuleNode(ZoneRule(dock))])

# declarative workflow file (shareable, no code)
pipe = Pipeline.from_json("workflow.json")   # see examples/workflow.json

node	what it does	wraps
DetectorNode	image → detections	any callable(image)->[Detection]
TrackerNode	detections → tracks	IoUTracker / NorfairTracker
RuleNode	tracks → events	ZoneRule / LineRule / CountRule
GateNode	drop uninteresting frames	any callable(image,t)->bool (e.g. MotionGate)
EnricherNode	attach context to frame.user	any callable(frame)->dict (VLM / V-JEPA)
SinkNode	emit events	JsonlSink / WebhookSink

Register your own for from_json with register_node("my_type", builder).

🎛️ supported models

Retina imports no model — any detector plugs in, and out comes one standard event stream. That seam is the point:

plug in any detector…	→	…out comes one retina.event stream
YOLO (Ultralytics: v5–v12, RT-DETR)	→	{"type":"zone.enter", "id":42, "label":"person", …}
any VLM (GPT-4o · Qwen-VL · Gemini · Claude)	→	{"type":"line.cross", "dir":"a_to_b", …}
Grounding DINO (open-vocab, no training)	→	{"type":"zone.dwell", "dur":31.0, …}
your existing sv.Detections (Supervision)	→	{"type":"count.threshold", "n":12, …}
any callable(image) -> [Detection]	→	…+ an optional latent vec on the same record

Supervision gives you boxes on a screen; Retina turns any of those into a serializable state + event stream the next layer (dynamics, twin, agent) can consume. Batteries-included adapters:

• YOLO family — YoloDetector("<weights>.pt") (Ultralytics): YOLOv5/8/9/10/11/12, RT-DETR. Open-vocab via YOLO-World.
• Open-vocab from text — GroundingDinoDetector(["forklift", "hard hat"]), no training.
• Any VLM — VlmDetector(client, prompt) (Qwen-VL / Gemini / GPT-4o / Claude / local), as a detector or an event-source enricher.
• Supervision interop — Detection.from_supervision(sv_detections) ingests a Roboflow sv.Detections, so anything that already converts to Supervision pipes straight into Retina's event layer.
• Latent producers (shipped) — DinoV2Embedder() fills per-object entity.vec (frozen DINOv2, pip install 'trio-retina[dino]'); VJepa2Embedder() fills the scene latent ws.scene from a rolling clip (frozen V-JEPA 2 video encoder, pip install 'trio-retina[vjepa]'). Swap either underneath the same fixed state schema — see examples/world_model/multi_encoder.py.

Trackers are pluggable too: IoUTracker (pure-Python default) or NorfairTracker.

📦 the event format

The retina.event standard is tiny, like a JWT — three required fields, everything else optional and omitted when absent. Full spec in SPEC.md.

{"type":"zone.dwell","t":1718254799.8,"src":"cam_01","id":42,"label":"person","zone":"dock","dur":31.0}

from retina import validate
validate(event)   # -> [] if valid, else a list of problems  (pure-Python, ships a JSON Schema)

🌍 the world-model stack

Retina is the encoder (s = Enc(x)) in a world model. It doesn't try to win any one vertical — driving, games, and robotics each already have their own stack; it's the neutral state layer those structured, multi-sensor world models plug into, in four ways:

• One contract for many sensors. A camera embedding, a radar return, an IMU stream, a WiFi CSI latent — heterogeneous encoders all land in the same WorldState, fused on a model-tagged vec and a typed locus / scene. Retina doesn't build the sensors; it normalizes any of them into one state.
• Swap the front or the back, never both. Encoder and dynamics meet on a frozen state contract — change the perception model without retraining the dynamics, or change the dynamics without touching perception. The seam is the product.
• A state you can read, log, and verify. Small, serializable, half-symbolic — the state doubles as the world model's observability layer: eval it against ground truth, stream it to a digital twin, alert on a retina.event, even when the dynamics itself is a black box.
• Cheap at the edge. The encode-to-state step runs on CPU; the heavy dynamics lives wherever you like. Retina is the lightweight front door that turns raw signals into state before the expensive layer sees them.

The scope is deliberate — Retina is for world models that reason over structured, multi-sensor state, not monolithic pixel-to-pixel video generators. With the latent producers shipped, that seam is now demonstrable end to end — on a synthetic scene, as a small but honest proof of concept (examples/world_model/):

1 · swap the encoder, the state is constant. The same pipeline, run three ways — symbolic-only, + DinoV2Embedder (per-object entity.vec), and + VJepa2Embedder (scene-level ws.scene) — yields the identical WorldState schema; only which model filled the latent changes. → multi_encoder.py

2 · a dynamics model imagines the future off that state. A small transformer trained offline on recorded WorldState sequences predicts where each entity is headed, and rolls out imagination trajectories inside the learned model. The honest ablation — does Retina's appearance latent actually help? — on held-out data with real DINOv2 vecs (Mac Studio, MPS), mean held-out 7-step position error (px, lower is better):

dynamics input	7-step error
constant-velocity baseline	7.68 px
learned, pos-only	1.45 px
learned, pos + appearance latent	1.33 px

The latent channel measurably improves prediction: +83% over constant-velocity, +8% over pos-only at horizon 7 — and the edge widens with the horizon, because that's where local velocity runs out and object type (legible only from appearance) decides the future. → dynamics.py, full grid in BENCHMARK.md

Raw video → one standardized Retina WorldState → predicted player runs. Left is a real broadcast clip (Roboflow's MIT-licensed sports sample, originally DFL Bundesliga). It goes through a real YOLO detector + tracker and a frozen DINOv2-small appearance encoder, and comes out as one model-agnostic WorldState; the right panel renders that state as a top-down tactical radar, and the small dynamics transformer — trained offline on those sequences — draws each player's predicted next run ahead in brand indigo (faint gray = where they came from). Teams are coloured by clustering the players' DINOv2 appearance vectors into two groups — the latent knows who's who. The radar is a stylized perspective top-down (no Roboflow pitch-keypoint weights on this host, so a fixed homography from the clip's pitch landmarks, not per-frame). Honest by design: player motion is stochastic, so at this short horizon the learned model roughly ties a constant-velocity baseline on held-out error — the appearance latent's measurable win lives in the cleaner synthetic ablation above, not on free-running humans. Real pipeline, end to end — examples/world_model/soccer/. The synthetic car rollout (held-out, where the latent earns its keep) lives in make_demo_gif.py · media/rollout.png.

3 · the state layer is signal-agnostic — the same schema carries a WiFi CSI world model. The WorldState/Vec that carried DINOv2 appearance above also carries a WiFi CSI channel latent, with zero core schema change to express an RF world model: the global channel latent drops into ws.scene, and the subject's metric room position rides the typed Entity.locus (metres, distinct from the pixel bbox). On synthetic CSI (a documented forward model, fully offline) we reproduce the recipe of two CSI world-model papers — a JEPA that predicts the next latent (arXiv:2409.10045) under an action-conditioned homomorphic transition (arXiv:2603.20048) — and route it through Retina's state layer. Real numbers from one run (seed 0, CPU): action-conditioning improves next-latent prediction +72.9% over an action-blind ablation; the latent self-organizes into a metric room map (~0.33 m probe error in a 6×5 m room); and the imagination rollout beats a constant-velocity baseline by ~22% over a 14-step horizon, with the edge growing past ~7 steps. Honest scope: those papers are channel/comms-side world models trained on real CSI; our contribution is showing the recipe runs through a signal-agnostic state layer — this is a synthetic-CSI proof of concept, not a real-WiFi result. → examples/world_model/csi/

4 · front + back compose through one standard. Any encoder in front, any dynamics behind, meeting on one serializable state — a pip-installable world-model seam you can run in one script. → end_to_end.py

pip install 'trio-retina[dynamics,dino]'
python examples/world_model/dataset.py --n 12 --len 24 --out examples/world_model/data/sequences.json
python examples/world_model/end_to_end.py   # encoder → WorldState → dynamics → imagined rollout

Honest scope: a synthetic scene, a tiny model, reproducible on MPS with run-to-run variance. The producers ship; the trained dynamics is a small example, not a product. The point is the seam — that front and back compose through one standardized state.

🎬 demos

Two more ways Retina's state feeds the next layer — same standard, different consumers:

Forecast — the dynamics layer on Retina One world-state → two dynamics models forecast where each entity is headed off the same state (gray = constant-velocity, magenta = learned). A dynamics model eats structured state, not pixels. → examples/forecast/

iTwin.js — a live, predictive digital twin Retina's entities, forecast arrows, and retina.event alerts on a real Bentley iTwin.js iModel (Baytown), one neutral JSON contract, fully headless — it gives the twin live eyes. → examples/itwin/

All examples

The examples live in this repo (not in the installed wheel) — git clone to run them. The top-level quickstarts run with no model and no GPU (synthetic detections):

python examples/quickstart.py          # zone / line / count / dwell events
python examples/three_apps.py          # one stream -> security, retail, safety
python examples/any_model.py           # swap the detector, rest unchanged
python examples/gate_savings.py        # a cheap gate cuts detector calls 100 -> 23
python examples/pipeline_compose.py    # compose with | (n8n without a GUI)
python examples/rtsp_to_webhook.py     # camera -> restricted-zone alert -> webhook
python examples/from_supervision.py    # ingest a Roboflow sv.Detections pipeline
python examples/latent_vec.py          # populate the latent vec channel by hand
python examples/dino_embeddings.py     # REAL DINOv2 per-object vecs (needs [dino])

Real-footage / dynamics demos need a clip and the extras — pip install 'trio-retina[all]':

python examples/yolo_video.py v.mp4    # YOLO on a video file
examples/forecast/                     # dynamics layer on the WorldState stream (needs [video] + a clip)
examples/itwin/                        # events + forecasts on a Bentley iTwin.js iModel

The world-model stack lives in examples/world_model/ (needs [dynamics], plus [dino]/[vjepa] for real encoders):

python examples/world_model/multi_encoder.py   # swap encoder, state schema stays constant
python examples/world_model/dynamics.py        # train + the honest appearance ablation
python examples/world_model/benchmark.py       # the front/back-end benchmark grid → BENCHMARK.md
python examples/world_model/end_to_end.py      # encoder → WorldState → dynamics → imagined rollout

Send events anywhere. WebhookSink(url) POSTs each event as JSON (stdlib urllib, no requests); JsonlSink(path) streams to a file. For a live camera, video_frames(src, live=True) reads RTSP / HLS / webcam with wall-clock timestamps — see examples/rtsp_to_webhook.py.

🎯 use cases

One state layer, many domains — the same retina.event stream, read differently above:

• Security & intrusion detection — zone.enter / line.cross on cameras and RTSP feeds.
• Retail analytics & people-counting — footfall, queue dwell, zone occupancy from any detector.
• Workplace safety — PPE, forklift, and restricted-zone alerts via open-vocab detectors.
• Smart city & traffic monitoring — vehicle/pedestrian counting and crossings at the edge.
• Industrial digital twins — feed live entities + forecasts into a twin (iTwin.js demo).

🧠 how it works

Everything flows through one append-only data unit, the Frame. Each stage enriches it and never overwrites upstream fields:

                      ┌──────────────── Frame (append-only) ───────────────┐
 frame ─► Detector ─► │ .detections ─► Tracker ─► .tracks ─► Rule ─► .events │ ─► Sink
   ▲        ▲         │                  ▲                    ▲              │     ▲
 source   any model   │   Gate (skip?)   tracker     zone/line/count/dwell  │  jsonl/
                      │   Enricher (VLM / V-JEPA → .user)                    │  webhook
                      └─────────────────────────────────────────────────────┘

• The detector is the model-agnostic seam: any callable(image) -> [Detection].
• The tracker gives objects identity over time; rules turn tracks into events; enrichers attach context; gates skip work; sinks push out.
• Output is dual: a readable symbolic stream and an optional model-tagged latent channel — never collapsed.

Why "encoder", the dual state, and how it compares to DeepStream / Supervision

Two senses of "encoder." Foundation backbones (V-JEPA, DINO, SAM, YOLO) turn pixels into features — that race is theirs, and Retina rides it. Retina is the encoder layer on top: it fuses many models into one record, gives objects persistent identity, structures it into entities + relations + events, carries the dual symbolic + latent channels, as an event-sourced stream — one small, serializable, model-agnostic standard.

Dual state. The same entities on two linked channels: symbolic (readable events / entity records, for rules / LLMs / dashboards) and latent (optional model-tagged embeddings, for a downstream dynamics model). Symbols you can read; vectors a model can predict on. The latent channel is a standardized, serializable interface — populate entity.vec with your own embedding (examples/latent_vec.py), or let a built-in producer fill it: DinoV2Embedder (per-object) and VJepa2Embedder (scene-level) both ship today.

vs DeepStream / Holoscan — same good ideas (event semantics, metadata model, composable graph), none of the weight:

	DeepStream / Holoscan	Retina
Install	CUDA + TensorRT + containers	pip install trio-retina
Hardware	NVIDIA / Jetson locked	any machine — CPU is fine
Model	tied to the NV stack	bring any model (or none)
Shape	a platform you build inside	a library you import
Core deps	a lot	numpy

vs Supervision — Supervision turns a model's output into detections + overlays (great toolbox, ends at the screen). Retina is a level up: it emits a serializable state + event stream that the next layer (dynamics, twin, agent) consumes. We compose Supervision / detectors, not compete with them.

Full rationale, references, and the world-model stack: DESIGN.md.

🗺️ roadmap

Early but real (v0.3.0). Stable: the event layer + JSON Schema/validator, the composable pipeline (| / list / JSON), YOLO + open-vocab + VLM detectors (plus from_supervision interop), IoU + Norfair trackers, and jitter-robust rules (exit_grace_s · anchor · min_frames).

Next: ByteTrack / OC-SORT · proximity / anomaly events · VLM-as-event-source · Kafka / MQTT sinks · more encoders behind the latent channel · model-based RL / latent-rollout imagination on the learned state · growing the front/back-end benchmark. See CHANGELOG.md.

Retina is the open perception encoder extracted from Trio; the layers above (dynamics, policy / judgment) are Trio's commercial platform. Retina is, and stays, model-agnostic and free.

🤝 contributing

Contributions that keep Retina small and beautiful are very welcome — see CONTRIBUTING.md for dev setup and how to add a detector / tracker / rule / sink. By participating you agree to the Code of Conduct; to report a vulnerability see SECURITY.md.

license

Apache-2.0.