Skip to content

README.md

Latest commit

 

History

History
259 lines (209 loc) · 12.1 KB

README.md

File metadata and controls

259 lines (209 loc) · 12.1 KB
Raw

opus_build — LLM-steered Omni3WD robot simulator

simulation overview

A PyBullet simulation of a 3-wheel omnidirectional robot driven by a local vision-language model (Gemma 4 via Ollama). Each cycle:

  1. Capture a 640×480 RGB frame from the robot's forward camera.
  2. Send the frame to Ollama; receive a JSON movement command.
  3. Execute the command (advance / strafe / rotate) for its duration.
  4. Check for coin pickups by proximity; loop.

The same Python module split is designed to map cleanly onto a Raspberry Pi + Arduino deployment later — see the hardware section below.

Requirements

  • Python 3.10+ (developed on 3.14 on Windows 11)
  • A local Ollama server with a multimodal model. Tested: gemma4:latest (8B, Q4_K_M, ~9.6 GB on disk).
  • A GPU for rendering and inference. Tested on an RTX 5060 Laptop (8 GB VRAM). Ollama handles partial CPU offload automatically.

Install

cd opus_build
python -m venv .venv
.venv/Scripts/python -m pip install -r requirements.txt

Make sure Ollama is running and the model is pulled:

ollama serve          # in one terminal
ollama pull gemma4:latest

Configure

Defaults live in config.py. Override any of these with an environment variable or a .env file next to config.py:

env var default what it does
OLLAMA_HOST http://localhost:11434 Ollama server URL
OLLAMA_MODEL gemma4:latest Any multimodal Ollama tag works
OLLAMA_TIMEOUT_S 180 Cold model loads can take ~30 s

Other knobs (speeds, camera FOV, world/coin geometry, sim tick rate) are plain Python constants — edit config.py directly.

Run

Command What it does
.venv/Scripts/python -m main Full end-to-end loop. GUI opens, Gemma drives until the coin is collected.
.venv/Scripts/python -m sim.world GUI preview of the world (no robot, no LLM).
.venv/Scripts/python -m sim.teleop Manual keyboard drive (W/A/S/D/Q/E/Space/R/+/-/Esc).
.venv/Scripts/python -m sim.camera Capture one frame from the start pose, save PNG to logs/.

Close the PyBullet window or press Esc to exit. The loop auto-exits on all coins collected.

What you'll see

  • A small room with a checkerboard floor, four walls, a wooden table, a blue box, a green cylinder, and a yellow sphere (the target).
  • The robot (dark cylinder, orange "nose" arrow, green "camera eye" on top-front) spawns in the corner opposite the coin.
  • In the terminal: each Gemma decision with its reasoning and the robot's pose before/after the command. Per-run transcripts go to logs/run-YYYYMMDD-HHMMSS.log.

Architecture

opus_build/
├── main.py               # live loop: capture -> LLM -> execute -> pickup check
├── config.py             # central constants + env var overrides
├── sim/
│   ├── world.py          # build_world() + check_collections()
│   ├── robot.py          # OmniRobot (chassis + camera mount + velocity control)
│   ├── camera.py         # image_cap(robot) -> np.ndarray
│   ├── movement.py       # 7 primitives + command_exec() + ACTION_DISPATCH
│   └── teleop.py         # keyboard driver for manual testing
├── llm/
│   ├── prompts.py        # SYSTEM_PROMPT + VALID_ACTIONS
│   └── client.py         # img_out(frame) -> {action, speed_mmps, duration_ms, reasoning}
├── assets/               # (placeholder — most assets come from pybullet_data)
├── logs/
│   ├── DEBUG_NOTES.md    # dated dev journal of every bug + fix
│   └── run-*.log         # per-run transcripts
└── tests/                # (scaffold only — not implemented in v1)

Data flow

  +--------------+         +---------------+         +-----------------+
  | sim.camera   |  frame  | llm.client    |   cmd   | sim.movement    |
  | image_cap()  | ------> | img_out()     | ------> | command_exec()  |
  +--------------+         +---------------+         +-----------------+
        ^                                                       |
        |                                                       v
  +-----+-----+                                          +------+------+
  | sim.world | <--- check_collections() <-------------- | sim.robot   |
  +-----------+                                          | OmniRobot   |
                                                         +-------------+

The dashed boundary between "sensing + thinking" (sim.camera + llm.client) and "acting" (sim.movement + sim.robot) is intentional — it's the same boundary we'd cross on real hardware.

Robot control functions

The seven movement primitives live in sim/movement.py and mirror the Arduino Omni3WD API 1:1. Each Python primitive takes (robot, speed_mmps); each Arduino method takes speedMMPS.

Python primitive Arduino equivalent Body-frame effect
advance(robot, v) Omni3WD::setCarAdvance(v) +X (forward)
backoff(robot, v) Omni3WD::setCarBackoff(v) −X (reverse)
left(robot, v) Omni3WD::setCarLeft(v) +Y (strafe left)
right(robot, v) Omni3WD::setCarRight(v) −Y (strafe right)
rotate_left(robot, v) Omni3WD::setCarRotateLeft(v) +Z angular (CCW from above)
rotate_right(robot, v) Omni3WD::setCarRotateRight(v) −Z angular (CW from above)
stop(robot) Omni3WD::setCarStop() zero

Dispatched by name via ACTION_DISPATCH (a dict keyed on the strings listed in llm/prompts.VALID_ACTIONS). command_exec(robot, command) applies the velocity every physics tick for duration_ms then stops.

Modification points

  • Add a new action (e.g. a diagonal advance_left): write the primitive in sim/movement.py, add it to ACTION_DISPATCH and to VALID_ACTIONS in llm/prompts.py, and describe it in SYSTEM_PROMPT so Gemma knows when to emit it. The JSON-schema validator in llm/client._coerce_command will then accept it.
  • Change speed/duration limits: edit config.SPEED_MAX_MMPS (used by _coerce_command to clamp speed_mmps) and the duration clamp constants at the top of _coerce_command in llm/client.py (currently [300, 3000] ms).
  • Change the robot geometry (bigger chassis, different wheel layout, different camera offset): edit ROBOT_RADIUS, ROBOT_HEIGHT, CAMERA_OFFSET_X, CAMERA_OFFSET_Z in config.py. Visuals are composed in sim/robot.OmniRobot._spawn.
  • Change the world: obstacles live in sim/world._build_obstacles; walls in _build_walls; the coin in _build_collectibles plus config.COLLECTIBLE_POSITIONS. Add multiple entries to spawn multiple coins — check_collections() already handles the list.
  • Swap the LLM: any multimodal model served by Ollama with JSON output works. Set OLLAMA_MODEL to e.g. llava:latest or qwen2.5vl:7b. If you change models, re-tune the MOTION MODEL block of the system prompt — the numbers there are calibrated for the current robot + physics (rotate at 120 mm/s for 1500 ms ≈ 40° of turn after friction damping).
  • Switch off the PID-lite damping: sim/robot.OmniRobot.__init__ sets lateralFriction=0.3, linearDamping=0.1, angularDamping=0.1. Raising friction makes the robot more sluggish; lowering it makes commanded velocity closer to nominal.

Deployment on real hardware (plan — not tested)

The Python module boundaries were drawn to match a Raspberry Pi + Arduino split. Nothing below has been tested end-to-end on hardware yet; the original MotorWheel / Omni3WD firmware lives at ../rb-nex-02/lib/MotorWheel/ in this repo.

What moves where

Module Sim Real hardware
sim.camera.image_cap PyBullet getCameraImage Pi: OpenCV VideoCapture or picamera2
llm.client.img_out HTTP to local Ollama Pi (local Ollama) or remote server (cloud API)
sim.movement.command_exec Apply velocity in PyBullet Pi encodes command → Arduino executes it
sim.movement primitives resetBaseVelocity on the body Arduino: Omni3WD::setCar*() methods
sim.world.check_collections Proximity despawn External pickup detector (ultrasonic / IR) — or drop

Suggested protocol (RPi → Arduino)

A line-oriented text protocol is simple and matches the Arduino's serial loop. Example mapping of the command dict:

{"action": "advance", "speed_mmps": 180, "duration_ms": 2000}
    -->  A180 2000\n

On the Arduino side, parse the first char as the action, the two ints as speed/duration, then:

switch (action) {
  case 'A': omni->setCarAdvance(speed);      break;
  case 'B': omni->setCarBackoff(speed);      break;
  case 'L': omni->setCarLeft(speed);         break;
  case 'R': omni->setCarRight(speed);        break;
  case 'Q': omni->setCarRotateLeft(speed);   break;
  case 'E': omni->setCarRotateRight(speed);  break;
  case 'S': omni->setCarStop();              break;
}
omni->delayMS(duration);   // runs the PID loop during the wait
omni->setCarStop();

Omni3WD::delayMS() calls PIDRegulate() every sample period, so the wheel controllers keep the commanded speeds stable for the full duration — functionally equivalent to the sim's "reapply velocity each physics tick" trick.

What stays unchanged

Everything in llm/ (client code + prompt) is hardware-agnostic. Even the system prompt's MOTION MODEL numbers may transfer — they describe what the commands should achieve, not what the sim does internally. On real hardware you will want to recalibrate those numbers once with a measured drive, since the friction profile is different.

Known quirks

Full dev journal at logs/DEBUG_NOTES.md. The three things most likely to bite a future dev:

  1. Gemma latency is 15-20 s per call and ~30-50 s on cold starts. The loop prints "gemma (+Ns)" so you know what's normal.
  2. Friction damps commanded speeds to ~68% of nominal. The motion model in the prompt is calibrated to actual damped numbers, not nominal — if you change lateralFriction in sim/robot.py, re-measure before editing the prompt.
  3. Gemma can be noisy on small targets at the edge of frame. The coin is deliberately sized at 18 cm radius for reliable detection; a smaller coin will trigger spin-and-search oscillation loops. If you shrink the target, tighten the prompt's "roughly centered" threshold.

Milestone history

Git tags map to milestones:

m1-scaffold   project skeleton + git init
m2-world      textured room, walls, obstacles
m3-robot      OmniRobot + 7 primitives + teleop
m4-camera     image_cap 640x480 RGB
m5-llm        Ollama client with JSON output
m6-e2e        end-to-end live loop (with earlier red-cube target)
m6b-coin      replaced target with collectible yellow sphere
v1.0          documented + cleaned up, successful end-to-end collection run

Each tag points at a commit with a clear scope; bug-fix commits sit between them.