Work type
feature
Area
other
Problem
Many terrain-locomotion and parkour tasks need exteroceptive observations beyond proprioception and local heightfield sampling. Existing heightfield scanning is useful but does not cover robot-mounted ray/LiDAR sensors, depth cameras, configurable intrinsics/extrinsics, or sensor update rates that differ from control frequency.
Deliverable
Add a backend-level batched exteroceptive sensor contract for reusable ray/LiDAR and depth-camera observations, with backend implementations where available and explicit unsupported behavior elsewhere.
Definition of done
- SimBackend exposes a stable sensor contract.
- Batched ray/LiDAR observations support configurable attachment frame, ray pattern, max distance, filters, and stable output shape.
- Batched depth observations support pinhole intrinsics, clipping, attachment frame, and stable output shape.
- Existing heightfield scanner behavior is unchanged.
- Unsupported backends fail explicitly.
- Smoke tests and docs/examples are added.
Dependencies and blockers
Related to #627, which covers batch raycaster support for an mjlab rough-terrain migration. This issue tracks the more general reusable exteroceptive sensor layer, including depth-camera support.
Proposed owner
No response
Validation plan
- Unit tests on simple plane/box scenes for shape, hit/no-hit, cutoff, filtering, and reset behavior.
- Env smoke test with sensor terms included in observations.
- Performance benchmark comparing batched implementation against per-env Python loops.
- Minimal documentation/example for adding a robot-mounted ray sensor and depth camera.
Work type
feature
Area
other
Problem
Many terrain-locomotion and parkour tasks need exteroceptive observations beyond proprioception and local heightfield sampling. Existing heightfield scanning is useful but does not cover robot-mounted ray/LiDAR sensors, depth cameras, configurable intrinsics/extrinsics, or sensor update rates that differ from control frequency.
Deliverable
Add a backend-level batched exteroceptive sensor contract for reusable ray/LiDAR and depth-camera observations, with backend implementations where available and explicit unsupported behavior elsewhere.
Definition of done
Dependencies and blockers
Related to #627, which covers batch raycaster support for an mjlab rough-terrain migration. This issue tracks the more general reusable exteroceptive sensor layer, including depth-camera support.
Proposed owner
No response
Validation plan