Add mesh picking backend and MeshRayCast system parameter (#15800)

# Objective

Closes #15545.

`bevy_picking` supports UI and sprite picking, but not mesh picking.
Being able to pick meshes would be extremely useful for various games,
tools, and our own examples, as well as scene editors and inspectors.
So, we need a mesh picking backend!

Luckily,
[`bevy_mod_picking`](https://github.com/aevyrie/bevy_mod_picking) (which
`bevy_picking` is based on) by @aevyrie already has a [backend for
it](https://github.com/aevyrie/bevy_mod_picking/blob/74f0c3c0fbc8048632ba46fd8f14e26aaea9c76c/backends/bevy_picking_raycast/src/lib.rs)
using [`bevy_mod_raycast`](https://github.com/aevyrie/bevy_mod_raycast).
As a side product of adding mesh picking, we also get support for
performing ray casts on meshes!

## Solution

Upstream a large chunk of the immediate-mode ray casting functionality
from `bevy_mod_raycast`, and add a mesh picking backend based on
`bevy_mod_picking`. Huge thanks to @aevyrie who did all the hard work on
these incredible crates!

All meshes are pickable by default. Picking can be disabled for
individual entities by adding `PickingBehavior::IGNORE`, like normal.
Or, if you want mesh picking to be entirely opt-in, you can set
`MeshPickingBackendSettings::require_markers` to `true` and add a
`RayCastPickable` component to the desired camera and target entities.

You can also use the new `MeshRayCast` system parameter to cast rays
into the world manually:

```rust
fn ray_cast_system(mut ray_cast: MeshRayCast, foo_query: Query<(), With<Foo>>) {
    let ray = Ray3d::new(Vec3::ZERO, Dir3::X);

    // Only ray cast against entities with the `Foo` component.
    let filter = |entity| foo_query.contains(entity);

    // Never early-exit. Note that you can change behavior per-entity.
    let early_exit_test = |_entity| false;

    // Ignore the visibility of entities. This allows ray casting hidden entities.
    let visibility = RayCastVisibility::Any;

    let settings = RayCastSettings::default()
        .with_filter(&filter)
        .with_early_exit_test(&early_exit_test)
        .with_visibility(visibility);

    // Cast the ray with the settings, returning a list of intersections.
    let hits = ray_cast.cast_ray(ray, &settings);
}
```

This is largely a direct port, but I did make several changes to match
our APIs better, remove things we don't need or that I think are
unnecessary, and do some general improvements to code quality and
documentation.

### Changes Relative to `bevy_mod_raycast` and `bevy_mod_picking`

- Every `Raycast` and "raycast" has been renamed to `RayCast` and "ray
cast" (similar reasoning as the "Naming" section in #15724)
- `Raycast` system param has been renamed to `MeshRayCast` to avoid
naming conflicts and to be explicit that it is not for colliders
- `RaycastBackend` has been renamed to `MeshPickingBackend`
- `RayCastVisibility` variants are now `Any`, `Visible`, and
`VisibleInView` instead of `Ignore`, `MustBeVisible`, and
`MustBeVisibleAndInView`
- `NoBackfaceCulling` has been renamed to `RayCastBackfaces`, to avoid
implying that it affects the rendering of backfaces for meshes (it
doesn't)
- `SimplifiedMesh` and `RayCastBackfaces` live near other ray casting
API types, not in their own 10 LoC module
- All intersection logic and types are in the same `intersections`
module, not split across several modules
- Some intersection types have been renamed to be clearer and more
consistent
	- `IntersectionData` -> `RayMeshHit` 
	- `RayHit` -> `RayTriangleHit`
- General documentation and code quality improvements

### Removed / Not Ported

- Removed unused ray helpers and types, like `PrimitiveIntersection`
- Removed getters on intersection types, and made their properties
public
- There is no `2d` feature, and `Raycast::mesh_query` and
`Raycast::mesh2d_query` have been merged into `MeshRayCast::mesh_query`,
which handles both 2D and 3D
- I assume this existed previously because `Mesh2dHandle` used to be in
`bevy_sprite`. Now both the 2D and 3D mesh are in `bevy_render`.
- There is no `debug` feature or ray debug rendering
- There is no deferred API (`RaycastSource`)
- There is no `CursorRayPlugin` (the picking backend handles this)

### Note for Reviewers

In case it's helpful, the [first
commit](281638e)
here is essentially a one-to-one port. The rest of the commits are
primarily refactoring and cleaning things up in the ways listed earlier,
as well as changes to the module structure.

It may also be useful to compare the original [picking
backend](https://github.com/aevyrie/bevy_mod_picking/blob/74f0c3c0fbc8048632ba46fd8f14e26aaea9c76c/backends/bevy_picking_raycast/src/lib.rs)
and [`bevy_mod_raycast`](https://github.com/aevyrie/bevy_mod_raycast) to
this PR. Feel free to mention if there are any changes that I should
revert or something I should not include in this PR.

## Testing

I tested mesh picking and relevant components in some examples, for both
2D and 3D meshes, and added a new `mesh_picking` example. I also
~~stole~~ ported over the [ray-mesh intersection
benchmark](https://github.com/aevyrie/bevy_mod_raycast/blob/dbc5ef32fe48997a1a7eeec7434d9dd8b829e52e/benches/ray_mesh_intersection.rs)
from `bevy_mod_raycast`.

---

## Showcase

Below is a version of the `2d_shapes` example modified to demonstrate 2D
mesh picking. This is not included in this PR.


https://github.com/user-attachments/assets/7742528c-8630-4c00-bacd-81576ac432bf

And below is the new `mesh_picking` example:


https://github.com/user-attachments/assets/b65c7a5a-fa3a-4c2d-8bbd-e7a2c772986e

There is also a really cool new `mesh_ray_cast` example ported over from
`bevy_mod_raycast`:


https://github.com/user-attachments/assets/3c5eb6c0-bd94-4fb0-bec6-8a85668a06c9

---------

Co-authored-by: Aevyrie <[email protected]>
Co-authored-by: Trent <[email protected]>
Co-authored-by: François Mockers <[email protected]>

Author: 4 people

Cargo.toml

@@ -111,6 +111,7 @@ default = [
   "bevy_gilrs",
   "bevy_gizmos",
   "bevy_gltf",
+  "bevy_mesh_picking_backend",
   "bevy_pbr",
   "bevy_picking",
   "bevy_remote",
@@ -136,6 +137,9 @@ default = [
   "x11",
 ]
 
+# Provides an implementation for picking meshes
+bevy_mesh_picking_backend = ["bevy_picking"]
+
 # Provides an implementation for picking sprites
 bevy_sprite_picking_backend = ["bevy_picking"]
 
@@ -1213,6 +1217,17 @@ setup = [
   ],
 ]
 
+[[example]]
+name = "mesh_ray_cast"
+path = "examples/3d/mesh_ray_cast.rs"
+doc-scrape-examples = true
+
+[package.metadata.example.mesh_ray_cast]
+name = "Mesh Ray Cast"
+description = "Demonstrates ray casting with the `MeshRayCast` system parameter"
+category = "3D Rendering"
+wasm = true
+
 [[example]]
 name = "lightmaps"
 path = "examples/3d/lightmaps.rs"
@@ -3695,6 +3710,18 @@ description = "Demonstrates how to rotate the skybox and the environment map sim
 category = "3D Rendering"
 wasm = false
 
+[[example]]
+name = "mesh_picking"
+path = "examples/picking/mesh_picking.rs"
+doc-scrape-examples = true
+required-features = ["bevy_mesh_picking_backend"]
+
+[package.metadata.example.mesh_picking]
+name = "Mesh Picking"
+description = "Demonstrates picking meshes"
+category = "Picking"
+wasm = true
+
 [[example]]
 name = "simple_picking"
 path = "examples/picking/simple_picking.rs"

benches/Cargo.toml

@@ -14,6 +14,7 @@ bevy_app = { path = "../crates/bevy_app" }
 bevy_ecs = { path = "../crates/bevy_ecs", features = ["multi_threaded"] }
 bevy_hierarchy = { path = "../crates/bevy_hierarchy" }
 bevy_math = { path = "../crates/bevy_math" }
+bevy_picking = { path = "../crates/bevy_picking", features = ["bevy_mesh"] }
 bevy_reflect = { path = "../crates/bevy_reflect", features = ["functions"] }
 bevy_render = { path = "../crates/bevy_render" }
 bevy_tasks = { path = "../crates/bevy_tasks" }
@@ -37,6 +38,11 @@ name = "ecs"
 path = "benches/bevy_ecs/benches.rs"
 harness = false
 
+[[bench]]
+name = "ray_mesh_intersection"
+path = "benches/bevy_picking/ray_mesh_intersection.rs"
+harness = false
+
 [[bench]]
 name = "reflect_function"
 path = "benches/bevy_reflect/function.rs"

benches/benches/bevy_picking/ray_mesh_intersection.rs

@@ -0,0 +1,120 @@
+use bevy_math::{Dir3, Mat4, Ray3d, Vec3};
+use bevy_picking::{mesh_picking::ray_cast, prelude::*};
+use criterion::{black_box, criterion_group, criterion_main, Criterion};
+
+fn ptoxznorm(p: u32, size: u32) -> (f32, f32) {
+    let ij = (p / (size), p % (size));
+    (ij.0 as f32 / size as f32, ij.1 as f32 / size as f32)
+}
+
+struct SimpleMesh {
+    positions: Vec<[f32; 3]>,
+    normals: Vec<[f32; 3]>,
+    indices: Vec<u32>,
+}
+
+fn mesh_creation(vertices_per_side: u32) -> SimpleMesh {
+    let mut positions = Vec::new();
+    let mut normals = Vec::new();
+    for p in 0..vertices_per_side.pow(2) {
+        let xz = ptoxznorm(p, vertices_per_side);
+        positions.push([xz.0 - 0.5, 0.0, xz.1 - 0.5]);
+        normals.push([0.0, 1.0, 0.0]);
+    }
+
+    let mut indices = vec![];
+    for p in 0..vertices_per_side.pow(2) {
+        if p % (vertices_per_side) != vertices_per_side - 1
+            && p / (vertices_per_side) != vertices_per_side - 1
+        {
+            indices.extend_from_slice(&[p, p + 1, p + vertices_per_side]);
+            indices.extend_from_slice(&[p + vertices_per_side, p + 1, p + vertices_per_side + 1]);
+        }
+    }
+
+    SimpleMesh {
+        positions,
+        normals,
+        indices,
+    }
+}
+
+fn ray_mesh_intersection(c: &mut Criterion) {
+    let mut group = c.benchmark_group("ray_mesh_intersection");
+    group.warm_up_time(std::time::Duration::from_millis(500));
+
+    for vertices_per_side in [10_u32, 100, 1000] {
+        group.bench_function(format!("{}_vertices", vertices_per_side.pow(2)), |b| {
+            let ray = Ray3d::new(Vec3::new(0.0, 1.0, 0.0), Dir3::NEG_Y);
+            let mesh_to_world = Mat4::IDENTITY;
+            let mesh = mesh_creation(vertices_per_side);
+
+            b.iter(|| {
+                black_box(ray_cast::ray_mesh_intersection(
+                    ray,
+                    &mesh_to_world,
+                    &mesh.positions,
+                    Some(&mesh.normals),
+                    Some(&mesh.indices),
+                    ray_cast::Backfaces::Cull,
+                ));
+            });
+        });
+    }
+}
+
+fn ray_mesh_intersection_no_cull(c: &mut Criterion) {
+    let mut group = c.benchmark_group("ray_mesh_intersection_no_cull");
+    group.warm_up_time(std::time::Duration::from_millis(500));
+
+    for vertices_per_side in [10_u32, 100, 1000] {
+        group.bench_function(format!("{}_vertices", vertices_per_side.pow(2)), |b| {
+            let ray = Ray3d::new(Vec3::new(0.0, 1.0, 0.0), Dir3::NEG_Y);
+            let mesh_to_world = Mat4::IDENTITY;
+            let mesh = mesh_creation(vertices_per_side);
+
+            b.iter(|| {
+                black_box(ray_cast::ray_mesh_intersection(
+                    ray,
+                    &mesh_to_world,
+                    &mesh.positions,
+                    Some(&mesh.normals),
+                    Some(&mesh.indices),
+                    ray_cast::Backfaces::Include,
+                ));
+            });
+        });
+    }
+}
+
+fn ray_mesh_intersection_no_intersection(c: &mut Criterion) {
+    let mut group = c.benchmark_group("ray_mesh_intersection_no_intersection");
+    group.warm_up_time(std::time::Duration::from_millis(500));
+
+    for vertices_per_side in [10_u32, 100, 1000] {
+        group.bench_function(format!("{}_vertices", (vertices_per_side).pow(2)), |b| {
+            let ray = Ray3d::new(Vec3::new(0.0, 1.0, 0.0), Dir3::X);
+            let mesh_to_world = Mat4::IDENTITY;
+            let mesh = mesh_creation(vertices_per_side);
+
+            b.iter(|| {
+                black_box(ray_cast::ray_mesh_intersection(
+                    ray,
+                    &mesh_to_world,
+                    &mesh.positions,
+                    Some(&mesh.normals),
+                    Some(&mesh.indices),
+                    ray_cast::Backfaces::Cull,
+                ));
+            });
+        });
+    }
+}
+
+criterion_group!(
+    benches,
+    ray_mesh_intersection,
+    ray_mesh_intersection_no_cull,
+    ray_mesh_intersection_no_intersection
+);
+criterion_main!(benches);

crates/bevy_internal/Cargo.toml

@@ -214,9 +214,10 @@ bevy_dev_tools = ["dep:bevy_dev_tools"]
 # Enable support for the Bevy Remote Protocol
 bevy_remote = ["dep:bevy_remote"]
 
-# Provides a picking functionality
+# Provides picking functionality
 bevy_picking = [
   "dep:bevy_picking",
+  "bevy_picking/bevy_mesh",
   "bevy_ui?/bevy_picking",
   "bevy_sprite?/bevy_picking",
 ]

crates/bevy_picking/Cargo.toml

@@ -7,6 +7,10 @@ homepage = "https://bevyengine.org"
 repository = "https://github.com/bevyengine/bevy"
 license = "MIT OR Apache-2.0"
 
+[features]
+# Provides a mesh picking backend
+bevy_mesh = ["dep:bevy_mesh", "dep:crossbeam-channel"]
+
 [dependencies]
 bevy_app = { path = "../bevy_app", version = "0.15.0-dev" }
 bevy_asset = { path = "../bevy_asset", version = "0.15.0-dev" }
@@ -15,13 +19,15 @@ bevy_ecs = { path = "../bevy_ecs", version = "0.15.0-dev" }
 bevy_hierarchy = { path = "../bevy_hierarchy", version = "0.15.0-dev" }
 bevy_input = { path = "../bevy_input", version = "0.15.0-dev" }
 bevy_math = { path = "../bevy_math", version = "0.15.0-dev" }
+bevy_mesh = { path = "../bevy_mesh", version = "0.15.0-dev", optional = true }
 bevy_reflect = { path = "../bevy_reflect", version = "0.15.0-dev" }
 bevy_render = { path = "../bevy_render", version = "0.15.0-dev" }
 bevy_time = { path = "../bevy_time", version = "0.15.0-dev" }
 bevy_transform = { path = "../bevy_transform", version = "0.15.0-dev" }
 bevy_utils = { path = "../bevy_utils", version = "0.15.0-dev" }
 bevy_window = { path = "../bevy_window", version = "0.15.0-dev" }
 
+crossbeam-channel = { version = "0.5", optional = true }
 uuid = { version = "1.1", features = ["v4"] }
 
 [lints]

crates/bevy_picking/src/lib.rs

@@ -156,6 +156,8 @@ pub mod backend;
 pub mod events;
 pub mod focus;
 pub mod input;
+#[cfg(feature = "bevy_mesh")]
+pub mod mesh_picking;
 pub mod pointer;
 
 use bevy_app::prelude::*;
@@ -166,6 +168,12 @@ use bevy_reflect::prelude::*;
 ///
 /// This includes the most common types in this crate, re-exported for your convenience.
 pub mod prelude {
+    #[cfg(feature = "bevy_mesh")]
+    #[doc(hidden)]
+    pub use crate::mesh_picking::{
+        ray_cast::{MeshRayCast, RayCastBackfaces, RayCastSettings, RayCastVisibility},
+        MeshPickingBackend, MeshPickingBackendSettings, RayCastPickable,
+    };
     #[doc(hidden)]
     pub use crate::{
         events::*, input::PointerInputPlugin, pointer::PointerButton, DefaultPickingPlugins,
@@ -274,6 +282,8 @@ impl Plugin for DefaultPickingPlugins {
             PickingPlugin::default(),
             InteractionPlugin,
         ));
+        #[cfg(feature = "bevy_mesh")]
+        app.add_plugins(mesh_picking::MeshPickingBackend);
     }
 }
 

crates/bevy_picking/src/mesh_picking/mod.rs

@@ -0,0 +1,133 @@
+//! A [mesh ray casting](ray_cast) backend for [`bevy_picking`](crate).
+//!
+//! By default, all meshes are pickable. Picking can be disabled for individual entities
+//! by adding [`PickingBehavior::IGNORE`].
+//!
+//! To make mesh picking entirely opt-in, set [`MeshPickingBackendSettings::require_markers`]
+//! to `true` and add a [`RayCastPickable`] component to the desired camera and target entities.
+//!
+//! To manually perform mesh ray casts independent of picking, use the [`MeshRayCast`] system parameter.
+
+pub mod ray_cast;
+
+use crate::{
+    backend::{ray::RayMap, HitData, PointerHits},
+    prelude::*,
+    PickSet,
+};
+use bevy_app::prelude::*;
+use bevy_ecs::prelude::*;
+use bevy_reflect::prelude::*;
+use bevy_render::{prelude::*, view::RenderLayers};
+use ray_cast::{MeshRayCast, RayCastSettings, RayCastVisibility, SimplifiedMesh};
+
+/// Runtime settings for the [`MeshPickingBackend`].
+#[derive(Resource, Reflect)]
+#[reflect(Resource, Default)]
+pub struct MeshPickingBackendSettings {
+    /// When set to `true` ray casting will only happen between cameras and entities marked with
+    /// [`RayCastPickable`]. `false` by default.
+    ///
+    /// This setting is provided to give you fine-grained control over which cameras and entities
+    /// should be used by the mesh picking backend at runtime.
+    pub require_markers: bool,
+
+    /// Determines how mesh picking should consider [`Visibility`]. When set to [`RayCastVisibility::Any`],
+    /// ray casts can be performed against both visible and hidden entities.
+    ///
+    /// Defaults to [`RayCastVisibility::VisibleInView`], only performing picking against visible entities
+    /// that are in the view of a camera.
+    pub ray_cast_visibility: RayCastVisibility,
+}
+
+impl Default for MeshPickingBackendSettings {
+    fn default() -> Self {
+        Self {
+            require_markers: false,
+            ray_cast_visibility: RayCastVisibility::VisibleInView,
+        }
+    }
+}
+
+/// An optional component that marks cameras and target entities that should be used in the [`MeshPickingBackend`].
+/// Only needed if [`MeshPickingBackendSettings::require_markers`] is set to `true`, and ignored otherwise.
+#[derive(Debug, Clone, Default, Component, Reflect)]
+#[reflect(Component, Default)]
+pub struct RayCastPickable;
+
+/// Adds the mesh picking backend to your app.
+#[derive(Clone, Default)]
+pub struct MeshPickingBackend;
+
+impl Plugin for MeshPickingBackend {
+    fn build(&self, app: &mut App) {
+        app.init_resource::<MeshPickingBackendSettings>()
+            .register_type::<(RayCastPickable, MeshPickingBackendSettings, SimplifiedMesh)>()
+            .add_systems(PreUpdate, update_hits.in_set(PickSet::Backend));
+    }
+}
+
+/// Casts rays into the scene using [`MeshPickingBackendSettings`] and sends [`PointerHits`] events.
+#[allow(clippy::too_many_arguments)]
+pub fn update_hits(
+    backend_settings: Res<MeshPickingBackendSettings>,
+    ray_map: Res<RayMap>,
+    picking_cameras: Query<(&Camera, Option<&RayCastPickable>, Option<&RenderLayers>)>,
+    pickables: Query<&PickingBehavior>,
+    marked_targets: Query<&RayCastPickable>,
+    layers: Query<&RenderLayers>,
+    mut ray_cast: MeshRayCast,
+    mut output: EventWriter<PointerHits>,
+) {
+    for (&ray_id, &ray) in ray_map.map().iter() {
+        let Ok((camera, cam_pickable, cam_layers)) = picking_cameras.get(ray_id.camera) else {
+            continue;
+        };
+        if backend_settings.require_markers && cam_pickable.is_none() {
+            continue;
+        }
+
+        let cam_layers = cam_layers.to_owned().unwrap_or_default();
+
+        let settings = RayCastSettings {
+            visibility: backend_settings.ray_cast_visibility,
+            filter: &|entity| {
+                let marker_requirement =
+                    !backend_settings.require_markers || marked_targets.get(entity).is_ok();
+
+                // Other entities missing render layers are on the default layer 0
+                let entity_layers = layers.get(entity).cloned().unwrap_or_default();
+                let render_layers_match = cam_layers.intersects(&entity_layers);
+
+                let is_pickable = pickables
+                    .get(entity)
+                    .map(|p| p.is_hoverable)
+                    .unwrap_or(true);
+
+                marker_requirement && render_layers_match && is_pickable
+            },
+            early_exit_test: &|entity_hit| {
+                pickables
+                    .get(entity_hit)
+                    .is_ok_and(|pickable| pickable.should_block_lower)
+            },
+        };
+        let picks = ray_cast
+            .cast_ray(ray, &settings)
+            .iter()
+            .map(|(entity, hit)| {
+                let hit_data = HitData::new(
+                    ray_id.camera,
+                    hit.distance,
+                    Some(hit.point),
+                    Some(hit.normal),
+                );
+                (*entity, hit_data)
+            })
+            .collect::<Vec<_>>();
+        let order = camera.order as f32;
+        if !picks.is_empty() {
+            output.send(PointerHits::new(ray_id.pointer, picks, order));
+        }
+    }
+}