Overlay flipping when directly above

[meir]

When laying down in VR, putting a window above your head flips it around when passing behind your headset on a vertical axis. As if the location of the overlay is based on a cylinder with the ends stitched.

Ideally the overlay would stay in orientation of your view and not based on location.

[galister]

this is because of the realign function (which sets the overlay rotation upon/after grab). this uses a manual implementation of look_at_rh, which is bound to the global up vector.

would need to come up with a way that handles well in both mostly-upright cases (overlay needs to snap to horizon), but also doesn't break at extremities like what you've mentioned.

this is the part, if you want to try your hand at it:

wlx-overlay-s/src/backend/overlay.rs

Lines 175 to 190 in 659f149

	let dot = to_hmd.normalize().dot(hmd.z_axis);
	let z_dist = to_hmd.length();
	let y_dist = (self.transform.translation.y - hmd.translation.y).abs();
	let x_angle = (y_dist / z_dist).asin();
	if dot < -f32::EPSILON {
	// facing down
	let up_point = hmd.translation + z_dist / x_angle.cos() * Vec3A::Y;
	up_dir = (up_point - self.transform.translation).normalize();
	} else if dot > f32::EPSILON {
	// facing up
	let dn_point = hmd.translation + z_dist / x_angle.cos() * Vec3A::NEG_Y;
	up_dir = (self.transform.translation - dn_point).normalize();
	} else {
	// perfectly upright
	up_dir = Vec3A::Y;

[meir]

yes thank you, that helps quite a bit, i'll look at it later this week