Use cases for DOM Overlays in VR

[AdaRoseCannon]

It would be useful to collect some use cases for how DOM Overlay's are being used in VR for potential implementations.

This example by @mrdoob is a great use case: https://twitter.com/mrdoob/status/1385184290867187715

[digisomni]

So for me, a major part of this is the ability to just make UI using web technology instead of trying to make UI components from scratch. I can create a Vue app with Vuetify and have that serve as a control panel for desktop and VR users alike.

Another major feature is the ability to display and have interactive media, such as screenshare or synced video watching. This allows us to rely on existing web implementations instead of trying to manually program systems for use in virtual worlds.

For example, here is a web app for street videos being used in a VR world. We can and do also throw up movies on similar screens.

[AdaRoseCannon]

Making UI elements with Vue is a good usecase, for the other ones:

In this case videos are best done using WebXR Layers for best performance and user experience.

Screenshare is interesting, @cabanier do you think a WebRTC Video can be put through a WebXR layer? Since you set the playback on the video element using:

videoEl.srcObject = remoteStream;

[cabanier]

Screenshare is interesting, @cabanier do you think a WebRTC Video can be put through a WebXR layer?

Yes, any video element can become the source for a video layer.

[felixmariotto]

Some broad use cases :

• forms ( login, messaging UI... )
• caption ( painting title, nametag above player... )
• site navigation tools and information ( current page, links... )
• input tools ( checkboxes, virtual keyboard... )
• media display ( image, video, iframes... )
• big text display ( IDE, ebook, blog... )
• file system ( import a file, save as... )

[cabanier]

A big issue with today's definition of DOM Overlays is that they are drawn on top of the VR scene.
Unless we change that, I suspect that they can only be used for non-interactive content.

[felixmariotto]

Unless we change that, I suspect that they can only be used for non-interactive content.

@cabanier why ? because the controllers would be drawn behind the UI ? A solution may be for the browsers to provide a stencil buffer so we can write on it to tell which parts of the top-drawn UI should be trimmed out to show what's behind ? Blind guess, I don't know if it's feasible.

[klausw]

@cabanier - https://immersive-web.github.io/dom-overlays/#xrsessioninit currently says "The DOM content MUST be composited as if it were the topmost content layer. It MUST NOT be occluded by content from the XRWebGLLayer or by images from a passthrough camera for an AR device."

This requirement intentionally does NOT apply to UI elements drawn by the UA directly. For example, it would be OK for the UA to provide a visible pointer ray, target reticle on the DOM layer, and/or a hand/controller model while the user appears to be interacting with the DOM overlay. This is also the only way to support interactions with cross-origin content in an iframe since the application isn't allowed to get poses in that case, so the application couldn't draw a pointer ray that intersects interactive cross-origin content. (See Event handling for cross-origin content. Note that the pose restriction doesn't apply if the UA treats cross-origin content as noninteractive.)

For use cases, in general browsers provide a lot of functionality for HTML pages that would be difficult to replicate in applications, for example:

• Accessibility features including text-to-speech support such as screen readers. (This can be helpful for visually impaired users, including someone who can't use their prescription lenses in a headset.)
• Automatic text translation. (The application may not directly support the user's native language.)
• Displaying third-party content that can't be accessed by the application due to cross-origin restrictions. (Example: live chat from a streaming site.)

[klausw]

In general, the floating-screen use case isn't covered in much detail in the current version of the specification. This is open for suggestions and/or additional features, for example it could be useful to provide a per-frame status value if the UA and application should cooperate on drawing pointer rays, or a stencil mask if appropriate.

[cabanier]

This requirement intentionally does NOT apply to UI elements drawn by the UA directly. For example, it would be OK for the UA to provide a visible pointer ray, target reticle on the DOM layer, and/or a hand/controller model while the user appears to be interacting with the DOM overlay.

I agree but we would need to change the spec to allow that.
There's also the issue that people will want more than one surface which is problematic because of DOM Overlay's use of the full screen API

[klausw]

I agree but we would need to change the spec to allow that.

Changing the spec is definitely an option as long as it's not inherently incompatible with current usage. For example, the spec could easily be augmented to say how such elements should be handled for the "floating" type, and clarifying that the UA isn't expected to draw such affordances for the "screen" type where there's no ambiguity about the touch location.

There's also the issue that people will want more than one surface which is problematic because of DOM Overlay's use of the full screen API

DOM Overlay doesn't require using the Fullscreen API, but it is restricted to a single element / surface. Input disambiguation is already quite complex with just one element. As we had discussed during previous occasions, it seems feasible to add a DOM surface type to the Layers API for noninteractive elements, but fully interactive content on multiple arbitrarily-placed surfaces seems quite tricky.

[digisomni]

A big issue with today's definition of DOM Overlays is that they are drawn on top of the VR scene.
Unless we change that, I suspect that they can only be used for non-interactive content.

More than that, it's not possible to make apps and other nifty things and toss them in the world then. For example, there are movie screens and radios.

One of the greatest things about being able to add websites into a scene is that you can take traditional (WebXR unrelated) web content and make it relevant to the world. From livestreams and screenshare to things like internet radio. e.g. http://radio.garden/

[cabanier]

There's also the issue that people will want more than one surface which is problematic because of DOM Overlay's use of the full screen API

DOM Overlay doesn't require using the Fullscreen API, but it is restricted to a single element / surface. Input disambiguation is already quite complex with just one element. As we had discussed during previous occasions, it seems feasible to add a DOM surface type to the Layers API for noninteractive elements, but fully interactive content on multiple arbitrarily-placed surfaces seems quite tricky.

No, DOM Layers will allow for fully interactive content as long as its same origin.
Input and the drawing of the controllers is handled by the UA.

[digisomni]

In general, the floating-screen use case isn't covered in much detail in the current version of the specification. This is open for suggestions and/or additional features, for example it could be useful to provide a per-frame status value if the UA and application should cooperate on drawing pointer rays, or a stencil mask if appropriate.

If it helps any: I will say the floating screen use case is extremely important. It allows us to make interesting interactive UI out of 2D web elements.

Something that's always been quite good and interesting was the pioneering of Microsoft with the Cliff House. It took the Hololens interactivity and put that into WMR. I think more UIs should have the ability to snap and place windows but also get them to follow you, letting the user pick and choose what they would like to make a UI element. Think of the power of being able to play with your screen space on Windows or Linux and then imagine that level of control in VR. Spatial web windows allow for us to do something akin to a window manager for VR.

[klausw]

There's also the issue that people will want more than one surface which is problematic because of DOM Overlay's use of the full screen API

DOM Overlay doesn't require using the Fullscreen API, but it is restricted to a single element / surface. Input disambiguation is already quite complex with just one element. As we had discussed during previous occasions, it seems feasible to add a DOM surface type to the Layers API for noninteractive elements, but fully interactive content on multiple arbitrarily-placed surfaces seems quite tricky.

No, DOM Layers will allow for fully interactive content as long as its same origin.
Input and the drawing of the controllers is handled by the UA.

Thanks for clarifying. Would it be more accurate to say that an implementation or API can basically choose two out of these three features for DOM content, but not all at once?

• interactive
• cross-origin
• arbitrary layer placement and occlusion

[cabanier]

Thanks for clarifying. Would it be more accurate to say that an implementation or API can basically choose two out of these three features for DOM content, but not all at once?

• interactive
• cross-origin
• arbitrary layer placement and occlusion

Yes :-)