Notes from Face To Face 4/21/2022

[nbutko]

We discussed the current shape of the API at the Immersive Web Working Group Face to Face.

There’s a growing recognition in the working group that a lack of camera access has held back adoption of WebXR compared to alternatives like getUserMedia. The ironic effect of having an artificially limited api is that it pushes people toward less privacy sensitive alternatives. The goal should be to have user consent that is potentially improved from existing flows (whatever that entails) but is powerful enough to empower the gamut of valuable developer use cases.

In order to make the current proposal fully compatible with headsets, the following changes might be needed

• frames are not coupled to XR frames, but come on a callback or some other mechanism
• frames are timestamped
• camera to viewer transform
• field of view for camera is provided

Here are some examples of reasons developers choose use alternatives to WebXR because this API is lacking:

• Custom Visual Effects: https://diamonds.samsmithworld.com/ renders diamonds with custom refraction of the camera feed.
• High quality reflections: https://www.8thwall.com/alivenow/freefire renders a metallic paint car with the users' environment reflected.
• Responsive Scale: https://www.8thwall.com/beemwebar/hearst-philips shows a hologram discussing body positivity that can be viewed comfortably at room scale or table scale.
• Photo Sharing: https://www.8thwall.com/aircards/dune-avatar enables photo sharing of a custom Dune avatar, with a Dune watermark embedded in the photo.
• Video Sharing: https://www.8thwall.com/alivenow/cadbury-fc-webar has users record and share a video with audio and embedded images painting their face with colors of their favorite soccer teams.
• Image Target Effects: https://www.8thwall.com/toolofnorthamerica/pizza-hut-pacman is a Pac Man game on top of a pizza box.
• Curved Image Target Effects: https://www.8thwall.com/tactic/white-castle animates comic on White Castle cups.
• Face Effects: https://www.jlab.com/pages/fitting-room lets users try on various audio accessories before purchasing.
• QR Code Scanning: https://8th.io/scan is a web assembly QR code scanner.
• Custom TensorFlow Models: https://www.campaignasia.com/article/heineken-asks-singapore-to-race-an-f1-car-in-ar/453689 detects when two Heineken bottles "cheers".
• Hand Effects: https://blog.tensorflow.org/2021/11/3D-handpose.html allows pages to detect hand position.
• Foot Effects: https://app2.vyking.io/vyking-web-ar/#/web lets users try on Air Jordans.
• Body Effects: https://www.auganix.org/geenee-announces-webar-full-body-tracking-sdk-for-3d-augmented-reality-try-on-and-avatars/ enables 3D WebAR try-on.

[tangobravo]

Thanks Nick. I believe this is one of two major barriers to adoption of WebXR on handheld devices - the other is around the "mode switch" of the immersive-ar session.

Camera access is a requirement for some (probably the majority) of our experiences for many of the same reasons as those you listed. However even with camera access, the limitations of the immersive-ar session would still prevent us from adopting WebXR in our tools and libraries - I'll talk about this more in the F2F session today.

In terms of the camera access API shape, the current proposal is acknowledged as being smartphone-only. I guess the overall question is do we attempt to move this current proposal forward (ie trying to work through the TAG issues and making it available without a flag in Chrome), or consider a more universal API for both headsets and phones. Or both?

From our side if the API meets our needs I can commit to integrating into Zappar's tools and libraries, which would probably result in immediate and significant usage of WebXR in the wild in our commercial projects and those of our partners. I think a simplified camera-first API that would also be applicable to headsets is achievable, but it would obviously need some interest from a browser vendor (likely Chromium in the first instance) to work on an alternative API.