CR Request for Web Neural Network API webnn

[dontcallmedom]

Document title, URLs, estimated publication date

Title: Web Neural Network API
URL: https://www.w3.org/TR/webnn/
Date: whenever transition is approved

Abstract

This document describes a dedicated low-level API for neural network inference hardware acceleration.

Status

https://www.w3.org/TR/webnn/#sotd amended for CR, and with additional note prepared in https://github.com/webmachinelearning/webnn/pull/340/files

Link to group's decision to request transition

https://lists.w3.org/Archives/Public/public-webmachinelearning-wg/2023Mar/0000.html

Changes

N/A (1st CR)

Requirements satisfied

As documented in webmachinelearning/webnn#240

Dependencies met (or not)

The normative dependencies haven't grown since FPWD (dependency to WebGL has actually been removed). The most significant dependency is to WebGPU and the spec may have to evolve significantly based on experience of implementing bridges between WebNN and WebGPU.

Wide Review

Documented in webmachinelearning/webnn#239

Issues addressed

The group address 95 issues; 68 remain open, including:

• 14 editorial
• 7 targeted for a v2
• 2 related to testing
• 4 are used for process related tracking
• 3 are use cases discussion

The rest we expect to be informed by implementation and adoption experience (in particular the ones on defining the right set of operations to support).

Formal Objections

None

Implementation

• web-platform-tests test suite /
  WPT tests tracker webmachinelearning/webnn#265
• Preliminary implementation reports based on one or more of:
  • JS implementations: webnn-baseline (pure JS), webnn-polyfill (use TF.js backends)
  • Chromium implementation: Intent to Prototype, source, tracker bug, issues, patches
  • Chromium backends: XNNPACK and DirectML (design doc)

Patent disclosures

https://www.w3.org/groups/wg/webmachinelearning/ipr/

cc @anssiko

[marcoscaceres]

Entering CR based on a single browser prototype implementation and some JS libraries seems somewhat concerning/premature. Is the working group sure this can be interoperably implemented across many platforms? (the two backends seem insufficient, tbh)

[anssiko]

@marcoscaceres, I’m happy to see also you are interested in this API and its implementations! In short, cross-platform interoperability has been a key consideration for this API from the beginning due to the nature of this domain.

The API is not tied to any particular platform and the WG has paid attention that the API is implementable on top of existing major platform APIs, such as Android Neural Networks API, Windows DirectML, Core ML and macOS/iOS Metal Performance Shaders and Basic Neural Network Subroutines as codified in the WG Charter. As a bonus, the API is polyfillable on top of WebGL, WebGPU and WebAssembly. There’s also a pure-JS implementation.

The WG has also developed a test suite with domain experts that goes beyond your typical expectations. It pays special attention to numerical precision.

If you’re interested in learning more about the implementations of this API, I’m happy to invite you as a guest to our future calls so you can ask questions from the experts in this domain. We have already received encouraging feedback from your colleagues at the recent F2F. We discussed some of that feedback at our bi-weekly yesterday and welcome you folks on board in an official capacity.

[marcoscaceres]

As stated by Chai

… the core feedback is the explicit device type may not work well with Apple's architecture, secondly, Core ML works on textures, WebNN on textures and buffers

That's concerning 😢 I'm not sure who gave that feedback, but that sounds like something pretty critical to fix before going to CR.

And then:

Apple discussed a chopping concept for running a graph across multiple devices
… excluding a GPU use case was mentioned by Apple, did you Chai consider that in your open PR?
… Chai: that open PR just consolidates the GPU code path it does not tackle that specific feedback explicitly

Which again is concerning.

Out of interest, who was it from Apple that gave the feedback and where? I can ping them for details.

[anssiko]

@marcoscaceres those quotes (I minuted, possible errors mine) are nothing that you should worry about in the context of this CR transition. Please provide your feedback in webmachinelearning/webnn#350 and we’ll invite you to our next meeting and help you understand this API so also you can start make helpful contributions. I’m sorry to say, but currently you are just confusing things here.

[marcoscaceres]

are nothing that you should worry about in the context of this CR transition.

Sorry, I'm confused. The above suggests the spec is not implementable on Apple's platforms and you are saying I shouldn't be worried? I think Apple's AC rep would formally object to the spec proceeding to CR if Apple folks are not confident it's implementable on multiple backends (specially if Apple's has provided feedback already with those concerns).

We risk again getting into a single-engine situation, which would be extremely unfortunate and something we should avoid. I can try to get WebKit folks to provide more feedback in the meantime - but it would send the wrong message to publish a CR if the WG has received feedback already the it may not work with one of the platforms/frameworks (per webmachinelearning/webnn#350).

[anssiko]

@marcoscaceres with all due respect, this is a Candidate Recommendation transition, not a Proposed Recommendation transition.

[marcoscaceres]

The Process document for Transitioning to Candidate Recommendation say:

Advisory Committee representatives may initiate an Advisory Committee Appeal of the decision to advance the technical report.

If the document proceeds to CR it means that the group believes it's implementable. Folks from Apple are saying it's not implementable (webmachinelearning/webnn#350) so this shouldn't go to CR (because it will just get kicked back to WD again). It's sending the wrong signal that the document is ready, when folks from Apple are saying it is not.

It would be better to keep it at WD until webmachinelearning/webnn#350 is addressed.

[anssiko]

This CR allows the WG to formally collect implementation experience to demonstrate that the specification works in practice. Helpful contributions at this stage are technical contributions toward the specification, further implementation experience and user feedback. An appeal is not considered a helpful contribution.

The WG is specifically interested in further implementation feedback with regard to the MLCommandEncoder interface and continues to gather further implementation experience to enable more efficient WebGPU integration. The spec may evolve based on this experience. This is all normal for a CR and is all noted in the Status section.

The information brought to the WG's attention suggests the API is implementable on all major platforms (I want to clarify this because I see confusing out of context quotes misrepresented in this issue). The WG has concrete evidence of this already: XNNPACK that is the backend of the first implementation is supported across architectures, including ARM64 on iOS and macOS. More implementations are in development.

[marcoscaceres]

Sure, but webmachinelearning/webnn#350 remains an issue. If the working group knows it's going to go back to CR (and possibly having to completely refactor the API), then why go to CR already? Saying it works with XNNPACK isn't particularly helpful.

[plehegar]

webmachinelearning/webnn#350 is vague and it's not clear if this is a use case which isn't addressed by the specification, or if it is purely a concern about the specification being implementable. Depending on which way it is, it is a different conversation.

[marcoscaceres]

It's a concern about the specification [not] being implementable on Apple's platforms/frameworks. WebKit folks are evaluating how/if we can join the working group and hopefully provide guidance towards addressing the issues. However, we are concerned about the spec going to CR prematurely for the reasons I already gave: It sends a bad signal and we don't want the design/API shape to be locked in before an opportunity to provide adequate feedback.

[plehegar]

The definition of a CR says:
[[
A Candidate Recommendation is a document that satisfies the technical requirements of the Working Group that produced it and their dependencies, and has already received wide review. W3C publishes a Candidate Recommendation to

• signal to the wider community that it is time to do a final review
• gather implementation experience
  ]]
  Candidate Recommendation (CR)

Unless webmachinelearning/webnn#350 gets refined within a week, I don't see a reason to block this transition, which fits the purpose of moving to CR. If implementation feedback is raised which require revising the design, it is expected to send the document back to Working Draft status.

cc @marcoscaceres

[plehegar]

One more thought:

I understand that the concern is about native implementation of this specification on Apple's platforms/frameworks and the integration with WebGPU.

Tt may be that we need to refine the CR exit criteria for this specification.

What are the proposed CR exit criteria for this specification and what is the minimum duration for this CR?

[anssiko]

@plehegar, your questions were probably more geared toward @dontcallmedom, but here's my input for your consideration:

I'd prefer to use the success criteria from the proposed charter that adds an additional explicit expectation with regard to open test suites.

I'll defer to @dontcallmedom for the minimum duration for this CR. From the WG leadership point of view, the WG is not in a hurry to exit CR and wants to give all interested implementers ample time to join the effort and provide feedback.

[dontcallmedom]

Looking at the draft charter under AC review, I think the following lines would make sense to surface in the CR exit criteria:

implementable on top of existing major platform APIs, such as Android Neural Networks API, Windows DirectML, and macOS/iOS Metal Performance Shaders and Basic Neural Network Subroutines.

at least two independent, interoperable implementations of every feature defined in the specification, where interoperability can be verified by passing open test suites, and two or more implementations interoperating with each other. In order to advance to Proposed Recommendation, each normative specification must have an open test suite of every feature defined in the specification.

In terms of minimum CR duration, we could probably go with 6 months?

[plehegar]

Approved with the updated exit criteria.

[wchao1115]

FWIW, @marcoscaceres and @plehegar, I'm the author of WebNN issue #350. My apologies that the issue lacks some details. Will update it soon. Since it's related to what being discussed here regarding Apple's position on WebNN's implementability, please allow me to clarify. Overall, Apple believes that WebNN is implementable with ANE (Apple Neural Engine) b/c WebNN is format-agnostic and that its interface is flexible enough to allow ANE to target compile the ML graph at the right time. They, however, noted an issue that we should also consider, which has to do with an ability to target multiple devices within the same graph. This is a legitimate feedback that we're looking into, but is not considered an implementation blocker.

As for the GPU texture support issue, it wasn't directed at WebNN but rather at DirectML as an implementation note i.e. a comparison was made relative to DirectML exclusive support of GPU buffers. The current WebNN spec defines WebGPU interop support for both GPU buffer and texture. Hope this clears up some confusion.

[dontcallmedom]

CR published https://www.w3.org/TR/2023/CR-webnn-20230330/

[marcoscaceres]

Pretty disappointing tbh :( Don't complain when we need to go back to WD when it doesn't work with Apple's frameworks. We did warn you.