[JS] Implement ASRPipeline#4132
Draft
Retribution98 wants to merge 1 commit into
Draft
Conversation
Signed-off-by: Kirill Suvorov <kirill.suvorov@intel.com>
github-actions
Bot
added
no-match-files
category: GH Pages Docs
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Node.js (N-API + TypeScript) support for ov::genai::ASRPipeline, exposing a JS/TS ASRPipeline that dispatches to Whisper or Qwen3-ASR based on model config.json, including streaming, tokenizer access, generation config access, decoded-result marshalling, and ASR-specific performance metrics.
Changes:
- Introduces native N-API wrappers for
ASRPipeline+ASRPerfMetrics, plus JS/TS API, decoded results/types, and addon registration. - Adds JS sample for automatic speech recognition (WAV decoding helper + README) and updates Python sample test to run the new JS sample.
- Adds JS tests covering creation, generate/stream, tokenizer, get/set generation config, timestamps, perf metrics, and Qwen3-ASR dispatch; adds a tiny-random Qwen3-ASR model to JS test setup.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/python_tests/samples/test_automatic_speech_recognition.py | Switches JS sample invocation to the new ASR JS sample. |
| src/js/tests/setup.py | Adds tiny-random Qwen3-ASR model download/conversion for JS tests. |
| src/js/tests/asrPipeline.test.js | Adds ASRPipeline JS test coverage (Whisper + Qwen3-ASR dispatch). |
| src/js/src/helper.cpp | Adds ASR config marshalling, decoded-result conversion, and ASR perf-metrics unwrap. |
| src/js/src/asr_pipeline/pipeline_wrapper.cpp | Implements N-API wrapper for ASRPipeline (async init, threaded generate + streamer). |
| src/js/src/asr_pipeline/perf_metrics.cpp | Implements N-API wrapper for ASRPerfMetrics with ASR-specific metrics accessors. |
| src/js/src/asr_pipeline/init_worker.cpp | Adds async init worker creating ov::genai::ASRPipeline. |
| src/js/src/addon.cpp | Registers ASRPipeline + ASRPerfMetrics in the addon module. |
| src/js/lib/utils.ts | Adds ASRGenerationConfig and ASRPipelineProperties TS types. |
| src/js/lib/pipelines/asrPipeline.ts | Adds TS ASRPipeline class + factory-facing API (generate, stream, config/tokenizer access). |
| src/js/lib/perfMetrics.ts | Adds TS types/interfaces for ASR perf metrics and raw metrics. |
| src/js/lib/index.ts | Exposes ASRPipeline + related decoded results and perf metrics from the public JS entrypoint. |
| src/js/lib/decodedResults.ts | Adds ASRDecodedResults + timestamp chunk types. |
| src/js/lib/addon.ts | Adds TS addon interface typing for native ASRPipeline binding. |
| src/js/include/helper.hpp | Declares ASR perf-metrics unwrap and ASR decoded-result conversion helpers. |
| src/js/include/asr_pipeline/pipeline_wrapper.hpp | Declares ASRPipeline N-API wrapper. |
| src/js/include/asr_pipeline/perf_metrics.hpp | Declares ASRPerfMetrics N-API wrapper. |
| src/js/include/asr_pipeline/init_worker.hpp | Declares ASRInitWorker. |
| src/js/include/addon.hpp | Extends AddonData with ASR pipeline + perf metrics prototypes. |
| site/docs/bindings/node-js.md | Documents ASRPipeline availability in Node.js bindings. |
| samples/js/automatic_speech_recognition/wav_utils.js | Adds WAV PCM16 mono/stereo 16 kHz reader for sample. |
| samples/js/automatic_speech_recognition/README.md | Adds documentation for the new JS ASR sample usage. |
| samples/js/automatic_speech_recognition/automatic_speech_recognition.js | Adds runnable JS ASR sample using ASRPipeline + WAV helper. |
Comment on lines
+204
to
+208
| context->callback_tsfn = | ||
| Napi::ThreadSafeFunction::New(env, callback, "ASR_generate_callback", 0, 1, [context](Napi::Env) { | ||
| context->native_thread.join(); | ||
| delete context; | ||
| }); |
Comment on lines
+143
to
+146
| async return() { | ||
| streamingStatus = StreamingStatus.CANCEL; | ||
| return { done: true, value: "" }; | ||
| }, |
|
|
||
| ## Prepare audio file | ||
|
|
||
| Prepare audio file in wav format with sampling rate 16k Hz. |
|
|
||
| :::warning API Coverage | ||
| Node.js bindings currently provide a subset of the full OpenVINO GenAI API available in C++ and Python. The focus is on core text generation (`LLMPipeline`), vision language models (`VLMPipeline`), text embedding (`TextEmbeddingPipeline`), text reranking (`TextRerankPipeline`), speech recognition (`WhisperPipeline`), speech generation (`Text2SpeechPipeline`), and image generation (`Text2ImagePipeline`, `Image2ImagePipeline`, `InpaintingPipeline`) functionality. | ||
| Node.js bindings currently provide a subset of the full OpenVINO GenAI API available in C++ and Python. The focus is on core text generation (`LLMPipeline`), vision language models (`VLMPipeline`), text embedding (`TextEmbeddingPipeline`), text reranking (`TextRerankPipeline`), speech recognition (`WhisperPipeline`, `ASRPipeline`), speech generation (`Text2SpeechPipeline`), and image generation (`Text2ImagePipeline`, `Image2ImagePipeline`, `InpaintingPipeline`) functionality. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add Node.js (N-API) bindings for
ov::genai::ASRPipeline, the automatic speechrecognition pipeline that dispatches to the concrete model implementation
(Whisper or Qwen3-ASR) based on the model's
config.json.What's included
src/js/src/asr_pipeline/*,src/js/include/asr_pipeline/*):ASRPipelineWrapper(asyncinit, threadedgeneratewith optional streamer,getTokenizer,get/setGenerationConfig),ASRInitWorker, andASRPerfMetricsWrapper.src/js/lib/pipelines/asrPipeline.ts+index.ts):ASRPipelinefactory withgenerate(),stream()(async iterator),getTokenizer(),get/setGenerationConfig().ASRDecodedResults(texts, scores, detectedlanguages, nestedsegment/word timestamps),
ASRDecodedResultChunk,ASRPerfMetrics(features-extraction / word-timestamps / encode / decode durations),
ASRGenerationConfig(Whisper + Qwen3-ASRcontext),ASRPipelineProperties.src/js/src/helper.cpp,include/helper.hpp):cpp_to_js<ASRGenerationConfig>,to_asr_decoded_result,ASRPerfMetricsunwrap, and registration in
addon.cpp/addon.hpp.samples/js/automatic_speech_recognition/) with WAV decodinghelper and README.
site/docs/bindings/node-js.mdlistsASRPipeline.Implementation notes
AudioInputsto avoid an extra full-sizecopy from the implicit
std::vector<float>→std::variantconversion.suppress_tokens/begin_suppress_tokensare marshalled asstd::vector<int64_t>, so empty arrays (used to clear suppression) no longerthrow during config conversion.
CVS-190146
Checklist:
src/js/tests/asrPipeline.test.js(creation,generate/stream,streamer, tokenizer, get/set generation config, segment/word timestamps,
ASRPerfMetrics, empty suppress-token arrays). Whisper-specific features arevalidated against a Whisper model; a dedicated block validates Qwen3-ASR
dispatch and its
contextoption.src/js/tests/setup.pyadds theASR_MODEL(tiny-random-qwen3-asr) test model.tests/python_tests/samples/test_automatic_speech_recognition.pypoints tothe new JS sample.