Support workers in all major runtimes out of the box#687
Support workers in all major runtimes out of the box#687lgarron wants to merge 2 commits intoGoogleChromeLabs:mainfrom
Conversation
|
I thought I was working off of (I'm currently testing this PR in production at https://github.com/cubing/cdn.cubing.net via cubing/cubing.js@main...comlink ) |
003ee0a to
d464842
Compare
Okay, done.
I'm also testing the latest in production at https://cdn.cubing.net/ (temporary, but I might need to overwrite it for routine deployments at any time) and for testing at https://experiments.cubing.net/cdn/ (also temporary, but not subject to routine deployment overwrites). This works well, and you can verify in DevTools that it's running the code from this PR. <script type="module">
import { randomScrambleForEvent } from "https://experiments.cubing.net/cdn/comlink/v0/js/cubing/scramble";
const scramble = await randomScrambleForEvent("333");
scramble.log();
</script> |
686326a to
a7b203e
Compare
This contains code from GoogleChromeLabs/comlink#687 , with some refactoring. We could publish this as its own library, but that's unlikely to have enough short-term benefits over vendoring. If GoogleChromeLabs/comlink#687 gets merged we can use `comlink` upstream, else we can publish our own fork later.
This PR includes all the logic necessary to write portable workers, making it trivial to write code that can be used in all browsers and all major runtimes.
With this implementation, the following now works in `node`, `bun`, and `deno`:
```ts
// main.js
import { Worker as NodeWorker } from "node:worker_threads";
import { wrap } from "comlink";
const api = wrap(new NodeWorker(new URL(import.meta.resolve("./worker.js"))));
console.log(await api.add(6, 7));
```
```ts
// worker.js
import { expose } from "comlink";
expose({ add: (a, b) => a + b });
```
And the following works in all major browsers (in addition to `node`, `bun`, and `deno`):
```ts
// main.js
import { PortableWorker } from "comlink/examples/portable-worker";
import { wrap } from "comlink";
const api = wrap(new PortableWorker(import.meta.resolve("./worker.js")));
console.log(await api.add(6, 7));
```
This PR makes the following changes:
# Automatic `nodeEndpoint(…)` for workers
`expose(…)` and `wrap(…)` accept `node` workers as arguments and automatically wrap them by looking for an endpoint that looks an `EventEmitter` (has `.addListener()`) but not an `EventTarget` (has `.addEventListener()`). I believe I told @surma a long time ago I would send a PR for this. So here we go. 😁
This means `nodeEndpoint(…)` is no longer needed by default. However:
- `comlink` has not defined historically defined exports and provided files that can be (and are probably) used standalone by other projects. So we need to preserve the existing files and their APIs to avoid any risk of breakage, without any cross-imports in the build.
- Some projects may be relying on being able to call `addEventListener(…)` / `removeEventListener(…)` on the `nodeEndpoint(…)` output.
Therefore:
- Builds of `comlink.ts` now include the `nodeEndpoint(…)` implementation, so that these files can remain standalone.
- Builds of `comlink.ts` now *export* `nodeEndpoint` so that users can call this if needed, without code duplication.
- Duplicate implementations are still provided in the `node-adapter.ts` builds, albeit with a warning.
Note that the code seems to have previously been written under the assumption that `node` message ports do not satisfy the `Endpoint` interface, but at least as of 2026 they do. This means that `node` message ports are not singled out in the type signatures of `wrap(…)` and `expose(…)` like they were in `nodeEndpoint(…)`.
# Automatic `.unref()`
When using reffable workers (or other reffable endpoints), `wrap(…)` now uses pending requests to perform reference counting on the worker and calls `.unref(…)` when it is not waiting on the worker. For anyone who is familiar with web worker behaviour, this should do "the right thing" out of the box.
It solves the following issue: `node`-style workers automatically `.ref()` when a message is passed, which means that the process will hang by default. It is possible to prevent this by calling `.unref()` but this can allow the process to exit before all work is completed, unless the caller is very careful to track pending requests. (The caller can also call `.terminate()` on the worker or `exit(…)` on the process, but these tend to be heavy tools that can mask other issues.) Since `comlink` has a centralized place that can track these requests `wrap(…)`, it can make this easy.
`wrap(…)` now accepts an `options` argument which takes `{ refCount: false }` to opt out of this behaviour.
## Examples
- `06-node-example` has been updated so that it works when run from any working directory in any runtime.
- `08-portable-worker` includes an example of how to instantiate a worker cross-platform (`node`, `bun`, `deno`, and all modern browsers). This handles:
- Feature detection so that web workers are used when possible, and `node`-style workers when needed. Note that this is extra-subtle because `bun` and `deno` both support web workers but the latter only supports the reffable API on `node:worker_threads` workers.
- Instantiating a cross-origin trampoline that is needed when the worker code is not on the same origin as the main page.
- Accepting strings starting with `file://` so that a return value of [`import.meta.resolve(…)`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve) can be passed to the constructor directly.
# `package.json` fixups
I added `files` and `exports` to `package.json`. I found it difficult to test the package without these.
# Runtime testing
- `bun` and `deno` are installed as dev dependencies and used to test compatibility.
There are also a few more details based on over a decade of experience with web workers. However, I've tried to stay idiomatic to the existing codebase where possible (even if there are a lot of opportunities to modernize).
This at least allows some use cases to benefit from the workaround, and isolates the workaround to the adapter instead of the reference counting code.
a7b203e to
cedbc39
Compare
GoogleChromeLabs/comlink#687 is stalled (waiting for a review), unfortunately.
GoogleChromeLabs/comlink#687 is stalled (waiting for a review), unfortunately.
This PR includes all the logic necessary to write portable workers, making it trivial to write code that can be used in all browsers and all major runtimes.
With this implementation, the following now works in
node,bun, anddeno:And the following works in all major browsers (in addition to
node,bun, anddeno):This PR makes the following changes:
Automatic
nodeEndpoint(…)for workersexpose(…)andwrap(…)acceptnodeworkers as arguments and automatically wrap them by looking for an endpoint that looks anEventEmitter(has.addListener()) but not anEventTarget(has.addEventListener()). I believe I told @surma a long time ago I would send a PR for this. So here we go. 😁This means
nodeEndpoint(…)is no longer needed by default. However:comlinkhas not defined historically defined exports and provided files that can be (and are probably) used standalone by other projects. So we need to preserve the existing files and their APIs to avoid any risk of breakage, without any cross-imports in the build.addEventListener(…)/removeEventListener(…)on thenodeEndpoint(…)output.Therefore:
comlink.tsnow include thenodeEndpoint(…)implementation, so that these files can remain standalone.comlink.tsnow exportnodeEndpointso that users can call this if needed, without code duplication.node-adapter.tsbuilds, albeit with a warning.Note that the code seems to have previously been written under the assumption that
nodemessage ports do not satisfy theEndpointinterface, but at least as of 2026 they do. This means thatnodemessage ports are not singled out in the type signatures ofwrap(…)andexpose(…)like they were innodeEndpoint(…).Automatic
.unref()When using reffable workers (or other reffable endpoints),
wrap(…)now uses pending requests to perform reference counting on the worker and calls.unref(…)when it is not waiting on the worker. For anyone who is familiar with web worker behaviour, this should do "the right thing" out of the box.It solves the following issue:
node-style workers automatically.ref()when a message is passed, which means that the process will hang by default. It is possible to prevent this by calling.unref()but this can allow the process to exit before all work is completed, unless the caller is very careful to track pending requests. (The caller can also call.terminate()on the worker orexit(…)on the process, but these tend to be heavy tools that can mask other issues.) Sincecomlinkhas a centralized place that can track these requestswrap(…), it can make this easy.wrap(…)now accepts anoptionsargument which takes{ refCount: false }to opt out of this behaviour.Examples
06-node-examplehas been updated so that it works when run from any working directory in any runtime.08-portable-workerincludes an example of how to instantiate a worker cross-platform (node,bun,deno, and all modern browsers). This handles:node-style workers when needed. Note that this is extra-subtle becausebunanddenoboth support web workers but the latter only supports the reffable API onnode:worker_threadsworkers.file://so that a return value ofimport.meta.resolve(…)can be passed to the constructor directly.package.jsonfixupsI added
filesandexportstopackage.json. I found it difficult to test the package without these.Runtime testing
bunanddenoare installed as dev dependencies and used to test compatibility.There are also a few more details based on over a decade of experience with web workers. However, I've tried to stay idiomatic to the existing codebase where possible (even if there are a lot of opportunities to modernize).