Web Worker support

[donmccurdy]

With #108 we'll support using Web Workers to offload local calculation for tilesets and rasters from the main thread. Unfortunately, using Web Workers packaged in libraries — as opposed to final applications — has been a minefield in the JavaScript ecosystem for years. If there is a "right" way to package Web Workers so that they'll work in all combinations of ...

• environment: Browser, Node.js
• format: ESM, CJS
• bundler: Vite, Webpack 5+, Parcel 2+, ...

... I have never found it. When a particular solution fails, it doesn't necessarily throw a runtime error we can catch and respond to. In some cases the build step (notably Parcel and Vite) may simply fail if a Web Worker cannot be resolved, which gives us no way to 'fall back' to a main-thread implementation.

I'm opening this thread to document the current state of support in @carto/api-client.

Environments

• ✅ Browser
• ❌ Node.js

Consider npm:web-worker if Node.js workers are needed.

Formats

• ✅ ESM
• ❌ CJS

There are a few ways to potentially add support for workers in CJS builds, but I'm not sure what environments/bundlers/users we'd be targeting, and trying to guess at the requirements here could add a lot of complexity. Consider dropping CJS support entirely after Node.js v18 EOL, the last version that cannot use require(...) to load ESM modules. Users bundling for CommonJS may be able to configure their bundlers compile ESM to CJS if needed, or to use require(esm) support in Node.js v20+.

Bundlers

• ✅ Vite
• ✅ Webpack 5
• ❌ Parcel 2

Currently Vite and Parcel appear to be mutually exclusive, each supporting only one of the statements below:

// ❌ vite, ✅ parcel
const worker = new Worker(new URL('./worker.js', import.meta.url), {type: 'module'});

// ✅ vite, ❌ parcel
const worker = new Worker(new URL('@carto/api-client/worker', import.meta.url), {type: 'module'});

User could configure Vite to handle './worker.js' by setting optimizeDeps.exclude = ['@carto/api-client'] in their vite.config.js file, but because Vite is used far more often, I think we have to use the Vite-compatible choice for now. I'm tracking these issues, which if resolved would allow us to support both Vite and Parcel:

• Workers are not imported correctly despite the worker syntax seemingly being correct, works in parcel vitejs/vite#11672
• new URL(foo, import.meta.url) doesn't work when dependency was optimized vitejs/vite#8427

We could also report the issue to Parcel — they support Web Workers but I haven't found an existing issue about the errors. With the current approach in @carto/api-client Parcel users will see this error on build:

@parcel/core: Failed to resolve '@carto/api-client/worker' from './node_modules/@carto/api-client/build/api-client.js'

[juandjara]

Hey I don't know I'm coming too late for this but have you considered creating the worker with

import Worker from '../workers/widget-tileset-worker.js?worker'

As far as I know, what this does is create a worker from the export default of that file, with automatic bundle entry separation. This way, you can defer URL creation to the bundler.

later, you can call this with

new Worker({
  name: 'cartowidgettileset',
})

I have run parcel build on this code and it did not gave any error, but I have yet to try it with a real example of integrating it into an app

[donmccurdy]

Hi @juandjara! It's not too late to change how the workers are constructed. I'm trying to do whatever is "most reliable" for different environments and bundlers now, ideally without relying on CDNs which would make maintenance harder. To my understanding the ?worker query suffix is a feature used by some bundlers to transform the code into equivalent code like...

const worker = new Worker(new URL('./worker.js', import.meta.url), {type: 'module'});

... at build time. Comparing web worker docs in different bundlers ...

• vite: https://v3.vitejs.dev/guide/features.html#web-workers
• webpack: https://webpack.js.org/guides/web-workers/
• parcel: https://parceljs.org/languages/javascript/#workers

... only Vite mentions this ?worker syntax, I wonder how standardized it is? But TBH I haven't tested yet in the bundlers, if it works in at least Vite and Webpack I think that could be worth switching over!

[sedghi]

Yeah, I think only Vite really understands the ?worker syntax.

We're also facing this issue in our library. We use the new URL method for reading workers, and this Vite issue is causing a lot of headaches and workarounds, like excluding it from the dependencies optimizer.

For now, the only thing we've found to do is document it until Vite fixes the problem.

[donmccurdy]

I've found one more issue that was a contributor to Web Workers breaking in Vite — it seems that having any CommonJS-only dependencies imported by the worker will cause Vite's dev server to just hang, as of Vite v6.2.0, so the Worker never initializes and no error is shown. I've inlined the CJS-only dependency and added a test to make sure we don't add more, which fixes the current issues we were seeing in Vite:

• fix(deps): Inline CommonJS dependencies quadbin-js#30

[donmccurdy]

Dependency on @turf/boolean-equal was also breaking Web Worker support in Vite. Can't see anything unusual about that package, it looks like a valid ESM/CJS dual package and we have other dependencies like that. In any case I'll have to inline it into this package to make Vite happy. Issue might be specific to Vite versions and/or config, since it wasn't happening in my standalone Vite tests.