Add a storage Worker and its client to @optuna/storage - #1322
Merged
Conversation
Preparation for running the storage backends in a Web Worker. `OptunaStorage` gains `close()`. Today nothing releases a storage: selecting another file in the standalone app only replaces a React state reference, and neither the SQLite connection nor the parsed Journal model is freed. A Worker makes that explicit, so the interface needs a way to say "this session is over" that both backends implement. `SQLite3Storage` also takes where to load sqlite-wasm from, as a URL or as bytes. It has been relying on the URL that sqlite-wasm derives from `import.meta.url`, which does not survive being bundled into a Worker: a Worker started from a blob: URL cannot resolve anything relative to itself. locateFile is therefore always set, and initialization failures now close the half-open database instead of leaking it. The initializer is imported from the bundler-friendly entrypoint of the dependency rather than its package entrypoint, which would pull in the optional Worker API this dashboard does not use.
The Worker owns one storage session: it detects the format from the first bytes of the transferred buffer, opens the matching backend, answers getStudies() / getStudy() / close(), and reports failures as structured errors. The client implements `OptunaStorage`, so a UI can treat it as any other storage, and takes the Worker through a factory because how a Worker is started differs per platform: a module Worker in a bundler that supports one, a blob: URL in a VS Code Webview. The factory hands back a dispose() so the client can release whatever the platform allocated, such as that blob: URL. Request and response types are derived from one request type to result map, and responses carry the request type so a mismatch is caught rather than handed to the caller as the wrong shape. Requests the Worker does not know about are answered with an error instead of silently leaving the caller's promise pending. Nothing calls this yet: the standalone app and the VS Code extension keep loading storages on the UI thread, and are moved over in later changes. The client is also exported as `@optuna/storage/worker-client` so that a UI can depend on it without importing the package root, which drags the SQLite backend into the bundle.
Each of the three entry points now says, at the top of the file, which context it belongs to and why the Worker entry is not part of `exports`. The comment is repeated verbatim in all three: whichever file you open first, the split is right there. SQLiteWasmOptions records why the wasm location is passed in at all, what the two forms are for, and why inlining it as base64 was not kept. The protocol also exported more than it needed to. StorageWorkerRequestOf was an alias for an Extract<> used in exactly one signature, StorageWorkerRequestWithoutId described itself rather than what it is, and the package re-exported the whole wire format although a consumer that reaches for it is talking to the Worker without the client.
Setting locateFile unconditionally broke the standalone app: with no wasm
location given it returned the bare path "sqlite3.wasm", so the browser resolved
it against the page, the dev server answered with index.html, and instantiation
failed on the HTML magic bytes ("expected magic word 00 61 73 6d, found 3c 21 44
4f"). Journal files were unaffected, which is why only the rdb e2e case failed.
The default path is the point: without locateFile, sqlite-wasm resolves the wasm
as `new URL("sqlite3.wasm", import.meta.url)`, which the bundler rewrites to the
asset it emitted. A built bundle now takes that branch again. locateFile is set
only when this call knows better, that is when a URL was passed, or when the
bytes were, where it exists solely to keep the fallback from being evaluated in
a Worker that cannot resolve anything relative to itself.
o-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c-bata
changed the title
StorageWorkerClient @optuna/storage
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contributor License Agreement
This repository (
optuna-dashboard) and Goptuna share common code.This pull request may therefore be ported to Goptuna.
Make sure that you understand the consequences concerning licenses and check the box below if you accept the term before creating this pull request.
Reference Issues/PRs
Follow-up of #1319. First step of moving SQLite / Journal parsing off the UI thread.
What does this implement/fix? Explain your changes.
SQLite3StorageandJournalFileStoragerun entirely in the caller's context:sqlite-wasm initialization,
sqlite3_deserialize(), every query, the UTF-8 decode and JSON parse of a whole Journal file, and the replay that rebuilds the Optuna objects. In the standalone app and in the VS Code Webview that caller is the UI thread, so selecting a large storage file freezes the page until parsing finishes.This PR adds the pieces needed to run that work in a Web Worker. Nothing calls them yet.
OptunaStorage.close()Nothing releases a storage today: selecting another file in the standalone app replaces a React state reference and leaves the SQLite connection and the parsed Journal model behind.
A Worker session has to be endable, so the interface gains
close()and both backends implement it. It is idempotent and does not fail, including after a failed open.An explicit sqlite-wasm source
SQLite3Storagerelied on the URL sqlite-wasm derives fromimport.meta.url, which does not survive bundling into a Worker: a Worker started from ablob:URL cannot resolve anything relative to itself.The constructor now takes the wasm as a URL or as bytes,
locateFileis always set so the fallback is never evaluated, and a failed initialization closes the half-open database instead of leaking it.The initializer is imported from the dependency's bundler-friendly entrypoint rather than its package entrypoint, which would pull in the optional Worker API this dashboard does not use.
The Worker and its client
storage_worker.tsowns one storage session. It detects the format from the first bytes of the transferred buffer, so callers no longer decide which backend to build, and answersgetStudies()/getStudy()/close(). Backend errors cross the boundary as structured errors rather than exception objects.StorageWorkerClientimplementsOptunaStorage, so a UI can use it like any other storage. It takes the Worker through a factory, because how a Worker is started differs per platform — a module Worker where the bundler supports one, ablob:URL in a VS Code Webview — and the factory returns adispose()so the client releases whatever the platform allocated. The storage buffer is transferred, not copied.Request and response types are derived from a single request-type-to-result map, responses carry the request type so a mismatch is reported instead of being handed to the caller as the wrong shape, and an unknown request is answered with an error rather than leaving the caller's promise pending forever.
Tests
worker_client.test.mjsdrives the client against a fake Worker: request ID matching, buffer transfer, double close, terminate, rejecting everything pending when the Worker fails, and rejecting a response that answers a different request type.storage_worker.test.mjsruns the real Worker onnode:worker_threadswith the existing fixtures: opening both formats, study summaries and details, transferring the wasm binary, an unsupported request, and a missing wasm asset.Follow-ups (not in this PR)
OpenStorageResultalready carries