Widget output rendered before its comm_open is processed fails permanently ("model not found" or stuck "Loading widget...")

[krassowski]

Description

Server-side-execution and RTC architectures (jupyter-server-nbmodel, jupyter-server-documents, jupyter-collaboration with jupyverse) deliver cell outputs to the browser through the shared-document (Yjs) websocket, while widget comms still travel over the kernel websocket. The two channels have no ordering guarantee, so an application/vnd.jupyter.widget-view+json output can reach the renderer a few milliseconds before the corresponding comm_open has been processed, something that could not happen when outputs and comms shared the ordered kernel iopub stream.

Two behaviors in jupyterlab_widgets turn this transient reordering into a permanent failure.

1. WidgetRenderer.renderModel has no recovery once restoredStatus is true.

ipywidgets/python/jupyterlab_widgets/src/renderer.ts

Lines 54 to 70 in ffd754b

	let wModel: DOMWidgetModel;
	try {
	// Presume we have a DOMWidgetModel. Should we check for sure?
	wModel = (await manager.get_model(source.model_id)) as DOMWidgetModel;
	} catch (err) {
	if (manager.restoredStatus) {
	// The manager has been restored, so this error won't be going away.
	this.node.textContent = 'Error displaying widget: model not found';
	this.addClass('jupyter-widgets');
	console.error(err);
	return;
	}
	// Store the model for a possible rerender
	this._rerenderMimeModel = model;
	return;
	}

get_model throws synchronously for unknown ids; if the manager already finished its initial restore, the renderer permanently shows Error displaying widget: model not found even though the comm_open for that very model may already be queued on the kernel websocket. The stash-and-re-render path only reacts to the one-shot restored signal, not to an individual model being registered later:

ipywidgets/python/jupyterlab_widgets/src/renderer.ts

Lines 109 to 118 in ffd754b

	private _rerender(): void {
	if (this._rerenderMimeModel) {
	// Clear the error message
	this.node.textContent = '';
	this.removeClass('jupyter-widgets');
	// Attempt to rerender.
	this.renderModel(this._rerenderMimeModel);
	}
	}

2. restoreWidgets awaits requestCommInfo without any timeout.

ipywidgets/python/jupyterlab_widgets/src/manager.ts

Lines 494 to 516 in ffd754b

	async restoreWidgets(
	notebook: INotebookModel,
	{ loadKernel, loadNotebook } = { loadKernel: true, loadNotebook: true }
	): Promise<void> {
	try {
	await this.context.sessionContext.ready;
	if (loadKernel) {
	try {
	this._kernelRestoreInProgress = true;
	await this._loadFromKernel();
	} finally {
	this._kernelRestoreInProgress = false;
	}
	}
	if (loadNotebook) {
	await this._loadFromNotebook(notebook);
	}
	// If the restore worked above, then update our state.
	this._restoredStatus = true;
	this._restored.emit();
	} catch (err) {
	// Do nothing if the restore did not work.

_loadFromKernel() calls _get_comm_info(), which awaits kernel.requestCommInfo(...):

ipywidgets/python/jupyterlab_widgets/src/manager.ts

Lines 136 to 149 in ffd754b

	async _get_comm_info(): Promise<any> {
	const kernel = this.kernel;
	if (!kernel) {
	throw new Error('No current kernel');
	}
	const reply = await kernel.requestCommInfo({
	target_name: this.comm_target_name,
	});
	if (reply.content.status === 'ok') {
	return (reply.content as any).comms;
	} else {
	return {};
	}
	}

If the comm_info_reply never arrives (kernel died at the wrong moment, or, as in the downstream report, a proxying server misrouting the shell reply to another websocket connection), the promise never settles: _restoredStatus stays false, restored never fires, and every renderer that stashed a mime model waits forever showing Loading widget....

Suggested direction (happy to contribute either patch):

• re-render on late model registration, e.g. emit a signal from register_model that pending renderers subscribe to, or give get_model an optional bounded wait for a not-yet-registered id;
• bound the kernel restore (timeout plus retry on reconnect), or at minimum resolve restoredStatus and emit restored on failure so stashed renderers re-render and can show a meaningful error instead of Loading widget....

Reproduce

1. Install jupyterlab and jupyter_server_documents (delivers outputs via the Yjs websocket; see the downstream issue for a Binder link), plus ipywidgets.
2. Create a fresh notebook with one cell:

   import ipywidgets
   ipywidgets.VBox([ipywidgets.HTML("hi"), ipywidgets.HBox([ipywidgets.HTML("there")])])

3. Restart and clear outputs, reload the page (a fresh page load makes comm processing slowest because widget AMD modules load cold), Run All.
4. The creating cell shows Error displaying widget: model not found or stays at Loading widget..., while re-running, or displaying the widget in a later cell, works.

Expected behavior

A widget-view output whose model registers moments after the first render attempt should still render (or at worst show a retryable state), and a failed or unanswered kernel-state restore should not permanently wedge every widget renderer on the page.

Context

• ipywidgets version: 8.1.8 (jupyterlab_widgets 3.0.16)
• JupyterLab 4.5.10, jupyter_server 2.20.0, jupyter_server_documents 0.3.3

Issues:

• Widgets don't load with nbmodel 0.1.1-a4, JupyterLab 4.5.7 and ipywidgets 8.1.x datalayer/jupyter-server-nbmodel#67
• IPyWidgets support datalayer/jupyter-server-nbmodel#38
• widgets don't always display when using jupyter-server-documents jupyter-ai-contrib/jupyter-server-documents#297

[echarles]

@krassowski Is this call still open and can https://github.com/datalayer/jupyter-server-nbmodel be a candidate for funding?