Async ipywidgets.Output updates from background task only render after next cell in JupyterLab

[decitre]

Description

Widget updates from the async consumer task appear to be buffered until another cell finishes.

This happens in JupyterLab, but not in JupyterLite (Pyodide/Xeus), where the same pattern works as expected.

Minimal notebook repro

import asyncio
import ipywidgets as W
from IPython.display import display

q = asyncio.Queue()
out = W.Output()
button = W.Button(description="Click me")

def on_click(_):
    q.put_nowait("click")

button.on_click(on_click)

async def consumer():
    counter = 0
    while True:
        item = await q.get()
        counter += 1
        with out:
            out.clear_output(wait=True)
            print(f"GOT click {counter}")

loop = asyncio.get_event_loop()
task = loop.create_task(consumer())

display(button, out)

Observed behavior

1. Run the cell above.
2. Click the "Click me" button several times.
3. Notice that no "GOT click N" output appears in the Output widget.
4. Execute a next cell with any valid Python
5. Notice that only then does the previous cell's output update and show:

   GOT click 1
   

So widget updates from the async consumer task appear to be buffered until another cell finishes.

Expected behavior

Each button click should immediately cause the consumer task to update the Output widget, and the new content should be visible right away in JupyterLab, just like it is in JupyterLite (same pattern works there).

Context

• Python: 3.11.9
• ipywidgets: 8.1.8
• jupyterlab_widgets: 3.0.16
• JupyterLab: 4.5.1
• ipykernel: 7.1.0
• Labextensions:
  • jupyterlab_pygments v0.3.0 enabled OK (python, jupyterlab_pygments)
  • jupyter-leaflet v0.20.0 enabled OK
  • jupyterlab-filesystem-access v0.7.0 enabled OK (python, jupyterlab-filesystem-access)
  • @jupyter-widgets/jupyterlab-manager v5.0.15 enabled OK (python, jupyterlab_widgets)
  • @jupyterlite/pyodide-kernel-extension v0.7.0 enabled OK (python, jupyterlite_pyodide_kernel)
  • @jupyterlite/xeus-extension v4.3.7 enabled OK (python, jupyterlite_xeus)

[krassowski]

ipykernel 7 or 6?

[decitre]

ipykernel 7 or 6?

ipykernel: 7.1.0

@krassowski Thx for the hint/question. The issue does not occur when I swap back to ipykernel==6.31.0.

CC @ianthomas23

[krassowski]

I believe this is because of ipython/ipykernel#1451 (which I believe should have been made as opt-in or at very minimum have an opt-out reverting back to previous behaviour via env variable; anyways, it could be re-implemented to work in ipywidgets, the original code which was automatically doing it was removed)

[decitre]

I believe this is because of ipython/ipykernel#1451 (which I believe should have been made as opt-in or at very minimum have an opt-out reverting back to previous behaviour via env variable; anyways, it could be re-implemented to work in ipywidgets, the original code which was automatically doing it was removed)

Thanks @krassowski for the quick feedback. Setting the parent to the current cell as shown below didn’t seem to work. Could I have misunderstood your suggestion?

import asyncio
import ipywidgets as W
from IPython.display import display
from IPython import get_ipython

q = asyncio.Queue()
out = W.Output()
button = W.Button(description="Click me")
current_cell_parent = get_ipython().get_parent()

def on_click(_):
    q.put_nowait("click")

button.on_click(on_click)

async def consumer():
    counter = 0
    get_ipython().set_parent(current_cell_parent)
    while True:
        item = await q.get()
        counter += 1
        with out:
            out.clear_output(wait=True)
            print(f"GOT click {counter}")

loop = asyncio.get_event_loop()
task = loop.create_task(consumer())

display(button, out)

[decitre]

Update: This is not just an output issue—async tasks are not scheduled in ipykernel 7.x.

Apologies for my earlier focus on output widgets. Further investigation shows the issue is broader: in ipykernel 7.x, background async tasks (e.g., using asyncio.Queue) are not scheduled while the notebook is idle. As a result, even side effects like file writing do not occur until another cell is executed.

Minimal reproducible example:

import asyncio
import ipywidgets as W
from IPython.display import display
from IPython import get_ipython

q = asyncio.Queue()
button = W.Button(description="Click me")
current_cell_parent = get_ipython().get_parent()

def on_click(_):
    q.put_nowait("click")

button.on_click(on_click)

async def consumer():
    get_ipython().set_parent(current_cell_parent)
    while True:
        await q.get()
        with open("consumer_log.txt", "a") as f:
            f.write(f"GOT click\n")

asyncio.ensure_future(consumer())

display(button)

Observed behavior:
With ipykernel==6.3.1, clicking the button immediately creates/updates consumer_log.txt.
With ipykernel==7.1.0, the file is only created or updated after running another cell.

Conclusion:
This shows that in ipykernel 7.x, async tasks do not run after the cell finishes. The issue is about async scheduling, not just output.
Is this intended in ipykernel 7.x? If so, is there a recommended workaround for async background tasks?
If not, I can update the issue summary to reflect this new understanding if helpful.