`serial_connected` never resets to False, causing `print()` to block when buffer is full

[BMDan]

CircuitPython version and board name

Adafruit CircuitPython 10.0.3 on 2025-10-17; FeatherS3 with ESP32S3

Code/REPL

import time
import supervisor

i=0
while True:
    i += 1
    print(f"[{time.monotonic_ns()}] [{supervisor.runtime.serial_connected}] {i}...")
    time.sleep(0.1)

Behavior

This code will run just fine on a fresh start. However, once a serial console has been connected (supervisor.runtime.serial_connected becomes True), even after that console disconnects, print() will become blocking any time the output buffer is full. This will be evidenced by, upon reconnection of the console, multiple older messages filling the screen, and then a large jump in the outputted time.monotonic_ns() as the program execution resumes. You will also note that the outputs that occurred while the console was disconnected still indicate that supervisor.runtime.serial_connected is True.

Description

I suspect this is the true root cause of #9150, and possibly #10500. I've been able to reproduce it with minicom, screen, and even cat as serial consoles.

My workaround is to create a ring buffer, but that's a lot of heavy lifting to paper over the fact that serial_connected apparently never changes back to False.

Additional information

In case anyone wants my ring buffer code; it's not the prettiest, but it works, and allows easy toggling in and out of ring mode; press CTRL-C once to toggle, and optionally quickly press CTRL-C again to fully exit.

_ring_drops: int = 0
_ring: list[tuple] = []
def ring_print(fmt: str, *args) -> None:
    global _ring
    global _ring_drops
    if len(_ring) > RING_SIZE:
        _ring.pop(0)
        _ring_drops += 1
    _ring.append((fmt, args))

# Monkeypatch print()
print("Patching print(); use ^C to see output.")
_orig_print = print
print = ring_print

while True: # Core kernel
    try:
        # <... do stuff...>
    except KeyboardInterrupt as e:
        _out = ""
        _orig_print(f"\033[2m[{time.time()}] Got keyboard interrupt at:")
        traceback.print_exception(e)  # So we know what our CTRL-C interrupted
        _orig_print("\033[0;0m", end="")
        if print is ring_print:
            if _ring_drops:
                _orig_print(f"<... dropped {_ring_drops} messages ...>")
            for fmt, args in _ring:
                _orig_print(fmt, *args)
            _ring = [] 
            _ring_drops = 0
            _out = f"\033[2m[{time.time()}] Switching to live output; press CTRL-C to toggle back to ring.\033[0;0m"      
            print = _orig_print
        else:
            _out = f"\033[2m[{time.time()}] Switching to ring-buffered output; press CTRL-C to toggle back to live.\033[0;0m"
            print = ring_print
        time.sleep(0.5)  # hit CTRL-C again and you'll exit the loop.
        _orig_print(_out)

[BMDan]

(Yes, I know it's technically not a real ring buffer, but that was a quick way to get the idea across.)

[dhalbert]

Could you describe in more detail how you provoke this? Are you shutting down the terminal program? Could you show some output that shows the printout including time values on disconnect and then after, on reconnect?

.serial_connected doesn't know that the terminal program has gone away. There is no USB indication on the CDC device that I know of. Instead, the host is going to buffer input on /dev/ttywhatever until someone reads it.But I think the serial output to the host will cause a USB stall when the buffer on the host fills up.
EDIT: This is wrong.

However, there may be an issue in the USB code that something about a USB stall is not undone.

(Aside: collections.deque would probably make a good ring buffer for this.)

[BMDan]

Could you describe in more detail how you provoke this? Are you shutting down the terminal program? Could you show some output that shows the printout including time values on disconnect and then after, on reconnect?

• screen: CTRL-A, k
• minicom: CTRL-A, x or CTRL-A, q [same effect in both cases]
• cat: SIGINT

Some sample output; I disconnected, waited a while, and then reconnected; you can see the jump caused by the stall at serial number 116:

[208679626477] [True] 101...
[208780212415] [True] 102...
[208880859375] [True] 103...
[208981506348] [True] 104...
[209082153322] [True] 105...
[209182800295] [True] 106...
[209283447269] [True] 107...
[209384094242] [True] 108...
[209484741216] [True] 109...
[209585388189] [True] 110...
[209686035163] [True] 111...
[209786682136] [True] 112...
[209887329110] [True] 113...
[209987945565] [True] 114...
[210088592539] [True] 115...    <-- from here
[235747833262] [True] 116...    <-- to here
[235848632812] [True] 117...
[235949432376] [True] 118...
[236050231939] [True] 119...
[236151031503] [True] 120...
[236251831066] [True] 121...
[236352630616] [True] 122...
[236453430180] [True] 123...
[236554229743] [True] 124...
[236655029307] [True] 125...
[236755828870] [True] 126...
[236856597902] [True] 127...

.serial_connected doesn't know that the terminal program has gone away. There is no USB indication on the CDC device that I know of. Instead, the host is going to buffer input on /dev/ttywhatever until someone reads it.But I think the serial output to the host will cause a USB stall when the buffer on the host fills up.

However, there may be an issue in the USB code that something about a USB stall is not undone.

What I find interesting is that, without a terminal connected at startup, .serial_connected is properly False, and print() output is silently discarded. Given that we're willing to do that, it seems to me that a simple solution would be to inspect the output buffer and, if a message would overflow it, to simply discard that message.

(Aside: collections.deque would probably make a good ring buffer for this.)

Ooooh! Didn't know we had collections available. BRB, replaced a bunch of halfassed things with proper defaultdicts. 😆

[tannewt]

Do we know if serial programs will set CDC line state when they exit? They might set the UART control flow lines.

[tannewt]

I don't want to discard when full because print() should be reliable when connected.

[BMDan]

I don't want to discard when full because print() should be reliable when connected.

That's fair. The consequence of that is kind of unfortunate, though. Especially in a case where print() is called rarely, it results in a bit of a time-bomb; you set up a device, check its console to make sure it came up cleanly, walk away, and at some point—hours, days, or weeks later—it'll just quietly lock up forever. Worse, it'll look like a Heisenbug, because as soon as you connect to it to debug it, it'll suddenly start working again.

[BMDan]

Do we know if serial programs will set CDC line state when they exit? They might set the UART control flow lines.

I don't think the serial program would be the one doing it; even if it could, something like cat /dev/ttyACM0 wouldn't. Any disconnection-signalling seems like it'd need to originate in the kernel. The relevant driver is https://github.com/torvalds/linux/blob/-/drivers/usb/class/cdc-acm.c . It looks like...

1. When the port is closed, it calls tty_port_close(), which...
2. If this is the last open FD on that TTY, calls tty_port_shutdown(), which
3. Drops DTR/RTS lines low if C_HUPCL is set...
4. which it is.

So if I'm reading it right, that would imply that the device should be able to see that there's no longer anyone connected on the other end of the line.

[dhalbert]

usb_cdc.Serial has a .out_waiting attribute which says whether bytes are waiting to be written. I think you could use that to check whether usb_cdc.console (same as a default print() stream) is blocked, and not write in that case.

I cannot find a similar API in CPython for, say, sys.stdout, to whether there is output pending. I did some experiments in Linux desktop Python writing to to one of two serial ports connected to two USB-serial adapters. I saw blocking, but then saw it continue, and then block again. I don't know what timeout is going on. I can't find anything in the documentation about this. Or maybe it is growing the internal buffers.

[dhalbert]

I will look at what is going on with TinyUSB and see what we might determine about .serial_connected re DTR/RTS.

[dhalbert]

What I said above about .serial_connected not knowing the state of the USB connection is wrong. It can in fact return False if there's no terminal program connected. It calls tud_cdc_connected() which in turns tud_cdc_get_line_state(), which is 0 if both are unasserted and 3 if both are asserted. If the terminal program is closed then DTR/RTS are both unasserted.

However, .serial_connected is more complicated than the CDC state. It returns what serial_connected() in supervisor/shared/serial.c returns. It returns true if there is any serial console connection, including BLE or web workflow, or even an onboard display.

For instance, on a PyPortal, a program will show print() output on the display. On a board with no display, like a Metro M4, there is no terminal display. Here is a simple test program which lights up board.LED if .serial_connected is True.

import board
import digitalio
import supervisor
import time

led = digitalio.DigitalInOut(board.LED)
led.switch_to_output(value=supervisor.runtime.serial_connected)

count = 0
while True:
    print(count, f"{supervisor.runtime.serial_connected=}")
    led.value = supervisor.runtime.serial_connected
    count += 1
    time.sleep(2)

On a PyPortal the LED is illuminated all the time, meaning .serial_connected is True all the time. On the Metro M4 the LED goes out when I quit the terminal program (I'm using tio). I also tried on a UM FeatherS2 (I don't have a FeatherS3), and the blue LED goes out when tio is quit.

Serial data out is written by serial_write_substring() in serial.c. It doesn't try write to USB when tud_cdc_connected() is false. However, it tries to write to all the places it can, including web workflow, etc.

---

Sorry for the somewhat long-winded explication. I cannot figure out a case where a write is blocked in the manner you described above. If the USB CDC connection is not active, it should just discard the output.

Could you try the test program in this post on your FeatherS3 and see whether the LED goes out when the terminal program is disconnected? Do you have web workflow active, or any other non-USB workflow?