httpcore.AsyncConnectionPool poisoning due to asyncio.CancelledError

[nikitagashkov]

Hello,

I’d like to report an issue we are encountering with the httpx/httpcore stack.

In a real-world scenario, our application (O) concurrently calls two downstream services (T1 and T2). We observed the following sequence:

1. T1 began returning 5xx errors while T2 remained healthy.
2. After some time, T1 recovered, but all requests to T2 started failing with httpx.PoolTimeout.
3. Restarting the O pods immediately resolved the httpx.PoolTimeout errors.

A simplified version of the real code looks like this:

@router.get("/")
async def view(
    t1_client: Annotated[httpx.AsyncClient, Depends(get_t1_client)],
    t2_client: Annotated[httpx.AsyncClient, Depends(get_t2_client)],
) -> dict[str, str]:
    async with asyncio.TaskGroup() as tg:
        t1_task = tg.create_task(http_get(t1_client, "/data"))
        t2_task = tg.create_task(http_get(t2_client, "/more-data"))
    return t1_task.result() | t2_task.result()


async def http_get(client: httpx.AsyncClient, path: str) -> dict[str, str]:
    resp = await client.get(path)
    resp.raise_for_status()
    return resp.json()

Our hypothesis is that t2_task starts execution but gets cancelled due to an exception in t1_task. As a result, some connections may not be cleaned up properly, eventually exhausting the connection pool.

I was able to reproduce what appears to be the same underlying issue with the following minimal example:

import asyncio
import httpcore


async def main() -> None:
    async with httpcore.AsyncConnectionPool(max_connections=1) as pool:
        tasks = [asyncio.create_task(pool.request("GET", f"http://localhost:8888/{i}")) for i in range(2)]
        await asyncio.sleep(0)
        for task in tasks:
            task.cancel()
        await asyncio.gather(*tasks, return_exceptions=True)

        print(pool)  # <AsyncConnectionPool [Requests: 0 active, 0 queued | Connections: 1 active, 0 idle]>
        await pool.request("GET", "http://localhost:8888/", extensions={"timeout": {"pool": 1}})  # httpcore.PoolTimeout


if __name__ == "__main__":
    asyncio.run(main())

At this point, the pool reports one active connection even though there are no active or queued requests, and subsequent requests fail with httpcore.PoolTimeout. This suggests that task cancellation during pool.request() can leave the pool in an inconsistent state where connections are not properly released.

Thank you for taking a look! Let me know if additional details or diagnostics would be helpful.