From ad14bfd8143d4486a7ce277de78e8f2b7daf4157 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 09:02:21 +0300 Subject: [PATCH 01/20] temporarily stip away thread safety --- CONTRIBUTORS.md | 4 +- promising/promise.py | 7 +-- promising/promising_context.py | 57 +++++++------------ .../hierarchy/test_unregister_from_parent.py | 6 +- .../test_unregister_on_cancellation.py | 2 +- 5 files changed, 29 insertions(+), 47 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index ffddafcc1..b8c6a8b70 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -51,7 +51,7 @@ Tests use `pytest-asyncio` in auto mode — all async test functions are automat The base class for hierarchical context management. Manages parent-child relationships, namespacing (`namespace` parameter), configuration inheritance (`children_start_soon`, `start_soon_default`, `collapse_tracebacks`, `thread_pool`), child-waiting (`await_children` / `await_children_sync`), child inspection (`collect_unsettled_children`), and trace/debugging (`get_trace`, `format_trace`, `print_trace`). Also provides `get_parent_promise()` to walk up past non-Promise contexts. Uses a `ContextVar` (`PromisingContext.__active_context`) to track the currently active context. -**Lifecycle and child tracking.** Each `PromisingContext` keeps an `_unsettled_children: set[PromisingContext]` (a strong-ref set) protected by a `threading.Lock`. Children are added via `_register_children_threadsafe()` when they are constructed (unless born closed via `close_context_immediately=True`, in which case registration is skipped entirely) and removed via `_unregister_children_threadsafe()` once they are both *closed* and have no remaining unsettled descendants. A context is "closed" when its `with` block has exited (`close_context_threadsafe()` runs in `__exit__`'s `finally`); for a `Promise`, the `with self:` block lives inside `_unpack_once_from_loop`, so the context closes the moment the wrapped awaitable produces its first result (intermediate Promise or final value). Closed contexts that still have unsettled descendants stay registered until those descendants drain — this is what `collect_unsettled_children` traverses recursively. Attempting to register a child on an already-closed context raises `ContextAlreadyClosedError`; re-entering an already-closed context raises the same error. +**Lifecycle and child tracking.** Each `PromisingContext` keeps an `_unsettled_children: set[PromisingContext]` (a strong-ref set) protected by a `threading.Lock`. Children are added via `_register_children()` when they are constructed (unless born closed via `close_context_immediately=True`, in which case registration is skipped entirely) and removed via `_unregister_children()` once they are both *closed* and have no remaining unsettled descendants. A context is "closed" when its `with` block has exited (`close_context()` runs in `__exit__`'s `finally`); for a `Promise`, the `with self:` block lives inside `_unpack_once_from_loop`, so the context closes the moment the wrapped awaitable produces its first result (intermediate Promise or final value). Closed contexts that still have unsettled descendants stay registered until those descendants drain — this is what `collect_unsettled_children` traverses recursively. Attempting to register a child on an already-closed context raises `ContextAlreadyClosedError`; re-entering an already-closed context raises the same error. `PromisingContext` exposes two doneness predicates: `closed()` tracks the context-manager lifecycle (`__exit__` flips it to `True`), and `done()` is what `await_children()` actually waits on. By default `done()` simply delegates to `closed()`. Subclasses can override `done()` to track a non-lifecycle condition — `Promise` does exactly this (it ties `done()` to its own result/cancellation state machine, since "fully unpacked" can come *after* the `with self:` block has already exited). @@ -76,7 +76,7 @@ Scheduling is driven by `_ensure_from_loop_single_unpacking_scheduled()` and `_e **Prefilled Promises.** A Promise constructed without an `awaitable` (using `prefilled_result` or `prefilled_exception`) passes `close_context_immediately=True` to `PromisingContext.__init__`, so it is born already closed and immediately set to `_FINISHED` — there is no coroutine to run inside a `with self:` block, and no parent registration happens. -**Late parent registration.** `Promise.__init__` passes `register_with_parent=False` to `PromisingContext.__init__` and only calls `_register_with_parent_thread_unsafe()` at the very end of its own constructor, after the state machine has been seeded (including the prefilled `_FINISHED` / exception path). This guarantees that a Promise whose construction raises is never visible to its parent's child set, and that the prefilled-Promise case described above falls out naturally — by the time `_register_with_parent_thread_unsafe` runs, `done()` is already `True`, so the registration is skipped. +**Late parent registration.** `Promise.__init__` passes `register_with_parent=False` to `PromisingContext.__init__` and only calls `_register_with_parent()` at the very end of its own constructor, after the state machine has been seeded (including the prefilled `_FINISHED` / exception path). This guarantees that a Promise whose construction raises is never visible to its parent's child set, and that the prefilled-Promise case described above falls out naturally — by the time `_register_with_parent` runs, `done()` is already `True`, so the registration is skipped. **Exception breadcrumbs.** `try_to_link_exception` attaches the `PromisingContext` to an exception as `__promising_context__` and stamps `__promising_collapse_traceback__` (a boolean snapshot of the context's resolved `collapse_tracebacks` setting) alongside it — only at the deepest level (skips if `__promising_context__` is already set, so a nested context that already attributed itself is preserved; the two attributes are always stamped as a pair). Primary attribution happens in `PromisingContext.__exit__`; `_set_exception_from_loop` and `_force_internal_error_finish_from_loop` also call it as a safety net for paths that don't pass through `__exit__`. The `sys.excepthook` / `threading.excepthook` overrides in `promising/errors.py` use `__promising_context__` to walk the ancestor chain and render each `Promise`'s `frame_summary_tuple` snapshot, and read `__promising_collapse_traceback__` (a boolean) to decide whether to collapse promising-internal frames in those stacks (and in the exception's own traceback) or print them in full. diff --git a/promising/promise.py b/promising/promise.py index 83d6093e3..551978a79 100644 --- a/promising/promise.py +++ b/promising/promise.py @@ -292,8 +292,7 @@ def __init__( # safe side" self.loop.call_soon_threadsafe(self._ensure_from_loop_full_unpacking_scheduled_wrapper) - # TODO Activate the threading lock ? - self._register_with_parent_thread_unsafe() + self._register_with_parent() @classmethod def get_active_promise(cls, *, raise_if_none: bool = True) -> "Promise[Any] | None": @@ -1047,7 +1046,7 @@ def _synthesize_cancellation_from_loop(self, msg: str | None = None) -> None: # `_unpack_once_from_loop` would normally close the context via # `with self:`. Without this, `_context_closed` stays False and the # child never unregisters from its parent. - self.close_context_threadsafe() + self.close_context() self._set_exception_from_loop(asyncio.CancelledError(msg) if msg is not None else asyncio.CancelledError()) @@ -1074,7 +1073,7 @@ def _set_state(self, new_state: Sentinel) -> None: # the `with` block already, but it might also have been # `_force_internal_error_finish_from_loop`) and unregister from parent # "if time": - self.close_context_threadsafe() + self.close_context() def _resolve_start_soon(self, start_soon: bool | None | Sentinel) -> bool: """ diff --git a/promising/promising_context.py b/promising/promising_context.py index ebaf081b3..bb4727c6b 100644 --- a/promising/promising_context.py +++ b/promising/promising_context.py @@ -4,7 +4,6 @@ import functools import inspect import logging -import threading from asyncio import AbstractEventLoop from contextvars import ContextVar from types import TracebackType @@ -430,23 +429,11 @@ def __init__( self._context_closed = close_context_immediately self._unsettled_children = set[PromisingContext]() - # TODO To simplify safe-guarding against race conditions, do EVERYTHING - # on the event loop, instead of using threading locks or any other - # kinds of synchronization techniques (except for the ones that are - # designed for operation within the same async event loop). For any - # public method that can be invoked both, from the event loop and from - # a different thread, just check what thread we are in and either do - # the operation directly or schedule it with - # `asyncio.run_coroutine_threadsafe` and read the concurrent future - # result instead. Do all this after you take care of the following - # issue: - # https://github.com/teremterem/Promising/issues/104 - self._unsettled_children_lock = threading.Lock() if register_with_parent: # No other code has a reference to this PromisingContext yet, so we # can just register it with the parent in a thread-unsafe manner - self._register_with_parent_thread_unsafe() + self._register_with_parent() @property def loop(self) -> AbstractEventLoop: @@ -479,7 +466,7 @@ def closed(self) -> bool: Whether this context is closed. A ``PromisingContext`` is "open" from the moment it is constructed - until ``close_context_threadsafe()`` runs (which happens automatically + until ``close_context()`` runs (which happens automatically when the ``with`` block exits). Closed contexts are still kept around in their parent's ``_unsettled_children`` until their own unsettled descendants drain (they do not accept new children anymore). @@ -732,8 +719,7 @@ def collect_unsettled_children( Returns: Set of child PromisingContexts matching the filter criteria. """ - with self._unsettled_children_lock: - children = list[PromisingContext](self._unsettled_children) + children = list[PromisingContext](self._unsettled_children) if awaitables_only: result = {child for child in children if inspect.isawaitable(child)} @@ -797,11 +783,11 @@ def __exit__( raise exc from exc_value finally: - self.close_context_threadsafe() + self.close_context() return False # Let's not suppress any exceptions - def close_context_threadsafe(self) -> None: + def close_context(self) -> None: """ Mark this context as closed and unregister it from its parent if no unsettled descendants remain. Safe to call from any thread. @@ -814,8 +800,7 @@ def close_context_threadsafe(self) -> None: result. After this runs, any further attempt to enter the context or to register children on it raises ``ContextAlreadyClosedError``. """ - with self._unsettled_children_lock: - self._context_closed = True + self._context_closed = True self._unregister_from_parent_if_time() def try_to_link_exception(self, exception: BaseException) -> None: @@ -868,18 +853,18 @@ def __repr__(self) -> str: namespace_prefix = "" if self.namespace is None else f"{self.namespace!r} " return f"<{namespace_prefix}{self.__class__.__name__} id={id(self)}>" - def _register_with_parent_thread_unsafe(self) -> None: + def _register_with_parent(self) -> None: # It is thread-safe for the parent but is unsafe for the child itself if self._parent is not None and not self.done(): - self._parent._register_children_threadsafe(self) + self._parent._register_children(self) def _unregister_from_parent_if_time(self) -> None: if self.done() and self._parent is not None and not self._unsettled_children: _hierarchy_logger.log_unregistering_from_parent(parent=self._parent, child=self) - self._parent._unregister_children_threadsafe(self) + self._parent._unregister_children(self) - def _register_children_threadsafe(self, *children: "PromisingContext") -> None: + def _register_children(self, *children: "PromisingContext") -> None: for child in children: if not isinstance(child, PromisingContext): raise TypeError( @@ -887,21 +872,19 @@ def _register_children_threadsafe(self, *children: "PromisingContext") -> None: f"Context: {self!r}\nChild: {child!r}" ) - with self._unsettled_children_lock: - if self.closed(): - raise ContextAlreadyClosedError( - f"Cannot register children in a context that has already been closed.\n" - f"Context: {self!r}\nChildren: {children!r}" - ) - self._unsettled_children.update(children) + if self.closed(): + raise ContextAlreadyClosedError( + f"Cannot register children in a context that has already been closed.\n" + f"Context: {self!r}\nChildren: {children!r}" + ) + self._unsettled_children.update(children) - _hierarchy_logger.log_children_registered(parent=self, children=children) + _hierarchy_logger.log_children_registered(parent=self, children=children) - def _unregister_children_threadsafe(self, *children: "PromisingContext") -> None: - with self._unsettled_children_lock: - self._unsettled_children.difference_update(children) + def _unregister_children(self, *children: "PromisingContext") -> None: + self._unsettled_children.difference_update(children) - _hierarchy_logger.log_children_unregistered(parent=self, children=children) + _hierarchy_logger.log_children_unregistered(parent=self, children=children) self._unregister_from_parent_if_time() diff --git a/tests/hierarchy/test_unregister_from_parent.py b/tests/hierarchy/test_unregister_from_parent.py index e952478dd..544188e2b 100644 --- a/tests/hierarchy/test_unregister_from_parent.py +++ b/tests/hierarchy/test_unregister_from_parent.py @@ -61,7 +61,7 @@ async def test_unregisters_from_parent_when_last_child_is_unregistered() -> None assert parent in grandparent._unsettled_children # Removing the last child triggers deferred unregistration - parent._unregister_children_threadsafe(grandchild) + parent._unregister_children(grandchild) assert grandchild not in parent._unsettled_children assert parent not in grandparent._unsettled_children @@ -81,11 +81,11 @@ async def test_does_not_unregister_while_other_children_remain() -> None: assert parent in grandparent._unsettled_children # Removing one child — parent should stay registered - parent._unregister_children_threadsafe(child_a) + parent._unregister_children(child_a) assert parent in grandparent._unsettled_children # Removing the last child triggers deferred unregistration - parent._unregister_children_threadsafe(child_b) + parent._unregister_children(child_b) assert parent not in grandparent._unsettled_children diff --git a/tests/hierarchy/test_unregister_on_cancellation.py b/tests/hierarchy/test_unregister_on_cancellation.py index fea518405..f73bdcdb2 100644 --- a/tests/hierarchy/test_unregister_on_cancellation.py +++ b/tests/hierarchy/test_unregister_on_cancellation.py @@ -20,7 +20,7 @@ async def test_cancel_pending_promise_unregisters_from_parent() -> None: """ Cancelling a never-started Promise (no underlying task — synthesize path in ``_cancel_from_loop``) must close its context so that the - Promise unregisters from its parent. Without ``close_context_threadsafe()`` + Promise unregisters from its parent. Without ``close_context()`` on that path, ``_context_closed`` stays False and the child is leaked in the parent's ``_unsettled_children``. """ From 13bd539bcaaae9ecce525d4bc3257c6a72a4f5f1 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 10:25:14 +0300 Subject: [PATCH 02/20] temporarily strip away thread safety --- CONTRIBUTORS.md | 14 +-- promising/errors.py | 2 +- promising/promise.py | 115 ++++++------------ promising/promising_context.py | 2 +- .../test_unregister_on_cancellation.py | 4 +- tests/resolution/test_promise_cancellation.py | 4 +- 6 files changed, 47 insertions(+), 94 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index b8c6a8b70..a7d8e7f35 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -51,7 +51,7 @@ Tests use `pytest-asyncio` in auto mode — all async test functions are automat The base class for hierarchical context management. Manages parent-child relationships, namespacing (`namespace` parameter), configuration inheritance (`children_start_soon`, `start_soon_default`, `collapse_tracebacks`, `thread_pool`), child-waiting (`await_children` / `await_children_sync`), child inspection (`collect_unsettled_children`), and trace/debugging (`get_trace`, `format_trace`, `print_trace`). Also provides `get_parent_promise()` to walk up past non-Promise contexts. Uses a `ContextVar` (`PromisingContext.__active_context`) to track the currently active context. -**Lifecycle and child tracking.** Each `PromisingContext` keeps an `_unsettled_children: set[PromisingContext]` (a strong-ref set) protected by a `threading.Lock`. Children are added via `_register_children()` when they are constructed (unless born closed via `close_context_immediately=True`, in which case registration is skipped entirely) and removed via `_unregister_children()` once they are both *closed* and have no remaining unsettled descendants. A context is "closed" when its `with` block has exited (`close_context()` runs in `__exit__`'s `finally`); for a `Promise`, the `with self:` block lives inside `_unpack_once_from_loop`, so the context closes the moment the wrapped awaitable produces its first result (intermediate Promise or final value). Closed contexts that still have unsettled descendants stay registered until those descendants drain — this is what `collect_unsettled_children` traverses recursively. Attempting to register a child on an already-closed context raises `ContextAlreadyClosedError`; re-entering an already-closed context raises the same error. +**Lifecycle and child tracking.** Each `PromisingContext` keeps an `_unsettled_children: set[PromisingContext]` (a strong-ref set) protected by a `threading.Lock`. Children are added via `_register_children()` when they are constructed (unless born closed via `close_context_immediately=True`, in which case registration is skipped entirely) and removed via `_unregister_children()` once they are both *closed* and have no remaining unsettled descendants. A context is "closed" when its `with` block has exited (`close_context()` runs in `__exit__`'s `finally`); for a `Promise`, the `with self:` block lives inside `_unpack_once`, so the context closes the moment the wrapped awaitable produces its first result (intermediate Promise or final value). Closed contexts that still have unsettled descendants stay registered until those descendants drain — this is what `collect_unsettled_children` traverses recursively. Attempting to register a child on an already-closed context raises `ContextAlreadyClosedError`; re-entering an already-closed context raises the same error. `PromisingContext` exposes two doneness predicates: `closed()` tracks the context-manager lifecycle (`__exit__` flips it to `True`), and `done()` is what `await_children()` actually waits on. By default `done()` simply delegates to `closed()`. Subclasses can override `done()` to track a non-lifecycle condition — `Promise` does exactly this (it ties `done()` to its own result/cancellation state machine, since "fully unpacked" can come *after* the `with self:` block has already exited). @@ -65,20 +65,20 @@ This file also contains the `context` class — a context manager / decorator th **Two-step unpacking on the loop.** Resolution is split into two cooperating tasks, both pinned to `self.loop`: -- `_unpack_once_from_loop()` — drives a single unpacking step. It enters the `with self:` block, awaits the wrapped `_awaitable`, and either records an intermediate `Promise` (via `_set_intermediate_promise_from_loop`, transition to `_UNPACKED_ONCE`) or stores the final value/exception (via `_set_result_from_loop` / `_set_exception_from_loop`, transition to `_FINISHED`). This is the task `unpack_once()` waits on. -- `_fully_unpack_from_loop()` — drives the Promise to completion. It ensures the single-unpacking task is scheduled, awaits it, then walks the chain of intermediate Promises (`while isinstance(result, Promise): result = await result`) until a non-Promise value is reached, and records that value as the final result. This is the task `__await__` (and, indirectly, `sync()`) waits on. +- `_unpack_once()` — drives a single unpacking step. It enters the `with self:` block, awaits the wrapped `_awaitable`, and either records an intermediate `Promise` (via `_set_intermediate_promise_from_loop`, transition to `_UNPACKED_ONCE`) or stores the final value/exception (via `_set_result` / `_set_exception`, transition to `_FINISHED`). This is the task `unpack_once()` waits on. +- `_unpack_fully()` — drives the Promise to completion. It ensures the single-unpacking task is scheduled, awaits it, then walks the chain of intermediate Promises (`while isinstance(result, Promise): result = await result`) until a non-Promise value is reached, and records that value as the final result. This is the task `__await__` (and, indirectly, `sync()`) waits on. -Scheduling is driven by `_ensure_from_loop_single_unpacking_scheduled()` and `_ensure_from_loop_full_unpacking_scheduled()`, both of which create the underlying `loop.create_task(...)` lazily on first need. `__init__` schedules `_fully_unpack_from_loop` via `call_soon_threadsafe` when `start_soon` is `True`, so eager Promises start as soon as the loop is reachable; deferred Promises (`start_soon=False`) are scheduled the first time anyone consumes them (`__await__`, `sync()`, `unpack_once()`, `unpack_once_sync()`). +Scheduling is driven by `_ensure_single_unpacking_scheduled()` and `_ensure_from_unpacking_scheduled()`, both of which create the underlying `loop.create_task(...)` lazily on first need. `__init__` schedules `_unpack_fully` when `start_soon` is `True`, so eager Promises start as soon as the loop is reachable; deferred Promises (`start_soon=False`) are scheduled the first time anyone consumes them (`__await__`, `sync()`, `unpack_once()`, `unpack_once_sync()`). **Sync and thread-safe consumption.** `sync()` and `unpack_once_sync()` dispatch onto the Promise's own event loop via `asyncio.run_coroutine_threadsafe` and block the calling thread on the resulting `concurrent.futures.Future`. Both refuse to run on the Promise's loop thread (`_assert_no_sync_usage_deadlock` → `SyncUsageError`). `cancel()` is similarly thread-safe: when called from outside the loop, it dispatches `_cancel_from_loop` via `call_soon_threadsafe` and blocks only long enough for the dispatched `_cancel_from_loop` call to return — it does not wait for the `CancelledError` itself to land (mirroring `asyncio.Future.cancel()` / `asyncio.Task.cancel()` semantics: the return value reports whether cancellation was *requested*). -**Cancellation mechanics.** `_cancel_from_loop` requests cancellation of any running unpacking task(s) via `Task.cancel(msg)`; the `CancelledError` then propagates through `_unpack_once_from_loop` / `_fully_unpack_from_loop` and is stored via `_set_exception_from_loop`, which picks the terminal state (`_CANCELLED_BEFORE_UNPACKED_ONCE` vs. `_CANCELLED_AFTER_UNPACKED_ONCE`) based on whether the first unpacking step had completed. When no task has been scheduled yet (e.g. `start_soon=False` and never awaited), `_synthesize_cancellation_from_loop` closes the context, stores a `CancelledError` directly, and closes the wrapped awaitable to silence the "coroutine was never awaited" warning. A `_unpacking_task_done_callback` covers the edge case where `Task.cancel()` lands between `create_task` and the first `__step`: asyncio resumes a cancelled task by throwing `CancelledError` into the coroutine, but on a coroutine that has never been stepped into there is no suspension point to throw at, so the exception is raised at function entry — *before* the `try` block — and propagates straight out of the task without the body's `try/except BaseException` ever seeing it. The Task ends up `cancelled()` but the Promise is still `_PENDING`, so the callback synthesizes the terminal state from the Task's recorded `CancelledError`. +**Cancellation mechanics.** `_cancel_from_loop` requests cancellation of any running unpacking task(s) via `Task.cancel(msg)`; the `CancelledError` then propagates through `_unpack_once` / `_unpack_fully` and is stored via `_set_exception`, which picks the terminal state (`_CANCELLED_BEFORE_UNPACKED_ONCE` vs. `_CANCELLED_AFTER_UNPACKED_ONCE`) based on whether the first unpacking step had completed. When no task has been scheduled yet (e.g. `start_soon=False` and never awaited), `_synthesize_cancellation_from_loop` closes the context, stores a `CancelledError` directly, and closes the wrapped awaitable to silence the "coroutine was never awaited" warning. A `_unpacking_task_done_callback` covers the edge case where `Task.cancel()` lands between `create_task` and the first `__step`: asyncio resumes a cancelled task by throwing `CancelledError` into the coroutine, but on a coroutine that has never been stepped into there is no suspension point to throw at, so the exception is raised at function entry — *before* the `try` block — and propagates straight out of the task without the body's `try/except BaseException` ever seeing it. The Task ends up `cancelled()` but the Promise is still `_PENDING`, so the callback synthesizes the terminal state from the Task's recorded `CancelledError`. **Prefilled Promises.** A Promise constructed without an `awaitable` (using `prefilled_result` or `prefilled_exception`) passes `close_context_immediately=True` to `PromisingContext.__init__`, so it is born already closed and immediately set to `_FINISHED` — there is no coroutine to run inside a `with self:` block, and no parent registration happens. **Late parent registration.** `Promise.__init__` passes `register_with_parent=False` to `PromisingContext.__init__` and only calls `_register_with_parent()` at the very end of its own constructor, after the state machine has been seeded (including the prefilled `_FINISHED` / exception path). This guarantees that a Promise whose construction raises is never visible to its parent's child set, and that the prefilled-Promise case described above falls out naturally — by the time `_register_with_parent` runs, `done()` is already `True`, so the registration is skipped. -**Exception breadcrumbs.** `try_to_link_exception` attaches the `PromisingContext` to an exception as `__promising_context__` and stamps `__promising_collapse_traceback__` (a boolean snapshot of the context's resolved `collapse_tracebacks` setting) alongside it — only at the deepest level (skips if `__promising_context__` is already set, so a nested context that already attributed itself is preserved; the two attributes are always stamped as a pair). Primary attribution happens in `PromisingContext.__exit__`; `_set_exception_from_loop` and `_force_internal_error_finish_from_loop` also call it as a safety net for paths that don't pass through `__exit__`. The `sys.excepthook` / `threading.excepthook` overrides in `promising/errors.py` use `__promising_context__` to walk the ancestor chain and render each `Promise`'s `frame_summary_tuple` snapshot, and read `__promising_collapse_traceback__` (a boolean) to decide whether to collapse promising-internal frames in those stacks (and in the exception's own traceback) or print them in full. +**Exception breadcrumbs.** `try_to_link_exception` attaches the `PromisingContext` to an exception as `__promising_context__` and stamps `__promising_collapse_traceback__` (a boolean snapshot of the context's resolved `collapse_tracebacks` setting) alongside it — only at the deepest level (skips if `__promising_context__` is already set, so a nested context that already attributed itself is preserved; the two attributes are always stamped as a pair). Primary attribution happens in `PromisingContext.__exit__`; `_set_exception` and `_force_internal_error_finish_from_loop` also call it as a safety net for paths that don't pass through `__exit__`. The `sys.excepthook` / `threading.excepthook` overrides in `promising/errors.py` use `__promising_context__` to walk the ancestor chain and render each `Promise`'s `frame_summary_tuple` snapshot, and read `__promising_collapse_traceback__` (a boolean) to decide whether to collapse promising-internal frames in those stacks (and in the exception's own traceback) or print them in full. **Unpacking semantics.** A `PromisingFunction` always returns a `Promise`, regardless of whether the underlying function returns a concrete value or another `Promise`. `await promise` and `promise.sync()` recursively chase nested `Promise`s until a non-`Promise` value is reached. `promise.unpack_once()` and `promise.unpack_once_sync()` unpack a single level — they return either a concrete value or the intermediate `Promise`. @@ -124,7 +124,7 @@ This module also defines the private state-machine sentinels used by `Promise` ( - `SentinelUsageError` — a `Sentinel` was used in a boolean context (e.g. `if INHERIT:`) - `SyncUsageError` — raised when a sync method (`promise.sync()`, `promise.unpack_once_sync()`, `await_children_sync()`) is called from the event loop thread -**Promising-traceback rendering.** `install_promising_tracebacks()` swaps in `_promising_sys_excepthook` and `_promising_threading_excepthook` (saving the previously installed hooks in `_excepthook_state` so they can be used as a fallback if rendering itself raises). Installation is idempotent and protected by `_excepthooks_lock`. `Promise._unpack_once_from_loop` calls `install_promising_tracebacks()` the first time it runs, so users typically don't have to install the hooks by hand. +**Promising-traceback rendering.** `install_promising_tracebacks()` swaps in `_promising_sys_excepthook` and `_promising_threading_excepthook` (saving the previously installed hooks in `_excepthook_state` so they can be used as a fallback if rendering itself raises). Installation is idempotent and protected by `_excepthooks_lock`. `Promise._unpack_once` calls `install_promising_tracebacks()` the first time it runs, so users typically don't have to install the hooks by hand. `_print_exception_with_promising_context` walks the standard `__cause__` / `__context__` chain (mirroring CPython's behavior, including `__suppress_context__`) via `_print_exception_chain`, and `_print_single_exception` prints each link of that chain with a per-link "promising trace". For each `PromisingContext` returned by `get_trace(ancestors_first=True)` that exposes a `frame_summary_tuple` (i.e. each `Promise`), the snapshot captured at construction time is rendered, with promising-internal frames optionally collapsed. diff --git a/promising/errors.py b/promising/errors.py index 20515e441..bd5feddd7 100644 --- a/promising/errors.py +++ b/promising/errors.py @@ -116,7 +116,7 @@ def install_promising_tracebacks() -> bool: successful installation are captured and used as a fallback if the promising renderer itself raises. - ``Promise._unpack_once_from_loop`` calls this function automatically + ``Promise._unpack_once`` calls this function automatically the first time a Promise runs, so applications rarely need to invoke it directly. It is exposed in the public API for cases where you want to enable promising tracebacks before any Promise has executed (for diff --git a/promising/promise.py b/promising/promise.py index 551978a79..018ef60dd 100644 --- a/promising/promise.py +++ b/promising/promise.py @@ -278,19 +278,13 @@ def __init__( self._single_unpacking_task: Task[T_co | Promise[Any]] | None = None if self._awaitable is None: - # No outside code has any reference to this Promise yet, so we can - # set the result/exception directly, no matter which thread the - # constructor is currently running in if prefilled_result is not UNCHANGED: - self._set_result_from_loop(prefilled_result) + self._set_result(prefilled_result) else: - self._set_exception_from_loop(prefilled_exception) + self._set_exception(prefilled_exception) if self._start_soon and self._awaitable is not None: - # We don't know which thread the Promise is created in, so we - # use the event loop's `call_soon_threadsafe` to "stay on the - # safe side" - self.loop.call_soon_threadsafe(self._ensure_from_loop_full_unpacking_scheduled_wrapper) + self._ensure_from_unpacking_scheduled() self._register_with_parent() @@ -325,7 +319,7 @@ def __await__(self) -> Generator[Any, None, T_co]: Await the Promise, fully unpacking all nested Promises. If the Promise hasn't started yet, starts execution via - ``_fully_unpack_from_loop()``. If already started via start_soon, + ``_unpack_fully()``. If already started via start_soon, waits for the existing task to complete. Once the Promise resolves, recursively awaits the result as long as it is itself a Promise, returning the final non-Promise value. @@ -341,7 +335,7 @@ def __await__(self) -> Generator[Any, None, T_co]: """ self._assert_awaiting_on_correct_event_loop() - self._ensure_from_loop_full_unpacking_scheduled() + self._ensure_from_unpacking_scheduled() if self._full_unpacking_task is not None: yield from self._full_unpacking_task @@ -404,7 +398,7 @@ async def unpack_once(self) -> "T_co | Promise[Any]": """ self._assert_awaiting_on_correct_event_loop() - self._ensure_from_loop_single_unpacking_scheduled() + self._ensure_single_unpacking_scheduled() if self._single_unpacking_task is not None: await self._single_unpacking_task @@ -478,7 +472,7 @@ def done(self) -> bool: ``_PENDING`` (to ``_UNPACKED_ONCE``, ``_FINISHED``, or one of the ``_CANCELLED_XX`` states), the state never moves backwards. The writers (``_set_intermediate_promise_from_loop`` / - ``_set_result_from_loop`` / ``_set_exception_from_loop``) write the + ``_set_result`` / ``_set_exception``) write the corresponding attribute (``_intermediate_promise``, ``_result``, ``_exception``) *before* advancing the state via ``_set_state``, so a reader that observes a state past ``_PENDING`` is guaranteed to also @@ -552,9 +546,6 @@ def result(self) -> T_co: raise self._exception if self._result is UNCHANGED: - # Should not happen: _assert_done() above guarantees a terminal - # state, and the only way to reach _FINISHED without an - # exception is via _set_result_from_loop (which sets _result). raise RuntimeError( f"Promise result is UNCHANGED even though the promise is done and there is no exception: {self!r}" ) @@ -622,13 +613,13 @@ def cancel(self, msg: str | None = None) -> bool: return value reports whether cancellation was *requested* — the Promise's terminal cancelled state is reached only once the ``CancelledError`` actually propagates through the underlying - unpacking task and is stored via ``_set_exception_from_loop``. Until + unpacking task and is stored via ``_set_exception``. Until then, ``cancelled()`` may still return ``False``. For a Promise whose underlying task hasn't been scheduled yet (e.g. ``start_soon=False`` and never awaited), the cancellation is synthesized as a ``CancelledError`` stored directly via - ``_set_exception_from_loop``, with no task involvement — analogous to + ``_set_exception``, with no task involvement — analogous to ``Future.cancel()`` on a not-yet-running future. When called from the Promise's own event loop thread the cancellation @@ -667,7 +658,7 @@ def callback(): self.loop.call_soon_threadsafe(callback) return future.result() - def _ensure_from_loop_single_unpacking_scheduled(self) -> None: + def _ensure_single_unpacking_scheduled(self) -> None: """ NOTE: This method can only be used from the event loop of the Promise. """ @@ -675,13 +666,13 @@ def _ensure_from_loop_single_unpacking_scheduled(self) -> None: if self._single_unpacking_task is None and not self.unpacked_once_or_done(): self._single_unpacking_task = self.loop.create_task( - self._unpack_once_from_loop(), name=str(self) + "-SingleUnpackingTask" + self._unpack_once(), name=str(self) + "-SingleUnpackingTask" ) self._single_unpacking_task.add_done_callback(self._unpacking_task_done_callback) _unpacking_logger.log_single_unpacking_scheduled(promise=self) - def _ensure_from_loop_full_unpacking_scheduled(self) -> None: + def _ensure_from_unpacking_scheduled(self) -> None: """ NOTE: This method can only be used from the event loop of the Promise. """ @@ -689,39 +680,19 @@ def _ensure_from_loop_full_unpacking_scheduled(self) -> None: if self._full_unpacking_task is None and not self.done(): self._full_unpacking_task = self.loop.create_task( - self._fully_unpack_from_loop(), name=str(self) + "-FullUnpackingTask" + self._unpack_fully(), name=str(self) + "-FullUnpackingTask" ) self._full_unpacking_task.add_done_callback(self._unpacking_task_done_callback) _unpacking_logger.log_full_unpacking_scheduled(promise=self) - def _ensure_from_loop_full_unpacking_scheduled_wrapper(self) -> None: - """ - ``call_soon_threadsafe``-safe wrapper around - ``_ensure_from_loop_full_unpacking_scheduled``. - - Used by the ``start_soon=True`` path in ``__init__``, where scheduling - is deferred to the event loop via ``call_soon_threadsafe``. Any - exception raised from that callback would otherwise propagate to the - loop's default exception handler and leave the Promise stuck in a - non-terminal state. This wrapper instead routes the exception through - ``_force_internal_error_finish_from_loop`` so the Promise is settled - as an internal error. - - NOTE: This method can only be used from the event loop of the Promise. - """ - try: - self._ensure_from_loop_full_unpacking_scheduled() - except BaseException as exc: - self._force_internal_error_finish_from_loop(exc) - def _unpacking_task_done_callback(self, task: Task[Any]) -> None: """ Bridge the case where ``task.cancel()`` lands between ``create_task`` and the first ``__step``: ``CancelledError`` is thrown into a not-yet-started coroutine and propagates out without entering the ``try/except BaseException`` inside - ``_unpack_once_from_loop`` / ``_fully_unpack_from_loop``, leaving + ``_unpack_once`` / ``_unpack_fully``, leaving the Promise non-terminal even though the Task ended cancelled. """ # Early return if the task wasn't cancelled, or if the Promise (self) @@ -742,7 +713,7 @@ def _unpacking_task_done_callback(self, task: Task[Any]) -> None: self._synthesize_cancellation_from_loop(msg) - async def _unpack_once_from_loop(self) -> None: + async def _unpack_once(self) -> None: """ Drive a single unpacking step on the event loop. @@ -750,11 +721,11 @@ async def _unpack_once_from_loop(self) -> None: promises created during this step are registered as its children), awaits the wrapped awaitable, and stores either an intermediate Promise or a final value/exception. The state machine is moved forward via - ``_set_intermediate_promise_from_loop`` / ``_set_result_from_loop`` / - ``_set_exception_from_loop``. + ``_set_intermediate_promise_from_loop`` / ``_set_result`` / + ``_set_exception``. Backs ``unpack_once()`` (and the first leg of - ``_fully_unpack_from_loop``). + ``_unpack_fully``). NOTE: This method can only be used from the event loop of the Promise. """ @@ -762,12 +733,8 @@ async def _unpack_once_from_loop(self) -> None: _unpacking_logger.log_single_unpacking_started(promise=self) if self.unpacked_once_or_done(): - # Should not happen: this method is only scheduled by - # _ensure_from_loop_single_unpacking_scheduled, which guards - # on `not unpacked_once_or_done()`. raise RuntimeError( - f"An attempt was made to _unpack_once_from_loop a Promise " - f"that was already unpacked once or done: {self!r}" + f"An attempt was made to _unpack_once a Promise that was already unpacked once or done: {self!r}" ) # TODO [TRACES] Introduce some sort of `DEBUG` boolean flag (like in @@ -782,16 +749,16 @@ async def _unpack_once_from_loop(self) -> None: except BaseException as exc: _unpacking_logger.log_unpacking_exception(promise=self, stage="unpack_once_from_loop", exc=exc) - self._set_exception_from_loop(exc) + self._set_exception(exc) else: if isinstance(result, Promise): self._set_intermediate_promise_from_loop(result) else: - self._set_result_from_loop(result) + self._set_result(result) _unpacking_logger.log_single_unpacking_finished(promise=self) - async def _fully_unpack_from_loop(self) -> None: + async def _unpack_fully(self) -> None: """ Drive the Promise to completion on the event loop, recursively unpacking nested Promises. @@ -800,7 +767,7 @@ async def _fully_unpack_from_loop(self) -> None: that produced an intermediate Promise, awaits it (and any further nested Promises) until a non-Promise value is reached, then stores that value as the final result. Any exception from the chain is - captured via ``_set_exception_from_loop``. + captured via ``_set_exception``. Backs ``__await__`` (and, indirectly, ``sync()``). @@ -814,7 +781,7 @@ async def _fully_unpack_from_loop(self) -> None: # becomes done already after unpack_once_from_loop completes return - self._ensure_from_loop_single_unpacking_scheduled() + self._ensure_single_unpacking_scheduled() if self._single_unpacking_task is not None: await self._single_unpacking_task @@ -843,9 +810,9 @@ async def _fully_unpack_from_loop(self) -> None: except BaseException as exc: _unpacking_logger.log_unpacking_exception(promise=self, stage="fully_unpack_from_loop", exc=exc) - self._set_exception_from_loop(exc) + self._set_exception(exc) else: - self._set_result_from_loop(result) + self._set_result(result) _unpacking_logger.log_full_unpacking_finished(promise=self) @@ -858,11 +825,6 @@ def _set_intermediate_promise_from_loop(self, promise: "Promise[Any]") -> None: """ try: if self._state is not _PENDING: - # Should not happen: only called from _unpack_once_from_loop - # when the awaitable resolved to a Promise. The only steps - # between the awaitable resolving and this call are the - # synchronous `with self:` exit and a logger call — - # neither yields, so state stays _PENDING. raise RuntimeError( f"Cannot set intermediate_promise on a promise because of the promise's current state: {self!r}" ) @@ -872,7 +834,7 @@ def _set_intermediate_promise_from_loop(self, promise: "Promise[Any]") -> None: except BaseException as internal_error: self._force_internal_error_finish_from_loop(internal_error) - def _set_result_from_loop(self, result: T_co) -> None: + def _set_result(self, result: T_co) -> None: """ Store the fully unpacked result. No-op if the Promise is already done (finished or cancelled). @@ -881,10 +843,6 @@ def _set_result_from_loop(self, result: T_co) -> None: """ try: if self._state not in (_PENDING, _UNPACKED_ONCE): - # Should not happen: all callsites reach this with state in - # (_PENDING, _UNPACKED_ONCE) — prefill in __init__, the - # non-Promise branch of _unpack_once_from_loop, or the end - # of _fully_unpack_from_loop's unwrap chain. raise RuntimeError(f"Cannot set result on a promise because of its current state: {self!r}") self._result = result self._set_state(_FINISHED) @@ -892,7 +850,7 @@ def _set_result_from_loop(self, result: T_co) -> None: except BaseException as internal_error: self._force_internal_error_finish_from_loop(internal_error) - def _set_exception_from_loop(self, exception: BaseException) -> None: + def _set_exception(self, exception: BaseException) -> None: """ Store the exception and move the Promise into a terminal state. The cancelled state is an *effect* of storing a ``CancelledError``, not a @@ -923,15 +881,10 @@ def _set_exception_from_loop(self, exception: BaseException) -> None: # that's already cancelled. Drop it; the original wins. return else: - # Should not happen: any non-CancelledError exception - # arriving on a non-_PENDING / non-_UNPACKED_ONCE Promise - # implies the framework's state machine is broken - # (legitimate user-triggered cancellation races are - # caught by the elif above). raise RuntimeError(f"Cannot set exception on a promise because of its current state: {self!r}") # The context was probably already attached to the exception by the - # ``with self:`` block of ``_unpack_once_from_loop``, but it is + # ``with self:`` block of ``_unpack_once``, but it is # also possible that the exception occurred outside the # ``with self:`` block (e.g. a framework bug), so lets try to # attach it here too. @@ -1003,9 +956,9 @@ def _cancel_from_loop(self, msg: str | None = None) -> bool: (see ``_synthesize_cancellation_from_loop``). The state machine is *not* moved here. Instead, the ``CancelledError`` - propagates through ``_unpack_once_from_loop`` / - ``_fully_unpack_from_loop`` (``except BaseException`` catches it) and - is stored via ``_set_exception_from_loop``. + propagates through ``_unpack_once`` / + ``_unpack_fully`` (``except BaseException`` catches it) and + is stored via ``_set_exception``. NOTE: This method can only be used from the event loop of the Promise. """ @@ -1043,12 +996,12 @@ def _synthesize_cancellation_from_loop(self, msg: str | None = None) -> None: NOTE: This method can only be used from the event loop of the Promise. """ - # `_unpack_once_from_loop` would normally close the context via + # `_unpack_once` would normally close the context via # `with self:`. Without this, `_context_closed` stays False and the # child never unregisters from its parent. self.close_context() - self._set_exception_from_loop(asyncio.CancelledError(msg) if msg is not None else asyncio.CancelledError()) + self._set_exception(asyncio.CancelledError(msg) if msg is not None else asyncio.CancelledError()) # Close the wrapped awaitable so a never-driven coroutine doesn't # trigger a "coroutine was never awaited" warning at GC time — diff --git a/promising/promising_context.py b/promising/promising_context.py index bb4727c6b..5588a4370 100644 --- a/promising/promising_context.py +++ b/promising/promising_context.py @@ -794,7 +794,7 @@ def close_context(self) -> None: Called automatically by ``__exit__`` (so a normal ``with`` block always closes the context). For a ``Promise``, the context is - also entered and exited from inside ``_unpack_once_from_loop`` + also entered and exited from inside ``_unpack_once`` around the awaiting of the wrapped awaitable, so the close happens in lockstep with the unpacking step that produced its first result. After this runs, any further attempt to enter the context diff --git a/tests/hierarchy/test_unregister_on_cancellation.py b/tests/hierarchy/test_unregister_on_cancellation.py index f73bdcdb2..cf7d5e842 100644 --- a/tests/hierarchy/test_unregister_on_cancellation.py +++ b/tests/hierarchy/test_unregister_on_cancellation.py @@ -83,7 +83,7 @@ async def test_coroutine_raising_cancelled_error_unregisters_from_parent() -> No """ When the coroutine itself raises ``CancelledError`` (no external cancel() call), the Promise still goes through the standard - ``_unpack_once_from_loop`` path whose ``with self:`` closes the + ``_unpack_once`` path whose ``with self:`` closes the context. Verify the cancelled Promise unregisters from its parent. """ with promising.context() as parent: @@ -110,7 +110,7 @@ async def test_cancel_full_unpacking_task_before_first_step_transitions_promise( ``create_task`` and its first ``__step`` throws ``CancelledError`` into a not-yet-started coroutine — Python propagates that exception out without entering the body's ``try/except BaseException``, so the - coroutine never calls ``_set_exception_from_loop`` itself. Without + coroutine never calls ``_set_exception`` itself. Without the done-callback bridge, the Task ends cancelled while the Promise stays ``_PENDING`` and leaks in its parent's ``_unsettled_children``. """ diff --git a/tests/resolution/test_promise_cancellation.py b/tests/resolution/test_promise_cancellation.py index 3b217c8aa..d4971e196 100644 --- a/tests/resolution/test_promise_cancellation.py +++ b/tests/resolution/test_promise_cancellation.py @@ -1,6 +1,6 @@ """ Tests for Promise.cancel() — modeled on asyncio.Future / asyncio.Task -semantics. Cancellation flows through ``_set_exception_from_loop``: the +semantics. Cancellation flows through ``_set_exception``: the ``CancelledError`` is stored first, the ``_CANCELLED_*`` state transition is its effect. """ @@ -173,7 +173,7 @@ async def coro() -> str: async def test_result_raises_not_done_before_cancel_propagates() -> None: """ ``cancel()`` on a running task only *requests* cancellation; until the - CancelledError lands and is stored via ``_set_exception_from_loop``, ``done()`` + CancelledError lands and is stored via ``_set_exception``, ``done()`` stays False and ``result()`` raises ``PromiseNotDoneError``. """ coro_started = asyncio.Event() From 4797c8a5d6745f46e468bd62ecb901ec8ef8905b Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 10:56:52 +0300 Subject: [PATCH 03/20] temporarily strip away thread safety --- CONTRIBUTORS.md | 6 +- promising/promise.py | 78 ++++++------------- .../test_unregister_on_cancellation.py | 6 +- 3 files changed, 27 insertions(+), 63 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index a7d8e7f35..ad50e5bef 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -65,14 +65,14 @@ This file also contains the `context` class — a context manager / decorator th **Two-step unpacking on the loop.** Resolution is split into two cooperating tasks, both pinned to `self.loop`: -- `_unpack_once()` — drives a single unpacking step. It enters the `with self:` block, awaits the wrapped `_awaitable`, and either records an intermediate `Promise` (via `_set_intermediate_promise_from_loop`, transition to `_UNPACKED_ONCE`) or stores the final value/exception (via `_set_result` / `_set_exception`, transition to `_FINISHED`). This is the task `unpack_once()` waits on. +- `_unpack_once()` — drives a single unpacking step. It enters the `with self:` block, awaits the wrapped `_awaitable`, and either records an intermediate `Promise` (via `_set_intermediate_promise`, transition to `_UNPACKED_ONCE`) or stores the final value/exception (via `_set_result` / `_set_exception`, transition to `_FINISHED`). This is the task `unpack_once()` waits on. - `_unpack_fully()` — drives the Promise to completion. It ensures the single-unpacking task is scheduled, awaits it, then walks the chain of intermediate Promises (`while isinstance(result, Promise): result = await result`) until a non-Promise value is reached, and records that value as the final result. This is the task `__await__` (and, indirectly, `sync()`) waits on. Scheduling is driven by `_ensure_single_unpacking_scheduled()` and `_ensure_from_unpacking_scheduled()`, both of which create the underlying `loop.create_task(...)` lazily on first need. `__init__` schedules `_unpack_fully` when `start_soon` is `True`, so eager Promises start as soon as the loop is reachable; deferred Promises (`start_soon=False`) are scheduled the first time anyone consumes them (`__await__`, `sync()`, `unpack_once()`, `unpack_once_sync()`). -**Sync and thread-safe consumption.** `sync()` and `unpack_once_sync()` dispatch onto the Promise's own event loop via `asyncio.run_coroutine_threadsafe` and block the calling thread on the resulting `concurrent.futures.Future`. Both refuse to run on the Promise's loop thread (`_assert_no_sync_usage_deadlock` → `SyncUsageError`). `cancel()` is similarly thread-safe: when called from outside the loop, it dispatches `_cancel_from_loop` via `call_soon_threadsafe` and blocks only long enough for the dispatched `_cancel_from_loop` call to return — it does not wait for the `CancelledError` itself to land (mirroring `asyncio.Future.cancel()` / `asyncio.Task.cancel()` semantics: the return value reports whether cancellation was *requested*). +**Sync consumption.** `sync()` and `unpack_once_sync()` dispatch onto the Promise's own event loop via `asyncio.run_coroutine_threadsafe` and block the calling thread on the resulting `concurrent.futures.Future`. Both refuse to run on the Promise's loop thread (`_assert_no_sync_usage_deadlock` → `SyncUsageError`). -**Cancellation mechanics.** `_cancel_from_loop` requests cancellation of any running unpacking task(s) via `Task.cancel(msg)`; the `CancelledError` then propagates through `_unpack_once` / `_unpack_fully` and is stored via `_set_exception`, which picks the terminal state (`_CANCELLED_BEFORE_UNPACKED_ONCE` vs. `_CANCELLED_AFTER_UNPACKED_ONCE`) based on whether the first unpacking step had completed. When no task has been scheduled yet (e.g. `start_soon=False` and never awaited), `_synthesize_cancellation_from_loop` closes the context, stores a `CancelledError` directly, and closes the wrapped awaitable to silence the "coroutine was never awaited" warning. A `_unpacking_task_done_callback` covers the edge case where `Task.cancel()` lands between `create_task` and the first `__step`: asyncio resumes a cancelled task by throwing `CancelledError` into the coroutine, but on a coroutine that has never been stepped into there is no suspension point to throw at, so the exception is raised at function entry — *before* the `try` block — and propagates straight out of the task without the body's `try/except BaseException` ever seeing it. The Task ends up `cancelled()` but the Promise is still `_PENDING`, so the callback synthesizes the terminal state from the Task's recorded `CancelledError`. +**Cancellation mechanics.** `cancel()` does not wait for the `CancelledError` itself to land (mirroring `asyncio.Future.cancel()` / `asyncio.Task.cancel()` semantics: the return value reports whether cancellation was *requested*). It requests cancellation of any running unpacking task(s) via `Task.cancel(msg)`; the `CancelledError` then propagates through `_unpack_once` / `_unpack_fully` and is stored via `_set_exception`, which picks the terminal state (`_CANCELLED_BEFORE_UNPACKED_ONCE` vs. `_CANCELLED_AFTER_UNPACKED_ONCE`) based on whether the first unpacking step had completed. When no task has been scheduled yet (e.g. `start_soon=False` and never awaited), `_synthesize_cancellation_from_loop` closes the context, stores a `CancelledError` directly, and closes the wrapped awaitable to silence the "coroutine was never awaited" warning. A `_unpacking_task_done_callback` covers the edge case where `Task.cancel()` lands between `create_task` and the first `__step`: asyncio resumes a cancelled task by throwing `CancelledError` into the coroutine, but on a coroutine that has never been stepped into there is no suspension point to throw at, so the exception is raised at function entry — *before* the `try` block — and propagates straight out of the task without the body's `try/except BaseException` ever seeing it. The Task ends up `cancelled()` but the Promise is still `_PENDING`, so the callback synthesizes the terminal state from the Task's recorded `CancelledError`. **Prefilled Promises.** A Promise constructed without an `awaitable` (using `prefilled_result` or `prefilled_exception`) passes `close_context_immediately=True` to `PromisingContext.__init__`, so it is born already closed and immediately set to `_FINISHED` — there is no coroutine to run inside a `with self:` block, and no parent registration happens. diff --git a/promising/promise.py b/promising/promise.py index 018ef60dd..86c71a21f 100644 --- a/promising/promise.py +++ b/promising/promise.py @@ -471,7 +471,7 @@ def done(self) -> bool: The Promise state machine is monotonic — once advanced past ``_PENDING`` (to ``_UNPACKED_ONCE``, ``_FINISHED``, or one of the ``_CANCELLED_XX`` states), the state never moves backwards. The - writers (``_set_intermediate_promise_from_loop`` / + writers (``_set_intermediate_promise`` / ``_set_result`` / ``_set_exception``) write the corresponding attribute (``_intermediate_promise``, ``_result``, ``_exception``) *before* advancing the state via ``_set_state``, so a @@ -637,26 +637,26 @@ def cancel(self, msg: str | None = None) -> bool: NOTE: This method is thread-safe, including from the event loop of the Promise. """ - if self.is_on_correct_running_loop(raise_thread_loop_not_running=False): - # We are on the event loop of the Promise, so we can cancel it - # directly - return self._cancel_from_loop(msg) + if self.done(): + return False - # We are on a different thread, so we need to use a thread-safe - # mechanism to cancel the Promise - self._assert_event_loop_running_for_sync() - future = concurrent.futures.Future() + cancellation_requested = False + if self._single_unpacking_task is not None and not self._single_unpacking_task.done(): + cancellation_requested |= self._single_unpacking_task.cancel(msg) + if self._full_unpacking_task is not None and not self._full_unpacking_task.done(): + cancellation_requested |= self._full_unpacking_task.cancel(msg) - def callback(): - try: - result = self._cancel_from_loop(msg) - except BaseException as exc: - future.set_exception(exc) - else: - future.set_result(result) + if cancellation_requested: + return True - self.loop.call_soon_threadsafe(callback) - return future.result() + # No task is currently running cancellation through — synthesize the + # CancelledError and store it directly. Covers the + # `start_soon=False`/never-awaited case as well as the rare race + # where every task has finished but the Promise hasn't transitioned + # to a terminal state yet. + self._synthesize_cancellation_from_loop(msg) + + return self.cancelled() def _ensure_single_unpacking_scheduled(self) -> None: """ @@ -721,7 +721,7 @@ async def _unpack_once(self) -> None: promises created during this step are registered as its children), awaits the wrapped awaitable, and stores either an intermediate Promise or a final value/exception. The state machine is moved forward via - ``_set_intermediate_promise_from_loop`` / ``_set_result`` / + ``_set_intermediate_promise`` / ``_set_result`` / ``_set_exception``. Backs ``unpack_once()`` (and the first leg of @@ -752,7 +752,7 @@ async def _unpack_once(self) -> None: self._set_exception(exc) else: if isinstance(result, Promise): - self._set_intermediate_promise_from_loop(result) + self._set_intermediate_promise(result) else: self._set_result(result) @@ -816,7 +816,7 @@ async def _unpack_fully(self) -> None: _unpacking_logger.log_full_unpacking_finished(promise=self) - def _set_intermediate_promise_from_loop(self, promise: "Promise[Any]") -> None: + def _set_intermediate_promise(self, promise: "Promise[Any]") -> None: """ Record the intermediate Promise returned by a single unpacking step. No-op if already unpacked once or done. @@ -949,47 +949,13 @@ def _assert_done(self) -> None: if not self.done(): raise PromiseNotDoneError(f"Promise is not done: {self!r}") - def _cancel_from_loop(self, msg: str | None = None) -> bool: - """ - Request cancellation of the underlying unpacking task(s) — or, when - no task has been scheduled yet, synthesize the cancellation directly - (see ``_synthesize_cancellation_from_loop``). - - The state machine is *not* moved here. Instead, the ``CancelledError`` - propagates through ``_unpack_once`` / - ``_unpack_fully`` (``except BaseException`` catches it) and - is stored via ``_set_exception``. - - NOTE: This method can only be used from the event loop of the Promise. - """ - if self.done(): - return False - - cancellation_requested = False - if self._single_unpacking_task is not None and not self._single_unpacking_task.done(): - cancellation_requested |= self._single_unpacking_task.cancel(msg) - if self._full_unpacking_task is not None and not self._full_unpacking_task.done(): - cancellation_requested |= self._full_unpacking_task.cancel(msg) - - if cancellation_requested: - return True - - # No task is currently running cancellation through — synthesize the - # CancelledError and store it directly. Covers the - # `start_soon=False`/never-awaited case as well as the rare race - # where every task has finished but the Promise hasn't transitioned - # to a terminal state yet. - self._synthesize_cancellation_from_loop(msg) - - return self.cancelled() - def _synthesize_cancellation_from_loop(self, msg: str | None = None) -> None: """ Drive the Promise into a cancelled terminal state without relying on a running unpacking task to surface the ``CancelledError``. Mirrors ``Future.cancel()`` on a not-yet-running future. - Shared by ``_cancel_from_loop`` (synthesize path, no task ever + Shared by ``cancel`` (synthesize path, no task ever scheduled) and ``_unpacking_task_done_callback`` (task cancelled between ``create_task`` and its first ``__step``, so the body's ``except BaseException`` never saw the ``CancelledError``). diff --git a/tests/hierarchy/test_unregister_on_cancellation.py b/tests/hierarchy/test_unregister_on_cancellation.py index cf7d5e842..14271d703 100644 --- a/tests/hierarchy/test_unregister_on_cancellation.py +++ b/tests/hierarchy/test_unregister_on_cancellation.py @@ -19,7 +19,7 @@ async def test_cancel_pending_promise_unregisters_from_parent() -> None: """ Cancelling a never-started Promise (no underlying task — synthesize - path in ``_cancel_from_loop``) must close its context so that the + path in ``cancel``) must close its context so that the Promise unregisters from its parent. Without ``close_context()`` on that path, ``_context_closed`` stays False and the child is leaked in the parent's ``_unsettled_children``. @@ -44,9 +44,7 @@ async def coro() -> str: async def test_cancel_pending_promise_from_other_thread_unregisters_from_parent() -> None: """ Synthesize path reached via the thread-safe dispatch: cancel() is - called from a non-loop thread, which schedules ``_cancel_from_loop`` - on the loop. The unregistration must still happen, just on the loop - thread. + called from a non-loop thread. The unregistration must still happen. """ with promising.context() as parent: From 7f029007b9b7bf6974c18b66bea10d02e9642fef Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 11:08:41 +0300 Subject: [PATCH 04/20] temporarily strip away thread safety --- CONTRIBUTORS.md | 4 ++-- promising/logging_utils.py | 20 ++++++++++---------- promising/promise.py | 22 +++++++++++----------- 3 files changed, 23 insertions(+), 23 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index ad50e5bef..cea27e3f6 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -72,13 +72,13 @@ Scheduling is driven by `_ensure_single_unpacking_scheduled()` and `_ensure_from **Sync consumption.** `sync()` and `unpack_once_sync()` dispatch onto the Promise's own event loop via `asyncio.run_coroutine_threadsafe` and block the calling thread on the resulting `concurrent.futures.Future`. Both refuse to run on the Promise's loop thread (`_assert_no_sync_usage_deadlock` → `SyncUsageError`). -**Cancellation mechanics.** `cancel()` does not wait for the `CancelledError` itself to land (mirroring `asyncio.Future.cancel()` / `asyncio.Task.cancel()` semantics: the return value reports whether cancellation was *requested*). It requests cancellation of any running unpacking task(s) via `Task.cancel(msg)`; the `CancelledError` then propagates through `_unpack_once` / `_unpack_fully` and is stored via `_set_exception`, which picks the terminal state (`_CANCELLED_BEFORE_UNPACKED_ONCE` vs. `_CANCELLED_AFTER_UNPACKED_ONCE`) based on whether the first unpacking step had completed. When no task has been scheduled yet (e.g. `start_soon=False` and never awaited), `_synthesize_cancellation_from_loop` closes the context, stores a `CancelledError` directly, and closes the wrapped awaitable to silence the "coroutine was never awaited" warning. A `_unpacking_task_done_callback` covers the edge case where `Task.cancel()` lands between `create_task` and the first `__step`: asyncio resumes a cancelled task by throwing `CancelledError` into the coroutine, but on a coroutine that has never been stepped into there is no suspension point to throw at, so the exception is raised at function entry — *before* the `try` block — and propagates straight out of the task without the body's `try/except BaseException` ever seeing it. The Task ends up `cancelled()` but the Promise is still `_PENDING`, so the callback synthesizes the terminal state from the Task's recorded `CancelledError`. +**Cancellation mechanics.** `cancel()` does not wait for the `CancelledError` itself to land (mirroring `asyncio.Future.cancel()` / `asyncio.Task.cancel()` semantics: the return value reports whether cancellation was *requested*). It requests cancellation of any running unpacking task(s) via `Task.cancel(msg)`; the `CancelledError` then propagates through `_unpack_once` / `_unpack_fully` and is stored via `_set_exception`, which picks the terminal state (`_CANCELLED_BEFORE_UNPACKED_ONCE` vs. `_CANCELLED_AFTER_UNPACKED_ONCE`) based on whether the first unpacking step had completed. When no task has been scheduled yet (e.g. `start_soon=False` and never awaited), `_synthesize_cancellation` closes the context, stores a `CancelledError` directly, and closes the wrapped awaitable to silence the "coroutine was never awaited" warning. A `_unpacking_task_done_callback` covers the edge case where `Task.cancel()` lands between `create_task` and the first `__step`: asyncio resumes a cancelled task by throwing `CancelledError` into the coroutine, but on a coroutine that has never been stepped into there is no suspension point to throw at, so the exception is raised at function entry — *before* the `try` block — and propagates straight out of the task without the body's `try/except BaseException` ever seeing it. The Task ends up `cancelled()` but the Promise is still `_PENDING`, so the callback synthesizes the terminal state from the Task's recorded `CancelledError`. **Prefilled Promises.** A Promise constructed without an `awaitable` (using `prefilled_result` or `prefilled_exception`) passes `close_context_immediately=True` to `PromisingContext.__init__`, so it is born already closed and immediately set to `_FINISHED` — there is no coroutine to run inside a `with self:` block, and no parent registration happens. **Late parent registration.** `Promise.__init__` passes `register_with_parent=False` to `PromisingContext.__init__` and only calls `_register_with_parent()` at the very end of its own constructor, after the state machine has been seeded (including the prefilled `_FINISHED` / exception path). This guarantees that a Promise whose construction raises is never visible to its parent's child set, and that the prefilled-Promise case described above falls out naturally — by the time `_register_with_parent` runs, `done()` is already `True`, so the registration is skipped. -**Exception breadcrumbs.** `try_to_link_exception` attaches the `PromisingContext` to an exception as `__promising_context__` and stamps `__promising_collapse_traceback__` (a boolean snapshot of the context's resolved `collapse_tracebacks` setting) alongside it — only at the deepest level (skips if `__promising_context__` is already set, so a nested context that already attributed itself is preserved; the two attributes are always stamped as a pair). Primary attribution happens in `PromisingContext.__exit__`; `_set_exception` and `_force_internal_error_finish_from_loop` also call it as a safety net for paths that don't pass through `__exit__`. The `sys.excepthook` / `threading.excepthook` overrides in `promising/errors.py` use `__promising_context__` to walk the ancestor chain and render each `Promise`'s `frame_summary_tuple` snapshot, and read `__promising_collapse_traceback__` (a boolean) to decide whether to collapse promising-internal frames in those stacks (and in the exception's own traceback) or print them in full. +**Exception breadcrumbs.** `try_to_link_exception` attaches the `PromisingContext` to an exception as `__promising_context__` and stamps `__promising_collapse_traceback__` (a boolean snapshot of the context's resolved `collapse_tracebacks` setting) alongside it — only at the deepest level (skips if `__promising_context__` is already set, so a nested context that already attributed itself is preserved; the two attributes are always stamped as a pair). Primary attribution happens in `PromisingContext.__exit__`; `_set_exception` and `_force_internal_error_finish` also call it as a safety net for paths that don't pass through `__exit__`. The `sys.excepthook` / `threading.excepthook` overrides in `promising/errors.py` use `__promising_context__` to walk the ancestor chain and render each `Promise`'s `frame_summary_tuple` snapshot, and read `__promising_collapse_traceback__` (a boolean) to decide whether to collapse promising-internal frames in those stacks (and in the exception's own traceback) or print them in full. **Unpacking semantics.** A `PromisingFunction` always returns a `Promise`, regardless of whether the underlying function returns a concrete value or another `Promise`. `await promise` and `promise.sync()` recursively chase nested `Promise`s until a non-`Promise` value is reached. `promise.unpack_once()` and `promise.unpack_once_sync()` unpack a single level — they return either a concrete value or the intermediate `Promise`. diff --git a/promising/logging_utils.py b/promising/logging_utils.py index d22015fed..dbae38fdc 100644 --- a/promising/logging_utils.py +++ b/promising/logging_utils.py @@ -127,16 +127,16 @@ def __init__(self, *, logger: logging.Logger | None = None, level: int) -> None: self.level = level def log_single_unpacking_scheduling(self, *, promise: "Promise") -> None: - self._log("ATTEMPTING TO SCHEDULE UNPACK_ONCE_FROM_LOOP", promise=promise) + self._log("ATTEMPTING TO SCHEDULE UNPACK_ONCE", promise=promise) def log_single_unpacking_scheduled(self, *, promise: "Promise") -> None: - self._log("SCHEDULED UNPACK_ONCE_FROM_LOOP", promise=promise) + self._log("SCHEDULED UNPACK_ONCE", promise=promise) def log_single_unpacking_started(self, *, promise: "Promise") -> None: - self._log("STARTED UNPACK_ONCE_FROM_LOOP", promise=promise) + self._log("STARTED UNPACK_ONCE", promise=promise) def log_single_unpacking_finished(self, *, promise: "Promise") -> None: - self._log("FINISHED UNPACK_ONCE_FROM_LOOP", promise=promise) + self._log("FINISHED UNPACK_ONCE", promise=promise) def log_single_unpacking_result(self, *, promise: "Promise", result: Any) -> None: if not self.logger.isEnabledFor(self.level): @@ -147,7 +147,7 @@ def log_single_unpacking_result(self, *, promise: "Promise", result: Any) -> Non kind = "Intermediate (Promise)" if isinstance(result, Promise) else "Final (non-Promise)" log_message = "\n".join( [ - f"UNPACK_ONCE_FROM_LOOP Result: {kind}", + f"UNPACK_ONCE Result: {kind}", f" promise: {promise}", f" result type: {type(result).__name__}", ] @@ -163,7 +163,7 @@ def log_unwrap_step(self, *, promise: "Promise", depth: int, result: Any) -> Non kind = "Promise (continue unwrap)" if isinstance(result, Promise) else "Non-Promise (stop)" log_message = "\n".join( [ - f"FULLY_UNPACK_FROM_LOOP Unwrap Step depth={depth}: {kind}", + f"UNPACK_FULLY Unwrap Step depth={depth}: {kind}", f" promise: {promise}", f" result type: {type(result).__name__}", ] @@ -184,16 +184,16 @@ def log_unpacking_exception(self, *, promise: "Promise", stage: str, exc: BaseEx self.logger.log(self.level, f"\n{log_message}\n") def log_full_unpacking_scheduling(self, *, promise: "Promise") -> None: - self._log("ATTEMPTING TO SCHEDULE FULLY_UNPACK_FROM_LOOP", promise=promise) + self._log("ATTEMPTING TO SCHEDULE UNPACK_FULLY", promise=promise) def log_full_unpacking_scheduled(self, *, promise: "Promise") -> None: - self._log("SCHEDULED FULLY_UNPACK_FROM_LOOP", promise=promise) + self._log("SCHEDULED UNPACK_FULLY", promise=promise) def log_full_unpacking_started(self, *, promise: "Promise") -> None: - self._log("STARTED FULLY_UNPACK_FROM_LOOP", promise=promise) + self._log("STARTED UNPACK_FULLY", promise=promise) def log_full_unpacking_finished(self, *, promise: "Promise") -> None: - self._log("FINISHED FULLY_UNPACK_FROM_LOOP", promise=promise) + self._log("FINISHED UNPACK_FULLY", promise=promise) def _log(self, headline: str, *, promise: "Promise") -> None: if not self.logger.isEnabledFor(self.level): diff --git a/promising/promise.py b/promising/promise.py index 86c71a21f..50c3e8ab2 100644 --- a/promising/promise.py +++ b/promising/promise.py @@ -654,7 +654,7 @@ def cancel(self, msg: str | None = None) -> bool: # `start_soon=False`/never-awaited case as well as the rare race # where every task has finished but the Promise hasn't transitioned # to a terminal state yet. - self._synthesize_cancellation_from_loop(msg) + self._synthesize_cancellation(msg) return self.cancelled() @@ -711,7 +711,7 @@ def _unpacking_task_done_callback(self, task: Task[Any]) -> None: if exc.args: msg = exc.args[0] - self._synthesize_cancellation_from_loop(msg) + self._synthesize_cancellation(msg) async def _unpack_once(self) -> None: """ @@ -748,7 +748,7 @@ async def _unpack_once(self) -> None: _unpacking_logger.log_single_unpacking_result(promise=self, result=result) except BaseException as exc: - _unpacking_logger.log_unpacking_exception(promise=self, stage="unpack_once_from_loop", exc=exc) + _unpacking_logger.log_unpacking_exception(promise=self, stage="_unpack_once", exc=exc) self._set_exception(exc) else: if isinstance(result, Promise): @@ -778,7 +778,7 @@ async def _unpack_fully(self) -> None: if self.done(): # When there are no more nested Promises to unpack, the Promise - # becomes done already after unpack_once_from_loop completes + # becomes done already after _unpack_once completes return self._ensure_single_unpacking_scheduled() @@ -809,7 +809,7 @@ async def _unpack_fully(self) -> None: _unpacking_logger.log_unwrap_step(promise=self, depth=depth, result=result) except BaseException as exc: - _unpacking_logger.log_unpacking_exception(promise=self, stage="fully_unpack_from_loop", exc=exc) + _unpacking_logger.log_unpacking_exception(promise=self, stage="_unpack_fully", exc=exc) self._set_exception(exc) else: self._set_result(result) @@ -832,7 +832,7 @@ def _set_intermediate_promise(self, promise: "Promise[Any]") -> None: self._set_state(_UNPACKED_ONCE) except BaseException as internal_error: - self._force_internal_error_finish_from_loop(internal_error) + self._force_internal_error_finish(internal_error) def _set_result(self, result: T_co) -> None: """ @@ -848,7 +848,7 @@ def _set_result(self, result: T_co) -> None: self._set_state(_FINISHED) except BaseException as internal_error: - self._force_internal_error_finish_from_loop(internal_error) + self._force_internal_error_finish(internal_error) def _set_exception(self, exception: BaseException) -> None: """ @@ -907,9 +907,9 @@ def _set_exception(self, exception: BaseException) -> None: # Contemplate on this GitHub issue along the way: # https://github.com/teremterem/Promising/issues/105 _logger.debug("Failed to chain original exception onto internal_error", exc_info=True) - self._force_internal_error_finish_from_loop(internal_error) + self._force_internal_error_finish(internal_error) - def _force_internal_error_finish_from_loop(self, error: BaseException) -> None: + def _force_internal_error_finish(self, error: BaseException) -> None: """ Last-resort recovery path. Force the Promise into _FINISHED with the given error, bypassing state validation. Each step is wrapped @@ -949,7 +949,7 @@ def _assert_done(self) -> None: if not self.done(): raise PromiseNotDoneError(f"Promise is not done: {self!r}") - def _synthesize_cancellation_from_loop(self, msg: str | None = None) -> None: + def _synthesize_cancellation(self, msg: str | None = None) -> None: """ Drive the Promise into a cancelled terminal state without relying on a running unpacking task to surface the ``CancelledError``. @@ -990,7 +990,7 @@ def _set_state(self, new_state: Sentinel) -> None: self._state = new_state # Force-close the context just in case (it was most likely closed by # the `with` block already, but it might also have been - # `_force_internal_error_finish_from_loop`) and unregister from parent + # `_force_internal_error_finish`) and unregister from parent # "if time": self.close_context() From a92eaf97df3e5c553ff8c121dc34ffa9f5dd707b Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 11:12:42 +0300 Subject: [PATCH 05/20] temporarily strip away thread safety --- promising/promise.py | 83 +------------------------------------------- 1 file changed, 1 insertion(+), 82 deletions(-) diff --git a/promising/promise.py b/promising/promise.py index 50c3e8ab2..f3f151eea 100644 --- a/promising/promise.py +++ b/promising/promise.py @@ -330,8 +330,6 @@ def __await__(self) -> Generator[Any, None, T_co]: Returns: The fully unpacked result of the Promise (no remaining nested Promises). - - NOTE: This method can only be used from the event loop of the Promise. """ self._assert_awaiting_on_correct_event_loop() @@ -364,9 +362,6 @@ def sync(self, *, timeout: float | None = None) -> T_co: SyncUsageError: If called from the same thread as the event loop, which would deadlock. TimeoutError: If timeout expires before completion. - - NOTE: This method is thread-safe, but it is unavailable from the event - loop of the Promise to avoid a deadlock. """ self._assert_no_sync_usage_deadlock() @@ -393,8 +388,6 @@ async def unpack_once(self) -> "T_co | Promise[Any]": Raises: EventLoopMismatchError: If awaited from a different event loop than the one this Promise belongs to. - - NOTE: This method can only be used from the event loop of the Promise. """ self._assert_awaiting_on_correct_event_loop() @@ -433,9 +426,6 @@ def unpack_once_sync(self, *, timeout: float | None = None) -> "T_co | Promise[A SyncUsageError: If called from the same thread as the event loop, which would deadlock. TimeoutError: If timeout expires before completion. - - NOTE: This method is thread-safe, but it is unavailable from the event - loop of the Promise to avoid a deadlock. """ self._assert_no_sync_usage_deadlock() @@ -461,32 +451,6 @@ def done(self) -> bool: Returns: Whether this Promise is "done". - - NOTE: This method is thread-safe, including from the event loop of the - Promise. - - Thread-safety contract for ``Promise`` state-reading methods (this - method and the ones below referencing it): - - The Promise state machine is monotonic — once advanced past - ``_PENDING`` (to ``_UNPACKED_ONCE``, ``_FINISHED``, or one of the - ``_CANCELLED_XX`` states), the state never moves backwards. The - writers (``_set_intermediate_promise`` / - ``_set_result`` / ``_set_exception``) write the - corresponding attribute (``_intermediate_promise``, ``_result``, - ``_exception``) *before* advancing the state via ``_set_state``, so a - reader that observes a state past ``_PENDING`` is guaranteed to also - observe the matching attribute. - - This relies on single-attribute reads and writes being atomic across - threads — which holds under CPython's reference (GIL-backed) - interpreter. Under a free-threaded CPython build the GIL no longer - provides that guarantee, and the reader/writer pair would need - explicit synchronization (e.g. a lock or memory fence) to remain - correct. Promising does not currently target free-threaded - interpreters. - # TODO Future-proof it ? - # https://github.com/teremterem/Promising/pull/102#discussion_r3197680342 """ state = self._state return state in (_FINISHED, _CANCELLED_BEFORE_UNPACKED_ONCE, _CANCELLED_AFTER_UNPACKED_ONCE) @@ -497,9 +461,6 @@ def unpacked_once(self) -> bool: either an intermediate Promise (which means a further unpacking step is still pending) or a final concrete value (in which case the Promise is also ``done()``). - - NOTE: This method is thread-safe, including from the event loop of the - Promise — see ``done()`` for the thread-safety contract. """ state = self._state return state in (_FINISHED, _UNPACKED_ONCE, _CANCELLED_AFTER_UNPACKED_ONCE) @@ -509,9 +470,6 @@ def unpacked_once_or_done(self) -> bool: Convenience predicate: True if the Promise is at least one-level unpacked, fully done, or cancelled. Used internally as the readiness check for one-level (non-recursive) consumers. - - NOTE: This method is thread-safe, including from the event loop of the - Promise — see ``done()`` for the thread-safety contract. """ state = self._state return state in (_FINISHED, _CANCELLED_BEFORE_UNPACKED_ONCE, _UNPACKED_ONCE, _CANCELLED_AFTER_UNPACKED_ONCE) @@ -520,9 +478,6 @@ def cancelled(self) -> bool: """ Whether the Promise has been cancelled (either before or after the first unpacking step). - - NOTE: This method is thread-safe, including from the event loop of the - Promise — see ``done()`` for the thread-safety contract. """ state = self._state return state in (_CANCELLED_BEFORE_UNPACKED_ONCE, _CANCELLED_AFTER_UNPACKED_ONCE) @@ -536,9 +491,6 @@ def result(self) -> T_co: asyncio.CancelledError: If the Promise was cancelled. BaseException: Re-raises whatever exception the Promise finished with (if any). - - NOTE: This method is thread-safe, including from the event loop of the - Promise — see ``done()`` for the thread-safety contract. """ self._assert_done() @@ -564,9 +516,6 @@ def intermediate_promise(self) -> "Promise[Any] | None": BaseException: Re-raises the underlying exception if the first unpacking step itself failed before producing an intermediate Promise. - - NOTE: This method is thread-safe, including from the event loop of the - Promise — see ``done()`` for the thread-safety contract. """ if not self.unpacked_once_or_done(): raise PromiseNotUnpackedError(f"Promise is not unpacked even once yet: {self!r}") @@ -592,9 +541,6 @@ def exception(self) -> BaseException | None: Raises: PromiseNotDoneError: If the Promise is not done yet. asyncio.CancelledError: If the Promise was cancelled. - - NOTE: This method is thread-safe, including from the event loop of the - Promise — see ``done()`` for the thread-safety contract. """ self._assert_done() @@ -633,9 +579,6 @@ def cancel(self, msg: str | None = None) -> bool: ``True`` if cancellation was requested for at least one underlying task, or synthesized for a not-yet-started Promise; ``False`` if the Promise was already done. - - NOTE: This method is thread-safe, including from the event loop of the - Promise. """ if self.done(): return False @@ -659,9 +602,6 @@ def cancel(self, msg: str | None = None) -> bool: return self.cancelled() def _ensure_single_unpacking_scheduled(self) -> None: - """ - NOTE: This method can only be used from the event loop of the Promise. - """ _unpacking_logger.log_single_unpacking_scheduling(promise=self) if self._single_unpacking_task is None and not self.unpacked_once_or_done(): @@ -673,9 +613,6 @@ def _ensure_single_unpacking_scheduled(self) -> None: _unpacking_logger.log_single_unpacking_scheduled(promise=self) def _ensure_from_unpacking_scheduled(self) -> None: - """ - NOTE: This method can only be used from the event loop of the Promise. - """ _unpacking_logger.log_full_unpacking_scheduling(promise=self) if self._full_unpacking_task is None and not self.done(): @@ -726,8 +663,6 @@ async def _unpack_once(self) -> None: Backs ``unpack_once()`` (and the first leg of ``_unpack_fully``). - - NOTE: This method can only be used from the event loop of the Promise. """ try: _unpacking_logger.log_single_unpacking_started(promise=self) @@ -770,8 +705,6 @@ async def _unpack_fully(self) -> None: captured via ``_set_exception``. Backs ``__await__`` (and, indirectly, ``sync()``). - - NOTE: This method can only be used from the event loop of the Promise. """ try: _unpacking_logger.log_full_unpacking_started(promise=self) @@ -791,7 +724,7 @@ async def _unpack_fully(self) -> None: result = self._intermediate_promise - # Note: cancelling this Promise does NOT propagate cancellation + # NOTE: Cancelling this Promise does NOT propagate cancellation # into the nested Promise being awaited below — asyncio's # task-cancellation lands on this task and unwinds upward; the # inner Promise's own task keeps running independently. @@ -820,8 +753,6 @@ def _set_intermediate_promise(self, promise: "Promise[Any]") -> None: """ Record the intermediate Promise returned by a single unpacking step. No-op if already unpacked once or done. - - NOTE: This method can only be used from the event loop of the Promise. """ try: if self._state is not _PENDING: @@ -838,8 +769,6 @@ def _set_result(self, result: T_co) -> None: """ Store the fully unpacked result. No-op if the Promise is already done (finished or cancelled). - - NOTE: This method can only be used from the event loop of the Promise. """ try: if self._state not in (_PENDING, _UNPACKED_ONCE): @@ -862,8 +791,6 @@ def _set_exception(self, exception: BaseException) -> None: A ``CancelledError`` arriving on an already-terminal Promise is silently dropped. Any other exception arriving in that state is treated as a framework bug and raises ``RuntimeError``. - - NOTE: This method can only be used from the event loop of the Promise. """ try: if self._state is _PENDING: @@ -921,8 +848,6 @@ def _force_internal_error_finish(self, error: BaseException) -> None: because parent unregistration raised. Treating such failures as bugs in the Promise class itself, this method prioritizes reaching a terminal state over surfacing further errors. - - NOTE: This method can only be used from the event loop of the Promise. """ try: _logger.debug("Force-finishing Promise %r with internal error", self, exc_info=error) @@ -942,10 +867,6 @@ def _force_internal_error_finish(self, error: BaseException) -> None: raise def _assert_done(self) -> None: - """ - NOTE: This method is thread-safe, including from the event loop of the - Promise — see ``done()`` for the thread-safety contract. - """ if not self.done(): raise PromiseNotDoneError(f"Promise is not done: {self!r}") @@ -959,8 +880,6 @@ def _synthesize_cancellation(self, msg: str | None = None) -> None: scheduled) and ``_unpacking_task_done_callback`` (task cancelled between ``create_task`` and its first ``__step``, so the body's ``except BaseException`` never saw the ``CancelledError``). - - NOTE: This method can only be used from the event loop of the Promise. """ # `_unpack_once` would normally close the context via # `with self:`. Without this, `_context_closed` stays False and the From 1e6befa6ab9da5eb8c95b7293074248ee81fc8ba Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 11:40:29 +0300 Subject: [PATCH 06/20] temporarily strip away thread safety --- CONTRIBUTORS.md | 4 +-- README.md | 32 +++++++++---------- promising/promise.py | 10 +----- promising/promising_context.py | 5 +-- .../test_unregister_on_cancellation.py | 10 ++---- tests/resolution/test_promise_cancellation.py | 5 ++- .../resolution/test_promising_function_run.py | 3 +- 7 files changed, 24 insertions(+), 45 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index cea27e3f6..2f64ef1b0 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -45,7 +45,7 @@ Tests use `pytest-asyncio` in auto mode — all async test functions are automat **Settings are frozen at creation time.** All configuration (`start_soon`, `children_start_soon`, `start_soon_default`, `thread_pool`, etc.) is fully resolved when a `Promise` or `PromisingContext` is constructed. Sentinels like `INHERIT` and `PROMISING_DEFAULT` are replaced with concrete values immediately — no deferred resolution happens at execution time. This is a core design principle: because a promise may run eagerly or be deferred, the user cannot predict *when* execution will happen, so settings must reflect the state of the world at the moment the promise was created. -**Core hierarchy flow:** `PromisingFunction` wraps an async or sync function → calling it creates a `Promise[T]` → during execution, the Promise sets itself as the current context via `ContextVar` → any Promises (and `PromisingContext` instances) created during that execution register themselves as its children via thread-safe strong-ref sets. +**Core hierarchy flow:** `PromisingFunction` wraps an async or sync function → calling it creates a `Promise[T]` → during execution, the Promise sets itself as the current context via `ContextVar` → any Promises (and `PromisingContext` instances) created during that execution register themselves as its children. ### PromisingContext (`promising/promising_context.py`) @@ -61,7 +61,7 @@ This file also contains the `context` class — a context manager / decorator th ### Promise (`promising/promise.py`) -`Promise[T_co]` is a direct subclass of `PromisingContext`. It owns a small state machine — `_PENDING` → `_UNPACKED_ONCE` → `_FINISHED`, with `_CANCELLED_BEFORE_UNPACKED_ONCE` / `_CANCELLED_AFTER_UNPACKED_ONCE` as alternative terminals — exposed through `done()`, `unpacked_once()`, `unpacked_once_or_done()`, and `cancelled()`, and queried for results via `result()`, `exception()`, and `intermediate_promise()`. All of those readers are thread-safe (state can only move forward). `loop`, `thread_pool`, and the rest of the configuration are inherited from `PromisingContext`. +`Promise[T_co]` is a direct subclass of `PromisingContext`. It owns a small state machine — `_PENDING` → `_UNPACKED_ONCE` → `_FINISHED`, with `_CANCELLED_BEFORE_UNPACKED_ONCE` / `_CANCELLED_AFTER_UNPACKED_ONCE` as alternative terminals — exposed through `done()`, `unpacked_once()`, `unpacked_once_or_done()`, and `cancelled()`, and queried for results via `result()`, `exception()`, and `intermediate_promise()`. `loop`, `thread_pool`, and the rest of the configuration are inherited from `PromisingContext`. **Two-step unpacking on the loop.** Resolution is split into two cooperating tasks, both pinned to `self.loop`: diff --git a/README.md b/README.md index 48bc080d7..bba5f86d5 100644 --- a/README.md +++ b/README.md @@ -346,9 +346,9 @@ promising.Defaults.START_SOON = True promising.Defaults.START_SOON = False ``` -## Thread-Safe Access +## Access from Non-Async Threads -`promise.sync()` (and `promise.unpack_once_sync()`) lets non-async threads block on a Promise's result. Both are thread-safe and schedule work on the Promise's event loop via `asyncio.run_coroutine_threadsafe`: +`promise.sync()` (and `promise.unpack_once_sync()`) lets non-async threads block on a Promise's result. Both schedule work on the Promise's event loop via `asyncio.run_coroutine_threadsafe`: ```python import threading @@ -357,7 +357,7 @@ async def main(): promise = fetch_data("https://example.com") def worker(): - # Blocks until the Promise resolves (thread-safe) + # Blocks until the Promise resolves result = promise.sync(timeout=5.0) print(result) @@ -367,12 +367,10 @@ async def main(): thread.join() ``` -Like `await`, calling `promise.sync()` automatically triggers the Promise's execution if it was created with `start_soon=False`. There is no need to start the Promise manually before blocking on it. +Like `await`, calling `promise.sync()` automatically triggers the Promise's execution if it was created with `start_soon=False`. `promise.sync()`, `promise.unpack_once_sync()`, and `await_children_sync()` all raise `SyncUsageError` if called from the event loop thread (which would deadlock). -`Promise.cancel()` is also thread-safe — it can be invoked from any thread and is dispatched onto the Promise's event loop when the caller is not already on it. - ## Result Unpacking A decorated function always returns a `Promise`, regardless of whether the underlying function returns a concrete value or another Promise. This means: @@ -470,7 +468,7 @@ Wrapping every async (or sync) operation in a `Promise` gives you: - **Effortless parallelism.** Call your decorated functions and they start running immediately — async on the event loop, sync in a thread pool (with `use_thread_pool=True`). Mix and match freely; the Promise abstraction papers over the difference. No manual `asyncio.gather`, no explicit executor management, no boilerplate to bridge async and threaded code. - **Multiple awaits.** A Promise caches its result. Any number of consumers can `await`, `.sync()`, `unpack_once()`, or `unpack_once_sync()` the same Promise and get the same value — the underlying function is never executed more than once. - **Automatic hierarchy.** Promises created during another Promise's execution become its children. You can wait for the entire subtree (`await_children()`), inspect what's still running (`collect_unsettled_children`), or scope configuration to a subtree — all without manual bookkeeping. -- **Thread-safe synchronous access.** Every Promise exposes `.sync()` and `.unpack_once_sync()`, so threads that can't `await` can still block on a Promise's result. Blocking automatically triggers execution of deferred (`start_soon=False`) Promises, just like `await` does. +- **Synchronous access.** Every Promise exposes `.sync()` and `.unpack_once_sync()`, so threads that can't `await` can still block on a Promise's result. Blocking automatically triggers execution of deferred (`start_soon=False`) Promises, just like `await` does. - **Consistent interface.** A decorated function always returns a `Promise` — whether the underlying function returns a concrete value or another Promise. `await` and `.sync()` recursively unpack nested Promises and return the final non-`Promise` value, so consumers get a uniform interface regardless of how deep the chain is. - **Configurable execution.** `start_soon`, `children_start_soon`, `thread_pool`, and other settings propagate through the hierarchy, letting you control eager vs. deferred execution and thread pool usage at any level. @@ -496,16 +494,16 @@ In short, a `Promise` turns a fire-and-forget coroutine into a first-class objec |---|---| | `await promise` | Wait for and return the result. Recursively unpacks nested Promises and always returns a concrete value. All consumption methods (`await`, `sync`, `unpack_once`, `unpack_once_sync`) can be called multiple times and always return the same cached result. Must be awaited on the Promise's own event loop (raises `EventLoopMismatchError` otherwise). | | `promise.unpack_once()` | Async — resolve the Promise but unpack only one level. Returns either a concrete value or another `Promise`. Must be awaited on the Promise's own event loop. | -| `promise.sync(timeout=None)` | Synchronous counterpart of `await promise` — blocks the calling thread, recursively unpacks nested Promises, and always returns a concrete value. Thread-safe; must not be called from the Promise's own event loop thread (raises `SyncUsageError`). | -| `promise.unpack_once_sync(timeout=None)` | Synchronous counterpart of `unpack_once` — blocks the calling thread and unpacks only one level. Returns either a concrete value or another `Promise`. Thread-safe; must not be called from the Promise's own event loop thread. | -| `promise.done()` | Whether the Promise is "done" — finished (with a result or exception) or cancelled. Thread-safe. Overrides `PromisingContext.done()`, which by default tracks the context-manager lifecycle. | -| `promise.cancelled()` | Whether the Promise has been cancelled. Thread-safe. | -| `promise.unpacked_once()` | Whether the Promise has been unpacked at least one level (i.e. its awaitable has produced either an intermediate Promise or a final value). Thread-safe. | -| `promise.unpacked_once_or_done()` | True if the Promise is at least one-level unpacked, fully done, or cancelled. Thread-safe. | -| `promise.result()` | The resolved value. Raises the underlying exception if the Promise finished with one, `PromiseNotDoneError` if it is not done yet, or `asyncio.CancelledError` if it was cancelled. Thread-safe. | -| `promise.exception()` | The exception that the Promise finished with, or `None`. Same readiness/cancellation rules as `result()`. Thread-safe. | -| `promise.intermediate_promise()` | The intermediate `Promise` produced by the first unpacking step, or `None` if the awaitable's result was already a non-Promise value. Raises `PromiseNotUnpackedError` if the Promise has not yet been unpacked once. Thread-safe. | -| `promise.cancel(msg=None)` | Request cancellation of the Promise. Mirrors `asyncio.Future.cancel()` / `asyncio.Task.cancel()`: the return value reports whether cancellation was *requested* — the Promise's terminal cancelled state is reached only once the `CancelledError` actually propagates through the underlying unpacking task(s). Thread-safe — when called from outside the Promise's event loop, the cancellation is dispatched onto that loop. | +| `promise.sync(timeout=None)` | Synchronous counterpart of `await promise` — blocks the calling thread, recursively unpacks nested Promises, and always returns a concrete value. Must not be called from the Promise's own event loop thread (raises `SyncUsageError`). | +| `promise.unpack_once_sync(timeout=None)` | Synchronous counterpart of `unpack_once` — blocks the calling thread and unpacks only one level. Returns either a concrete value or another `Promise`. Must not be called from the Promise's own event loop thread. | +| `promise.done()` | Whether the Promise is "done" — finished (with a result or exception) or cancelled. Overrides `PromisingContext.done()`, which by default tracks the context-manager lifecycle. | +| `promise.cancelled()` | Whether the Promise has been cancelled. | +| `promise.unpacked_once()` | Whether the Promise has been unpacked at least one level (i.e. its awaitable has produced either an intermediate Promise or a final value). | +| `promise.unpacked_once_or_done()` | True if the Promise is at least one-level unpacked, fully done, or cancelled. | +| `promise.result()` | The resolved value. Raises the underlying exception if the Promise finished with one, `PromiseNotDoneError` if it is not done yet, or `asyncio.CancelledError` if it was cancelled. | +| `promise.exception()` | The exception that the Promise finished with, or `None`. Same readiness/cancellation rules as `result()`. | +| `promise.intermediate_promise()` | The intermediate `Promise` produced by the first unpacking step, or `None` if the awaitable's result was already a non-Promise value. Raises `PromiseNotUnpackedError` if the Promise has not yet been unpacked once. | +| `promise.cancel(msg=None)` | Request cancellation of the Promise. Mirrors `asyncio.Future.cancel()` / `asyncio.Task.cancel()`: the return value reports whether cancellation was *requested* — the Promise's terminal cancelled state is reached only once the `CancelledError` actually propagates through the underlying unpacking task(s). | | `promise.loop` | The event loop this Promise is bound to (inherited from `PromisingContext`). | ### `wrap_awaitable` diff --git a/promising/promise.py b/promising/promise.py index f3f151eea..9bea54bc0 100644 --- a/promising/promise.py +++ b/promising/promise.py @@ -108,12 +108,11 @@ class Promise(PromisingContext, Generic[T_co]): Promise implements: - Asynchronous computation backed by an awaitable - Result/exception caching, with both async (``await``, ``unpack_once``) - and thread-safe sync (``sync``, ``unpack_once_sync``) consumption + and sync (``sync``, ``unpack_once_sync``) consumption - A two-step unpacking model: a single unpacking step that produces an intermediate Promise (if the awaitable returned one), and a full unpacking that recursively chases nested Promises down to a concrete value - - Cancellation that is safe to invoke from any thread - Construction-time stack capture (``frame_summary_tuple``) consumed by the ``sys.excepthook`` / ``threading.excepthook`` overrides installed via ``install_promising_tracebacks()`` to render @@ -568,13 +567,6 @@ def cancel(self, msg: str | None = None) -> bool: ``_set_exception``, with no task involvement — analogous to ``Future.cancel()`` on a not-yet-running future. - When called from the Promise's own event loop thread the cancellation - is dispatched directly. When called from any other thread it is - scheduled onto the Promise's event loop via - ``call_soon_threadsafe`` and the call blocks only long enough for the - scheduled dispatch to finish (it does not wait for the cancellation - itself to land). - Returns: ``True`` if cancellation was requested for at least one underlying task, or synthesized for a not-yet-started Promise; diff --git a/promising/promising_context.py b/promising/promising_context.py index 5588a4370..7a6cc567b 100644 --- a/promising/promising_context.py +++ b/promising/promising_context.py @@ -431,8 +431,6 @@ def __init__( self._unsettled_children = set[PromisingContext]() if register_with_parent: - # No other code has a reference to this PromisingContext yet, so we - # can just register it with the parent in a thread-unsafe manner self._register_with_parent() @property @@ -790,7 +788,7 @@ def __exit__( def close_context(self) -> None: """ Mark this context as closed and unregister it from its parent if - no unsettled descendants remain. Safe to call from any thread. + no unsettled descendants remain. Called automatically by ``__exit__`` (so a normal ``with`` block always closes the context). For a ``Promise``, the context is @@ -854,7 +852,6 @@ def __repr__(self) -> str: return f"<{namespace_prefix}{self.__class__.__name__} id={id(self)}>" def _register_with_parent(self) -> None: - # It is thread-safe for the parent but is unsafe for the child itself if self._parent is not None and not self.done(): self._parent._register_children(self) diff --git a/tests/hierarchy/test_unregister_on_cancellation.py b/tests/hierarchy/test_unregister_on_cancellation.py index 14271d703..f69989884 100644 --- a/tests/hierarchy/test_unregister_on_cancellation.py +++ b/tests/hierarchy/test_unregister_on_cancellation.py @@ -43,8 +43,8 @@ async def coro() -> str: async def test_cancel_pending_promise_from_other_thread_unregisters_from_parent() -> None: """ - Synthesize path reached via the thread-safe dispatch: cancel() is - called from a non-loop thread. The unregistration must still happen. + cancel() is called from a non-loop thread. The unregistration must still + happen. """ with promising.context() as parent: @@ -63,10 +63,6 @@ def cancel_in_thread() -> None: thread = threading.Thread(target=cancel_in_thread) thread.start() - # Yield so the threadsafe callback (and the thread blocked on its - # future) can run on this loop. Don't await the promise itself - # here — that would start an unpacking task and race with the - # synthesize path we're trying to exercise. while not cancel_result: await asyncio.sleep(0.1) thread.join(timeout=2) @@ -118,8 +114,6 @@ async def coro() -> str: return "unreachable" promise = Promise(coro(), start_soon=True) - # Let the threadsafe scheduling callback create the task without - # giving the task itself a chance to take its first step. await asyncio.sleep(0) full_task = promise._full_unpacking_task assert full_task is not None diff --git a/tests/resolution/test_promise_cancellation.py b/tests/resolution/test_promise_cancellation.py index d4971e196..cf4babaaa 100644 --- a/tests/resolution/test_promise_cancellation.py +++ b/tests/resolution/test_promise_cancellation.py @@ -200,14 +200,13 @@ async def coro() -> str: assert promise.done() is True -# ── Thread-safe cancel() ───────────────────────────────────────── +# ── cancel() from another thread ───────────────────────────────── async def test_cancel_from_another_thread() -> None: """ cancel() called from a thread other than the event loop's thread - dispatches via call_soon_threadsafe and reports True once the - cancellation request was scheduled. + reports True once the cancellation request was scheduled. """ coro_started = asyncio.Event() coro_finished = False diff --git a/tests/resolution/test_promising_function_run.py b/tests/resolution/test_promising_function_run.py index 5e145ecff..9ea95ad26 100644 --- a/tests/resolution/test_promising_function_run.py +++ b/tests/resolution/test_promising_function_run.py @@ -56,8 +56,7 @@ def test_promising_function_run_with_child_promise( ) -> None: """ Same as above but also creates a child Promise inside the function, which - exercises _call_soon_threadsafe and verifies the event loop is correctly - resolved at runtime. + verifies the event loop is correctly resolved at runtime. Runs in a separate thread to avoid interfering with the pytest-asyncio event loop. From 621e85dc4714c1325b6f527730fd4f8ca93033fd Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 11:40:53 +0300 Subject: [PATCH 07/20] uv lock --upgrade --- uv.lock | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/uv.lock b/uv.lock index 61cbff85c..88fa94182 100644 --- a/uv.lock +++ b/uv.lock @@ -408,14 +408,14 @@ wheels = [ [[package]] name = "click" -version = "8.3.3" +version = "8.4.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "colorama", marker = "sys_platform == 'win32'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/bb/63/f9e1ea081ce35720d8b92acde70daaedace594dc93b693c869e0d5910718/click-8.3.3.tar.gz", hash = "sha256:398329ad4837b2ff7cbe1dd166a4c0f8900c3ca3a218de04466f38f6497f18a2", size = 328061, upload-time = "2026-04-22T15:11:27.506Z" } +sdist = { url = "https://files.pythonhosted.org/packages/23/e4/796662cd90cf80e3a363c99db2b88e0e394b988a575f60a17e16440cd011/click-8.4.0.tar.gz", hash = "sha256:638f1338fe1235c8f4e008e4a8a254fb5c5fbdcbb40ece3c9142ebb78e792973", size = 350843, upload-time = "2026-05-17T00:47:58.425Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/ae/44/c1221527f6a71a01ec6fbad7fa78f1d50dfa02217385cf0fa3eec7087d59/click-8.3.3-py3-none-any.whl", hash = "sha256:a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613", size = 110502, upload-time = "2026-04-22T15:11:25.044Z" }, + { url = "https://files.pythonhosted.org/packages/ee/ae/8e92f8058baf87f6c7d86ee7e457668690195cc77efedb8d3797a06e3940/click-8.4.0-py3-none-any.whl", hash = "sha256:40c50b7c6c6adac2823d411041ec84f3f103f1b280d5e9ce0d7f998995832f81", size = 116147, upload-time = "2026-05-17T00:47:56.842Z" }, ] [[package]] @@ -581,11 +581,11 @@ wheels = [ [[package]] name = "decorator" -version = "5.2.1" +version = "5.3.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/43/fa/6d96a0978d19e17b68d634497769987b16c8f4cd0a7a05048bec693caa6b/decorator-5.2.1.tar.gz", hash = "sha256:65f266143752f734b0a7cc83c46f4618af75b8c5911b00ccb61d0ac9b6da0360", size = 56711, upload-time = "2025-02-24T04:41:34.073Z" } +sdist = { url = "https://files.pythonhosted.org/packages/5c/50/a39dd7ab407e93978dfa07d109b7d633e37958c89f30cbcec061b77b3ebc/decorator-5.3.0.tar.gz", hash = "sha256:95fda3122972c847cf0ff7e0ce2829bf25136f2526b627b3da85b60ca5f485c0", size = 58431, upload-time = "2026-05-17T06:59:57.258Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/4e/8c/f3147f5c4b73e7550fe5f9352eaa956ae838d5c51eb58e7a25b9f3e2643b/decorator-5.2.1-py3-none-any.whl", hash = "sha256:d316bb415a2d9e2d2b3abcc4084c6502fc09240e292cd76a76afc106a1c8e04a", size = 9190, upload-time = "2025-02-24T04:41:32.565Z" }, + { url = "https://files.pythonhosted.org/packages/d5/6f/f8d0bba4dc2a69817d74f640d504650241ebf2f9f7263426f1b953b344d4/decorator-5.3.0-py3-none-any.whl", hash = "sha256:f8c2d71ede92f073144ddd7f3e9fbbc3bd0f2f29522c9d75ee648d66553834f4", size = 11104, upload-time = "2026-05-17T06:59:54.676Z" }, ] [[package]] @@ -1224,7 +1224,7 @@ wheels = [ [[package]] name = "litellm" -version = "1.84.0" +version = "1.85.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "aiohttp" }, @@ -1240,9 +1240,9 @@ dependencies = [ { name = "tiktoken" }, { name = "tokenizers" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/dd/e9/8941b7e72a187000561d932c0f2f2ed2b0fd080dfc33ba6e05961d45ca7d/litellm-1.84.0.tar.gz", hash = "sha256:b8ad0cbea11a5941b18d5af973017a340abd3d3ab41cb86e5401b970626d71a6", size = 15103206, upload-time = "2026-05-14T05:45:53.017Z" } +sdist = { url = "https://files.pythonhosted.org/packages/3b/d5/3c9b560db2ffa9e498655d0dfd74f408bc5b32ede858b5731c2a5fa4c752/litellm-1.85.0.tar.gz", hash = "sha256:babdd569809af913d08a08a7eb55df1ed3e6a3960ee365c6cef4ad031c9bc72a", size = 15344387, upload-time = "2026-05-17T01:59:15.97Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/01/a6/77fa1bbf5e42eb596b06318b3f7e6af5d0f44028046d1d598c6a595d028f/litellm-1.84.0-py3-none-any.whl", hash = "sha256:2a58d6041e6aa27d1a28dc8d8828ab500fef1a00ef74ca65e60899035010c2f2", size = 16735062, upload-time = "2026-05-14T05:45:49.927Z" }, + { url = "https://files.pythonhosted.org/packages/1c/38/e6a4abb062e039d18d59538cc4e6fc370c2c10cd2bff4a2e546acb69dcb9/litellm-1.85.0-py3-none-any.whl", hash = "sha256:2bb449153610691faffd76f5b94a8c29e4b66fc5394156ebf54fd4fe92759b1a", size = 16978229, upload-time = "2026-05-17T01:59:11.902Z" }, ] [[package]] From 344a7efd650489b86997bcd49dcf4e9cf21d87e3 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 12:38:16 +0300 Subject: [PATCH 08/20] couple new todos in tests --- tests/hierarchy/test_unregister_on_cancellation.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/hierarchy/test_unregister_on_cancellation.py b/tests/hierarchy/test_unregister_on_cancellation.py index f69989884..96e6922c7 100644 --- a/tests/hierarchy/test_unregister_on_cancellation.py +++ b/tests/hierarchy/test_unregister_on_cancellation.py @@ -108,6 +108,9 @@ async def test_cancel_full_unpacking_task_before_first_step_transitions_promise( the done-callback bridge, the Task ends cancelled while the Promise stays ``_PENDING`` and leaks in its parent's ``_unsettled_children``. """ + # TODO [TESTS] This test is suspicious - we need a way to know that it + # actually tests what it's supposed to test and doesn't pass for unrelated + # reasons with promising.context() as parent: async def coro() -> str: @@ -138,6 +141,9 @@ async def test_cancel_single_unpacking_task_before_first_step_transitions_promis ``_single_unpacking_task`` created via ``unpack_once()``. Verifies the done-callback is wired on both task creation sites. """ + # TODO [TESTS] This test is suspicious - we need a way to know that it + # actually tests what it's supposed to test and doesn't pass for unrelated + # reasons with promising.context() as parent: async def coro() -> str: From f17f043d8a168c09bb97f3cef845d0e1c5622007 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 14:13:03 +0300 Subject: [PATCH 09/20] race condition tests --- tests/race_conditions/__init__.py | 0 tests/race_conditions/conftest.py | 37 ++ .../test_concurrent_consumption_race.py | 318 +++++++++++ .../test_context_lifecycle_race.py | 332 ++++++++++++ .../test_invariants_under_load.py | 412 +++++++++++++++ .../test_state_machine_race.py | 260 +++++++++ .../test_unsettled_children_set_race.py | 496 ++++++++++++++++++ 7 files changed, 1855 insertions(+) create mode 100644 tests/race_conditions/__init__.py create mode 100644 tests/race_conditions/conftest.py create mode 100644 tests/race_conditions/test_concurrent_consumption_race.py create mode 100644 tests/race_conditions/test_context_lifecycle_race.py create mode 100644 tests/race_conditions/test_invariants_under_load.py create mode 100644 tests/race_conditions/test_state_machine_race.py create mode 100644 tests/race_conditions/test_unsettled_children_set_race.py diff --git a/tests/race_conditions/__init__.py b/tests/race_conditions/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/race_conditions/conftest.py b/tests/race_conditions/conftest.py new file mode 100644 index 000000000..44af6a574 --- /dev/null +++ b/tests/race_conditions/conftest.py @@ -0,0 +1,37 @@ +""" +Race-condition stress tests use threads and high iteration counts to +surface unsynchronized state in the framework. + +We force a very short GIL switch interval (1µs) so the interpreter +preempts between almost every bytecode, dramatically widening the race +windows that the framework's unprotected read-then-act sequences +expose. Without this, CPython's default ~5ms switch interval often +hides the bugs. +""" + +import sys +from collections.abc import Iterator + +import pytest + +# Apply a longer per-test timeout to every test in this directory. +pytestmark = pytest.mark.timeout(30) + +_ORIGINAL_SWITCH_INTERVAL = sys.getswitchinterval() +_RACE_SWITCH_INTERVAL = 1e-6 # 1 microsecond + + +@pytest.fixture(autouse=True) +def _aggressive_gil_switching() -> Iterator[None]: + """Force frequent GIL hand-offs so reader/writer races surface.""" + sys.setswitchinterval(_RACE_SWITCH_INTERVAL) + try: + yield + finally: + sys.setswitchinterval(_ORIGINAL_SWITCH_INTERVAL) + + +def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None: + for item in items: + if "race_conditions" in str(item.fspath): + item.add_marker(pytest.mark.timeout(30)) diff --git a/tests/race_conditions/test_concurrent_consumption_race.py b/tests/race_conditions/test_concurrent_consumption_race.py new file mode 100644 index 000000000..63446dcd3 --- /dev/null +++ b/tests/race_conditions/test_concurrent_consumption_race.py @@ -0,0 +1,318 @@ +""" +Race conditions around multi-thread consumption of Promises and +context hierarchies via the sync API: + +- ``Promise.sync()`` / ``Promise.unpack_once_sync()`` — invoked by + multiple worker threads simultaneously on the same Promise. +- ``promising.await_children_sync()`` — invoked from one thread while + workers on other threads keep registering new child Promises under + the same parent. + +All of these end up reading/writing ``Promise._state``, +``Promise._full_unpacking_task``, ``Promise._single_unpacking_task``, +``PromisingContext._unsettled_children`` — none of which are protected. +""" + +import asyncio +import threading +from concurrent.futures import ThreadPoolExecutor + +import pytest + +import promising + +# ── concurrent sync() consumption ────────────────────────────── + + +async def test_many_threads_calling_sync_on_same_promise_consistent() -> None: + """ + N worker threads simultaneously call ``promise.sync()`` on the same + Promise. All must return the same value; none must hang or raise. + """ + loop = asyncio.get_running_loop() + + for _ in range(10): + + async def coro() -> int: + await asyncio.sleep(0.01) + return 1234 + + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) + + N = 32 + results: list[int] = [] + errors: list[BaseException] = [] + lock = threading.Lock() + + def consumer() -> None: + try: + value = promise.sync(timeout=5) + with lock: + results.append(value) + except BaseException as exc: # noqa: BLE001 + with lock: + errors.append(exc) + + with ThreadPoolExecutor(max_workers=N) as ex: + await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) + + assert not errors, errors + assert results == [1234] * N, f"inconsistent results from concurrent sync(): {set(results)}" + + +async def test_many_threads_calling_unpack_once_sync_on_same_promise_consistent() -> None: + """ + Same scenario as ``test_many_threads_calling_sync_on_same_promise`` + but using ``unpack_once_sync``. All threads should observe the same + one-level-unpacking outcome (here: a concrete value, since the + coroutine does not return a Promise). + """ + loop = asyncio.get_running_loop() + + async def coro() -> str: + await asyncio.sleep(0.01) + return "ok" + + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) + + N = 32 + results: list[object] = [] + errors: list[BaseException] = [] + lock = threading.Lock() + + def consumer() -> None: + try: + value = promise.unpack_once_sync(timeout=5) + with lock: + results.append(value) + except BaseException as exc: # noqa: BLE001 + with lock: + errors.append(exc) + + with ThreadPoolExecutor(max_workers=N) as ex: + await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) + + assert not errors, errors + assert results == ["ok"] * N + + +# ── sync() race with cancel() ────────────────────────────────── + + +async def test_sync_consumer_thread_observes_clean_terminal_after_cancel() -> None: + """ + One thread waits on ``promise.sync()`` while another fires + ``promise.cancel()``. The sync caller must observe a clean + terminal state — either the value (if cancel lost the race) or + ``CancelledError``. It must never observe an internal + ``RuntimeError`` from a racing state transition. + """ + loop = asyncio.get_running_loop() + + for _ in range(40): + cancel_event = asyncio.Event() + + async def coro() -> int: + try: + await cancel_event.wait() + except asyncio.CancelledError: + raise + return 9 + + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) + + consumer_done = threading.Event() + consumer_result: dict[str, object] = {} + + def consume() -> None: + try: + consumer_result["value"] = promise.sync(timeout=5) + except BaseException as exc: # noqa: BLE001 + consumer_result["exception"] = exc + finally: + consumer_done.set() + + consumer = threading.Thread(target=consume, daemon=True) + consumer.start() + + # Let the consumer thread get into run_coroutine_threadsafe() + await asyncio.sleep(0) + + promise.cancel() + + # Park the loop until the consumer thread is done. + while not consumer_done.is_set(): + await asyncio.sleep(0) + + consumer.join(timeout=5) + assert not consumer.is_alive() + + # The sync caller must end up with a clean terminal outcome: + # either the value (cancel lost the race) or a CancelledError. + # Anything else (RuntimeError, internal state errors, ...) means + # the racing state-machine paths leaked an internal exception. + if "exception" in consumer_result: + exc = consumer_result["exception"] + # Compare by class name to side-step asyncio vs. + # concurrent.futures CancelledError class differences. + assert type(exc).__name__ == "CancelledError", ( + f"sync() raised an unexpected internal exception during cancel race: " + f"{type(exc).__module__}.{type(exc).__qualname__}: {exc!r}" + ) + else: + assert consumer_result["value"] == 9 + + +# ── await_children_sync race with thread-side registration ────── + + +async def test_await_children_sync_during_thread_registration_does_not_raise() -> None: + """ + A sync promising function runs in the thread pool and calls + ``promising.await_children_sync()``; meanwhile additional threads + keep creating child Promises under the same parent. The sync + waiter must drain all children without raising "set changed size + during iteration" from the underlying ``collect_unsettled_children``. + """ + loop = asyncio.get_running_loop() + + @promising.function + async def root() -> None: + active = promising.get_active_promise() + stop = threading.Event() + thread_errors: list[BaseException] = [] + registered_children: list[promising.Promise] = [] + registered_lock = threading.Lock() + + async def _quick() -> int: + return 0 + + def writer() -> None: + try: + while not stop.is_set(): + p = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + with registered_lock: + registered_children.append(p) + except BaseException as exc: # noqa: BLE001 + thread_errors.append(exc) + + @promising.function(use_thread_pool=True) + def sync_waiter() -> str: + # Each iteration triggers a fresh collect_unsettled_children + # via await_children_sync. + for _ in range(20): + promising.await_children_sync(whole_subtree=True) + return "drained" + + writers = [threading.Thread(target=writer, daemon=True) for _ in range(4)] + for w in writers: + w.start() + try: + waiter_promise = sync_waiter() + try: + await waiter_promise + finally: + stop.set() + for w in writers: + w.join(timeout=5) + finally: + for c in registered_children: + c.cancel() + + assert not thread_errors, thread_errors + + await root() + + +# ── concurrent .sync() consumption *and* registration ─────────── + + +async def test_concurrent_sync_consumers_and_child_registrations() -> None: + """ + Realistic stress: half the workers consume a published Promise + via ``.sync()`` while the other half register fresh child Promises + on the same parent. The framework must keep both the consumption + path and the parent's set consistent. + """ + loop = asyncio.get_running_loop() + + @promising.function + async def root() -> int: + active = promising.get_active_promise() + + @promising.function + async def published() -> int: + await asyncio.sleep(0.01) + return 99 + + target = published() + + N_CONSUMERS = 16 + N_REGISTRARS = 16 + errors: list[BaseException] = [] + errors_lock = threading.Lock() + consumer_results: list[int] = [] + new_children: list[promising.Promise] = [] + + async def _quick() -> int: + return 0 + + def consumer() -> None: + try: + consumer_results.append(target.sync(timeout=5)) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + def registrar() -> None: + try: + for _ in range(20): + new_children.append( + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + ) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: + consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] + registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] + await asyncio.gather(*consumer_futs, *registrar_futs) + + assert not errors, errors + assert consumer_results == [99] * N_CONSUMERS + + # All registrars' children must be tracked. + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = set(new_children) - actual + assert not missing, f"{len(missing)} children were lost from the parent's set" + + for c in new_children: + c.cancel() + return 0 + + await root() + + +# ── sanity ────────────────────────────────────────────────────── + + +@pytest.mark.skip(reason="Sanity reference — single-thread consumption is fine.") +async def test_single_threaded_sync_consumption_works_sanity() -> None: + async def coro() -> int: + return 1 + + promise = promising.wrap_awaitable(coro(), parent=None, start_soon=True) + loop = asyncio.get_running_loop() + value = await loop.run_in_executor(None, promise.sync) + assert value == 1 diff --git a/tests/race_conditions/test_context_lifecycle_race.py b/tests/race_conditions/test_context_lifecycle_race.py new file mode 100644 index 000000000..3ef0ded8e --- /dev/null +++ b/tests/race_conditions/test_context_lifecycle_race.py @@ -0,0 +1,332 @@ +""" +Race conditions around ``PromisingContext``'s lifecycle: + +- ``__enter__`` reads ``self._previous_token`` and + ``self._context_closed`` before writing the token. Two threads + entering the same context can both pass the check and both call + ``ContextVar.set``, leaving the loser's token orphaned and the next + ``__exit__`` cross-thread ``reset()`` either no-op'ing or raising + ``ValueError``. +- ``close_context`` flips ``_context_closed`` and calls + ``_unregister_from_parent_if_time`` without coordinating with + ``_register_children`` on the same instance. + +These tests stress those paths via real ``with`` blocks and real +``PromisingContext`` instances. +""" + +import asyncio +import threading + +import pytest + +import promising + +# ── helpers ───────────────────────────────────────────────────── + + +def _make_dedicated_loop() -> asyncio.AbstractEventLoop: + return asyncio.new_event_loop() + + +# ── concurrent __enter__ of the same instance ─────────────────── + + +def test_concurrent_enter_same_context_only_one_succeeds() -> None: + """ + Two worker threads call ``ctx.__enter__()`` on the same instance + simultaneously. Exactly one should succeed; the other must see + ``ContextAlreadyActiveError``. Without a lock around the + ``_previous_token is not None`` check both can pass it and both + will overwrite ``_previous_token`` — silently leaking the + first-thread token. + """ + loop = _make_dedicated_loop() + try: + N = 8 + for _ in range(500): + ctx = promising.PromisingContext(loop=loop, parent=None) + barrier = threading.Barrier(N) + succeeded: list[str] = [] + already_active_errors: list[BaseException] = [] + other_errors: list[BaseException] = [] + list_lock = threading.Lock() + + def enter() -> None: + try: + barrier.wait() + ctx.__enter__() + with list_lock: + succeeded.append(threading.current_thread().name) + except promising.ContextAlreadyActiveError as exc: + with list_lock: + already_active_errors.append(exc) + except BaseException as exc: # noqa: BLE001 + with list_lock: + other_errors.append(exc) + + threads = [threading.Thread(target=enter, name=f"T{i}", daemon=True) for i in range(N)] + for t in threads: + t.start() + for t in threads: + t.join(timeout=5) + assert not t.is_alive() + + assert not other_errors, other_errors + assert len(succeeded) == 1, ( + f"{len(succeeded)} threads entered the same context concurrently " + f"(succeeded={succeeded}, already_active={len(already_active_errors)})" + ) + assert len(already_active_errors) == N - 1 + finally: + loop.close() + + +def test_concurrent_enter_then_concurrent_exit_does_not_corrupt_contextvar() -> None: + """ + When two threads enter the same context (currently both can pass + the check) and then both call ``__exit__``, one of them ends up + calling ``ContextVar.reset(token)`` with a token that was created + in the *other* thread's context. That cross-thread reset is at + minimum a no-op and at worst raises ``ValueError``. Either way + ``__exit__`` must not crash with an unrelated error. + """ + import time + + loop = _make_dedicated_loop() + try: + for _ in range(3000): + ctx = promising.PromisingContext(loop=loop, parent=None) + + enter_barrier = threading.Barrier(2) + errors: list[BaseException] = [] + errors_lock = threading.Lock() + + def routine() -> None: + try: + enter_barrier.wait() + try: + ctx.__enter__() + except promising.ContextAlreadyActiveError: + # Framework rejected this thread's entry: nothing to exit. + return + # Yield to give the other thread a chance to enter, too. + time.sleep(0.0005) + ctx.__exit__(None, None, None) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + t1 = threading.Thread(target=routine, daemon=True) + t2 = threading.Thread(target=routine, daemon=True) + t1.start() + t2.start() + t1.join(timeout=5) + t2.join(timeout=5) + + assert not errors, f"{len(errors)} unexpected exceptions: {errors!r}" + finally: + loop.close() + + +# ── register vs close on a single context ─────────────────────── + + +def test_register_child_during_close_does_not_silently_succeed() -> None: + """ + Thread A is in the middle of ``parent._register_children(child)``: + after the ``self.closed()`` check passes but before + ``self._unsettled_children.update(...)`` runs, Thread B calls + ``parent.close_context()``. Currently the child is then added to + a context that has already been closed. + + The expected contract: either the child registration raises + ``ContextAlreadyClosedError`` and the parent's set stays empty, + OR the registration completes and the child must still be reachable + via ``collect_unsettled_children`` until it is closed itself. + """ + loop = _make_dedicated_loop() + try: + for _ in range(3000): + parent = promising.PromisingContext(loop=loop, parent=None) + barrier = threading.Barrier(2) + outcome: dict[str, object] = {} + + def add_child() -> None: + barrier.wait() + try: + outcome["child"] = promising.PromisingContext(loop=loop, parent=parent) + except promising.ContextAlreadyClosedError as exc: + outcome["closed_error"] = exc + + def close_parent() -> None: + barrier.wait() + parent.close_context() + + t1 = threading.Thread(target=add_child, daemon=True) + t2 = threading.Thread(target=close_parent, daemon=True) + t1.start() + t2.start() + t1.join(timeout=5) + t2.join(timeout=5) + + if "child" in outcome: + child = outcome["child"] + # Invariant: a context that is closed cannot accept new + # children. If the race lets registration succeed against + # a closed parent, that contract is broken — the child + # has a parent reference that points at a context that + # will never drive its lifecycle. + assert not parent.closed(), ( + "child registration succeeded onto a parent that is now closed — " + "the `closed() → no new children` invariant was violated by a race" + ) + reachable = child in parent.collect_unsettled_children( + whole_subtree=False, + awaitables_only=False, + ) + assert reachable, ( + "child registered onto parent silently, but is missing from " + "parent._unsettled_children — torn update or lost child" + ) + finally: + loop.close() + + +# ── Promise context: with-block opening races with sync registration ─ + + +async def test_promise_context_open_races_with_external_child_registration() -> None: + """ + A Promise enters its own context (``with self:``) inside + ``_unpack_once`` on the loop thread. While that ``__enter__`` + runs, a worker thread tries to register a brand-new child Promise + against the same Promise (as parent). The child registration uses + ``self.closed()`` as the gate — that field flips to True only on + ``__exit__``, but ``_previous_token`` is being set in lockstep with + ``__enter__`` on the loop thread. + + Concurrent reads/writes of those instance attributes can: + + - raise ``ContextAlreadyClosedError`` even though the parent is + still open + - silently add a child to a parent whose lifecycle has already + moved on + """ + loop = asyncio.get_running_loop() + + @promising.function + async def parent_promise() -> int: + active = promising.get_active_promise() + errors: list[BaseException] = [] + children: list[promising.Promise] = [] + children_lock = threading.Lock() + stop = threading.Event() + + async def _quick() -> int: + return 1 + + def worker() -> None: + try: + while not stop.is_set(): + p = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + with children_lock: + children.append(p) + except BaseException as exc: # noqa: BLE001 + errors.append(exc) + + writers = [threading.Thread(target=worker, daemon=True) for _ in range(8)] + for w in writers: + w.start() + try: + for _ in range(200): + await asyncio.sleep(0) + finally: + stop.set() + for w in writers: + w.join(timeout=5) + + assert not errors, errors + + # Every child registered while the Promise was running must + # be tracked. + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = set(children) - actual + assert not missing, f"{len(missing)} children were lost from active Promise's set during the lifecycle race" + + # Clean up so asyncio doesn't warn about un-awaited coros. + for c in children: + c.cancel() + + return 0 + + await parent_promise() + + +# ── close_context idempotency under threads ───────────────────── + + +def test_close_context_concurrent_calls_must_be_idempotent() -> None: + """ + ``close_context`` flips ``_context_closed`` and conditionally + unregisters from the parent. Concurrent calls from many threads + on the same instance should converge on a single + "closed + unregistered" terminal state — but they all read the + fields, all decide they need to unregister, and all call + ``parent._unregister_children(self)``. With the underlying + ``set.difference_update`` being a no-op on missing elements, the + only visible damage is when several siblings finish in the same + cycle and trigger re-entrant cascading unregistration. This test + asserts the simpler property: no exceptions, and parent is empty. + """ + loop = _make_dedicated_loop() + try: + for _ in range(100): + parent = promising.PromisingContext(loop=loop, parent=None) + child = promising.PromisingContext(loop=loop, parent=parent) + + N = 16 + errors: list[BaseException] = [] + errors_lock = threading.Lock() + barrier = threading.Barrier(N) + + def close_target() -> None: + try: + barrier.wait() + child.close_context() + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + threads = [threading.Thread(target=close_target, daemon=True) for _ in range(N)] + for t in threads: + t.start() + for t in threads: + t.join(timeout=5) + assert not t.is_alive() + + assert not errors, errors + assert child.closed() + assert child not in parent.collect_unsettled_children( + whole_subtree=False, + awaitables_only=False, + ) + finally: + loop.close() + + +@pytest.mark.skip(reason="Sanity reference — single-threaded lifecycle is fine.") +def test_single_threaded_enter_exit_works() -> None: + loop = _make_dedicated_loop() + try: + ctx = promising.PromisingContext(loop=loop, parent=None) + ctx.__enter__() + ctx.__exit__(None, None, None) + assert ctx.closed() + finally: + loop.close() diff --git a/tests/race_conditions/test_invariants_under_load.py b/tests/race_conditions/test_invariants_under_load.py new file mode 100644 index 000000000..c2656e704 --- /dev/null +++ b/tests/race_conditions/test_invariants_under_load.py @@ -0,0 +1,412 @@ +""" +Heavier load tests that stress thread-safety invariants the framework +must maintain regardless of CPython's GIL atomicity guarantees. + +These tests do not target a single check-then-act window — they keep +many threads doing many operations until either: + +- the framework raises something unexpected (``RuntimeError`` from a + state-machine mismatch, ``ContextAlreadyClosedError`` on what should + be a still-open context, etc.), or +- an invariant on the *final* state is violated (lost children, + inconsistent ``done()``/``cancelled()``/``result()`` triplet, ...). + +The intent: even when individual race windows are tiny on CPython, a +high-volume test eventually lands on the wrong interleaving. +""" + +import asyncio +import threading +from concurrent.futures import ThreadPoolExecutor + +import promising + +# ── tree-shape invariants under heavy parallel construction ───── + + +async def test_deep_hierarchy_stress_keeps_tree_consistent() -> None: + """ + Many sync-pool workers build a 2-level promise hierarchy in + parallel. After all work completes the root must have awaited every + leaf, and every leaf's parent must have unregistered cleanly. + """ + + N_PARENTS = 16 + N_CHILDREN_PER_PARENT = 32 + + @promising.function + async def leaf(idx: int) -> int: + return idx + + @promising.function(use_thread_pool=True) + def sync_parent(parent_idx: int) -> int: + children = [leaf(parent_idx * 1000 + i) for i in range(N_CHILDREN_PER_PARENT)] + total = 0 + for c in children: + total += c.sync(timeout=5) + return total + + @promising.function + async def root() -> int: + parents = [sync_parent(i) for i in range(N_PARENTS)] + return sum(await asyncio.gather(*parents)) + + expected = sum(p * 1000 * N_CHILDREN_PER_PARENT + sum(range(N_CHILDREN_PER_PARENT)) for p in range(N_PARENTS)) + actual = await root() + assert actual == expected, ( + f"lost children or duplicated work in concurrent hierarchy build: {actual} vs {expected}" + ) + + +# ── many cancels racing with a fast natural completion ────────── + + +async def test_cancel_race_state_consistency_high_iterations() -> None: + """ + For a Promise whose coroutine completes after one tick, fire a + storm of ``cancel()`` calls from worker threads while the loop + drives the task to completion. After settling, the *triplet* + ``done()``/``cancelled()``/``exception()``/``result()`` must agree: + + - ``cancelled()`` True → ``result()`` raises ``CancelledError``, + ``exception()`` raises ``CancelledError`` + - ``cancelled()`` False → ``result()`` returns the value, + ``exception()`` returns ``None`` + + A torn state-machine update can put the Promise in a state where + ``cancelled()`` is False but ``result()`` raises + ``CancelledError`` (or vice-versa) — that's the bug we are hunting. + """ + loop = asyncio.get_running_loop() + + for _ in range(300): + + async def coro() -> int: + for _ in range(3): + await asyncio.sleep(0) + return 42 + + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) + + N = 8 + ready = threading.Barrier(N + 1) + + def canceller() -> None: + ready.wait() + promise.cancel() + + threads = [threading.Thread(target=canceller, daemon=True) for _ in range(N)] + for t in threads: + t.start() + + ready.wait() + + try: + value = await promise + saw_value = True + except BaseException: # noqa: BLE001 + value = None + saw_value = False + + for t in threads: + t.join(timeout=5) + assert not t.is_alive() + + assert promise.done(), f"promise did not reach a terminal state: {promise!r}" + + if saw_value: + # We got the natural value back. The Promise's terminal + # state must reflect that and ``result()``/``exception()`` + # must agree. + assert not promise.cancelled(), f"await returned value={value} but promise.cancelled()=True: {promise!r}" + assert promise.result() == value + assert promise.exception() is None + else: + # We got an exception. It must be a CancelledError and + # the Promise must be in the cancelled state. + assert promise.cancelled(), f"await raised an exception but promise.cancelled()=False: {promise!r}" + # result() / exception() must raise CancelledError, not + # RuntimeError or anything else. + raised_in_result: BaseException | None = None + try: + promise.result() + except BaseException as exc: # noqa: BLE001 + raised_in_result = exc + assert raised_in_result is not None and type(raised_in_result).__name__ == "CancelledError", ( + f"promise.result() raised the wrong kind of exception: {raised_in_result!r}; promise={promise!r}" + ) + + +# ── concurrent threads doing many `await_children` cycles ─────── + + +async def test_await_children_under_continuous_registration_load() -> None: + """ + Heavy load: from inside a parent Promise, many worker threads + register child Promises in tight loops. Concurrently the loop + drains via ``await_children`` repeatedly. After settling, all + children registered up to that point must be done and the parent + must have no unsettled awaitable descendants. + """ + + N_WRITERS = 6 + WRITES = 100 + + @promising.function + async def root() -> int: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() + + thread_errors: list[BaseException] = [] + thread_errors_lock = threading.Lock() + all_promises: list[promising.Promise] = [] + all_promises_lock = threading.Lock() + + async def _quick() -> int: + return 0 + + start = threading.Barrier(N_WRITERS + 1) + + def writer() -> None: + try: + start.wait() + local = [] + for _ in range(WRITES): + local.append( + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + ) + with all_promises_lock: + all_promises.extend(local) + except BaseException as exc: # noqa: BLE001 + with thread_errors_lock: + thread_errors.append(exc) + + writers = [threading.Thread(target=writer, daemon=True) for _ in range(N_WRITERS)] + for w in writers: + w.start() + + start.wait() + + # Drain while workers register. + for _ in range(30): + await promising.await_children(whole_subtree=True) + + for w in writers: + w.join(timeout=10) + assert not w.is_alive() + + await promising.await_children(whole_subtree=True) + + assert not thread_errors, thread_errors + + with all_promises_lock: + for p in all_promises: + assert p.done(), f"child promise not done after final drain: {p!r}" + + unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) + assert unsettled == set(), f"{len(unsettled)} awaitable descendants left after draining" + return 0 + + await root() + + +# ── concurrent get_active_context across threads ──────────────── + + +async def test_get_active_context_returns_correct_context_per_thread() -> None: + """ + Each ``@promising.function(use_thread_pool=True)`` body runs on a + worker thread; ``ctx.run(...)`` propagates the ``ContextVar`` so + ``get_active_context()`` should return *that worker's* parent + Promise inside the sync body. Many sync workers running + concurrently must each see their own parent — never another + worker's. A failure here points to ``ContextVar`` propagation + being broken under contention. + """ + + @promising.function(use_thread_pool=True) + def worker(label: str) -> tuple[str, str]: + expected_namespace_substring = label + active_promise = promising.get_active_promise() + # The active promise's namespace was set explicitly to label. + return (expected_namespace_substring, active_promise.namespace or "") + + @promising.function + async def root() -> None: + N = 32 + promises = [worker(f"label_{i}", namespace=f"label_{i}") for i in range(N)] + results = await asyncio.gather(*promises) + for expected, actual in results: + assert expected in actual, ( + f"worker thread observed wrong active promise: expected namespace to " + f"contain {expected!r}, got {actual!r}" + ) + + await root() + + +# ── massive concurrent close (cascading unregister) ───────────── + + +def test_massive_concurrent_close_grandparent_consistency() -> None: + """ + Three-level tree: grandparent → middle (closed) → many children. + Every child closes itself simultaneously. Cascading unregistration + must converge on an empty grandparent, with the middle context + unregistered exactly once. The race is around the read-then-act in + ``_unregister_from_parent_if_time``. + """ + loop = asyncio.new_event_loop() + try: + for _ in range(20): + grandparent = promising.PromisingContext(loop=loop, parent=None) + middle = promising.PromisingContext(loop=loop, parent=grandparent) + + N = 128 + children = [promising.PromisingContext(loop=loop, parent=middle) for _ in range(N)] + middle.close_context() + + barrier = threading.Barrier(N) + errors: list[BaseException] = [] + errors_lock = threading.Lock() + + def closer(c: promising.PromisingContext): + def _go() -> None: + try: + barrier.wait() + c.close_context() + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + return _go + + threads = [threading.Thread(target=closer(c), daemon=True) for c in children] + for t in threads: + t.start() + for t in threads: + t.join(timeout=10) + assert not t.is_alive() + + assert not errors, errors + + grand_unsettled = grandparent.collect_unsettled_children( + whole_subtree=False, + awaitables_only=False, + ) + assert grand_unsettled == set(), ( + f"grandparent leaked {len(grand_unsettled)} entries after cascading unregister storm" + ) + finally: + loop.close() + + +# ── concurrent Promise.run from multiple threads ──────────────── + + +def test_concurrent_independent_promise_run_invocations_isolate() -> None: + """ + ``PromisingFunction.run()`` creates its own event loop. Multiple + threads should be able to call ``.run()`` on the *same* + ``@promising.function`` concurrently without leaking state across + each other's hierarchies — the ``__active_context`` ContextVar + must be properly per-thread/per-loop. + """ + + @promising.function + async def task(idx: int) -> int: + return idx * 2 + + N = 8 + + results: list[int] = [] + errors: list[BaseException] = [] + lock = threading.Lock() + barrier = threading.Barrier(N) + + def runner(idx: int) -> None: + try: + barrier.wait() + value = task.run(idx) + with lock: + results.append(value) + except BaseException as exc: # noqa: BLE001 + with lock: + errors.append(exc) + + threads = [threading.Thread(target=runner, args=(i,), daemon=True) for i in range(N)] + for t in threads: + t.start() + for t in threads: + t.join(timeout=10) + assert not t.is_alive() + + assert not errors, errors + assert sorted(results) == [i * 2 for i in range(N)], ( + f"results mismatch — possible cross-thread leakage of active context: {sorted(results)}" + ) + + +# ── sync() race with cancel — RuntimeError must not surface ───── + + +async def test_sync_consumers_never_observe_internal_runtime_error() -> None: + """ + Consumers calling ``promise.sync()`` from threads must never see a + ``RuntimeError`` originating from the framework's own state + machine (``Cannot set result on a promise because of its current + state ...``). Only acceptable outcomes are the value or a + ``CancelledError``. + """ + loop = asyncio.get_running_loop() + + for _ in range(50): + + async def coro() -> str: + for _ in range(4): + await asyncio.sleep(0) + return "v" + + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) + + N_CONSUMERS = 8 + N_CANCELLERS = 4 + + outcomes: list[object] = [] + outcomes_lock = threading.Lock() + + start = threading.Barrier(N_CONSUMERS + N_CANCELLERS + 1) + + def consumer() -> None: + start.wait() + try: + outcomes.append(("value", promise.sync(timeout=5))) + except BaseException as exc: # noqa: BLE001 + with outcomes_lock: + outcomes.append(("error", exc)) + + def canceller() -> None: + start.wait() + promise.cancel() + + with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_CANCELLERS) as ex: + consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] + canceller_futs = [loop.run_in_executor(ex, canceller) for _ in range(N_CANCELLERS)] + start.wait() + await asyncio.gather(*consumer_futs, *canceller_futs) + + for kind, payload in outcomes: + if kind == "value": + assert payload == "v", f"unexpected value from sync(): {payload!r}" + else: + # Any exception must be a CancelledError — not RuntimeError, + # PromiseNotDoneError, etc. + assert type(payload).__name__ == "CancelledError", ( + f"sync() raised internal/unexpected error during cancel race: " + f"{type(payload).__module__}.{type(payload).__qualname__}: {payload!r}" + ) diff --git a/tests/race_conditions/test_state_machine_race.py b/tests/race_conditions/test_state_machine_race.py new file mode 100644 index 000000000..83ecc441c --- /dev/null +++ b/tests/race_conditions/test_state_machine_race.py @@ -0,0 +1,260 @@ +""" +Race conditions around ``Promise``'s state machine. + +``Promise._state`` is read and written by: + +- ``_unpack_once`` / ``_unpack_fully`` on the loop thread, via + ``_set_intermediate_promise`` / ``_set_result`` / ``_set_exception`` +- ``Promise.cancel()`` and ``_synthesize_cancellation`` from *any* + thread — public API mirrors ``Future.cancel()`` which is invokable + off-loop +- ``done()`` / ``cancelled()`` / ``result()`` consumers from any thread + +There is no lock guarding the transitions. ``_set_exception`` and +``_set_result`` perform a multi-step (read state → decide terminal +state → write state) sequence, so two threads can both observe a +``_PENDING`` state and both race to write a terminal state. The framework +either: + +- raises an internal ``RuntimeError`` ("Cannot set result on a + promise because of its current state ...") which is then swallowed + into the Promise via ``_force_internal_error_finish`` +- or silently transitions to a wrong terminal state (e.g. ``cancelled`` + after a successful ``_set_result``). + +These tests aim to surface those races. +""" + +import asyncio +import threading +from typing import Any + +import pytest + +import promising + +# ── helpers ───────────────────────────────────────────────────── + + +def _run_threads_with_barrier(targets: list, *, join_timeout: float = 10.0) -> list[BaseException]: + errors: list[BaseException] = [] + errors_lock = threading.Lock() + barrier = threading.Barrier(len(targets)) + + def _wrap(fn): + def _run() -> None: + try: + barrier.wait() + fn() + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + return _run + + threads = [threading.Thread(target=_wrap(fn), daemon=True) for fn in targets] + for t in threads: + t.start() + for t in threads: + t.join(timeout=join_timeout) + assert not t.is_alive(), "Worker thread did not finish in time" + return errors + + +# ── concurrent cancel ────────────────────────────────────────── + + +async def test_many_threads_cancelling_same_pending_promise_yield_consistent_state() -> None: + """ + Many worker threads race to cancel the same ``start_soon=False`` + Promise (no task scheduled yet → cancel goes through + ``_synthesize_cancellation`` which writes ``_state``). All workers + should agree on a single cancelled terminal state. + """ + for _ in range(50): + + async def coro() -> int: + return 1 + + promise = promising.wrap_awaitable(coro(), parent=None, start_soon=False) + + N = 32 + + def _cancel() -> None: + promise.cancel("from worker") + + errors = _run_threads_with_barrier([_cancel] * N) + assert not errors, errors + + assert promise.done(), "Promise must be done after concurrent cancellation" + assert promise.cancelled(), "Promise must be in the cancelled state" + + # exception() on a cancelled Promise re-raises the stored + # CancelledError — it must not raise RuntimeError or any other + # internal error. + with pytest.raises(asyncio.CancelledError): + promise.exception() + + +async def test_cancel_racing_with_natural_completion_keeps_state_consistent() -> None: + """ + Spawn a Promise that completes after a quick ``asyncio.sleep(0)``. + Right as the loop tries to call ``_set_result``, fire many cancels + from worker threads. The Promise must end up in *exactly one* + terminal state — either successfully ``finished`` with the value, + or ``cancelled``. It must never end up with a hybrid internal-error + state or surface ``RuntimeError`` to the caller. + """ + for _ in range(80): + loop = asyncio.get_running_loop() + + async def coro() -> int: + await asyncio.sleep(0) + return 7 + + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) + + N = 16 + started = threading.Barrier(N + 1) + + def _cancel() -> None: + started.wait() + try: + promise.cancel() + except BaseException: # noqa: BLE001 + # Some race losers can raise — that itself is a bug + # worth surfacing. + raise + + threads = [threading.Thread(target=_cancel, daemon=True) for _ in range(N)] + for t in threads: + t.start() + + # Race the workers against the loop step that resolves the task. + started.wait() + try: + value = await promise + outcome: dict[str, Any] = {"finished": value} + except asyncio.CancelledError: + outcome = {"cancelled": True} + + for t in threads: + t.join(timeout=5) + assert not t.is_alive() + + assert promise.done(), "Promise must reach a terminal state" + if "finished" in outcome: + assert not promise.cancelled() + assert promise.result() == 7 + assert promise.exception() is None + else: + assert promise.cancelled() + with pytest.raises(asyncio.CancelledError): + promise.result() + + +async def test_concurrent_cancel_with_full_unpacking_promise_chain() -> None: + """ + When a Promise has both a single-unpacking and a full-unpacking + task scheduled, ``cancel()`` calls ``cancel`` on both. Done callbacks + on those tasks then invoke ``_synthesize_cancellation`` via + ``_unpacking_task_done_callback``. Multiple worker threads cancelling + the same Promise simultaneously can cause both done-callbacks to + fire ``_synthesize_cancellation`` against an already-transitioning + state, triggering an internal ``RuntimeError``. + """ + for _ in range(40): + loop = asyncio.get_running_loop() + + async def inner_coro() -> str: + await asyncio.sleep(0.01) + return "ok" + + @promising.function + async def outer_func() -> promising.Promise[str]: + return promising.wrap_awaitable(inner_coro(), loop=loop, start_soon=True) + + promise = outer_func() + + # Schedule both unpacking paths + async def _kick() -> None: + await promise.unpack_once() # schedules single-unpacking + + kick_task = loop.create_task(_kick()) + try: + # Brief yield so the single unpacking task is in flight + await asyncio.sleep(0) + + def _cancel() -> None: + promise.cancel() + + errors = _run_threads_with_barrier([_cancel] * 8) + assert not errors, errors + + try: + await promise + except asyncio.CancelledError: + pass + except BaseException: + # The promise might also have completed before the + # cancel landed. + pass + finally: + kick_task.cancel() + try: + await kick_task + except (asyncio.CancelledError, BaseException): + pass + + assert promise.done(), f"Promise not done after cancel race: {promise!r}" + + # exception() must not raise a non-CancelledError exception. + if promise.cancelled(): + with pytest.raises(asyncio.CancelledError): + promise.exception() + else: + exc = promise.exception() + # Any *internal* error from racing _set_* paths would show + # up here. + assert exc is None or isinstance(exc, asyncio.CancelledError), ( + f"unexpected exception from racing state machine: {exc!r}" + ) + + +async def test_concurrent_set_state_does_not_double_unregister_from_parent() -> None: + """ + ``_set_state`` calls ``close_context()`` which calls + ``_unregister_from_parent_if_time`` which checks ``self.done()`` + and ``not self._unsettled_children`` and then calls + ``parent._unregister_children(self)``. Two threads transitioning + state in lockstep can both call into parent's unregister path, + triggering ``set.difference_update`` twice and (if cascading) racing + further up. + """ + loop = asyncio.get_running_loop() + + parent = promising.PromisingContext(loop=loop, parent=None) + + for _ in range(50): + + async def coro() -> int: + return 5 + + promise = promising.wrap_awaitable(coro(), parent=parent, start_soon=False) + + # Two threads call cancel — only one can actually set the + # terminal state, but with the race both think they can. + N = 4 + + def _cancel() -> None: + promise.cancel() + + errors = _run_threads_with_barrier([_cancel] * N) + assert not errors, errors + + assert promise.done() + assert promise.cancelled() + + # After all the promises cancelled, none should remain in parent. + remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert remaining == set(), f"parent leaked {len(remaining)} children after racing cancellations" diff --git a/tests/race_conditions/test_unsettled_children_set_race.py b/tests/race_conditions/test_unsettled_children_set_race.py new file mode 100644 index 000000000..b96e09b79 --- /dev/null +++ b/tests/race_conditions/test_unsettled_children_set_race.py @@ -0,0 +1,496 @@ +""" +Race conditions around ``PromisingContext._unsettled_children``. + +The set is mutated concurrently by: + +- worker threads that create new child contexts/promises (each new + child registers itself via ``parent._register_children(self)``) +- worker threads that close child contexts (which call + ``parent._unregister_children(self)``) +- consumers iterating the set via ``collect_unsettled_children`` / + ``await_children`` + +None of these paths take a lock, so concurrent access can: + +- raise ``RuntimeError: Set changed size during iteration`` +- silently lose children (registration overwritten by an unregistration + that happened in a stale snapshot) +- register children onto a parent that has *just* been closed, breaking + the ``closed() → no new children`` invariant +- corrupt the parent's idea of who its children are after a cascading + ``_unregister_from_parent_if_time`` re-entrance + +Every test below stresses one of these surfaces. While the framework is +unprotected, they are expected to fail; they exist so locks added later +can be verified end-to-end. +""" + +import asyncio +import threading +from concurrent.futures import ThreadPoolExecutor + +import pytest + +import promising + +# ── helpers ───────────────────────────────────────────────────── + + +def _make_dedicated_loop() -> asyncio.AbstractEventLoop: + """Brand-new event loop owned by the test, never started.""" + return asyncio.new_event_loop() + + +def _run_workers(targets: list, *, join_timeout: float = 15.0) -> list[BaseException]: + """ + Run a list of nullary callables in parallel threads behind a single + ``threading.Barrier`` so they all fire as close to simultaneously as + possible. Returns any exceptions raised. + """ + errors: list[BaseException] = [] + errors_lock = threading.Lock() + barrier = threading.Barrier(len(targets)) + + def _wrap(fn): + def _run(): + try: + barrier.wait() + fn() + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + return _run + + threads = [threading.Thread(target=_wrap(fn), daemon=True) for fn in targets] + for t in threads: + t.start() + for t in threads: + t.join(timeout=join_timeout) + assert not t.is_alive(), "Worker thread did not finish in time" + + return errors + + +# ── concurrent registration ────────────────────────────────────── + + +def test_concurrent_child_registration_keeps_all_children() -> None: + """ + Many worker threads simultaneously construct child ``PromisingContext`` + instances under one parent. After all workers finish, the parent's + ``_unsettled_children`` must contain *every* child that was created. + With unsynchronized ``set.update`` calls from many threads, + children can be lost. + """ + loop = _make_dedicated_loop() + try: + for _ in range(20): + parent = promising.PromisingContext(loop=loop, parent=None) + + N_THREADS = 64 + CHILDREN_PER_THREAD = 50 + + created: list[promising.PromisingContext] = [] + created_lock = threading.Lock() + + def worker() -> None: + local: list[promising.PromisingContext] = [] + for _ in range(CHILDREN_PER_THREAD): + child = promising.PromisingContext(loop=loop, parent=parent) + local.append(child) + with created_lock: + created.extend(local) + + errors = _run_workers([worker] * N_THREADS) + assert not errors, errors + + expected = set(created) + actual = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + missing = expected - actual + extra = actual - expected + + assert not missing, f"{len(missing)} children were lost from the parent's _unsettled_children set" + assert not extra, f"unexpected children appeared in the parent's _unsettled_children set: {extra!r}" + finally: + loop.close() + + +def test_collect_unsettled_children_during_concurrent_registration_does_not_raise() -> None: + """ + A reader thread iterates ``collect_unsettled_children`` in a tight + loop while writer threads keep registering new child contexts. With + no lock around the set, the reader's internal ``list(set)`` snapshot + races with mutations and can raise + ``RuntimeError: Set changed size during iteration``. + """ + loop = _make_dedicated_loop() + try: + parent = promising.PromisingContext(loop=loop, parent=None) + + N_WRITERS = 16 + WRITES_PER_WRITER = 200 + + writers_done = threading.Event() + writers_finished_count = 0 + writers_finished_lock = threading.Lock() + + def writer() -> None: + nonlocal writers_finished_count + try: + for _ in range(WRITES_PER_WRITER): + promising.PromisingContext(loop=loop, parent=parent) + finally: + with writers_finished_lock: + writers_finished_count += 1 + if writers_finished_count == N_WRITERS: + writers_done.set() + + def reader() -> None: + while not writers_done.is_set(): + parent.collect_unsettled_children(whole_subtree=True, awaitables_only=False) + + targets = [reader] + [writer] * N_WRITERS + errors = _run_workers(targets) + assert not errors, errors + finally: + loop.close() + + +# ── concurrent unregistration ──────────────────────────────────── + + +def test_concurrent_child_close_keeps_consistent_set() -> None: + """ + Many children close themselves concurrently (each call invokes + ``parent._unregister_children(self)``). After every worker has + finished, the parent must have an *empty* ``_unsettled_children`` + set — no leaked entries from races on ``set.difference_update``. + """ + loop = _make_dedicated_loop() + try: + parent = promising.PromisingContext(loop=loop, parent=None) + + N_CHILDREN = 256 + children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_CHILDREN)] + # Sanity check before the race + assert len(parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False)) == N_CHILDREN + + def make_closer(child: promising.PromisingContext): + def _close() -> None: + child.close_context() + + return _close + + errors = _run_workers([make_closer(c) for c in children]) + assert not errors, errors + + remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert remaining == set(), f"parent still holds {len(remaining)} stale child references after concurrent close" + finally: + loop.close() + + +def test_concurrent_registration_and_unregistration_keeps_set_intact() -> None: + """ + Two waves run simultaneously: half the threads register new + contexts under a parent, the other half close pre-existing children. + Reading the set with ``len()`` from a third thread must never blow + up and the final state must equal (pre-existing - closed + + newly-registered). + """ + loop = _make_dedicated_loop() + try: + parent = promising.PromisingContext(loop=loop, parent=None) + + N_PRE_EXISTING = 200 + pre_existing = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_PRE_EXISTING)] + + N_NEW = 200 + new_children: list[promising.PromisingContext] = [] + new_children_lock = threading.Lock() + + def closer(child: promising.PromisingContext): + def _close() -> None: + child.close_context() + + return _close + + def adder() -> None: + local = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_NEW // 50)] + with new_children_lock: + new_children.extend(local) + + targets = [closer(c) for c in pre_existing] + [adder] * 50 + errors = _run_workers(targets) + assert not errors, errors + + remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert remaining == set(new_children), ( + f"after concurrent add/close, parent's set has " + f"{len(remaining ^ set(new_children))} discrepancies " + f"(expected={len(new_children)}, got={len(remaining)})" + ) + finally: + loop.close() + + +# ── register-vs-close race on the *parent* ─────────────────────── + + +def test_register_child_after_parent_closed_must_be_rejected() -> None: + """ + ``_register_children`` reads ``self.closed()`` *before* it + ``self._unsettled_children.update(...)``. If another thread closes + the parent in between, the check passes but the close already + happened and the child is silently added to a closed parent — + breaking the ``closed() → cannot accept children`` contract. + + The framework promises ``ContextAlreadyClosedError`` when adding to + a closed context; in the post-close-but-update-still-happens window + we expect *either* that error, or no leaked child in the closed + parent's set — never both "no error" and "child present". + """ + loop = _make_dedicated_loop() + try: + # Run many short races to widen the window. + for _ in range(5000): + parent = promising.PromisingContext(loop=loop, parent=None) + + start = threading.Barrier(2) + outcome: dict[str, object] = {} + + def add_child() -> None: + start.wait() + try: + child = promising.PromisingContext(loop=loop, parent=parent) + outcome["child"] = child + except promising.ContextAlreadyClosedError as exc: + outcome["error"] = exc + + def close_parent() -> None: + start.wait() + parent.close_context() + + t1 = threading.Thread(target=add_child, daemon=True) + t2 = threading.Thread(target=close_parent, daemon=True) + t1.start() + t2.start() + t1.join(timeout=5) + t2.join(timeout=5) + + assert not t1.is_alive() and not t2.is_alive() + + if "child" in outcome: + child = outcome["child"] + # Invariant: a closed parent must not accept new children. + # If registration silently succeeded against a parent that + # is now closed, the framework's `closed() → no new + # children` contract has been broken by the race. + assert not parent.closed(), ( + "child registration silently succeeded after parent was closed — invariant violated" + ) + in_parent = child in parent.collect_unsettled_children( + whole_subtree=False, + awaitables_only=False, + ) + assert in_parent, ( + "registration silently succeeded but the child is missing from " + "the closed parent's _unsettled_children — torn write" + ) + finally: + loop.close() + + +# ── await_children race ────────────────────────────────────────── + + +async def test_await_children_during_concurrent_thread_registration() -> None: + """ + Inside an ``@promising.function``, a worker thread continuously + constructs child Promises (each ``Promise.__init__`` calls + ``parent._register_children(self)`` on the worker thread). The loop + thread, meanwhile, repeatedly ``await``s ``await_children()`` — + which iterates the same ``_unsettled_children`` set. + + With no lock, the loop-side iteration (``list(set)`` inside + ``collect_unsettled_children``) can raise + ``RuntimeError: Set changed size during iteration``, or + ``await_children`` can return while previously registered children + are still unsettled. + """ + + N_WRITERS = 4 + WRITES = 200 + + @promising.function + async def parent_func() -> None: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() + thread_errors: list[BaseException] = [] + thread_errors_lock = threading.Lock() + + async def _quick() -> int: + return 42 + + start_barrier = threading.Barrier(N_WRITERS + 1) + + def thread_writer() -> None: + try: + start_barrier.wait() + for _ in range(WRITES): + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + except BaseException as exc: # noqa: BLE001 + with thread_errors_lock: + thread_errors.append(exc) + + writers = [threading.Thread(target=thread_writer, daemon=True) for _ in range(N_WRITERS)] + for w in writers: + w.start() + + # Release the writers and immediately race them with await_children. + start_barrier.wait() + + # Drain children many times to keep the race window open. + for _ in range(20): + await promising.await_children(whole_subtree=True) + + for w in writers: + w.join(timeout=10) + assert not w.is_alive() + + # Final drain. + await promising.await_children(whole_subtree=True) + + assert not thread_errors, thread_errors + + unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) + assert unsettled == set(), ( + f"await_children returned with {len(unsettled)} unsettled awaitable children remaining" + ) + + await parent_func() + + +# ── cascading unregister race ──────────────────────────────────── + + +def test_cascading_unregister_keeps_grandparent_set_consistent() -> None: + """ + A grandparent owns a middle-tier parent which owns many children. + All children close simultaneously: each unregistration triggers + ``_unregister_from_parent_if_time`` on the middle context, which + can cascade up to the grandparent. + + The cascading path reads ``_unsettled_children`` membership and + calls ``_unregister_children`` on the grandparent — without locking, + two children finishing in lockstep can both observe an empty middle + set and both try to unregister the middle from the grandparent. + """ + loop = _make_dedicated_loop() + try: + for _ in range(50): + grandparent = promising.PromisingContext(loop=loop, parent=None) + middle = promising.PromisingContext(loop=loop, parent=grandparent) + + N = 64 + children = [promising.PromisingContext(loop=loop, parent=middle) for _ in range(N)] + # Close middle *after* attaching children, so it stays in + # grandparent's set until all of its children drain. + middle.close_context() + + def make_closer(c: promising.PromisingContext): + def _close() -> None: + c.close_context() + + return _close + + errors = _run_workers([make_closer(c) for c in children]) + assert not errors, errors + + grand_children = grandparent.collect_unsettled_children( + whole_subtree=False, + awaitables_only=False, + ) + # After every middle-child has closed, middle should have + # cascaded out of grandparent's set exactly once. + assert middle not in grand_children, ( + "middle context still appears in grandparent's set even though all of its children are closed" + ) + finally: + loop.close() + + +# ── load-bearing scenario: real Promises under a parent Promise ── + + +async def test_concurrent_promise_creation_from_threads_registers_all() -> None: + """ + Many worker threads inside the same parent Promise create child + Promises concurrently. Every newly-created child must end up in + the parent's ``_unsettled_children`` set; none must be lost. + """ + + @promising.function + async def parent_func() -> int: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() + + N_THREADS = 32 + CHILDREN_PER_THREAD = 20 + + created: list[promising.Promise] = [] + created_lock = threading.Lock() + + async def _quick() -> int: + return 0 + + def worker() -> None: + local = [] + for _ in range(CHILDREN_PER_THREAD): + child = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + local.append(child) + with created_lock: + created.extend(local) + + with ThreadPoolExecutor(max_workers=N_THREADS) as ex: + await asyncio.gather(*[loop.run_in_executor(ex, worker) for _ in range(N_THREADS)]) + + expected = set(created) + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = expected - actual + assert not missing, f"{len(missing)} child Promises lost from the parent's set" + + # Clean up the unused coroutines so asyncio doesn't warn. + for child in created: + child.cancel() + + return len(created) + + n = await parent_func() + assert n == 32 * 20 + + +# ── sanity (would-pass) test, included for self-verification ───── + + +@pytest.mark.skip(reason="Sanity reference; concurrency is fine when the set is touched from one thread.") +def test_single_thread_registration_keeps_all_children_sanity() -> None: + loop = _make_dedicated_loop() + try: + parent = promising.PromisingContext(loop=loop, parent=None) + children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(1000)] + actual = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert actual == set(children) + finally: + loop.close() From fce3109047672ec2526fe92bae3f0a827cf0a1e7 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 14:37:37 +0300 Subject: [PATCH 10/20] race condition tests --- .../test_concurrent_consumption_race.py | 22 +++++++++++++++++++ .../test_invariants_under_load.py | 13 +++++++++++ .../test_unsettled_children_set_race.py | 19 ++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/tests/race_conditions/test_concurrent_consumption_race.py b/tests/race_conditions/test_concurrent_consumption_race.py index 63446dcd3..0447c2357 100644 --- a/tests/race_conditions/test_concurrent_consumption_race.py +++ b/tests/race_conditions/test_concurrent_consumption_race.py @@ -28,6 +28,20 @@ async def test_many_threads_calling_sync_on_same_promise_consistent() -> None: """ N worker threads simultaneously call ``promise.sync()`` on the same Promise. All must return the same value; none must hang or raise. + + NOTE [race-injection / 2026-05-17]: this test could not be made to + fail even by deliberately introducing race-prone breakage in + ``promising/`` (lost children, non-atomic state writes, dropped + ContextVar copy, etc.). All ``.sync()`` callers dispatch their + awaiting onto the Promise's single event loop via + ``run_coroutine_threadsafe``, so the actual ``await self`` runs + serialized on the loop thread; only one ``_full_unpacking_task`` is + ever scheduled, every coroutine yields from the same task, and the + final ``_result`` is written once by the task itself. There is no + race surface to inject into without changing this serialization. + Keep as a regression guard against future refactors that try to + short-circuit the loop dispatch (e.g. a "fast path" returning a + cached ``_result`` directly from the caller thread). """ loop = asyncio.get_running_loop() @@ -66,6 +80,14 @@ async def test_many_threads_calling_unpack_once_sync_on_same_promise_consistent( but using ``unpack_once_sync``. All threads should observe the same one-level-unpacking outcome (here: a concrete value, since the coroutine does not return a Promise). + + NOTE [race-injection / 2026-05-17]: same finding as the ``.sync()`` + sibling above — could not be broken by deliberate framework + sabotage. ``unpack_once_sync`` also goes through + ``run_coroutine_threadsafe`` and the single + ``_single_unpacking_task``. Only one task ever drives the unpack, + all callers share its result. Keep as regression guard against + future refactors that bypass the loop dispatch. """ loop = asyncio.get_running_loop() diff --git a/tests/race_conditions/test_invariants_under_load.py b/tests/race_conditions/test_invariants_under_load.py index c2656e704..2d8dfda7b 100644 --- a/tests/race_conditions/test_invariants_under_load.py +++ b/tests/race_conditions/test_invariants_under_load.py @@ -316,6 +316,19 @@ def test_concurrent_independent_promise_run_invocations_isolate() -> None: ``@promising.function`` concurrently without leaking state across each other's hierarchies — the ``__active_context`` ContextVar must be properly per-thread/per-loop. + + NOTE [race-injection / 2026-05-17]: this test could not be made to + fail by injecting the obvious race bugs into ``promising/`` + (non-atomic set writes, dropped ``copy_context``, dropped state + guards, etc.). Each ``.run()`` allocates its own ``asyncio`` event + loop in its own thread, and ``PromisingContext.__active_context`` + is a ``ContextVar`` whose value is per-thread / per-asyncio-task — + so the threads never share an active-context slot in the first + place. Breaking it would require an architectural change (e.g. + replacing the ``ContextVar`` with a module-level global). Keep as + regression guard against exactly that kind of refactor: someone + "caching" the active promise in a global for perf would cause + cross-thread leakage and this test would catch it. """ @promising.function diff --git a/tests/race_conditions/test_unsettled_children_set_race.py b/tests/race_conditions/test_unsettled_children_set_race.py index b96e09b79..d4c3a0c97 100644 --- a/tests/race_conditions/test_unsettled_children_set_race.py +++ b/tests/race_conditions/test_unsettled_children_set_race.py @@ -123,6 +123,25 @@ def test_collect_unsettled_children_during_concurrent_registration_does_not_rais no lock around the set, the reader's internal ``list(set)`` snapshot races with mutations and can raise ``RuntimeError: Set changed size during iteration``. + + NOTE [race-injection / 2026-05-17]: could not be made to fail + simultaneously with the lost-children tests in this file. The two + bug surfaces are mutually exclusive: + + - "lost children" requires non-atomic read-modify-write on + ``_unsettled_children`` (i.e. rebinding to a fresh set), which + means the reader's iteration target is a *different object* than + the writer mutates — no in-place mutation, no + ``RuntimeError: set changed size during iteration``. + - "set changed size during iteration" requires in-place mutation + (``.add`` / ``.discard``) on the live set, which is atomic per + call in CPython and therefore would not lose children. + + Pick one bug pattern, surface the other. Keep this test as + forward-compat for nogil Python (3.13t) where set-iteration + atomicity weakens, and as a regression guard against refactors + that swap the atomic ``set.update`` / ``list(set)`` C calls for + Python-level loops over the live set. """ loop = _make_dedicated_loop() try: From 039225beef26d6b1acb1e2e63c91169a3a36eb55 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 14:45:24 +0300 Subject: [PATCH 11/20] race condition tests --- tests/race_conditions/__init__.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/race_conditions/__init__.py b/tests/race_conditions/__init__.py index e69de29bb..71c220121 100644 --- a/tests/race_conditions/__init__.py +++ b/tests/race_conditions/__init__.py @@ -0,0 +1 @@ +# TODO [TESTS] These tests haven't been reviewed by a human at all yet ! From 02834b6170e3d5568f6eddb899c1d3d809613954 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 15:58:01 +0300 Subject: [PATCH 12/20] experimental commit: bumped numbers of repetitions in race condition tests --- .../test_concurrent_consumption_race.py | 246 +++++++-------- .../test_context_lifecycle_race.py | 94 +++--- .../test_invariants_under_load.py | 173 ++++++----- .../test_state_machine_race.py | 8 +- .../test_unsettled_children_set_race.py | 289 +++++++++--------- 5 files changed, 417 insertions(+), 393 deletions(-) diff --git a/tests/race_conditions/test_concurrent_consumption_race.py b/tests/race_conditions/test_concurrent_consumption_race.py index 0447c2357..5aa179d08 100644 --- a/tests/race_conditions/test_concurrent_consumption_race.py +++ b/tests/race_conditions/test_concurrent_consumption_race.py @@ -45,7 +45,7 @@ async def test_many_threads_calling_sync_on_same_promise_consistent() -> None: """ loop = asyncio.get_running_loop() - for _ in range(10): + for _ in range(100): async def coro() -> int: await asyncio.sleep(0.01) @@ -91,31 +91,33 @@ async def test_many_threads_calling_unpack_once_sync_on_same_promise_consistent( """ loop = asyncio.get_running_loop() - async def coro() -> str: - await asyncio.sleep(0.01) - return "ok" + for _ in range(100): - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) + async def coro() -> str: + await asyncio.sleep(0.01) + return "ok" - N = 32 - results: list[object] = [] - errors: list[BaseException] = [] - lock = threading.Lock() + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) - def consumer() -> None: - try: - value = promise.unpack_once_sync(timeout=5) - with lock: - results.append(value) - except BaseException as exc: # noqa: BLE001 - with lock: - errors.append(exc) + N = 32 + results: list[object] = [] + errors: list[BaseException] = [] + lock = threading.Lock() - with ThreadPoolExecutor(max_workers=N) as ex: - await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) + def consumer() -> None: + try: + value = promise.unpack_once_sync(timeout=5) + with lock: + results.append(value) + except BaseException as exc: # noqa: BLE001 + with lock: + errors.append(exc) - assert not errors, errors - assert results == ["ok"] * N + with ThreadPoolExecutor(max_workers=N) as ex: + await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) + + assert not errors, errors + assert results == ["ok"] * N # ── sync() race with cancel() ────────────────────────────────── @@ -131,7 +133,7 @@ async def test_sync_consumer_thread_observes_clean_terminal_after_cancel() -> No """ loop = asyncio.get_running_loop() - for _ in range(40): + for _ in range(500): cancel_event = asyncio.Event() async def coro() -> int: @@ -198,57 +200,59 @@ async def test_await_children_sync_during_thread_registration_does_not_raise() - """ loop = asyncio.get_running_loop() - @promising.function - async def root() -> None: - active = promising.get_active_promise() - stop = threading.Event() - thread_errors: list[BaseException] = [] - registered_children: list[promising.Promise] = [] - registered_lock = threading.Lock() + for _ in range(30): - async def _quick() -> int: - return 0 + @promising.function + async def root() -> None: + active = promising.get_active_promise() + stop = threading.Event() + thread_errors: list[BaseException] = [] + registered_children: list[promising.Promise] = [] + registered_lock = threading.Lock() + + async def _quick() -> int: + return 0 + + def writer() -> None: + try: + while not stop.is_set(): + p = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + with registered_lock: + registered_children.append(p) + except BaseException as exc: # noqa: BLE001 + thread_errors.append(exc) + + @promising.function(use_thread_pool=True) + def sync_waiter() -> str: + # Each iteration triggers a fresh collect_unsettled_children + # via await_children_sync. + for _ in range(20): + promising.await_children_sync(whole_subtree=True) + return "drained" - def writer() -> None: - try: - while not stop.is_set(): - p = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - with registered_lock: - registered_children.append(p) - except BaseException as exc: # noqa: BLE001 - thread_errors.append(exc) - - @promising.function(use_thread_pool=True) - def sync_waiter() -> str: - # Each iteration triggers a fresh collect_unsettled_children - # via await_children_sync. - for _ in range(20): - promising.await_children_sync(whole_subtree=True) - return "drained" - - writers = [threading.Thread(target=writer, daemon=True) for _ in range(4)] - for w in writers: - w.start() - try: - waiter_promise = sync_waiter() + writers = [threading.Thread(target=writer, daemon=True) for _ in range(4)] + for w in writers: + w.start() try: - await waiter_promise + waiter_promise = sync_waiter() + try: + await waiter_promise + finally: + stop.set() + for w in writers: + w.join(timeout=5) finally: - stop.set() - for w in writers: - w.join(timeout=5) - finally: - for c in registered_children: - c.cancel() + for c in registered_children: + c.cancel() - assert not thread_errors, thread_errors + assert not thread_errors, thread_errors - await root() + await root() # ── concurrent .sync() consumption *and* registration ─────────── @@ -263,67 +267,69 @@ async def test_concurrent_sync_consumers_and_child_registrations() -> None: """ loop = asyncio.get_running_loop() - @promising.function - async def root() -> int: - active = promising.get_active_promise() + for _ in range(30): @promising.function - async def published() -> int: - await asyncio.sleep(0.01) - return 99 - - target = published() - - N_CONSUMERS = 16 - N_REGISTRARS = 16 - errors: list[BaseException] = [] - errors_lock = threading.Lock() - consumer_results: list[int] = [] - new_children: list[promising.Promise] = [] - - async def _quick() -> int: - return 0 - - def consumer() -> None: - try: - consumer_results.append(target.sync(timeout=5)) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - def registrar() -> None: - try: - for _ in range(20): - new_children.append( - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, + async def root() -> int: + active = promising.get_active_promise() + + @promising.function + async def published() -> int: + await asyncio.sleep(0.01) + return 99 + + target = published() + + N_CONSUMERS = 16 + N_REGISTRARS = 16 + errors: list[BaseException] = [] + errors_lock = threading.Lock() + consumer_results: list[int] = [] + new_children: list[promising.Promise] = [] + + async def _quick() -> int: + return 0 + + def consumer() -> None: + try: + consumer_results.append(target.sync(timeout=5)) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + def registrar() -> None: + try: + for _ in range(20): + new_children.append( + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) ) - ) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) - with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: - consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] - registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] - await asyncio.gather(*consumer_futs, *registrar_futs) + with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: + consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] + registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] + await asyncio.gather(*consumer_futs, *registrar_futs) - assert not errors, errors - assert consumer_results == [99] * N_CONSUMERS + assert not errors, errors + assert consumer_results == [99] * N_CONSUMERS - # All registrars' children must be tracked. - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = set(new_children) - actual - assert not missing, f"{len(missing)} children were lost from the parent's set" + # All registrars' children must be tracked. + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = set(new_children) - actual + assert not missing, f"{len(missing)} children were lost from the parent's set" - for c in new_children: - c.cancel() - return 0 + for c in new_children: + c.cancel() + return 0 - await root() + await root() # ── sanity ────────────────────────────────────────────────────── diff --git a/tests/race_conditions/test_context_lifecycle_race.py b/tests/race_conditions/test_context_lifecycle_race.py index 3ef0ded8e..86f2cc493 100644 --- a/tests/race_conditions/test_context_lifecycle_race.py +++ b/tests/race_conditions/test_context_lifecycle_race.py @@ -44,7 +44,7 @@ def test_concurrent_enter_same_context_only_one_succeeds() -> None: loop = _make_dedicated_loop() try: N = 8 - for _ in range(500): + for _ in range(3000): ctx = promising.PromisingContext(loop=loop, parent=None) barrier = threading.Barrier(N) succeeded: list[str] = [] @@ -215,57 +215,61 @@ async def test_promise_context_open_races_with_external_child_registration() -> """ loop = asyncio.get_running_loop() - @promising.function - async def parent_promise() -> int: - active = promising.get_active_promise() - errors: list[BaseException] = [] - children: list[promising.Promise] = [] - children_lock = threading.Lock() - stop = threading.Event() + for _ in range(30): - async def _quick() -> int: - return 1 + @promising.function + async def parent_promise() -> int: + active = promising.get_active_promise() + errors: list[BaseException] = [] + children: list[promising.Promise] = [] + children_lock = threading.Lock() + stop = threading.Event() - def worker() -> None: - try: - while not stop.is_set(): - p = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - with children_lock: - children.append(p) - except BaseException as exc: # noqa: BLE001 - errors.append(exc) - - writers = [threading.Thread(target=worker, daemon=True) for _ in range(8)] - for w in writers: - w.start() - try: - for _ in range(200): - await asyncio.sleep(0) - finally: - stop.set() + async def _quick() -> int: + return 1 + + def worker() -> None: + try: + while not stop.is_set(): + p = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + with children_lock: + children.append(p) + except BaseException as exc: # noqa: BLE001 + errors.append(exc) + + writers = [threading.Thread(target=worker, daemon=True) for _ in range(8)] for w in writers: - w.join(timeout=5) + w.start() + try: + for _ in range(200): + await asyncio.sleep(0) + finally: + stop.set() + for w in writers: + w.join(timeout=5) - assert not errors, errors + assert not errors, errors - # Every child registered while the Promise was running must - # be tracked. - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = set(children) - actual - assert not missing, f"{len(missing)} children were lost from active Promise's set during the lifecycle race" + # Every child registered while the Promise was running must + # be tracked. + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = set(children) - actual + assert not missing, ( + f"{len(missing)} children were lost from active Promise's set during the lifecycle race" + ) - # Clean up so asyncio doesn't warn about un-awaited coros. - for c in children: - c.cancel() + # Clean up so asyncio doesn't warn about un-awaited coros. + for c in children: + c.cancel() - return 0 + return 0 - await parent_promise() + await parent_promise() # ── close_context idempotency under threads ───────────────────── @@ -286,7 +290,7 @@ def test_close_context_concurrent_calls_must_be_idempotent() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(100): + for _ in range(2000): parent = promising.PromisingContext(loop=loop, parent=None) child = promising.PromisingContext(loop=loop, parent=parent) diff --git a/tests/race_conditions/test_invariants_under_load.py b/tests/race_conditions/test_invariants_under_load.py index 2d8dfda7b..8f3196942 100644 --- a/tests/race_conditions/test_invariants_under_load.py +++ b/tests/race_conditions/test_invariants_under_load.py @@ -52,10 +52,11 @@ async def root() -> int: return sum(await asyncio.gather(*parents)) expected = sum(p * 1000 * N_CHILDREN_PER_PARENT + sum(range(N_CHILDREN_PER_PARENT)) for p in range(N_PARENTS)) - actual = await root() - assert actual == expected, ( - f"lost children or duplicated work in concurrent hierarchy build: {actual} vs {expected}" - ) + for _ in range(30): + actual = await root() + assert actual == expected, ( + f"lost children or duplicated work in concurrent hierarchy build: {actual} vs {expected}" + ) # ── many cancels racing with a fast natural completion ────────── @@ -79,7 +80,7 @@ async def test_cancel_race_state_consistency_high_iterations() -> None: """ loop = asyncio.get_running_loop() - for _ in range(300): + for _ in range(2000): async def coro() -> int: for _ in range(3): @@ -152,67 +153,69 @@ async def test_await_children_under_continuous_registration_load() -> None: N_WRITERS = 6 WRITES = 100 - @promising.function - async def root() -> int: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() - - thread_errors: list[BaseException] = [] - thread_errors_lock = threading.Lock() - all_promises: list[promising.Promise] = [] - all_promises_lock = threading.Lock() - - async def _quick() -> int: - return 0 - - start = threading.Barrier(N_WRITERS + 1) - - def writer() -> None: - try: - start.wait() - local = [] - for _ in range(WRITES): - local.append( - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, + for _ in range(20): + + @promising.function + async def root() -> int: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() + + thread_errors: list[BaseException] = [] + thread_errors_lock = threading.Lock() + all_promises: list[promising.Promise] = [] + all_promises_lock = threading.Lock() + + async def _quick() -> int: + return 0 + + start = threading.Barrier(N_WRITERS + 1) + + def writer() -> None: + try: + start.wait() + local = [] + for _ in range(WRITES): + local.append( + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) ) - ) - with all_promises_lock: - all_promises.extend(local) - except BaseException as exc: # noqa: BLE001 - with thread_errors_lock: - thread_errors.append(exc) + with all_promises_lock: + all_promises.extend(local) + except BaseException as exc: # noqa: BLE001 + with thread_errors_lock: + thread_errors.append(exc) - writers = [threading.Thread(target=writer, daemon=True) for _ in range(N_WRITERS)] - for w in writers: - w.start() + writers = [threading.Thread(target=writer, daemon=True) for _ in range(N_WRITERS)] + for w in writers: + w.start() - start.wait() + start.wait() - # Drain while workers register. - for _ in range(30): - await promising.await_children(whole_subtree=True) + # Drain while workers register. + for _ in range(30): + await promising.await_children(whole_subtree=True) - for w in writers: - w.join(timeout=10) - assert not w.is_alive() + for w in writers: + w.join(timeout=10) + assert not w.is_alive() - await promising.await_children(whole_subtree=True) + await promising.await_children(whole_subtree=True) - assert not thread_errors, thread_errors + assert not thread_errors, thread_errors - with all_promises_lock: - for p in all_promises: - assert p.done(), f"child promise not done after final drain: {p!r}" + with all_promises_lock: + for p in all_promises: + assert p.done(), f"child promise not done after final drain: {p!r}" - unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) - assert unsettled == set(), f"{len(unsettled)} awaitable descendants left after draining" - return 0 + unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) + assert unsettled == set(), f"{len(unsettled)} awaitable descendants left after draining" + return 0 - await root() + await root() # ── concurrent get_active_context across threads ──────────────── @@ -247,7 +250,8 @@ async def root() -> None: f"contain {expected!r}, got {actual!r}" ) - await root() + for _ in range(100): + await root() # ── massive concurrent close (cascading unregister) ───────────── @@ -263,7 +267,7 @@ def test_massive_concurrent_close_grandparent_consistency() -> None: """ loop = asyncio.new_event_loop() try: - for _ in range(20): + for _ in range(100): grandparent = promising.PromisingContext(loop=loop, parent=None) middle = promising.PromisingContext(loop=loop, parent=grandparent) @@ -337,32 +341,33 @@ async def task(idx: int) -> int: N = 8 - results: list[int] = [] - errors: list[BaseException] = [] - lock = threading.Lock() - barrier = threading.Barrier(N) + for _ in range(50): + results: list[int] = [] + errors: list[BaseException] = [] + lock = threading.Lock() + barrier = threading.Barrier(N) - def runner(idx: int) -> None: - try: - barrier.wait() - value = task.run(idx) - with lock: - results.append(value) - except BaseException as exc: # noqa: BLE001 - with lock: - errors.append(exc) - - threads = [threading.Thread(target=runner, args=(i,), daemon=True) for i in range(N)] - for t in threads: - t.start() - for t in threads: - t.join(timeout=10) - assert not t.is_alive() - - assert not errors, errors - assert sorted(results) == [i * 2 for i in range(N)], ( - f"results mismatch — possible cross-thread leakage of active context: {sorted(results)}" - ) + def runner(idx: int) -> None: + try: + barrier.wait() + value = task.run(idx) + with lock: + results.append(value) + except BaseException as exc: # noqa: BLE001 + with lock: + errors.append(exc) + + threads = [threading.Thread(target=runner, args=(i,), daemon=True) for i in range(N)] + for t in threads: + t.start() + for t in threads: + t.join(timeout=10) + assert not t.is_alive() + + assert not errors, errors + assert sorted(results) == [i * 2 for i in range(N)], ( + f"results mismatch — possible cross-thread leakage of active context: {sorted(results)}" + ) # ── sync() race with cancel — RuntimeError must not surface ───── @@ -378,7 +383,7 @@ async def test_sync_consumers_never_observe_internal_runtime_error() -> None: """ loop = asyncio.get_running_loop() - for _ in range(50): + for _ in range(500): async def coro() -> str: for _ in range(4): diff --git a/tests/race_conditions/test_state_machine_race.py b/tests/race_conditions/test_state_machine_race.py index 83ecc441c..ebaa55cda 100644 --- a/tests/race_conditions/test_state_machine_race.py +++ b/tests/race_conditions/test_state_machine_race.py @@ -71,7 +71,7 @@ async def test_many_threads_cancelling_same_pending_promise_yield_consistent_sta ``_synthesize_cancellation`` which writes ``_state``). All workers should agree on a single cancelled terminal state. """ - for _ in range(50): + for _ in range(2000): async def coro() -> int: return 1 @@ -105,7 +105,7 @@ async def test_cancel_racing_with_natural_completion_keeps_state_consistent() -> or ``cancelled``. It must never end up with a hybrid internal-error state or surface ``RuntimeError`` to the caller. """ - for _ in range(80): + for _ in range(1000): loop = asyncio.get_running_loop() async def coro() -> int: @@ -163,7 +163,7 @@ async def test_concurrent_cancel_with_full_unpacking_promise_chain() -> None: fire ``_synthesize_cancellation`` against an already-transitioning state, triggering an internal ``RuntimeError``. """ - for _ in range(40): + for _ in range(500): loop = asyncio.get_running_loop() async def inner_coro() -> str: @@ -235,7 +235,7 @@ async def test_concurrent_set_state_does_not_double_unregister_from_parent() -> parent = promising.PromisingContext(loop=loop, parent=None) - for _ in range(50): + for _ in range(1000): async def coro() -> int: return 5 diff --git a/tests/race_conditions/test_unsettled_children_set_race.py b/tests/race_conditions/test_unsettled_children_set_race.py index d4c3a0c97..abcd15348 100644 --- a/tests/race_conditions/test_unsettled_children_set_race.py +++ b/tests/race_conditions/test_unsettled_children_set_race.py @@ -85,7 +85,7 @@ def test_concurrent_child_registration_keeps_all_children() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(20): + for _ in range(200): parent = promising.PromisingContext(loop=loop, parent=None) N_THREADS = 64 @@ -145,33 +145,34 @@ def test_collect_unsettled_children_during_concurrent_registration_does_not_rais """ loop = _make_dedicated_loop() try: - parent = promising.PromisingContext(loop=loop, parent=None) + for _ in range(30): + parent = promising.PromisingContext(loop=loop, parent=None) - N_WRITERS = 16 - WRITES_PER_WRITER = 200 + N_WRITERS = 16 + WRITES_PER_WRITER = 200 - writers_done = threading.Event() - writers_finished_count = 0 - writers_finished_lock = threading.Lock() + writers_done = threading.Event() + writers_finished_count = 0 + writers_finished_lock = threading.Lock() - def writer() -> None: - nonlocal writers_finished_count - try: - for _ in range(WRITES_PER_WRITER): - promising.PromisingContext(loop=loop, parent=parent) - finally: - with writers_finished_lock: - writers_finished_count += 1 - if writers_finished_count == N_WRITERS: - writers_done.set() - - def reader() -> None: - while not writers_done.is_set(): - parent.collect_unsettled_children(whole_subtree=True, awaitables_only=False) - - targets = [reader] + [writer] * N_WRITERS - errors = _run_workers(targets) - assert not errors, errors + def writer() -> None: + nonlocal writers_finished_count + try: + for _ in range(WRITES_PER_WRITER): + promising.PromisingContext(loop=loop, parent=parent) + finally: + with writers_finished_lock: + writers_finished_count += 1 + if writers_finished_count == N_WRITERS: + writers_done.set() + + def reader() -> None: + while not writers_done.is_set(): + parent.collect_unsettled_children(whole_subtree=True, awaitables_only=False) + + targets = [reader] + [writer] * N_WRITERS + errors = _run_workers(targets) + assert not errors, errors finally: loop.close() @@ -188,24 +189,27 @@ def test_concurrent_child_close_keeps_consistent_set() -> None: """ loop = _make_dedicated_loop() try: - parent = promising.PromisingContext(loop=loop, parent=None) + for _ in range(50): + parent = promising.PromisingContext(loop=loop, parent=None) - N_CHILDREN = 256 - children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_CHILDREN)] - # Sanity check before the race - assert len(parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False)) == N_CHILDREN + N_CHILDREN = 256 + children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_CHILDREN)] + # Sanity check before the race + assert len(parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False)) == N_CHILDREN - def make_closer(child: promising.PromisingContext): - def _close() -> None: - child.close_context() + def make_closer(child: promising.PromisingContext): + def _close() -> None: + child.close_context() - return _close + return _close - errors = _run_workers([make_closer(c) for c in children]) - assert not errors, errors + errors = _run_workers([make_closer(c) for c in children]) + assert not errors, errors - remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert remaining == set(), f"parent still holds {len(remaining)} stale child references after concurrent close" + remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert remaining == set(), ( + f"parent still holds {len(remaining)} stale child references after concurrent close" + ) finally: loop.close() @@ -220,36 +224,37 @@ def test_concurrent_registration_and_unregistration_keeps_set_intact() -> None: """ loop = _make_dedicated_loop() try: - parent = promising.PromisingContext(loop=loop, parent=None) + for _ in range(50): + parent = promising.PromisingContext(loop=loop, parent=None) - N_PRE_EXISTING = 200 - pre_existing = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_PRE_EXISTING)] + N_PRE_EXISTING = 200 + pre_existing = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_PRE_EXISTING)] - N_NEW = 200 - new_children: list[promising.PromisingContext] = [] - new_children_lock = threading.Lock() + N_NEW = 200 + new_children: list[promising.PromisingContext] = [] + new_children_lock = threading.Lock() - def closer(child: promising.PromisingContext): - def _close() -> None: - child.close_context() + def closer(child: promising.PromisingContext): + def _close() -> None: + child.close_context() - return _close + return _close - def adder() -> None: - local = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_NEW // 50)] - with new_children_lock: - new_children.extend(local) + def adder() -> None: + local = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_NEW // 50)] + with new_children_lock: + new_children.extend(local) - targets = [closer(c) for c in pre_existing] + [adder] * 50 - errors = _run_workers(targets) - assert not errors, errors + targets = [closer(c) for c in pre_existing] + [adder] * 50 + errors = _run_workers(targets) + assert not errors, errors - remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert remaining == set(new_children), ( - f"after concurrent add/close, parent's set has " - f"{len(remaining ^ set(new_children))} discrepancies " - f"(expected={len(new_children)}, got={len(remaining)})" - ) + remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert remaining == set(new_children), ( + f"after concurrent add/close, parent's set has " + f"{len(remaining ^ set(new_children))} discrepancies " + f"(expected={len(new_children)}, got={len(remaining)})" + ) finally: loop.close() @@ -342,58 +347,60 @@ async def test_await_children_during_concurrent_thread_registration() -> None: N_WRITERS = 4 WRITES = 200 - @promising.function - async def parent_func() -> None: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() - thread_errors: list[BaseException] = [] - thread_errors_lock = threading.Lock() + for _ in range(20): - async def _quick() -> int: - return 42 + @promising.function + async def parent_func() -> None: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() + thread_errors: list[BaseException] = [] + thread_errors_lock = threading.Lock() - start_barrier = threading.Barrier(N_WRITERS + 1) + async def _quick() -> int: + return 42 - def thread_writer() -> None: - try: - start_barrier.wait() - for _ in range(WRITES): - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - except BaseException as exc: # noqa: BLE001 - with thread_errors_lock: - thread_errors.append(exc) - - writers = [threading.Thread(target=thread_writer, daemon=True) for _ in range(N_WRITERS)] - for w in writers: - w.start() + start_barrier = threading.Barrier(N_WRITERS + 1) - # Release the writers and immediately race them with await_children. - start_barrier.wait() - - # Drain children many times to keep the race window open. - for _ in range(20): + def thread_writer() -> None: + try: + start_barrier.wait() + for _ in range(WRITES): + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + except BaseException as exc: # noqa: BLE001 + with thread_errors_lock: + thread_errors.append(exc) + + writers = [threading.Thread(target=thread_writer, daemon=True) for _ in range(N_WRITERS)] + for w in writers: + w.start() + + # Release the writers and immediately race them with await_children. + start_barrier.wait() + + # Drain children many times to keep the race window open. + for _ in range(20): + await promising.await_children(whole_subtree=True) + + for w in writers: + w.join(timeout=10) + assert not w.is_alive() + + # Final drain. await promising.await_children(whole_subtree=True) - for w in writers: - w.join(timeout=10) - assert not w.is_alive() - - # Final drain. - await promising.await_children(whole_subtree=True) + assert not thread_errors, thread_errors - assert not thread_errors, thread_errors - - unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) - assert unsettled == set(), ( - f"await_children returned with {len(unsettled)} unsettled awaitable children remaining" - ) + unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) + assert unsettled == set(), ( + f"await_children returned with {len(unsettled)} unsettled awaitable children remaining" + ) - await parent_func() + await parent_func() # ── cascading unregister race ──────────────────────────────────── @@ -413,7 +420,7 @@ def test_cascading_unregister_keeps_grandparent_set_consistent() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(50): + for _ in range(300): grandparent = promising.PromisingContext(loop=loop, parent=None) middle = promising.PromisingContext(loop=loop, parent=grandparent) @@ -455,49 +462,51 @@ async def test_concurrent_promise_creation_from_threads_registers_all() -> None: the parent's ``_unsettled_children`` set; none must be lost. """ - @promising.function - async def parent_func() -> int: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() + for _ in range(20): - N_THREADS = 32 - CHILDREN_PER_THREAD = 20 + @promising.function + async def parent_func() -> int: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() - created: list[promising.Promise] = [] - created_lock = threading.Lock() + N_THREADS = 32 + CHILDREN_PER_THREAD = 20 - async def _quick() -> int: - return 0 + created: list[promising.Promise] = [] + created_lock = threading.Lock() - def worker() -> None: - local = [] - for _ in range(CHILDREN_PER_THREAD): - child = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - local.append(child) - with created_lock: - created.extend(local) + async def _quick() -> int: + return 0 + + def worker() -> None: + local = [] + for _ in range(CHILDREN_PER_THREAD): + child = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + local.append(child) + with created_lock: + created.extend(local) - with ThreadPoolExecutor(max_workers=N_THREADS) as ex: - await asyncio.gather(*[loop.run_in_executor(ex, worker) for _ in range(N_THREADS)]) + with ThreadPoolExecutor(max_workers=N_THREADS) as ex: + await asyncio.gather(*[loop.run_in_executor(ex, worker) for _ in range(N_THREADS)]) - expected = set(created) - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = expected - actual - assert not missing, f"{len(missing)} child Promises lost from the parent's set" + expected = set(created) + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = expected - actual + assert not missing, f"{len(missing)} child Promises lost from the parent's set" - # Clean up the unused coroutines so asyncio doesn't warn. - for child in created: - child.cancel() + # Clean up the unused coroutines so asyncio doesn't warn. + for child in created: + child.cancel() - return len(created) + return len(created) - n = await parent_func() - assert n == 32 * 20 + n = await parent_func() + assert n == 32 * 20 # ── sanity (would-pass) test, included for self-verification ───── From 58a78182391e611bde77231f2b461d2a5178913c Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 17:19:34 +0300 Subject: [PATCH 13/20] Revert "experimental commit: bumped numbers of repetitions in race condition tests" This reverts commit 02834b6170e3d5568f6eddb899c1d3d809613954. --- .../test_concurrent_consumption_race.py | 246 ++++++++------- .../test_context_lifecycle_race.py | 94 +++--- .../test_invariants_under_load.py | 173 +++++------ .../test_state_machine_race.py | 8 +- .../test_unsettled_children_set_race.py | 289 +++++++++--------- 5 files changed, 393 insertions(+), 417 deletions(-) diff --git a/tests/race_conditions/test_concurrent_consumption_race.py b/tests/race_conditions/test_concurrent_consumption_race.py index 5aa179d08..0447c2357 100644 --- a/tests/race_conditions/test_concurrent_consumption_race.py +++ b/tests/race_conditions/test_concurrent_consumption_race.py @@ -45,7 +45,7 @@ async def test_many_threads_calling_sync_on_same_promise_consistent() -> None: """ loop = asyncio.get_running_loop() - for _ in range(100): + for _ in range(10): async def coro() -> int: await asyncio.sleep(0.01) @@ -91,33 +91,31 @@ async def test_many_threads_calling_unpack_once_sync_on_same_promise_consistent( """ loop = asyncio.get_running_loop() - for _ in range(100): + async def coro() -> str: + await asyncio.sleep(0.01) + return "ok" - async def coro() -> str: - await asyncio.sleep(0.01) - return "ok" + promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) + N = 32 + results: list[object] = [] + errors: list[BaseException] = [] + lock = threading.Lock() - N = 32 - results: list[object] = [] - errors: list[BaseException] = [] - lock = threading.Lock() + def consumer() -> None: + try: + value = promise.unpack_once_sync(timeout=5) + with lock: + results.append(value) + except BaseException as exc: # noqa: BLE001 + with lock: + errors.append(exc) - def consumer() -> None: - try: - value = promise.unpack_once_sync(timeout=5) - with lock: - results.append(value) - except BaseException as exc: # noqa: BLE001 - with lock: - errors.append(exc) + with ThreadPoolExecutor(max_workers=N) as ex: + await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) - with ThreadPoolExecutor(max_workers=N) as ex: - await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) - - assert not errors, errors - assert results == ["ok"] * N + assert not errors, errors + assert results == ["ok"] * N # ── sync() race with cancel() ────────────────────────────────── @@ -133,7 +131,7 @@ async def test_sync_consumer_thread_observes_clean_terminal_after_cancel() -> No """ loop = asyncio.get_running_loop() - for _ in range(500): + for _ in range(40): cancel_event = asyncio.Event() async def coro() -> int: @@ -200,59 +198,57 @@ async def test_await_children_sync_during_thread_registration_does_not_raise() - """ loop = asyncio.get_running_loop() - for _ in range(30): + @promising.function + async def root() -> None: + active = promising.get_active_promise() + stop = threading.Event() + thread_errors: list[BaseException] = [] + registered_children: list[promising.Promise] = [] + registered_lock = threading.Lock() - @promising.function - async def root() -> None: - active = promising.get_active_promise() - stop = threading.Event() - thread_errors: list[BaseException] = [] - registered_children: list[promising.Promise] = [] - registered_lock = threading.Lock() - - async def _quick() -> int: - return 0 - - def writer() -> None: - try: - while not stop.is_set(): - p = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - with registered_lock: - registered_children.append(p) - except BaseException as exc: # noqa: BLE001 - thread_errors.append(exc) - - @promising.function(use_thread_pool=True) - def sync_waiter() -> str: - # Each iteration triggers a fresh collect_unsettled_children - # via await_children_sync. - for _ in range(20): - promising.await_children_sync(whole_subtree=True) - return "drained" + async def _quick() -> int: + return 0 - writers = [threading.Thread(target=writer, daemon=True) for _ in range(4)] - for w in writers: - w.start() + def writer() -> None: + try: + while not stop.is_set(): + p = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + with registered_lock: + registered_children.append(p) + except BaseException as exc: # noqa: BLE001 + thread_errors.append(exc) + + @promising.function(use_thread_pool=True) + def sync_waiter() -> str: + # Each iteration triggers a fresh collect_unsettled_children + # via await_children_sync. + for _ in range(20): + promising.await_children_sync(whole_subtree=True) + return "drained" + + writers = [threading.Thread(target=writer, daemon=True) for _ in range(4)] + for w in writers: + w.start() + try: + waiter_promise = sync_waiter() try: - waiter_promise = sync_waiter() - try: - await waiter_promise - finally: - stop.set() - for w in writers: - w.join(timeout=5) + await waiter_promise finally: - for c in registered_children: - c.cancel() + stop.set() + for w in writers: + w.join(timeout=5) + finally: + for c in registered_children: + c.cancel() - assert not thread_errors, thread_errors + assert not thread_errors, thread_errors - await root() + await root() # ── concurrent .sync() consumption *and* registration ─────────── @@ -267,69 +263,67 @@ async def test_concurrent_sync_consumers_and_child_registrations() -> None: """ loop = asyncio.get_running_loop() - for _ in range(30): + @promising.function + async def root() -> int: + active = promising.get_active_promise() @promising.function - async def root() -> int: - active = promising.get_active_promise() - - @promising.function - async def published() -> int: - await asyncio.sleep(0.01) - return 99 - - target = published() - - N_CONSUMERS = 16 - N_REGISTRARS = 16 - errors: list[BaseException] = [] - errors_lock = threading.Lock() - consumer_results: list[int] = [] - new_children: list[promising.Promise] = [] - - async def _quick() -> int: - return 0 - - def consumer() -> None: - try: - consumer_results.append(target.sync(timeout=5)) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - def registrar() -> None: - try: - for _ in range(20): - new_children.append( - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) + async def published() -> int: + await asyncio.sleep(0.01) + return 99 + + target = published() + + N_CONSUMERS = 16 + N_REGISTRARS = 16 + errors: list[BaseException] = [] + errors_lock = threading.Lock() + consumer_results: list[int] = [] + new_children: list[promising.Promise] = [] + + async def _quick() -> int: + return 0 + + def consumer() -> None: + try: + consumer_results.append(target.sync(timeout=5)) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + def registrar() -> None: + try: + for _ in range(20): + new_children.append( + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, ) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) + ) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) - with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: - consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] - registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] - await asyncio.gather(*consumer_futs, *registrar_futs) + with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: + consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] + registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] + await asyncio.gather(*consumer_futs, *registrar_futs) - assert not errors, errors - assert consumer_results == [99] * N_CONSUMERS + assert not errors, errors + assert consumer_results == [99] * N_CONSUMERS - # All registrars' children must be tracked. - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = set(new_children) - actual - assert not missing, f"{len(missing)} children were lost from the parent's set" + # All registrars' children must be tracked. + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = set(new_children) - actual + assert not missing, f"{len(missing)} children were lost from the parent's set" - for c in new_children: - c.cancel() - return 0 + for c in new_children: + c.cancel() + return 0 - await root() + await root() # ── sanity ────────────────────────────────────────────────────── diff --git a/tests/race_conditions/test_context_lifecycle_race.py b/tests/race_conditions/test_context_lifecycle_race.py index 86f2cc493..3ef0ded8e 100644 --- a/tests/race_conditions/test_context_lifecycle_race.py +++ b/tests/race_conditions/test_context_lifecycle_race.py @@ -44,7 +44,7 @@ def test_concurrent_enter_same_context_only_one_succeeds() -> None: loop = _make_dedicated_loop() try: N = 8 - for _ in range(3000): + for _ in range(500): ctx = promising.PromisingContext(loop=loop, parent=None) barrier = threading.Barrier(N) succeeded: list[str] = [] @@ -215,61 +215,57 @@ async def test_promise_context_open_races_with_external_child_registration() -> """ loop = asyncio.get_running_loop() - for _ in range(30): - - @promising.function - async def parent_promise() -> int: - active = promising.get_active_promise() - errors: list[BaseException] = [] - children: list[promising.Promise] = [] - children_lock = threading.Lock() - stop = threading.Event() + @promising.function + async def parent_promise() -> int: + active = promising.get_active_promise() + errors: list[BaseException] = [] + children: list[promising.Promise] = [] + children_lock = threading.Lock() + stop = threading.Event() - async def _quick() -> int: - return 1 - - def worker() -> None: - try: - while not stop.is_set(): - p = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - with children_lock: - children.append(p) - except BaseException as exc: # noqa: BLE001 - errors.append(exc) + async def _quick() -> int: + return 1 - writers = [threading.Thread(target=worker, daemon=True) for _ in range(8)] - for w in writers: - w.start() + def worker() -> None: try: - for _ in range(200): - await asyncio.sleep(0) - finally: - stop.set() - for w in writers: - w.join(timeout=5) + while not stop.is_set(): + p = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + with children_lock: + children.append(p) + except BaseException as exc: # noqa: BLE001 + errors.append(exc) + + writers = [threading.Thread(target=worker, daemon=True) for _ in range(8)] + for w in writers: + w.start() + try: + for _ in range(200): + await asyncio.sleep(0) + finally: + stop.set() + for w in writers: + w.join(timeout=5) - assert not errors, errors + assert not errors, errors - # Every child registered while the Promise was running must - # be tracked. - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = set(children) - actual - assert not missing, ( - f"{len(missing)} children were lost from active Promise's set during the lifecycle race" - ) + # Every child registered while the Promise was running must + # be tracked. + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = set(children) - actual + assert not missing, f"{len(missing)} children were lost from active Promise's set during the lifecycle race" - # Clean up so asyncio doesn't warn about un-awaited coros. - for c in children: - c.cancel() + # Clean up so asyncio doesn't warn about un-awaited coros. + for c in children: + c.cancel() - return 0 + return 0 - await parent_promise() + await parent_promise() # ── close_context idempotency under threads ───────────────────── @@ -290,7 +286,7 @@ def test_close_context_concurrent_calls_must_be_idempotent() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(2000): + for _ in range(100): parent = promising.PromisingContext(loop=loop, parent=None) child = promising.PromisingContext(loop=loop, parent=parent) diff --git a/tests/race_conditions/test_invariants_under_load.py b/tests/race_conditions/test_invariants_under_load.py index 8f3196942..2d8dfda7b 100644 --- a/tests/race_conditions/test_invariants_under_load.py +++ b/tests/race_conditions/test_invariants_under_load.py @@ -52,11 +52,10 @@ async def root() -> int: return sum(await asyncio.gather(*parents)) expected = sum(p * 1000 * N_CHILDREN_PER_PARENT + sum(range(N_CHILDREN_PER_PARENT)) for p in range(N_PARENTS)) - for _ in range(30): - actual = await root() - assert actual == expected, ( - f"lost children or duplicated work in concurrent hierarchy build: {actual} vs {expected}" - ) + actual = await root() + assert actual == expected, ( + f"lost children or duplicated work in concurrent hierarchy build: {actual} vs {expected}" + ) # ── many cancels racing with a fast natural completion ────────── @@ -80,7 +79,7 @@ async def test_cancel_race_state_consistency_high_iterations() -> None: """ loop = asyncio.get_running_loop() - for _ in range(2000): + for _ in range(300): async def coro() -> int: for _ in range(3): @@ -153,69 +152,67 @@ async def test_await_children_under_continuous_registration_load() -> None: N_WRITERS = 6 WRITES = 100 - for _ in range(20): - - @promising.function - async def root() -> int: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() - - thread_errors: list[BaseException] = [] - thread_errors_lock = threading.Lock() - all_promises: list[promising.Promise] = [] - all_promises_lock = threading.Lock() - - async def _quick() -> int: - return 0 - - start = threading.Barrier(N_WRITERS + 1) - - def writer() -> None: - try: - start.wait() - local = [] - for _ in range(WRITES): - local.append( - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - ) - with all_promises_lock: - all_promises.extend(local) - except BaseException as exc: # noqa: BLE001 - with thread_errors_lock: - thread_errors.append(exc) + @promising.function + async def root() -> int: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() - writers = [threading.Thread(target=writer, daemon=True) for _ in range(N_WRITERS)] - for w in writers: - w.start() + thread_errors: list[BaseException] = [] + thread_errors_lock = threading.Lock() + all_promises: list[promising.Promise] = [] + all_promises_lock = threading.Lock() - start.wait() + async def _quick() -> int: + return 0 + + start = threading.Barrier(N_WRITERS + 1) - # Drain while workers register. - for _ in range(30): - await promising.await_children(whole_subtree=True) + def writer() -> None: + try: + start.wait() + local = [] + for _ in range(WRITES): + local.append( + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + ) + with all_promises_lock: + all_promises.extend(local) + except BaseException as exc: # noqa: BLE001 + with thread_errors_lock: + thread_errors.append(exc) - for w in writers: - w.join(timeout=10) - assert not w.is_alive() + writers = [threading.Thread(target=writer, daemon=True) for _ in range(N_WRITERS)] + for w in writers: + w.start() + start.wait() + + # Drain while workers register. + for _ in range(30): await promising.await_children(whole_subtree=True) - assert not thread_errors, thread_errors + for w in writers: + w.join(timeout=10) + assert not w.is_alive() - with all_promises_lock: - for p in all_promises: - assert p.done(), f"child promise not done after final drain: {p!r}" + await promising.await_children(whole_subtree=True) - unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) - assert unsettled == set(), f"{len(unsettled)} awaitable descendants left after draining" - return 0 + assert not thread_errors, thread_errors + + with all_promises_lock: + for p in all_promises: + assert p.done(), f"child promise not done after final drain: {p!r}" - await root() + unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) + assert unsettled == set(), f"{len(unsettled)} awaitable descendants left after draining" + return 0 + + await root() # ── concurrent get_active_context across threads ──────────────── @@ -250,8 +247,7 @@ async def root() -> None: f"contain {expected!r}, got {actual!r}" ) - for _ in range(100): - await root() + await root() # ── massive concurrent close (cascading unregister) ───────────── @@ -267,7 +263,7 @@ def test_massive_concurrent_close_grandparent_consistency() -> None: """ loop = asyncio.new_event_loop() try: - for _ in range(100): + for _ in range(20): grandparent = promising.PromisingContext(loop=loop, parent=None) middle = promising.PromisingContext(loop=loop, parent=grandparent) @@ -341,33 +337,32 @@ async def task(idx: int) -> int: N = 8 - for _ in range(50): - results: list[int] = [] - errors: list[BaseException] = [] - lock = threading.Lock() - barrier = threading.Barrier(N) - - def runner(idx: int) -> None: - try: - barrier.wait() - value = task.run(idx) - with lock: - results.append(value) - except BaseException as exc: # noqa: BLE001 - with lock: - errors.append(exc) - - threads = [threading.Thread(target=runner, args=(i,), daemon=True) for i in range(N)] - for t in threads: - t.start() - for t in threads: - t.join(timeout=10) - assert not t.is_alive() + results: list[int] = [] + errors: list[BaseException] = [] + lock = threading.Lock() + barrier = threading.Barrier(N) - assert not errors, errors - assert sorted(results) == [i * 2 for i in range(N)], ( - f"results mismatch — possible cross-thread leakage of active context: {sorted(results)}" - ) + def runner(idx: int) -> None: + try: + barrier.wait() + value = task.run(idx) + with lock: + results.append(value) + except BaseException as exc: # noqa: BLE001 + with lock: + errors.append(exc) + + threads = [threading.Thread(target=runner, args=(i,), daemon=True) for i in range(N)] + for t in threads: + t.start() + for t in threads: + t.join(timeout=10) + assert not t.is_alive() + + assert not errors, errors + assert sorted(results) == [i * 2 for i in range(N)], ( + f"results mismatch — possible cross-thread leakage of active context: {sorted(results)}" + ) # ── sync() race with cancel — RuntimeError must not surface ───── @@ -383,7 +378,7 @@ async def test_sync_consumers_never_observe_internal_runtime_error() -> None: """ loop = asyncio.get_running_loop() - for _ in range(500): + for _ in range(50): async def coro() -> str: for _ in range(4): diff --git a/tests/race_conditions/test_state_machine_race.py b/tests/race_conditions/test_state_machine_race.py index ebaa55cda..83ecc441c 100644 --- a/tests/race_conditions/test_state_machine_race.py +++ b/tests/race_conditions/test_state_machine_race.py @@ -71,7 +71,7 @@ async def test_many_threads_cancelling_same_pending_promise_yield_consistent_sta ``_synthesize_cancellation`` which writes ``_state``). All workers should agree on a single cancelled terminal state. """ - for _ in range(2000): + for _ in range(50): async def coro() -> int: return 1 @@ -105,7 +105,7 @@ async def test_cancel_racing_with_natural_completion_keeps_state_consistent() -> or ``cancelled``. It must never end up with a hybrid internal-error state or surface ``RuntimeError`` to the caller. """ - for _ in range(1000): + for _ in range(80): loop = asyncio.get_running_loop() async def coro() -> int: @@ -163,7 +163,7 @@ async def test_concurrent_cancel_with_full_unpacking_promise_chain() -> None: fire ``_synthesize_cancellation`` against an already-transitioning state, triggering an internal ``RuntimeError``. """ - for _ in range(500): + for _ in range(40): loop = asyncio.get_running_loop() async def inner_coro() -> str: @@ -235,7 +235,7 @@ async def test_concurrent_set_state_does_not_double_unregister_from_parent() -> parent = promising.PromisingContext(loop=loop, parent=None) - for _ in range(1000): + for _ in range(50): async def coro() -> int: return 5 diff --git a/tests/race_conditions/test_unsettled_children_set_race.py b/tests/race_conditions/test_unsettled_children_set_race.py index abcd15348..d4c3a0c97 100644 --- a/tests/race_conditions/test_unsettled_children_set_race.py +++ b/tests/race_conditions/test_unsettled_children_set_race.py @@ -85,7 +85,7 @@ def test_concurrent_child_registration_keeps_all_children() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(200): + for _ in range(20): parent = promising.PromisingContext(loop=loop, parent=None) N_THREADS = 64 @@ -145,34 +145,33 @@ def test_collect_unsettled_children_during_concurrent_registration_does_not_rais """ loop = _make_dedicated_loop() try: - for _ in range(30): - parent = promising.PromisingContext(loop=loop, parent=None) + parent = promising.PromisingContext(loop=loop, parent=None) - N_WRITERS = 16 - WRITES_PER_WRITER = 200 + N_WRITERS = 16 + WRITES_PER_WRITER = 200 - writers_done = threading.Event() - writers_finished_count = 0 - writers_finished_lock = threading.Lock() + writers_done = threading.Event() + writers_finished_count = 0 + writers_finished_lock = threading.Lock() - def writer() -> None: - nonlocal writers_finished_count - try: - for _ in range(WRITES_PER_WRITER): - promising.PromisingContext(loop=loop, parent=parent) - finally: - with writers_finished_lock: - writers_finished_count += 1 - if writers_finished_count == N_WRITERS: - writers_done.set() - - def reader() -> None: - while not writers_done.is_set(): - parent.collect_unsettled_children(whole_subtree=True, awaitables_only=False) - - targets = [reader] + [writer] * N_WRITERS - errors = _run_workers(targets) - assert not errors, errors + def writer() -> None: + nonlocal writers_finished_count + try: + for _ in range(WRITES_PER_WRITER): + promising.PromisingContext(loop=loop, parent=parent) + finally: + with writers_finished_lock: + writers_finished_count += 1 + if writers_finished_count == N_WRITERS: + writers_done.set() + + def reader() -> None: + while not writers_done.is_set(): + parent.collect_unsettled_children(whole_subtree=True, awaitables_only=False) + + targets = [reader] + [writer] * N_WRITERS + errors = _run_workers(targets) + assert not errors, errors finally: loop.close() @@ -189,27 +188,24 @@ def test_concurrent_child_close_keeps_consistent_set() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(50): - parent = promising.PromisingContext(loop=loop, parent=None) + parent = promising.PromisingContext(loop=loop, parent=None) - N_CHILDREN = 256 - children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_CHILDREN)] - # Sanity check before the race - assert len(parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False)) == N_CHILDREN + N_CHILDREN = 256 + children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_CHILDREN)] + # Sanity check before the race + assert len(parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False)) == N_CHILDREN - def make_closer(child: promising.PromisingContext): - def _close() -> None: - child.close_context() + def make_closer(child: promising.PromisingContext): + def _close() -> None: + child.close_context() - return _close + return _close - errors = _run_workers([make_closer(c) for c in children]) - assert not errors, errors + errors = _run_workers([make_closer(c) for c in children]) + assert not errors, errors - remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert remaining == set(), ( - f"parent still holds {len(remaining)} stale child references after concurrent close" - ) + remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert remaining == set(), f"parent still holds {len(remaining)} stale child references after concurrent close" finally: loop.close() @@ -224,37 +220,36 @@ def test_concurrent_registration_and_unregistration_keeps_set_intact() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(50): - parent = promising.PromisingContext(loop=loop, parent=None) + parent = promising.PromisingContext(loop=loop, parent=None) - N_PRE_EXISTING = 200 - pre_existing = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_PRE_EXISTING)] + N_PRE_EXISTING = 200 + pre_existing = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_PRE_EXISTING)] - N_NEW = 200 - new_children: list[promising.PromisingContext] = [] - new_children_lock = threading.Lock() + N_NEW = 200 + new_children: list[promising.PromisingContext] = [] + new_children_lock = threading.Lock() - def closer(child: promising.PromisingContext): - def _close() -> None: - child.close_context() + def closer(child: promising.PromisingContext): + def _close() -> None: + child.close_context() - return _close + return _close - def adder() -> None: - local = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_NEW // 50)] - with new_children_lock: - new_children.extend(local) + def adder() -> None: + local = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_NEW // 50)] + with new_children_lock: + new_children.extend(local) - targets = [closer(c) for c in pre_existing] + [adder] * 50 - errors = _run_workers(targets) - assert not errors, errors + targets = [closer(c) for c in pre_existing] + [adder] * 50 + errors = _run_workers(targets) + assert not errors, errors - remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert remaining == set(new_children), ( - f"after concurrent add/close, parent's set has " - f"{len(remaining ^ set(new_children))} discrepancies " - f"(expected={len(new_children)}, got={len(remaining)})" - ) + remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) + assert remaining == set(new_children), ( + f"after concurrent add/close, parent's set has " + f"{len(remaining ^ set(new_children))} discrepancies " + f"(expected={len(new_children)}, got={len(remaining)})" + ) finally: loop.close() @@ -347,60 +342,58 @@ async def test_await_children_during_concurrent_thread_registration() -> None: N_WRITERS = 4 WRITES = 200 - for _ in range(20): + @promising.function + async def parent_func() -> None: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() + thread_errors: list[BaseException] = [] + thread_errors_lock = threading.Lock() - @promising.function - async def parent_func() -> None: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() - thread_errors: list[BaseException] = [] - thread_errors_lock = threading.Lock() + async def _quick() -> int: + return 42 - async def _quick() -> int: - return 42 + start_barrier = threading.Barrier(N_WRITERS + 1) - start_barrier = threading.Barrier(N_WRITERS + 1) + def thread_writer() -> None: + try: + start_barrier.wait() + for _ in range(WRITES): + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + except BaseException as exc: # noqa: BLE001 + with thread_errors_lock: + thread_errors.append(exc) - def thread_writer() -> None: - try: - start_barrier.wait() - for _ in range(WRITES): - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - except BaseException as exc: # noqa: BLE001 - with thread_errors_lock: - thread_errors.append(exc) - - writers = [threading.Thread(target=thread_writer, daemon=True) for _ in range(N_WRITERS)] - for w in writers: - w.start() - - # Release the writers and immediately race them with await_children. - start_barrier.wait() - - # Drain children many times to keep the race window open. - for _ in range(20): - await promising.await_children(whole_subtree=True) - - for w in writers: - w.join(timeout=10) - assert not w.is_alive() - - # Final drain. + writers = [threading.Thread(target=thread_writer, daemon=True) for _ in range(N_WRITERS)] + for w in writers: + w.start() + + # Release the writers and immediately race them with await_children. + start_barrier.wait() + + # Drain children many times to keep the race window open. + for _ in range(20): await promising.await_children(whole_subtree=True) - assert not thread_errors, thread_errors + for w in writers: + w.join(timeout=10) + assert not w.is_alive() - unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) - assert unsettled == set(), ( - f"await_children returned with {len(unsettled)} unsettled awaitable children remaining" - ) + # Final drain. + await promising.await_children(whole_subtree=True) - await parent_func() + assert not thread_errors, thread_errors + + unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) + assert unsettled == set(), ( + f"await_children returned with {len(unsettled)} unsettled awaitable children remaining" + ) + + await parent_func() # ── cascading unregister race ──────────────────────────────────── @@ -420,7 +413,7 @@ def test_cascading_unregister_keeps_grandparent_set_consistent() -> None: """ loop = _make_dedicated_loop() try: - for _ in range(300): + for _ in range(50): grandparent = promising.PromisingContext(loop=loop, parent=None) middle = promising.PromisingContext(loop=loop, parent=grandparent) @@ -462,51 +455,49 @@ async def test_concurrent_promise_creation_from_threads_registers_all() -> None: the parent's ``_unsettled_children`` set; none must be lost. """ - for _ in range(20): + @promising.function + async def parent_func() -> int: + active = promising.get_active_promise() + loop = asyncio.get_running_loop() - @promising.function - async def parent_func() -> int: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() + N_THREADS = 32 + CHILDREN_PER_THREAD = 20 - N_THREADS = 32 - CHILDREN_PER_THREAD = 20 + created: list[promising.Promise] = [] + created_lock = threading.Lock() - created: list[promising.Promise] = [] - created_lock = threading.Lock() - - async def _quick() -> int: - return 0 + async def _quick() -> int: + return 0 - def worker() -> None: - local = [] - for _ in range(CHILDREN_PER_THREAD): - child = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - local.append(child) - with created_lock: - created.extend(local) + def worker() -> None: + local = [] + for _ in range(CHILDREN_PER_THREAD): + child = promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) + local.append(child) + with created_lock: + created.extend(local) - with ThreadPoolExecutor(max_workers=N_THREADS) as ex: - await asyncio.gather(*[loop.run_in_executor(ex, worker) for _ in range(N_THREADS)]) + with ThreadPoolExecutor(max_workers=N_THREADS) as ex: + await asyncio.gather(*[loop.run_in_executor(ex, worker) for _ in range(N_THREADS)]) - expected = set(created) - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = expected - actual - assert not missing, f"{len(missing)} child Promises lost from the parent's set" + expected = set(created) + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = expected - actual + assert not missing, f"{len(missing)} child Promises lost from the parent's set" - # Clean up the unused coroutines so asyncio doesn't warn. - for child in created: - child.cancel() + # Clean up the unused coroutines so asyncio doesn't warn. + for child in created: + child.cancel() - return len(created) + return len(created) - n = await parent_func() - assert n == 32 * 20 + n = await parent_func() + assert n == 32 * 20 # ── sanity (would-pass) test, included for self-verification ───── From 80ab5dcedccf669fa50deb9112e196fa4cdae27a Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 17:25:29 +0300 Subject: [PATCH 14/20] repeat two more race condition tests multiple times --- .../test_concurrent_consumption_race.py | 82 ++++++++++--------- .../test_invariants_under_load.py | 3 +- 2 files changed, 44 insertions(+), 41 deletions(-) diff --git a/tests/race_conditions/test_concurrent_consumption_race.py b/tests/race_conditions/test_concurrent_consumption_race.py index 0447c2357..3b8d65f20 100644 --- a/tests/race_conditions/test_concurrent_consumption_race.py +++ b/tests/race_conditions/test_concurrent_consumption_race.py @@ -272,55 +272,57 @@ async def published() -> int: await asyncio.sleep(0.01) return 99 - target = published() - - N_CONSUMERS = 16 - N_REGISTRARS = 16 - errors: list[BaseException] = [] - errors_lock = threading.Lock() - consumer_results: list[int] = [] - new_children: list[promising.Promise] = [] - async def _quick() -> int: return 0 - def consumer() -> None: - try: - consumer_results.append(target.sync(timeout=5)) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) + N_CONSUMERS = 16 + N_REGISTRARS = 16 - def registrar() -> None: - try: - for _ in range(20): - new_children.append( - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, + for _ in range(30): + target = published() + + errors: list[BaseException] = [] + errors_lock = threading.Lock() + consumer_results: list[int] = [] + new_children: list[promising.Promise] = [] + + def consumer() -> None: + try: + consumer_results.append(target.sync(timeout=5)) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) + + def registrar() -> None: + try: + for _ in range(20): + new_children.append( + promising.wrap_awaitable( + _quick(), + parent=active, + loop=loop, + start_soon=False, + ) ) - ) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) + except BaseException as exc: # noqa: BLE001 + with errors_lock: + errors.append(exc) - with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: - consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] - registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] - await asyncio.gather(*consumer_futs, *registrar_futs) + with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: + consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] + registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] + await asyncio.gather(*consumer_futs, *registrar_futs) - assert not errors, errors - assert consumer_results == [99] * N_CONSUMERS + assert not errors, errors + assert consumer_results == [99] * N_CONSUMERS - # All registrars' children must be tracked. - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = set(new_children) - actual - assert not missing, f"{len(missing)} children were lost from the parent's set" + # All registrars' children must be tracked. + actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) + missing = set(new_children) - actual + assert not missing, f"{len(missing)} children were lost from the parent's set" - for c in new_children: - c.cancel() + for c in new_children: + c.cancel() return 0 await root() diff --git a/tests/race_conditions/test_invariants_under_load.py b/tests/race_conditions/test_invariants_under_load.py index 2d8dfda7b..d5bb989d5 100644 --- a/tests/race_conditions/test_invariants_under_load.py +++ b/tests/race_conditions/test_invariants_under_load.py @@ -247,7 +247,8 @@ async def root() -> None: f"contain {expected!r}, got {actual!r}" ) - await root() + for _ in range(30): + await root() # ── massive concurrent close (cascading unregister) ───────────── From ee01f760d7bae66b630d10d4d7dfd0a4c1c2f881 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 23:08:47 +0300 Subject: [PATCH 15/20] in progress: review auto-generated race condition tests --- .../test_context_lifecycle_race.py | 19 +++++++++++++------ .../test_unsettled_children_set_race.py | 7 +++++++ 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/tests/race_conditions/test_context_lifecycle_race.py b/tests/race_conditions/test_context_lifecycle_race.py index 3ef0ded8e..4cfa25032 100644 --- a/tests/race_conditions/test_context_lifecycle_race.py +++ b/tests/race_conditions/test_context_lifecycle_race.py @@ -84,12 +84,12 @@ def enter() -> None: def test_concurrent_enter_then_concurrent_exit_does_not_corrupt_contextvar() -> None: """ - When two threads enter the same context (currently both can pass - the check) and then both call ``__exit__``, one of them ends up - calling ``ContextVar.reset(token)`` with a token that was created - in the *other* thread's context. That cross-thread reset is at - minimum a no-op and at worst raises ``ValueError``. Either way - ``__exit__`` must not crash with an unrelated error. + When two threads enter the same context (provided such a race condition + exists) and then both call ``__exit__``, one of them ends up calling + ``ContextVar.reset(token)`` with a token that was created in the *other* + thread's context. That cross-thread reset is at minimum a no-op and at + worst raises ``ValueError``. Either way ``__exit__`` must not crash with an + unrelated error. """ import time @@ -145,6 +145,9 @@ def test_register_child_during_close_does_not_silently_succeed() -> None: OR the registration completes and the child must still be reachable via ``collect_unsettled_children`` until it is closed itself. """ + # TODO [TESTS] How is this test different from + # tests/race_conditions/test_unsettled_children_set_race.py + # ::test_register_child_after_parent_closed_must_be_rejected ? loop = _make_dedicated_loop() try: for _ in range(3000): @@ -177,6 +180,10 @@ def close_parent() -> None: # a closed parent, that contract is broken — the child # has a parent reference that points at a context that # will never drive its lifecycle. + # TODO [TESTS] How do we know it happened AFTER the parent was + # closed ? I'm struggling to spot the part of test that + # insures things happened in that order and not the other way + # around assert not parent.closed(), ( "child registration succeeded onto a parent that is now closed — " "the `closed() → no new children` invariant was violated by a race" diff --git a/tests/race_conditions/test_unsettled_children_set_race.py b/tests/race_conditions/test_unsettled_children_set_race.py index d4c3a0c97..4b61a3b6d 100644 --- a/tests/race_conditions/test_unsettled_children_set_race.py +++ b/tests/race_conditions/test_unsettled_children_set_race.py @@ -270,6 +270,9 @@ def test_register_child_after_parent_closed_must_be_rejected() -> None: we expect *either* that error, or no leaked child in the closed parent's set — never both "no error" and "child present". """ + # TODO [TESTS] How is this test different from + # tests/race_conditions/test_context_lifecycle_race.py + # ::test_register_child_during_close_does_not_silently_succeed ? loop = _make_dedicated_loop() try: # Run many short races to widen the window. @@ -306,6 +309,10 @@ def close_parent() -> None: # If registration silently succeeded against a parent that # is now closed, the framework's `closed() → no new # children` contract has been broken by the race. + # TODO [TESTS] How do we know it happened AFTER the parent was + # closed ? I'm struggling to spot the part of test that + # insures things happened in that order and not the other way + # around assert not parent.closed(), ( "child registration silently succeeded after parent was closed — invariant violated" ) From 32a32107cf080736272fa069b0b99f39750d74dc Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Sun, 17 May 2026 23:12:32 +0300 Subject: [PATCH 16/20] in progress: review auto-generated race condition tests --- tests/race_conditions/test_context_lifecycle_race.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/race_conditions/test_context_lifecycle_race.py b/tests/race_conditions/test_context_lifecycle_race.py index 4cfa25032..2008a749f 100644 --- a/tests/race_conditions/test_context_lifecycle_race.py +++ b/tests/race_conditions/test_context_lifecycle_race.py @@ -91,6 +91,9 @@ def test_concurrent_enter_then_concurrent_exit_does_not_corrupt_contextvar() -> worst raises ``ValueError``. Either way ``__exit__`` must not crash with an unrelated error. """ + # TODO [TESTS] Is this really a useful test ? Stress-testing `__enter__` is + # important, but `__exit__` ? On the condition that there is already a + # race condition upon `__enter__` ? Who cares at that point ? import time loop = _make_dedicated_loop() From d3404c1cc681003a67a6136f739d027944dd4d8f Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Tue, 19 May 2026 22:25:57 +0300 Subject: [PATCH 17/20] drop low quality race condition tests entirely --- tests/race_conditions/__init__.py | 1 - tests/race_conditions/conftest.py | 37 -- .../test_concurrent_consumption_race.py | 342 ------------ .../test_context_lifecycle_race.py | 342 ------------ .../test_invariants_under_load.py | 426 -------------- .../test_state_machine_race.py | 260 --------- .../test_unsettled_children_set_race.py | 522 ------------------ 7 files changed, 1930 deletions(-) delete mode 100644 tests/race_conditions/__init__.py delete mode 100644 tests/race_conditions/conftest.py delete mode 100644 tests/race_conditions/test_concurrent_consumption_race.py delete mode 100644 tests/race_conditions/test_context_lifecycle_race.py delete mode 100644 tests/race_conditions/test_invariants_under_load.py delete mode 100644 tests/race_conditions/test_state_machine_race.py delete mode 100644 tests/race_conditions/test_unsettled_children_set_race.py diff --git a/tests/race_conditions/__init__.py b/tests/race_conditions/__init__.py deleted file mode 100644 index 71c220121..000000000 --- a/tests/race_conditions/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# TODO [TESTS] These tests haven't been reviewed by a human at all yet ! diff --git a/tests/race_conditions/conftest.py b/tests/race_conditions/conftest.py deleted file mode 100644 index 44af6a574..000000000 --- a/tests/race_conditions/conftest.py +++ /dev/null @@ -1,37 +0,0 @@ -""" -Race-condition stress tests use threads and high iteration counts to -surface unsynchronized state in the framework. - -We force a very short GIL switch interval (1µs) so the interpreter -preempts between almost every bytecode, dramatically widening the race -windows that the framework's unprotected read-then-act sequences -expose. Without this, CPython's default ~5ms switch interval often -hides the bugs. -""" - -import sys -from collections.abc import Iterator - -import pytest - -# Apply a longer per-test timeout to every test in this directory. -pytestmark = pytest.mark.timeout(30) - -_ORIGINAL_SWITCH_INTERVAL = sys.getswitchinterval() -_RACE_SWITCH_INTERVAL = 1e-6 # 1 microsecond - - -@pytest.fixture(autouse=True) -def _aggressive_gil_switching() -> Iterator[None]: - """Force frequent GIL hand-offs so reader/writer races surface.""" - sys.setswitchinterval(_RACE_SWITCH_INTERVAL) - try: - yield - finally: - sys.setswitchinterval(_ORIGINAL_SWITCH_INTERVAL) - - -def pytest_collection_modifyitems(config: pytest.Config, items: list[pytest.Item]) -> None: - for item in items: - if "race_conditions" in str(item.fspath): - item.add_marker(pytest.mark.timeout(30)) diff --git a/tests/race_conditions/test_concurrent_consumption_race.py b/tests/race_conditions/test_concurrent_consumption_race.py deleted file mode 100644 index 3b8d65f20..000000000 --- a/tests/race_conditions/test_concurrent_consumption_race.py +++ /dev/null @@ -1,342 +0,0 @@ -""" -Race conditions around multi-thread consumption of Promises and -context hierarchies via the sync API: - -- ``Promise.sync()`` / ``Promise.unpack_once_sync()`` — invoked by - multiple worker threads simultaneously on the same Promise. -- ``promising.await_children_sync()`` — invoked from one thread while - workers on other threads keep registering new child Promises under - the same parent. - -All of these end up reading/writing ``Promise._state``, -``Promise._full_unpacking_task``, ``Promise._single_unpacking_task``, -``PromisingContext._unsettled_children`` — none of which are protected. -""" - -import asyncio -import threading -from concurrent.futures import ThreadPoolExecutor - -import pytest - -import promising - -# ── concurrent sync() consumption ────────────────────────────── - - -async def test_many_threads_calling_sync_on_same_promise_consistent() -> None: - """ - N worker threads simultaneously call ``promise.sync()`` on the same - Promise. All must return the same value; none must hang or raise. - - NOTE [race-injection / 2026-05-17]: this test could not be made to - fail even by deliberately introducing race-prone breakage in - ``promising/`` (lost children, non-atomic state writes, dropped - ContextVar copy, etc.). All ``.sync()`` callers dispatch their - awaiting onto the Promise's single event loop via - ``run_coroutine_threadsafe``, so the actual ``await self`` runs - serialized on the loop thread; only one ``_full_unpacking_task`` is - ever scheduled, every coroutine yields from the same task, and the - final ``_result`` is written once by the task itself. There is no - race surface to inject into without changing this serialization. - Keep as a regression guard against future refactors that try to - short-circuit the loop dispatch (e.g. a "fast path" returning a - cached ``_result`` directly from the caller thread). - """ - loop = asyncio.get_running_loop() - - for _ in range(10): - - async def coro() -> int: - await asyncio.sleep(0.01) - return 1234 - - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) - - N = 32 - results: list[int] = [] - errors: list[BaseException] = [] - lock = threading.Lock() - - def consumer() -> None: - try: - value = promise.sync(timeout=5) - with lock: - results.append(value) - except BaseException as exc: # noqa: BLE001 - with lock: - errors.append(exc) - - with ThreadPoolExecutor(max_workers=N) as ex: - await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) - - assert not errors, errors - assert results == [1234] * N, f"inconsistent results from concurrent sync(): {set(results)}" - - -async def test_many_threads_calling_unpack_once_sync_on_same_promise_consistent() -> None: - """ - Same scenario as ``test_many_threads_calling_sync_on_same_promise`` - but using ``unpack_once_sync``. All threads should observe the same - one-level-unpacking outcome (here: a concrete value, since the - coroutine does not return a Promise). - - NOTE [race-injection / 2026-05-17]: same finding as the ``.sync()`` - sibling above — could not be broken by deliberate framework - sabotage. ``unpack_once_sync`` also goes through - ``run_coroutine_threadsafe`` and the single - ``_single_unpacking_task``. Only one task ever drives the unpack, - all callers share its result. Keep as regression guard against - future refactors that bypass the loop dispatch. - """ - loop = asyncio.get_running_loop() - - async def coro() -> str: - await asyncio.sleep(0.01) - return "ok" - - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=False) - - N = 32 - results: list[object] = [] - errors: list[BaseException] = [] - lock = threading.Lock() - - def consumer() -> None: - try: - value = promise.unpack_once_sync(timeout=5) - with lock: - results.append(value) - except BaseException as exc: # noqa: BLE001 - with lock: - errors.append(exc) - - with ThreadPoolExecutor(max_workers=N) as ex: - await asyncio.gather(*[loop.run_in_executor(ex, consumer) for _ in range(N)]) - - assert not errors, errors - assert results == ["ok"] * N - - -# ── sync() race with cancel() ────────────────────────────────── - - -async def test_sync_consumer_thread_observes_clean_terminal_after_cancel() -> None: - """ - One thread waits on ``promise.sync()`` while another fires - ``promise.cancel()``. The sync caller must observe a clean - terminal state — either the value (if cancel lost the race) or - ``CancelledError``. It must never observe an internal - ``RuntimeError`` from a racing state transition. - """ - loop = asyncio.get_running_loop() - - for _ in range(40): - cancel_event = asyncio.Event() - - async def coro() -> int: - try: - await cancel_event.wait() - except asyncio.CancelledError: - raise - return 9 - - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) - - consumer_done = threading.Event() - consumer_result: dict[str, object] = {} - - def consume() -> None: - try: - consumer_result["value"] = promise.sync(timeout=5) - except BaseException as exc: # noqa: BLE001 - consumer_result["exception"] = exc - finally: - consumer_done.set() - - consumer = threading.Thread(target=consume, daemon=True) - consumer.start() - - # Let the consumer thread get into run_coroutine_threadsafe() - await asyncio.sleep(0) - - promise.cancel() - - # Park the loop until the consumer thread is done. - while not consumer_done.is_set(): - await asyncio.sleep(0) - - consumer.join(timeout=5) - assert not consumer.is_alive() - - # The sync caller must end up with a clean terminal outcome: - # either the value (cancel lost the race) or a CancelledError. - # Anything else (RuntimeError, internal state errors, ...) means - # the racing state-machine paths leaked an internal exception. - if "exception" in consumer_result: - exc = consumer_result["exception"] - # Compare by class name to side-step asyncio vs. - # concurrent.futures CancelledError class differences. - assert type(exc).__name__ == "CancelledError", ( - f"sync() raised an unexpected internal exception during cancel race: " - f"{type(exc).__module__}.{type(exc).__qualname__}: {exc!r}" - ) - else: - assert consumer_result["value"] == 9 - - -# ── await_children_sync race with thread-side registration ────── - - -async def test_await_children_sync_during_thread_registration_does_not_raise() -> None: - """ - A sync promising function runs in the thread pool and calls - ``promising.await_children_sync()``; meanwhile additional threads - keep creating child Promises under the same parent. The sync - waiter must drain all children without raising "set changed size - during iteration" from the underlying ``collect_unsettled_children``. - """ - loop = asyncio.get_running_loop() - - @promising.function - async def root() -> None: - active = promising.get_active_promise() - stop = threading.Event() - thread_errors: list[BaseException] = [] - registered_children: list[promising.Promise] = [] - registered_lock = threading.Lock() - - async def _quick() -> int: - return 0 - - def writer() -> None: - try: - while not stop.is_set(): - p = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - with registered_lock: - registered_children.append(p) - except BaseException as exc: # noqa: BLE001 - thread_errors.append(exc) - - @promising.function(use_thread_pool=True) - def sync_waiter() -> str: - # Each iteration triggers a fresh collect_unsettled_children - # via await_children_sync. - for _ in range(20): - promising.await_children_sync(whole_subtree=True) - return "drained" - - writers = [threading.Thread(target=writer, daemon=True) for _ in range(4)] - for w in writers: - w.start() - try: - waiter_promise = sync_waiter() - try: - await waiter_promise - finally: - stop.set() - for w in writers: - w.join(timeout=5) - finally: - for c in registered_children: - c.cancel() - - assert not thread_errors, thread_errors - - await root() - - -# ── concurrent .sync() consumption *and* registration ─────────── - - -async def test_concurrent_sync_consumers_and_child_registrations() -> None: - """ - Realistic stress: half the workers consume a published Promise - via ``.sync()`` while the other half register fresh child Promises - on the same parent. The framework must keep both the consumption - path and the parent's set consistent. - """ - loop = asyncio.get_running_loop() - - @promising.function - async def root() -> int: - active = promising.get_active_promise() - - @promising.function - async def published() -> int: - await asyncio.sleep(0.01) - return 99 - - async def _quick() -> int: - return 0 - - N_CONSUMERS = 16 - N_REGISTRARS = 16 - - for _ in range(30): - target = published() - - errors: list[BaseException] = [] - errors_lock = threading.Lock() - consumer_results: list[int] = [] - new_children: list[promising.Promise] = [] - - def consumer() -> None: - try: - consumer_results.append(target.sync(timeout=5)) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - def registrar() -> None: - try: - for _ in range(20): - new_children.append( - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - ) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_REGISTRARS) as ex: - consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] - registrar_futs = [loop.run_in_executor(ex, registrar) for _ in range(N_REGISTRARS)] - await asyncio.gather(*consumer_futs, *registrar_futs) - - assert not errors, errors - assert consumer_results == [99] * N_CONSUMERS - - # All registrars' children must be tracked. - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = set(new_children) - actual - assert not missing, f"{len(missing)} children were lost from the parent's set" - - for c in new_children: - c.cancel() - return 0 - - await root() - - -# ── sanity ────────────────────────────────────────────────────── - - -@pytest.mark.skip(reason="Sanity reference — single-thread consumption is fine.") -async def test_single_threaded_sync_consumption_works_sanity() -> None: - async def coro() -> int: - return 1 - - promise = promising.wrap_awaitable(coro(), parent=None, start_soon=True) - loop = asyncio.get_running_loop() - value = await loop.run_in_executor(None, promise.sync) - assert value == 1 diff --git a/tests/race_conditions/test_context_lifecycle_race.py b/tests/race_conditions/test_context_lifecycle_race.py deleted file mode 100644 index 2008a749f..000000000 --- a/tests/race_conditions/test_context_lifecycle_race.py +++ /dev/null @@ -1,342 +0,0 @@ -""" -Race conditions around ``PromisingContext``'s lifecycle: - -- ``__enter__`` reads ``self._previous_token`` and - ``self._context_closed`` before writing the token. Two threads - entering the same context can both pass the check and both call - ``ContextVar.set``, leaving the loser's token orphaned and the next - ``__exit__`` cross-thread ``reset()`` either no-op'ing or raising - ``ValueError``. -- ``close_context`` flips ``_context_closed`` and calls - ``_unregister_from_parent_if_time`` without coordinating with - ``_register_children`` on the same instance. - -These tests stress those paths via real ``with`` blocks and real -``PromisingContext`` instances. -""" - -import asyncio -import threading - -import pytest - -import promising - -# ── helpers ───────────────────────────────────────────────────── - - -def _make_dedicated_loop() -> asyncio.AbstractEventLoop: - return asyncio.new_event_loop() - - -# ── concurrent __enter__ of the same instance ─────────────────── - - -def test_concurrent_enter_same_context_only_one_succeeds() -> None: - """ - Two worker threads call ``ctx.__enter__()`` on the same instance - simultaneously. Exactly one should succeed; the other must see - ``ContextAlreadyActiveError``. Without a lock around the - ``_previous_token is not None`` check both can pass it and both - will overwrite ``_previous_token`` — silently leaking the - first-thread token. - """ - loop = _make_dedicated_loop() - try: - N = 8 - for _ in range(500): - ctx = promising.PromisingContext(loop=loop, parent=None) - barrier = threading.Barrier(N) - succeeded: list[str] = [] - already_active_errors: list[BaseException] = [] - other_errors: list[BaseException] = [] - list_lock = threading.Lock() - - def enter() -> None: - try: - barrier.wait() - ctx.__enter__() - with list_lock: - succeeded.append(threading.current_thread().name) - except promising.ContextAlreadyActiveError as exc: - with list_lock: - already_active_errors.append(exc) - except BaseException as exc: # noqa: BLE001 - with list_lock: - other_errors.append(exc) - - threads = [threading.Thread(target=enter, name=f"T{i}", daemon=True) for i in range(N)] - for t in threads: - t.start() - for t in threads: - t.join(timeout=5) - assert not t.is_alive() - - assert not other_errors, other_errors - assert len(succeeded) == 1, ( - f"{len(succeeded)} threads entered the same context concurrently " - f"(succeeded={succeeded}, already_active={len(already_active_errors)})" - ) - assert len(already_active_errors) == N - 1 - finally: - loop.close() - - -def test_concurrent_enter_then_concurrent_exit_does_not_corrupt_contextvar() -> None: - """ - When two threads enter the same context (provided such a race condition - exists) and then both call ``__exit__``, one of them ends up calling - ``ContextVar.reset(token)`` with a token that was created in the *other* - thread's context. That cross-thread reset is at minimum a no-op and at - worst raises ``ValueError``. Either way ``__exit__`` must not crash with an - unrelated error. - """ - # TODO [TESTS] Is this really a useful test ? Stress-testing `__enter__` is - # important, but `__exit__` ? On the condition that there is already a - # race condition upon `__enter__` ? Who cares at that point ? - import time - - loop = _make_dedicated_loop() - try: - for _ in range(3000): - ctx = promising.PromisingContext(loop=loop, parent=None) - - enter_barrier = threading.Barrier(2) - errors: list[BaseException] = [] - errors_lock = threading.Lock() - - def routine() -> None: - try: - enter_barrier.wait() - try: - ctx.__enter__() - except promising.ContextAlreadyActiveError: - # Framework rejected this thread's entry: nothing to exit. - return - # Yield to give the other thread a chance to enter, too. - time.sleep(0.0005) - ctx.__exit__(None, None, None) - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - t1 = threading.Thread(target=routine, daemon=True) - t2 = threading.Thread(target=routine, daemon=True) - t1.start() - t2.start() - t1.join(timeout=5) - t2.join(timeout=5) - - assert not errors, f"{len(errors)} unexpected exceptions: {errors!r}" - finally: - loop.close() - - -# ── register vs close on a single context ─────────────────────── - - -def test_register_child_during_close_does_not_silently_succeed() -> None: - """ - Thread A is in the middle of ``parent._register_children(child)``: - after the ``self.closed()`` check passes but before - ``self._unsettled_children.update(...)`` runs, Thread B calls - ``parent.close_context()``. Currently the child is then added to - a context that has already been closed. - - The expected contract: either the child registration raises - ``ContextAlreadyClosedError`` and the parent's set stays empty, - OR the registration completes and the child must still be reachable - via ``collect_unsettled_children`` until it is closed itself. - """ - # TODO [TESTS] How is this test different from - # tests/race_conditions/test_unsettled_children_set_race.py - # ::test_register_child_after_parent_closed_must_be_rejected ? - loop = _make_dedicated_loop() - try: - for _ in range(3000): - parent = promising.PromisingContext(loop=loop, parent=None) - barrier = threading.Barrier(2) - outcome: dict[str, object] = {} - - def add_child() -> None: - barrier.wait() - try: - outcome["child"] = promising.PromisingContext(loop=loop, parent=parent) - except promising.ContextAlreadyClosedError as exc: - outcome["closed_error"] = exc - - def close_parent() -> None: - barrier.wait() - parent.close_context() - - t1 = threading.Thread(target=add_child, daemon=True) - t2 = threading.Thread(target=close_parent, daemon=True) - t1.start() - t2.start() - t1.join(timeout=5) - t2.join(timeout=5) - - if "child" in outcome: - child = outcome["child"] - # Invariant: a context that is closed cannot accept new - # children. If the race lets registration succeed against - # a closed parent, that contract is broken — the child - # has a parent reference that points at a context that - # will never drive its lifecycle. - # TODO [TESTS] How do we know it happened AFTER the parent was - # closed ? I'm struggling to spot the part of test that - # insures things happened in that order and not the other way - # around - assert not parent.closed(), ( - "child registration succeeded onto a parent that is now closed — " - "the `closed() → no new children` invariant was violated by a race" - ) - reachable = child in parent.collect_unsettled_children( - whole_subtree=False, - awaitables_only=False, - ) - assert reachable, ( - "child registered onto parent silently, but is missing from " - "parent._unsettled_children — torn update or lost child" - ) - finally: - loop.close() - - -# ── Promise context: with-block opening races with sync registration ─ - - -async def test_promise_context_open_races_with_external_child_registration() -> None: - """ - A Promise enters its own context (``with self:``) inside - ``_unpack_once`` on the loop thread. While that ``__enter__`` - runs, a worker thread tries to register a brand-new child Promise - against the same Promise (as parent). The child registration uses - ``self.closed()`` as the gate — that field flips to True only on - ``__exit__``, but ``_previous_token`` is being set in lockstep with - ``__enter__`` on the loop thread. - - Concurrent reads/writes of those instance attributes can: - - - raise ``ContextAlreadyClosedError`` even though the parent is - still open - - silently add a child to a parent whose lifecycle has already - moved on - """ - loop = asyncio.get_running_loop() - - @promising.function - async def parent_promise() -> int: - active = promising.get_active_promise() - errors: list[BaseException] = [] - children: list[promising.Promise] = [] - children_lock = threading.Lock() - stop = threading.Event() - - async def _quick() -> int: - return 1 - - def worker() -> None: - try: - while not stop.is_set(): - p = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - with children_lock: - children.append(p) - except BaseException as exc: # noqa: BLE001 - errors.append(exc) - - writers = [threading.Thread(target=worker, daemon=True) for _ in range(8)] - for w in writers: - w.start() - try: - for _ in range(200): - await asyncio.sleep(0) - finally: - stop.set() - for w in writers: - w.join(timeout=5) - - assert not errors, errors - - # Every child registered while the Promise was running must - # be tracked. - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = set(children) - actual - assert not missing, f"{len(missing)} children were lost from active Promise's set during the lifecycle race" - - # Clean up so asyncio doesn't warn about un-awaited coros. - for c in children: - c.cancel() - - return 0 - - await parent_promise() - - -# ── close_context idempotency under threads ───────────────────── - - -def test_close_context_concurrent_calls_must_be_idempotent() -> None: - """ - ``close_context`` flips ``_context_closed`` and conditionally - unregisters from the parent. Concurrent calls from many threads - on the same instance should converge on a single - "closed + unregistered" terminal state — but they all read the - fields, all decide they need to unregister, and all call - ``parent._unregister_children(self)``. With the underlying - ``set.difference_update`` being a no-op on missing elements, the - only visible damage is when several siblings finish in the same - cycle and trigger re-entrant cascading unregistration. This test - asserts the simpler property: no exceptions, and parent is empty. - """ - loop = _make_dedicated_loop() - try: - for _ in range(100): - parent = promising.PromisingContext(loop=loop, parent=None) - child = promising.PromisingContext(loop=loop, parent=parent) - - N = 16 - errors: list[BaseException] = [] - errors_lock = threading.Lock() - barrier = threading.Barrier(N) - - def close_target() -> None: - try: - barrier.wait() - child.close_context() - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - threads = [threading.Thread(target=close_target, daemon=True) for _ in range(N)] - for t in threads: - t.start() - for t in threads: - t.join(timeout=5) - assert not t.is_alive() - - assert not errors, errors - assert child.closed() - assert child not in parent.collect_unsettled_children( - whole_subtree=False, - awaitables_only=False, - ) - finally: - loop.close() - - -@pytest.mark.skip(reason="Sanity reference — single-threaded lifecycle is fine.") -def test_single_threaded_enter_exit_works() -> None: - loop = _make_dedicated_loop() - try: - ctx = promising.PromisingContext(loop=loop, parent=None) - ctx.__enter__() - ctx.__exit__(None, None, None) - assert ctx.closed() - finally: - loop.close() diff --git a/tests/race_conditions/test_invariants_under_load.py b/tests/race_conditions/test_invariants_under_load.py deleted file mode 100644 index d5bb989d5..000000000 --- a/tests/race_conditions/test_invariants_under_load.py +++ /dev/null @@ -1,426 +0,0 @@ -""" -Heavier load tests that stress thread-safety invariants the framework -must maintain regardless of CPython's GIL atomicity guarantees. - -These tests do not target a single check-then-act window — they keep -many threads doing many operations until either: - -- the framework raises something unexpected (``RuntimeError`` from a - state-machine mismatch, ``ContextAlreadyClosedError`` on what should - be a still-open context, etc.), or -- an invariant on the *final* state is violated (lost children, - inconsistent ``done()``/``cancelled()``/``result()`` triplet, ...). - -The intent: even when individual race windows are tiny on CPython, a -high-volume test eventually lands on the wrong interleaving. -""" - -import asyncio -import threading -from concurrent.futures import ThreadPoolExecutor - -import promising - -# ── tree-shape invariants under heavy parallel construction ───── - - -async def test_deep_hierarchy_stress_keeps_tree_consistent() -> None: - """ - Many sync-pool workers build a 2-level promise hierarchy in - parallel. After all work completes the root must have awaited every - leaf, and every leaf's parent must have unregistered cleanly. - """ - - N_PARENTS = 16 - N_CHILDREN_PER_PARENT = 32 - - @promising.function - async def leaf(idx: int) -> int: - return idx - - @promising.function(use_thread_pool=True) - def sync_parent(parent_idx: int) -> int: - children = [leaf(parent_idx * 1000 + i) for i in range(N_CHILDREN_PER_PARENT)] - total = 0 - for c in children: - total += c.sync(timeout=5) - return total - - @promising.function - async def root() -> int: - parents = [sync_parent(i) for i in range(N_PARENTS)] - return sum(await asyncio.gather(*parents)) - - expected = sum(p * 1000 * N_CHILDREN_PER_PARENT + sum(range(N_CHILDREN_PER_PARENT)) for p in range(N_PARENTS)) - actual = await root() - assert actual == expected, ( - f"lost children or duplicated work in concurrent hierarchy build: {actual} vs {expected}" - ) - - -# ── many cancels racing with a fast natural completion ────────── - - -async def test_cancel_race_state_consistency_high_iterations() -> None: - """ - For a Promise whose coroutine completes after one tick, fire a - storm of ``cancel()`` calls from worker threads while the loop - drives the task to completion. After settling, the *triplet* - ``done()``/``cancelled()``/``exception()``/``result()`` must agree: - - - ``cancelled()`` True → ``result()`` raises ``CancelledError``, - ``exception()`` raises ``CancelledError`` - - ``cancelled()`` False → ``result()`` returns the value, - ``exception()`` returns ``None`` - - A torn state-machine update can put the Promise in a state where - ``cancelled()`` is False but ``result()`` raises - ``CancelledError`` (or vice-versa) — that's the bug we are hunting. - """ - loop = asyncio.get_running_loop() - - for _ in range(300): - - async def coro() -> int: - for _ in range(3): - await asyncio.sleep(0) - return 42 - - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) - - N = 8 - ready = threading.Barrier(N + 1) - - def canceller() -> None: - ready.wait() - promise.cancel() - - threads = [threading.Thread(target=canceller, daemon=True) for _ in range(N)] - for t in threads: - t.start() - - ready.wait() - - try: - value = await promise - saw_value = True - except BaseException: # noqa: BLE001 - value = None - saw_value = False - - for t in threads: - t.join(timeout=5) - assert not t.is_alive() - - assert promise.done(), f"promise did not reach a terminal state: {promise!r}" - - if saw_value: - # We got the natural value back. The Promise's terminal - # state must reflect that and ``result()``/``exception()`` - # must agree. - assert not promise.cancelled(), f"await returned value={value} but promise.cancelled()=True: {promise!r}" - assert promise.result() == value - assert promise.exception() is None - else: - # We got an exception. It must be a CancelledError and - # the Promise must be in the cancelled state. - assert promise.cancelled(), f"await raised an exception but promise.cancelled()=False: {promise!r}" - # result() / exception() must raise CancelledError, not - # RuntimeError or anything else. - raised_in_result: BaseException | None = None - try: - promise.result() - except BaseException as exc: # noqa: BLE001 - raised_in_result = exc - assert raised_in_result is not None and type(raised_in_result).__name__ == "CancelledError", ( - f"promise.result() raised the wrong kind of exception: {raised_in_result!r}; promise={promise!r}" - ) - - -# ── concurrent threads doing many `await_children` cycles ─────── - - -async def test_await_children_under_continuous_registration_load() -> None: - """ - Heavy load: from inside a parent Promise, many worker threads - register child Promises in tight loops. Concurrently the loop - drains via ``await_children`` repeatedly. After settling, all - children registered up to that point must be done and the parent - must have no unsettled awaitable descendants. - """ - - N_WRITERS = 6 - WRITES = 100 - - @promising.function - async def root() -> int: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() - - thread_errors: list[BaseException] = [] - thread_errors_lock = threading.Lock() - all_promises: list[promising.Promise] = [] - all_promises_lock = threading.Lock() - - async def _quick() -> int: - return 0 - - start = threading.Barrier(N_WRITERS + 1) - - def writer() -> None: - try: - start.wait() - local = [] - for _ in range(WRITES): - local.append( - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - ) - with all_promises_lock: - all_promises.extend(local) - except BaseException as exc: # noqa: BLE001 - with thread_errors_lock: - thread_errors.append(exc) - - writers = [threading.Thread(target=writer, daemon=True) for _ in range(N_WRITERS)] - for w in writers: - w.start() - - start.wait() - - # Drain while workers register. - for _ in range(30): - await promising.await_children(whole_subtree=True) - - for w in writers: - w.join(timeout=10) - assert not w.is_alive() - - await promising.await_children(whole_subtree=True) - - assert not thread_errors, thread_errors - - with all_promises_lock: - for p in all_promises: - assert p.done(), f"child promise not done after final drain: {p!r}" - - unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) - assert unsettled == set(), f"{len(unsettled)} awaitable descendants left after draining" - return 0 - - await root() - - -# ── concurrent get_active_context across threads ──────────────── - - -async def test_get_active_context_returns_correct_context_per_thread() -> None: - """ - Each ``@promising.function(use_thread_pool=True)`` body runs on a - worker thread; ``ctx.run(...)`` propagates the ``ContextVar`` so - ``get_active_context()`` should return *that worker's* parent - Promise inside the sync body. Many sync workers running - concurrently must each see their own parent — never another - worker's. A failure here points to ``ContextVar`` propagation - being broken under contention. - """ - - @promising.function(use_thread_pool=True) - def worker(label: str) -> tuple[str, str]: - expected_namespace_substring = label - active_promise = promising.get_active_promise() - # The active promise's namespace was set explicitly to label. - return (expected_namespace_substring, active_promise.namespace or "") - - @promising.function - async def root() -> None: - N = 32 - promises = [worker(f"label_{i}", namespace=f"label_{i}") for i in range(N)] - results = await asyncio.gather(*promises) - for expected, actual in results: - assert expected in actual, ( - f"worker thread observed wrong active promise: expected namespace to " - f"contain {expected!r}, got {actual!r}" - ) - - for _ in range(30): - await root() - - -# ── massive concurrent close (cascading unregister) ───────────── - - -def test_massive_concurrent_close_grandparent_consistency() -> None: - """ - Three-level tree: grandparent → middle (closed) → many children. - Every child closes itself simultaneously. Cascading unregistration - must converge on an empty grandparent, with the middle context - unregistered exactly once. The race is around the read-then-act in - ``_unregister_from_parent_if_time``. - """ - loop = asyncio.new_event_loop() - try: - for _ in range(20): - grandparent = promising.PromisingContext(loop=loop, parent=None) - middle = promising.PromisingContext(loop=loop, parent=grandparent) - - N = 128 - children = [promising.PromisingContext(loop=loop, parent=middle) for _ in range(N)] - middle.close_context() - - barrier = threading.Barrier(N) - errors: list[BaseException] = [] - errors_lock = threading.Lock() - - def closer(c: promising.PromisingContext): - def _go() -> None: - try: - barrier.wait() - c.close_context() - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - return _go - - threads = [threading.Thread(target=closer(c), daemon=True) for c in children] - for t in threads: - t.start() - for t in threads: - t.join(timeout=10) - assert not t.is_alive() - - assert not errors, errors - - grand_unsettled = grandparent.collect_unsettled_children( - whole_subtree=False, - awaitables_only=False, - ) - assert grand_unsettled == set(), ( - f"grandparent leaked {len(grand_unsettled)} entries after cascading unregister storm" - ) - finally: - loop.close() - - -# ── concurrent Promise.run from multiple threads ──────────────── - - -def test_concurrent_independent_promise_run_invocations_isolate() -> None: - """ - ``PromisingFunction.run()`` creates its own event loop. Multiple - threads should be able to call ``.run()`` on the *same* - ``@promising.function`` concurrently without leaking state across - each other's hierarchies — the ``__active_context`` ContextVar - must be properly per-thread/per-loop. - - NOTE [race-injection / 2026-05-17]: this test could not be made to - fail by injecting the obvious race bugs into ``promising/`` - (non-atomic set writes, dropped ``copy_context``, dropped state - guards, etc.). Each ``.run()`` allocates its own ``asyncio`` event - loop in its own thread, and ``PromisingContext.__active_context`` - is a ``ContextVar`` whose value is per-thread / per-asyncio-task — - so the threads never share an active-context slot in the first - place. Breaking it would require an architectural change (e.g. - replacing the ``ContextVar`` with a module-level global). Keep as - regression guard against exactly that kind of refactor: someone - "caching" the active promise in a global for perf would cause - cross-thread leakage and this test would catch it. - """ - - @promising.function - async def task(idx: int) -> int: - return idx * 2 - - N = 8 - - results: list[int] = [] - errors: list[BaseException] = [] - lock = threading.Lock() - barrier = threading.Barrier(N) - - def runner(idx: int) -> None: - try: - barrier.wait() - value = task.run(idx) - with lock: - results.append(value) - except BaseException as exc: # noqa: BLE001 - with lock: - errors.append(exc) - - threads = [threading.Thread(target=runner, args=(i,), daemon=True) for i in range(N)] - for t in threads: - t.start() - for t in threads: - t.join(timeout=10) - assert not t.is_alive() - - assert not errors, errors - assert sorted(results) == [i * 2 for i in range(N)], ( - f"results mismatch — possible cross-thread leakage of active context: {sorted(results)}" - ) - - -# ── sync() race with cancel — RuntimeError must not surface ───── - - -async def test_sync_consumers_never_observe_internal_runtime_error() -> None: - """ - Consumers calling ``promise.sync()`` from threads must never see a - ``RuntimeError`` originating from the framework's own state - machine (``Cannot set result on a promise because of its current - state ...``). Only acceptable outcomes are the value or a - ``CancelledError``. - """ - loop = asyncio.get_running_loop() - - for _ in range(50): - - async def coro() -> str: - for _ in range(4): - await asyncio.sleep(0) - return "v" - - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) - - N_CONSUMERS = 8 - N_CANCELLERS = 4 - - outcomes: list[object] = [] - outcomes_lock = threading.Lock() - - start = threading.Barrier(N_CONSUMERS + N_CANCELLERS + 1) - - def consumer() -> None: - start.wait() - try: - outcomes.append(("value", promise.sync(timeout=5))) - except BaseException as exc: # noqa: BLE001 - with outcomes_lock: - outcomes.append(("error", exc)) - - def canceller() -> None: - start.wait() - promise.cancel() - - with ThreadPoolExecutor(max_workers=N_CONSUMERS + N_CANCELLERS) as ex: - consumer_futs = [loop.run_in_executor(ex, consumer) for _ in range(N_CONSUMERS)] - canceller_futs = [loop.run_in_executor(ex, canceller) for _ in range(N_CANCELLERS)] - start.wait() - await asyncio.gather(*consumer_futs, *canceller_futs) - - for kind, payload in outcomes: - if kind == "value": - assert payload == "v", f"unexpected value from sync(): {payload!r}" - else: - # Any exception must be a CancelledError — not RuntimeError, - # PromiseNotDoneError, etc. - assert type(payload).__name__ == "CancelledError", ( - f"sync() raised internal/unexpected error during cancel race: " - f"{type(payload).__module__}.{type(payload).__qualname__}: {payload!r}" - ) diff --git a/tests/race_conditions/test_state_machine_race.py b/tests/race_conditions/test_state_machine_race.py deleted file mode 100644 index 83ecc441c..000000000 --- a/tests/race_conditions/test_state_machine_race.py +++ /dev/null @@ -1,260 +0,0 @@ -""" -Race conditions around ``Promise``'s state machine. - -``Promise._state`` is read and written by: - -- ``_unpack_once`` / ``_unpack_fully`` on the loop thread, via - ``_set_intermediate_promise`` / ``_set_result`` / ``_set_exception`` -- ``Promise.cancel()`` and ``_synthesize_cancellation`` from *any* - thread — public API mirrors ``Future.cancel()`` which is invokable - off-loop -- ``done()`` / ``cancelled()`` / ``result()`` consumers from any thread - -There is no lock guarding the transitions. ``_set_exception`` and -``_set_result`` perform a multi-step (read state → decide terminal -state → write state) sequence, so two threads can both observe a -``_PENDING`` state and both race to write a terminal state. The framework -either: - -- raises an internal ``RuntimeError`` ("Cannot set result on a - promise because of its current state ...") which is then swallowed - into the Promise via ``_force_internal_error_finish`` -- or silently transitions to a wrong terminal state (e.g. ``cancelled`` - after a successful ``_set_result``). - -These tests aim to surface those races. -""" - -import asyncio -import threading -from typing import Any - -import pytest - -import promising - -# ── helpers ───────────────────────────────────────────────────── - - -def _run_threads_with_barrier(targets: list, *, join_timeout: float = 10.0) -> list[BaseException]: - errors: list[BaseException] = [] - errors_lock = threading.Lock() - barrier = threading.Barrier(len(targets)) - - def _wrap(fn): - def _run() -> None: - try: - barrier.wait() - fn() - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - return _run - - threads = [threading.Thread(target=_wrap(fn), daemon=True) for fn in targets] - for t in threads: - t.start() - for t in threads: - t.join(timeout=join_timeout) - assert not t.is_alive(), "Worker thread did not finish in time" - return errors - - -# ── concurrent cancel ────────────────────────────────────────── - - -async def test_many_threads_cancelling_same_pending_promise_yield_consistent_state() -> None: - """ - Many worker threads race to cancel the same ``start_soon=False`` - Promise (no task scheduled yet → cancel goes through - ``_synthesize_cancellation`` which writes ``_state``). All workers - should agree on a single cancelled terminal state. - """ - for _ in range(50): - - async def coro() -> int: - return 1 - - promise = promising.wrap_awaitable(coro(), parent=None, start_soon=False) - - N = 32 - - def _cancel() -> None: - promise.cancel("from worker") - - errors = _run_threads_with_barrier([_cancel] * N) - assert not errors, errors - - assert promise.done(), "Promise must be done after concurrent cancellation" - assert promise.cancelled(), "Promise must be in the cancelled state" - - # exception() on a cancelled Promise re-raises the stored - # CancelledError — it must not raise RuntimeError or any other - # internal error. - with pytest.raises(asyncio.CancelledError): - promise.exception() - - -async def test_cancel_racing_with_natural_completion_keeps_state_consistent() -> None: - """ - Spawn a Promise that completes after a quick ``asyncio.sleep(0)``. - Right as the loop tries to call ``_set_result``, fire many cancels - from worker threads. The Promise must end up in *exactly one* - terminal state — either successfully ``finished`` with the value, - or ``cancelled``. It must never end up with a hybrid internal-error - state or surface ``RuntimeError`` to the caller. - """ - for _ in range(80): - loop = asyncio.get_running_loop() - - async def coro() -> int: - await asyncio.sleep(0) - return 7 - - promise = promising.wrap_awaitable(coro(), parent=None, loop=loop, start_soon=True) - - N = 16 - started = threading.Barrier(N + 1) - - def _cancel() -> None: - started.wait() - try: - promise.cancel() - except BaseException: # noqa: BLE001 - # Some race losers can raise — that itself is a bug - # worth surfacing. - raise - - threads = [threading.Thread(target=_cancel, daemon=True) for _ in range(N)] - for t in threads: - t.start() - - # Race the workers against the loop step that resolves the task. - started.wait() - try: - value = await promise - outcome: dict[str, Any] = {"finished": value} - except asyncio.CancelledError: - outcome = {"cancelled": True} - - for t in threads: - t.join(timeout=5) - assert not t.is_alive() - - assert promise.done(), "Promise must reach a terminal state" - if "finished" in outcome: - assert not promise.cancelled() - assert promise.result() == 7 - assert promise.exception() is None - else: - assert promise.cancelled() - with pytest.raises(asyncio.CancelledError): - promise.result() - - -async def test_concurrent_cancel_with_full_unpacking_promise_chain() -> None: - """ - When a Promise has both a single-unpacking and a full-unpacking - task scheduled, ``cancel()`` calls ``cancel`` on both. Done callbacks - on those tasks then invoke ``_synthesize_cancellation`` via - ``_unpacking_task_done_callback``. Multiple worker threads cancelling - the same Promise simultaneously can cause both done-callbacks to - fire ``_synthesize_cancellation`` against an already-transitioning - state, triggering an internal ``RuntimeError``. - """ - for _ in range(40): - loop = asyncio.get_running_loop() - - async def inner_coro() -> str: - await asyncio.sleep(0.01) - return "ok" - - @promising.function - async def outer_func() -> promising.Promise[str]: - return promising.wrap_awaitable(inner_coro(), loop=loop, start_soon=True) - - promise = outer_func() - - # Schedule both unpacking paths - async def _kick() -> None: - await promise.unpack_once() # schedules single-unpacking - - kick_task = loop.create_task(_kick()) - try: - # Brief yield so the single unpacking task is in flight - await asyncio.sleep(0) - - def _cancel() -> None: - promise.cancel() - - errors = _run_threads_with_barrier([_cancel] * 8) - assert not errors, errors - - try: - await promise - except asyncio.CancelledError: - pass - except BaseException: - # The promise might also have completed before the - # cancel landed. - pass - finally: - kick_task.cancel() - try: - await kick_task - except (asyncio.CancelledError, BaseException): - pass - - assert promise.done(), f"Promise not done after cancel race: {promise!r}" - - # exception() must not raise a non-CancelledError exception. - if promise.cancelled(): - with pytest.raises(asyncio.CancelledError): - promise.exception() - else: - exc = promise.exception() - # Any *internal* error from racing _set_* paths would show - # up here. - assert exc is None or isinstance(exc, asyncio.CancelledError), ( - f"unexpected exception from racing state machine: {exc!r}" - ) - - -async def test_concurrent_set_state_does_not_double_unregister_from_parent() -> None: - """ - ``_set_state`` calls ``close_context()`` which calls - ``_unregister_from_parent_if_time`` which checks ``self.done()`` - and ``not self._unsettled_children`` and then calls - ``parent._unregister_children(self)``. Two threads transitioning - state in lockstep can both call into parent's unregister path, - triggering ``set.difference_update`` twice and (if cascading) racing - further up. - """ - loop = asyncio.get_running_loop() - - parent = promising.PromisingContext(loop=loop, parent=None) - - for _ in range(50): - - async def coro() -> int: - return 5 - - promise = promising.wrap_awaitable(coro(), parent=parent, start_soon=False) - - # Two threads call cancel — only one can actually set the - # terminal state, but with the race both think they can. - N = 4 - - def _cancel() -> None: - promise.cancel() - - errors = _run_threads_with_barrier([_cancel] * N) - assert not errors, errors - - assert promise.done() - assert promise.cancelled() - - # After all the promises cancelled, none should remain in parent. - remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert remaining == set(), f"parent leaked {len(remaining)} children after racing cancellations" diff --git a/tests/race_conditions/test_unsettled_children_set_race.py b/tests/race_conditions/test_unsettled_children_set_race.py deleted file mode 100644 index 4b61a3b6d..000000000 --- a/tests/race_conditions/test_unsettled_children_set_race.py +++ /dev/null @@ -1,522 +0,0 @@ -""" -Race conditions around ``PromisingContext._unsettled_children``. - -The set is mutated concurrently by: - -- worker threads that create new child contexts/promises (each new - child registers itself via ``parent._register_children(self)``) -- worker threads that close child contexts (which call - ``parent._unregister_children(self)``) -- consumers iterating the set via ``collect_unsettled_children`` / - ``await_children`` - -None of these paths take a lock, so concurrent access can: - -- raise ``RuntimeError: Set changed size during iteration`` -- silently lose children (registration overwritten by an unregistration - that happened in a stale snapshot) -- register children onto a parent that has *just* been closed, breaking - the ``closed() → no new children`` invariant -- corrupt the parent's idea of who its children are after a cascading - ``_unregister_from_parent_if_time`` re-entrance - -Every test below stresses one of these surfaces. While the framework is -unprotected, they are expected to fail; they exist so locks added later -can be verified end-to-end. -""" - -import asyncio -import threading -from concurrent.futures import ThreadPoolExecutor - -import pytest - -import promising - -# ── helpers ───────────────────────────────────────────────────── - - -def _make_dedicated_loop() -> asyncio.AbstractEventLoop: - """Brand-new event loop owned by the test, never started.""" - return asyncio.new_event_loop() - - -def _run_workers(targets: list, *, join_timeout: float = 15.0) -> list[BaseException]: - """ - Run a list of nullary callables in parallel threads behind a single - ``threading.Barrier`` so they all fire as close to simultaneously as - possible. Returns any exceptions raised. - """ - errors: list[BaseException] = [] - errors_lock = threading.Lock() - barrier = threading.Barrier(len(targets)) - - def _wrap(fn): - def _run(): - try: - barrier.wait() - fn() - except BaseException as exc: # noqa: BLE001 - with errors_lock: - errors.append(exc) - - return _run - - threads = [threading.Thread(target=_wrap(fn), daemon=True) for fn in targets] - for t in threads: - t.start() - for t in threads: - t.join(timeout=join_timeout) - assert not t.is_alive(), "Worker thread did not finish in time" - - return errors - - -# ── concurrent registration ────────────────────────────────────── - - -def test_concurrent_child_registration_keeps_all_children() -> None: - """ - Many worker threads simultaneously construct child ``PromisingContext`` - instances under one parent. After all workers finish, the parent's - ``_unsettled_children`` must contain *every* child that was created. - With unsynchronized ``set.update`` calls from many threads, - children can be lost. - """ - loop = _make_dedicated_loop() - try: - for _ in range(20): - parent = promising.PromisingContext(loop=loop, parent=None) - - N_THREADS = 64 - CHILDREN_PER_THREAD = 50 - - created: list[promising.PromisingContext] = [] - created_lock = threading.Lock() - - def worker() -> None: - local: list[promising.PromisingContext] = [] - for _ in range(CHILDREN_PER_THREAD): - child = promising.PromisingContext(loop=loop, parent=parent) - local.append(child) - with created_lock: - created.extend(local) - - errors = _run_workers([worker] * N_THREADS) - assert not errors, errors - - expected = set(created) - actual = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - missing = expected - actual - extra = actual - expected - - assert not missing, f"{len(missing)} children were lost from the parent's _unsettled_children set" - assert not extra, f"unexpected children appeared in the parent's _unsettled_children set: {extra!r}" - finally: - loop.close() - - -def test_collect_unsettled_children_during_concurrent_registration_does_not_raise() -> None: - """ - A reader thread iterates ``collect_unsettled_children`` in a tight - loop while writer threads keep registering new child contexts. With - no lock around the set, the reader's internal ``list(set)`` snapshot - races with mutations and can raise - ``RuntimeError: Set changed size during iteration``. - - NOTE [race-injection / 2026-05-17]: could not be made to fail - simultaneously with the lost-children tests in this file. The two - bug surfaces are mutually exclusive: - - - "lost children" requires non-atomic read-modify-write on - ``_unsettled_children`` (i.e. rebinding to a fresh set), which - means the reader's iteration target is a *different object* than - the writer mutates — no in-place mutation, no - ``RuntimeError: set changed size during iteration``. - - "set changed size during iteration" requires in-place mutation - (``.add`` / ``.discard``) on the live set, which is atomic per - call in CPython and therefore would not lose children. - - Pick one bug pattern, surface the other. Keep this test as - forward-compat for nogil Python (3.13t) where set-iteration - atomicity weakens, and as a regression guard against refactors - that swap the atomic ``set.update`` / ``list(set)`` C calls for - Python-level loops over the live set. - """ - loop = _make_dedicated_loop() - try: - parent = promising.PromisingContext(loop=loop, parent=None) - - N_WRITERS = 16 - WRITES_PER_WRITER = 200 - - writers_done = threading.Event() - writers_finished_count = 0 - writers_finished_lock = threading.Lock() - - def writer() -> None: - nonlocal writers_finished_count - try: - for _ in range(WRITES_PER_WRITER): - promising.PromisingContext(loop=loop, parent=parent) - finally: - with writers_finished_lock: - writers_finished_count += 1 - if writers_finished_count == N_WRITERS: - writers_done.set() - - def reader() -> None: - while not writers_done.is_set(): - parent.collect_unsettled_children(whole_subtree=True, awaitables_only=False) - - targets = [reader] + [writer] * N_WRITERS - errors = _run_workers(targets) - assert not errors, errors - finally: - loop.close() - - -# ── concurrent unregistration ──────────────────────────────────── - - -def test_concurrent_child_close_keeps_consistent_set() -> None: - """ - Many children close themselves concurrently (each call invokes - ``parent._unregister_children(self)``). After every worker has - finished, the parent must have an *empty* ``_unsettled_children`` - set — no leaked entries from races on ``set.difference_update``. - """ - loop = _make_dedicated_loop() - try: - parent = promising.PromisingContext(loop=loop, parent=None) - - N_CHILDREN = 256 - children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_CHILDREN)] - # Sanity check before the race - assert len(parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False)) == N_CHILDREN - - def make_closer(child: promising.PromisingContext): - def _close() -> None: - child.close_context() - - return _close - - errors = _run_workers([make_closer(c) for c in children]) - assert not errors, errors - - remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert remaining == set(), f"parent still holds {len(remaining)} stale child references after concurrent close" - finally: - loop.close() - - -def test_concurrent_registration_and_unregistration_keeps_set_intact() -> None: - """ - Two waves run simultaneously: half the threads register new - contexts under a parent, the other half close pre-existing children. - Reading the set with ``len()`` from a third thread must never blow - up and the final state must equal (pre-existing - closed + - newly-registered). - """ - loop = _make_dedicated_loop() - try: - parent = promising.PromisingContext(loop=loop, parent=None) - - N_PRE_EXISTING = 200 - pre_existing = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_PRE_EXISTING)] - - N_NEW = 200 - new_children: list[promising.PromisingContext] = [] - new_children_lock = threading.Lock() - - def closer(child: promising.PromisingContext): - def _close() -> None: - child.close_context() - - return _close - - def adder() -> None: - local = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(N_NEW // 50)] - with new_children_lock: - new_children.extend(local) - - targets = [closer(c) for c in pre_existing] + [adder] * 50 - errors = _run_workers(targets) - assert not errors, errors - - remaining = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert remaining == set(new_children), ( - f"after concurrent add/close, parent's set has " - f"{len(remaining ^ set(new_children))} discrepancies " - f"(expected={len(new_children)}, got={len(remaining)})" - ) - finally: - loop.close() - - -# ── register-vs-close race on the *parent* ─────────────────────── - - -def test_register_child_after_parent_closed_must_be_rejected() -> None: - """ - ``_register_children`` reads ``self.closed()`` *before* it - ``self._unsettled_children.update(...)``. If another thread closes - the parent in between, the check passes but the close already - happened and the child is silently added to a closed parent — - breaking the ``closed() → cannot accept children`` contract. - - The framework promises ``ContextAlreadyClosedError`` when adding to - a closed context; in the post-close-but-update-still-happens window - we expect *either* that error, or no leaked child in the closed - parent's set — never both "no error" and "child present". - """ - # TODO [TESTS] How is this test different from - # tests/race_conditions/test_context_lifecycle_race.py - # ::test_register_child_during_close_does_not_silently_succeed ? - loop = _make_dedicated_loop() - try: - # Run many short races to widen the window. - for _ in range(5000): - parent = promising.PromisingContext(loop=loop, parent=None) - - start = threading.Barrier(2) - outcome: dict[str, object] = {} - - def add_child() -> None: - start.wait() - try: - child = promising.PromisingContext(loop=loop, parent=parent) - outcome["child"] = child - except promising.ContextAlreadyClosedError as exc: - outcome["error"] = exc - - def close_parent() -> None: - start.wait() - parent.close_context() - - t1 = threading.Thread(target=add_child, daemon=True) - t2 = threading.Thread(target=close_parent, daemon=True) - t1.start() - t2.start() - t1.join(timeout=5) - t2.join(timeout=5) - - assert not t1.is_alive() and not t2.is_alive() - - if "child" in outcome: - child = outcome["child"] - # Invariant: a closed parent must not accept new children. - # If registration silently succeeded against a parent that - # is now closed, the framework's `closed() → no new - # children` contract has been broken by the race. - # TODO [TESTS] How do we know it happened AFTER the parent was - # closed ? I'm struggling to spot the part of test that - # insures things happened in that order and not the other way - # around - assert not parent.closed(), ( - "child registration silently succeeded after parent was closed — invariant violated" - ) - in_parent = child in parent.collect_unsettled_children( - whole_subtree=False, - awaitables_only=False, - ) - assert in_parent, ( - "registration silently succeeded but the child is missing from " - "the closed parent's _unsettled_children — torn write" - ) - finally: - loop.close() - - -# ── await_children race ────────────────────────────────────────── - - -async def test_await_children_during_concurrent_thread_registration() -> None: - """ - Inside an ``@promising.function``, a worker thread continuously - constructs child Promises (each ``Promise.__init__`` calls - ``parent._register_children(self)`` on the worker thread). The loop - thread, meanwhile, repeatedly ``await``s ``await_children()`` — - which iterates the same ``_unsettled_children`` set. - - With no lock, the loop-side iteration (``list(set)`` inside - ``collect_unsettled_children``) can raise - ``RuntimeError: Set changed size during iteration``, or - ``await_children`` can return while previously registered children - are still unsettled. - """ - - N_WRITERS = 4 - WRITES = 200 - - @promising.function - async def parent_func() -> None: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() - thread_errors: list[BaseException] = [] - thread_errors_lock = threading.Lock() - - async def _quick() -> int: - return 42 - - start_barrier = threading.Barrier(N_WRITERS + 1) - - def thread_writer() -> None: - try: - start_barrier.wait() - for _ in range(WRITES): - promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - except BaseException as exc: # noqa: BLE001 - with thread_errors_lock: - thread_errors.append(exc) - - writers = [threading.Thread(target=thread_writer, daemon=True) for _ in range(N_WRITERS)] - for w in writers: - w.start() - - # Release the writers and immediately race them with await_children. - start_barrier.wait() - - # Drain children many times to keep the race window open. - for _ in range(20): - await promising.await_children(whole_subtree=True) - - for w in writers: - w.join(timeout=10) - assert not w.is_alive() - - # Final drain. - await promising.await_children(whole_subtree=True) - - assert not thread_errors, thread_errors - - unsettled = active.collect_unsettled_children(whole_subtree=True, awaitables_only=True) - assert unsettled == set(), ( - f"await_children returned with {len(unsettled)} unsettled awaitable children remaining" - ) - - await parent_func() - - -# ── cascading unregister race ──────────────────────────────────── - - -def test_cascading_unregister_keeps_grandparent_set_consistent() -> None: - """ - A grandparent owns a middle-tier parent which owns many children. - All children close simultaneously: each unregistration triggers - ``_unregister_from_parent_if_time`` on the middle context, which - can cascade up to the grandparent. - - The cascading path reads ``_unsettled_children`` membership and - calls ``_unregister_children`` on the grandparent — without locking, - two children finishing in lockstep can both observe an empty middle - set and both try to unregister the middle from the grandparent. - """ - loop = _make_dedicated_loop() - try: - for _ in range(50): - grandparent = promising.PromisingContext(loop=loop, parent=None) - middle = promising.PromisingContext(loop=loop, parent=grandparent) - - N = 64 - children = [promising.PromisingContext(loop=loop, parent=middle) for _ in range(N)] - # Close middle *after* attaching children, so it stays in - # grandparent's set until all of its children drain. - middle.close_context() - - def make_closer(c: promising.PromisingContext): - def _close() -> None: - c.close_context() - - return _close - - errors = _run_workers([make_closer(c) for c in children]) - assert not errors, errors - - grand_children = grandparent.collect_unsettled_children( - whole_subtree=False, - awaitables_only=False, - ) - # After every middle-child has closed, middle should have - # cascaded out of grandparent's set exactly once. - assert middle not in grand_children, ( - "middle context still appears in grandparent's set even though all of its children are closed" - ) - finally: - loop.close() - - -# ── load-bearing scenario: real Promises under a parent Promise ── - - -async def test_concurrent_promise_creation_from_threads_registers_all() -> None: - """ - Many worker threads inside the same parent Promise create child - Promises concurrently. Every newly-created child must end up in - the parent's ``_unsettled_children`` set; none must be lost. - """ - - @promising.function - async def parent_func() -> int: - active = promising.get_active_promise() - loop = asyncio.get_running_loop() - - N_THREADS = 32 - CHILDREN_PER_THREAD = 20 - - created: list[promising.Promise] = [] - created_lock = threading.Lock() - - async def _quick() -> int: - return 0 - - def worker() -> None: - local = [] - for _ in range(CHILDREN_PER_THREAD): - child = promising.wrap_awaitable( - _quick(), - parent=active, - loop=loop, - start_soon=False, - ) - local.append(child) - with created_lock: - created.extend(local) - - with ThreadPoolExecutor(max_workers=N_THREADS) as ex: - await asyncio.gather(*[loop.run_in_executor(ex, worker) for _ in range(N_THREADS)]) - - expected = set(created) - actual = active.collect_unsettled_children(whole_subtree=False, awaitables_only=True) - missing = expected - actual - assert not missing, f"{len(missing)} child Promises lost from the parent's set" - - # Clean up the unused coroutines so asyncio doesn't warn. - for child in created: - child.cancel() - - return len(created) - - n = await parent_func() - assert n == 32 * 20 - - -# ── sanity (would-pass) test, included for self-verification ───── - - -@pytest.mark.skip(reason="Sanity reference; concurrency is fine when the set is touched from one thread.") -def test_single_thread_registration_keeps_all_children_sanity() -> None: - loop = _make_dedicated_loop() - try: - parent = promising.PromisingContext(loop=loop, parent=None) - children = [promising.PromisingContext(loop=loop, parent=parent) for _ in range(1000)] - actual = parent.collect_unsettled_children(whole_subtree=False, awaitables_only=False) - assert actual == set(children) - finally: - loop.close() From 7ee0af1dde92c242e42416d3f50978ee61e9353f Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Tue, 19 May 2026 22:36:36 +0300 Subject: [PATCH 18/20] fix a typo --- CONTRIBUTORS.md | 2 +- promising/promise.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 2f64ef1b0..4be02be59 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -68,7 +68,7 @@ This file also contains the `context` class — a context manager / decorator th - `_unpack_once()` — drives a single unpacking step. It enters the `with self:` block, awaits the wrapped `_awaitable`, and either records an intermediate `Promise` (via `_set_intermediate_promise`, transition to `_UNPACKED_ONCE`) or stores the final value/exception (via `_set_result` / `_set_exception`, transition to `_FINISHED`). This is the task `unpack_once()` waits on. - `_unpack_fully()` — drives the Promise to completion. It ensures the single-unpacking task is scheduled, awaits it, then walks the chain of intermediate Promises (`while isinstance(result, Promise): result = await result`) until a non-Promise value is reached, and records that value as the final result. This is the task `__await__` (and, indirectly, `sync()`) waits on. -Scheduling is driven by `_ensure_single_unpacking_scheduled()` and `_ensure_from_unpacking_scheduled()`, both of which create the underlying `loop.create_task(...)` lazily on first need. `__init__` schedules `_unpack_fully` when `start_soon` is `True`, so eager Promises start as soon as the loop is reachable; deferred Promises (`start_soon=False`) are scheduled the first time anyone consumes them (`__await__`, `sync()`, `unpack_once()`, `unpack_once_sync()`). +Scheduling is driven by `_ensure_single_unpacking_scheduled()` and `_ensure_full_unpacking_scheduled()`, both of which create the underlying `loop.create_task(...)` lazily on first need. `__init__` schedules `_unpack_fully` when `start_soon` is `True`, so eager Promises start as soon as the loop is reachable; deferred Promises (`start_soon=False`) are scheduled the first time anyone consumes them (`__await__`, `sync()`, `unpack_once()`, `unpack_once_sync()`). **Sync consumption.** `sync()` and `unpack_once_sync()` dispatch onto the Promise's own event loop via `asyncio.run_coroutine_threadsafe` and block the calling thread on the resulting `concurrent.futures.Future`. Both refuse to run on the Promise's loop thread (`_assert_no_sync_usage_deadlock` → `SyncUsageError`). diff --git a/promising/promise.py b/promising/promise.py index 9bea54bc0..2e7fa89f1 100644 --- a/promising/promise.py +++ b/promising/promise.py @@ -283,7 +283,7 @@ def __init__( self._set_exception(prefilled_exception) if self._start_soon and self._awaitable is not None: - self._ensure_from_unpacking_scheduled() + self._ensure_full_unpacking_scheduled() self._register_with_parent() @@ -332,7 +332,7 @@ def __await__(self) -> Generator[Any, None, T_co]: """ self._assert_awaiting_on_correct_event_loop() - self._ensure_from_unpacking_scheduled() + self._ensure_full_unpacking_scheduled() if self._full_unpacking_task is not None: yield from self._full_unpacking_task @@ -604,7 +604,7 @@ def _ensure_single_unpacking_scheduled(self) -> None: _unpacking_logger.log_single_unpacking_scheduled(promise=self) - def _ensure_from_unpacking_scheduled(self) -> None: + def _ensure_full_unpacking_scheduled(self) -> None: _unpacking_logger.log_full_unpacking_scheduling(promise=self) if self._full_unpacking_task is None and not self.done(): From d5e39b006bb3c657cee42964756befde91b7077d Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Tue, 19 May 2026 22:44:13 +0300 Subject: [PATCH 19/20] RACE_CONDITION_INVARIANTS.md --- RACE_CONDITION_INVARIANTS.md | 443 +++++++++++++++++++++++++++++++++++ 1 file changed, 443 insertions(+) create mode 100644 RACE_CONDITION_INVARIANTS.md diff --git a/RACE_CONDITION_INVARIANTS.md b/RACE_CONDITION_INVARIANTS.md new file mode 100644 index 000000000..3ef82a4c3 --- /dev/null +++ b/RACE_CONDITION_INVARIANTS.md @@ -0,0 +1,443 @@ +# Race-Condition Invariants for the `promising` Framework + +A catalogue of properties that must hold even when multiple threads / tasks +hit the same `Promise` or `PromisingContext` concurrently. Each entry is +shaped so a test can target it: **invariant → why it can break → suggested +stressor**. + +The framework intentionally exposes cross-thread access (sync functions in +a `ThreadPoolExecutor`, `.sync()` / `await_children_sync()` from arbitrary +threads, cross-thread cancellation, etc.), so most of these invariants are +*not* protected by the single-event-loop assumption — they need real +locking, atomic reads/writes, or careful state-machine design. + +--- + +## 1. `Promise` state machine + +The legal transitions are: + +``` +_PENDING ──► _UNPACKED_ONCE ──► _FINISHED + │ │ + │ └──► _CANCELLED_AFTER_UNPACKED_ONCE + └──► _CANCELLED_BEFORE_UNPACKED_ONCE + └──► _FINISHED (exception before unpack) +``` + +### 1.1. Terminal states are absorbing +Once `_state` is `_FINISHED`, `_CANCELLED_BEFORE_UNPACKED_ONCE`, or +`_CANCELLED_AFTER_UNPACKED_ONCE`, no further transition is allowed. +- **Break vector:** `_set_result` / `_set_exception` / + `_set_intermediate_promise` racing each other from the single-unpack + task, the full-unpack task, `cancel()`'s synthesize path, and + `_unpacking_task_done_callback`. +- **Stressor:** schedule `cancel()` from N threads at random points during + `_unpack_once`/`_unpack_fully`; assert final state is one of the legal + terminals and matches the first writer's intent. + +### 1.2. No "spontaneous" transition out of `_PENDING` +`_state` only leaves `_PENDING` via a `_set_*` call that observed +`_state is _PENDING` (or `_UNPACKED_ONCE`) under the framework's own +serialisation. There must be no path where two writers both observe +`_PENDING`, both proceed, and both mutate state. +- **Break vector:** `_set_intermediate_promise` checks `is _PENDING` then + writes `_intermediate_promise` then writes `_state`. A concurrent + `_set_exception(CancelledError)` could interleave between the check and + the writes. +- **Stressor:** monkey-patch `_set_state` to `await asyncio.sleep(0)` + between assignment and `close_context()`; trigger cancellation in that + window. + +### 1.3. `done()`, `cancelled()`, `unpacked_once()` are monotonic +`done()` may only flip `False → True`. Same for `cancelled()` once True, +and for `unpacked_once()` once True. +- **Break vector:** an in-progress `_set_*` that fails midway through (the + state-machine assertion raises) calls `_force_internal_error_finish`, + which re-writes `_state = _FINISHED` and `_exception`. If + `done()` was already True via a different path the second write must + not regress any predicate. +- **Stressor:** poll `done()` / `cancelled()` from a hot loop on another + thread while the Promise resolves; assert no observed regression. + +### 1.4. Predicate coherence +At any single read, the *combination* of `done()` / `cancelled()` / +`unpacked_once()` must be consistent with one of the five legal +`_state` values. Equivalently: `done() implies unpacked_once_or_done()`; +`cancelled() implies done()`; if `_intermediate_promise is None` then +`unpacked_once() implies (exception is not None or _state is _FINISHED)`. +- **Break vector:** `_set_state` writes `_state` then calls + `close_context()` (non-atomic with the surrounding `_result` / + `_exception` / `_intermediate_promise` write). +- **Stressor:** read all five accessors back-to-back from another thread + while resolution is in flight; check the snapshot maps to one valid + state row in a truth table. + +### 1.5. `_result` is set iff `_state is _FINISHED` and `_exception is None` +`result()` must never raise `RuntimeError("Promise result is UNCHANGED…")` +in a healthy run. The internal `_result == UNCHANGED` guard in `result()` +is a tripwire for a state/result write reordering. +- **Stressor:** call `.result()` on a hot loop while many concurrent + consumers `await` the same Promise. + +### 1.6. `_exception` and `_result` are mutually exclusive +After `done()`, exactly one of `_exception` / `_result` is the "real" +value. Specifically: `_exception is not None` ⇔ `_result is UNCHANGED` ⇔ +`result()` raises. Holds even across the `_force_internal_error_finish` +path. + +--- + +## 2. Unpacking-task lifecycle + +### 2.1. At most one `_single_unpacking_task` ever exists per Promise +`_ensure_single_unpacking_scheduled` is the only place that creates one +and gates on `_single_unpacking_task is None and not unpacked_once_or_done()`. +Two callers must never both pass that gate. +- **Break vector:** `unpack_once()` is async (runs on the loop thread, no + preemption between Python statements) — but it can also be triggered + indirectly via `unpack_once_sync()` from another thread, which uses + `run_coroutine_threadsafe` to dispatch back to the loop, and via + `_unpack_fully` calling `_ensure_single_unpacking_scheduled` after an + `await`. +- **Stressor:** N coroutines each `await promise.unpack_once()` on the + same Promise concurrently; assert exactly one `_single_unpacking_task` + object identity ever existed. + +### 2.2. At most one `_full_unpacking_task` ever exists per Promise +Symmetric to 2.1; gated by `_full_unpacking_task is None and not done()`. +- **Stressor:** N consumers `await promise` + M threads call + `.sync()` simultaneously on the same Promise; assert single full-task + identity. + +### 2.3. The two task slots, once written, are never overwritten +`_single_unpacking_task` / `_full_unpacking_task` are written exactly +once, in the `None → Task` direction. + +### 2.4. A Task scheduled by `_ensure_*_scheduled` must run its done-callback +`_unpacking_task_done_callback` is the only thing that catches the +"cancelled between `create_task` and the first `__step`" race. It must +fire even if the loop is being torn down — otherwise the Promise stays +non-terminal. + +### 2.5. `unpacked_once_or_done()` flips True before the +`_single_unpacking_task` becomes `done()` +The body of `_unpack_once` calls `_set_*` *before* returning. Any +consumer that observes `_single_unpacking_task.done()` must already see +`unpacked_once_or_done()`. + +--- + +## 3. Cancellation + +### 3.1. A single `cancel()` call drives the Promise terminal +After `cancel()` returns `True`, eventually `done()` becomes `True`. If +no underlying task is running, `cancel()` makes that transition happen +synchronously via `_synthesize_cancellation`. There must be no path +where `cancel()` returns `True` but the Promise stays `_PENDING` +forever. + +### 3.2. Concurrent `cancel()` calls don't corrupt state +N threads call `.cancel()` simultaneously. Outcome: +- exactly one stored exception (the first one wins; later + `CancelledError`s are silently dropped per `_set_exception`) +- `_state` is one of `_CANCELLED_BEFORE_UNPACKED_ONCE` / + `_CANCELLED_AFTER_UNPACKED_ONCE` +- the Promise is unregistered from its parent exactly once. + +### 3.3. `cancel()` racing with natural completion +If `cancel()` lands the same instant the body completes successfully, +the final state is *either* `_FINISHED` with a result *or* a cancelled +state — never both, never neither, and never `_FINISHED` with a stored +`CancelledError` masquerading as a real exception. +- **Break vector:** `_set_result` is called by the body, `cancel()` then + synthesises a `CancelledError` after `done()` already flipped. + `_set_exception` has an explicit branch for "already-terminal + + CancelledError → drop". Verify that branch covers every interleaving. + +### 3.4. `_synthesize_cancellation` always closes the context +The comment in `_synthesize_cancellation` is load-bearing: without +`close_context()` the child never unregisters. Invariant: every code +path that lands in `_CANCELLED_*` must have run `close_context()` +at least once. +- **Stressor:** cancel a `start_soon=False` Promise that was never + awaited; assert it unregisters from its parent immediately. + +### 3.5. Cancel of parent does NOT cancel nested ("returned") Promise +The TODO around `_unpack_fully` notes this intentional design. The +inner Promise's task keeps running independently. Test that cancelling +the outer leaves the inner's `_state` untouched. + +--- + +## 4. Parent / child hierarchy (`_unsettled_children`) + +### 4.1. A child is registered exactly once and unregistered exactly once +Independent of how many tasks/threads race through its construction and +resolution. +- **Break vector:** `_register_with_parent` runs at the end of + `__init__` (after super init). For a sync `@promising.function` + promise, that body runs on a worker thread while the parent's + `_unsettled_children.update(...)` mutates a `set` from a non-loop + thread. +- **Stressor:** spawn K sync child promises in parallel inside one + parent's body; after parent finishes assert `len(_unsettled_children) == 0` + and that the set never raised `RuntimeError: Set changed size during + iteration` during a concurrent `collect_unsettled_children`. + +### 4.2. `_unsettled_children` is never read inconsistently +`collect_unsettled_children` builds a `list[…](self._unsettled_children)` +from a `set`. On CPython this snapshot is safe only because the GIL +serializes `set.__iter__`'s C-level iteration with `set.add` / `set.discard` +*at most* — there is still a documented `RuntimeError: Set changed size +during iteration` window if the iteration is interleaved at the Python +level (i.e. across more than one bytecode boundary). +- **Stressor:** repeatedly call `collect_unsettled_children(whole_subtree=True)` + while children are being registered/unregistered from worker threads; + assert no exception escapes. + +### 4.3. `closed()` parents reject new children +`_register_children` raises `ContextAlreadyClosedError` if `closed()` is +True. Invariant: there is no interleaving in which a child gets added to +a parent *after* the parent's `close_context()` ran. +- **Break vector:** `close_context()` sets `_context_closed = True` then + unregisters. A child being constructed concurrently can read + `_context_closed == False` between those statements? It runs + before — but child registration happens *after* parent unregister + attempt, so a worker thread mid-construction could miss the flag flip. +- **Stressor:** start a parent Promise that finishes very fast, fire off + many sync grandchildren from a worker thread mid-finish, assert each + either succeeds *or* gets a clean `ContextAlreadyClosedError`. + +### 4.4. `await_children()` does terminate when all descendants settle +The outer `while children := …` loop terminates iff no descendant ever +escapes registration. Tests should provoke deeply nested cross-thread +spawns and confirm `await_children()` always returns in finite time. +- **Break vector:** a grandchild registers with its parent *after* the + parent's set was last sampled by `collect_unsettled_children`, but + *before* the parent finished `done()`. The outer loop should still + pick it up on the next iteration. Verify with adversarial scheduling. + +### 4.5. Parent's `_unregister_from_parent_if_time` is correct under races +The guard `done() and not _unsettled_children` is checked +non-atomically. If a child finishes (calls +`parent._unregister_children(self)` → `parent._unregister_from_parent_if_time`) +exactly as a new grandchild is added, the parent must not unregister +prematurely and orphan the grandchild. + +### 4.6. Re-entry protection +`PromisingContext.__enter__` checks `_previous_token is not None` and +`_context_closed`. A context must never be successfully entered twice +concurrently from different tasks/threads (the framework comment at +`promising_context.py:749` flags this as a known race). + +--- + +## 5. `__active_context` ContextVar + +### 5.1. After every balanced `with ctx:` block, the active context is restored +Even if the block ran on a worker thread (with `contextvars.copy_context()` +propagation), or if multiple async tasks race over the same +`PromisingContext` instance. + +### 5.2. No leak across thread-pool boundaries +A sync `@promising.function` body executes in a thread-pool thread with +the active-context ContextVar set via `ctx.run(...)`. The worker thread's +default contextvars must be unaffected when the next executor task uses +the same worker thread. +- **Stressor:** submit N sync functions to a 1-worker pool; between + jobs, assert `PromisingContext.get_active_context(raise_if_none=False)` + on that worker thread is `None`. + +### 5.3. Concurrent enters on distinct contexts in the same task don't lose tokens +Stacked `with` blocks must restore in LIFO order even if intermediate +asynchronous suspensions happened. + +--- + +## 6. Cross-thread sync APIs + +### 6.1. `promise.sync()` from many threads on the same Promise returns the same value +Each caller drives `run_coroutine_threadsafe(awaitable_as_coroutine(self), +self.loop)` and blocks. Invariant: every caller observes the cached +result; the underlying function executes exactly once. +- **Stressor:** N threads `.sync()` the same not-yet-started + (`start_soon=False`) Promise; assert the body ran exactly once and all + returned the same value. + +### 6.2. `.sync()` deadlock guard fires under every interleaving +`_assert_no_sync_usage_deadlock` raises `SyncUsageError` if called on +the loop thread. There must be no race where the check observes "not on +loop" but the actual `concurrent_future.result()` blocks the loop. + +### 6.3. `unpack_once_sync()` fast-path is consistent +The fast-path returns directly when `unpacked_once_or_done()`. If two +threads call `unpack_once_sync()` and one takes the slow path, both must +end up observing the same `_intermediate_promise` (or final value / +exception). + +### 6.4. `await_children_sync()` is a faithful sync mirror of `await_children()` +Driven via `run_coroutine_threadsafe` — must terminate iff the async +version would; must not deadlock when the loop is healthy. + +--- + +## 7. Eager scheduling (`start_soon=True`) + +### 7.1. `start_soon=True` schedules the full-unpack task before `__init__` returns +For `start_soon=True`, the call to `_ensure_full_unpacking_scheduled` +happens before `_register_with_parent`. Invariant: by the time the +parent sees the child in `_unsettled_children`, the child's task is +already on the loop. + +### 7.2. Eager + deferred mix doesn't drop work +A parent with `children_start_soon=False` whose body spawns 100 children +and then exits without awaiting them must still see all children in its +`_unsettled_children`, and the parent's `await_children()` (called by +`protected_run`) must execute every one exactly once. + +### 7.3. `_resolve_start_soon` snapshots are not torn +The decision tree in `_resolve_start_soon` reads +`parent_context._children_start_soon`. The parent's settings are +"frozen at creation time" per README §"Settings Are Frozen at Creation +Time". A child being constructed concurrently with the parent's +constructor finishing must still read fully-initialized parent settings. + +--- + +## 8. Settings inheritance / global `Defaults` + +### 8.1. A Promise's resolved settings are immutable after construction +After `__init__` returns, `_start_soon`, `_start_soon_default`, +`_children_start_soon`, `_collapse_tracebacks`, `_thread_pool`, `_loop`, +`_parent` never change. Tests should assert this under concurrent +mutation of `Defaults.*` and concurrent context entry. + +### 8.2. Mutating `Defaults.START_SOON` mid-flight does not retroactively change behaviour +A test flips `Defaults.START_SOON` from `True` to `False` while a tree +of promises is being constructed and resolved; the already-created +promises continue with their captured value, while new ones reflect the +new default. + +### 8.3. `Defaults.PROMISING_THREAD_POOL` swap is safe +Replacing the global pool while promises are running must not strand +any submitted callable. + +--- + +## 9. Excepthook installation + +### 9.1. `install_promising_tracebacks()` is idempotent +Called inside `_unpack_once` on every first run, often from many +promises in parallel. Must produce stable `sys.excepthook` / +`threading.excepthook` values regardless of interleaving. +- **Stressor:** create thousands of root-level promises in parallel; + capture `sys.excepthook` at the end; assert it equals the installed + promising excepthook (not the default and not double-wrapped). + +--- + +## 10. Cross-event-loop guards + +### 10.1. `_assert_awaiting_on_correct_event_loop` always sees the right loop +When a Promise's `loop` was inherited from its parent and the parent's +loop is no longer running (or a different loop is current), `await +promise` raises `EventLoopMismatchError` *deterministically* — not +"sometimes hangs, sometimes raises". + +### 10.2. A Promise created on one loop, awaited on another, raises +Even when both loops are alive simultaneously on different threads. + +--- + +## 11. Result-caching idempotence + +### 11.1. The wrapped callable runs at most once per Promise +Even under N concurrent `await` + `.sync()` + `unpack_once()` consumers. +- **Stressor:** wrap a callable that increments a counter; saturate it + with all four consumption APIs from multiple threads; assert + counter == 1. + +### 11.2. Identity invariant: `await p` and `p.sync()` and `p.result()` +return the *same* object (not just equal) +For non-Promise return values, the cached `_result` is returned by +identity. Concurrent consumers must all get `is`-equal results. + +--- + +## 12. `intermediate_promise()` visibility + +### 12.1. After `unpacked_once_or_done()` returns True, `intermediate_promise()` +either returns the stored intermediate Promise or raises the stored +exception +There must be no race where `unpacked_once_or_done()` is True but +`intermediate_promise()` raises `PromiseNotUnpackedError`. + +### 12.2. The intermediate Promise's parent linkage is set before it is reachable +The intermediate Promise is created inside the outer Promise's +`with self:` block, so its parent is the outer. Concurrent +`get_trace()` on the inner must see the outer as ancestor immediately. + +--- + +## 13. Tracing / observability + +### 13.1. `get_trace()` is consistent +A concurrent reader walking parents via `get_parent_context()` always +sees an acyclic chain ending at a root. Even during mass +register/unregister churn, no cycle ever appears, and `get_trace()` +terminates. + +### 13.2. `format_trace()` / `print_trace()` don't raise on a context being +unregistered concurrently +`__repr__` reads `self.namespace` and `id(self)` — both immutable — +so the only race surface is the parent chain walk in `get_trace`. Fuzz +test confirming. + +--- + +## 14. Memory / leak invariants + +### 14.1. After `await promise` returns, the awaitable is released +The promise holds `_awaitable`, which after consumption serves no +purpose. Verify (with `weakref`) that the awaitable's referents become +collectable promptly after settlement, including the `cancel()`-pre-start +path that calls `awaitable.close()` in `_synthesize_cancellation`. + +### 14.2. After a parent fully resolves and `await_children()` returns, +`_unsettled_children` is empty +And the parent itself is unregistered from *its* parent. Otherwise the +tree leaks across runs. + +--- + +## 15. Frame-summary capture race + +### 15.1. `frame_summary_tuple` reflects the constructor's caller +`traceback.walk_stack` runs on the constructing thread before +`super().__init__` returns. If the constructor is called from a thread +pool worker, the captured stack must be that worker's stack (not the +loop's). Test by spawning from a sync function and inspecting frames. + +--- + +## Practical guidance for tests + +- Use `asyncio.sleep(0)` injected via monkey-patch into the framework's + state-transition methods (`_set_state`, `_set_exception`, + `_register_with_parent`, `close_context`) to widen race windows + deterministically. +- Pair every "happy path" race test with an "exception path" twin: the + most likely state-machine corruption sites are the `try/except + BaseException` blocks that funnel into + `_force_internal_error_finish`. +- For thread-pool stress, prefer a custom small pool (1–2 workers) and + many submissions — that maximises contention on `_unsettled_children` + and `__active_context`. +- For cancellation stress, alternate `cancel()` callers between the loop + thread and a non-loop thread, and target the four phases (before + schedule, after schedule but before first `__step`, mid-await, after + unpack_once). +- For each invariant, assert both the *terminal* property (final state + is legal) and the *transient* property (no intermediate observation + violated a monotonicity / coherence rule). From b3738130bba17cf0bbebf48caa629fa4ba902e45 Mon Sep 17 00:00:00 2001 From: Oleksandr Tereshchenko Date: Wed, 20 May 2026 09:28:13 +0300 Subject: [PATCH 20/20] two versions of RACE_CONDITION_INVARIANTS.md --- RACE_CONDITION_INVARIANTS_A.md | 414 ++++++++++++++++++ ...IANTS.md => RACE_CONDITION_INVARIANTS_B.md | 0 2 files changed, 414 insertions(+) create mode 100644 RACE_CONDITION_INVARIANTS_A.md rename RACE_CONDITION_INVARIANTS.md => RACE_CONDITION_INVARIANTS_B.md (100%) diff --git a/RACE_CONDITION_INVARIANTS_A.md b/RACE_CONDITION_INVARIANTS_A.md new file mode 100644 index 000000000..a7e5f2ef1 --- /dev/null +++ b/RACE_CONDITION_INVARIANTS_A.md @@ -0,0 +1,414 @@ +# Race-Condition Invariants for Promising + +A catalogue of properties that must hold under arbitrary thread/event-loop +interleavings. Each invariant is phrased as a postcondition that a test can +assert after exercising a chosen race window. The list is intended as a +checklist for building a `tests/race_conditions/` suite that hammers each +invariant with many concurrent actors, repeats, and (where useful) +`hypothesis` schedules. + +Scope notes: + +- Promising explicitly targets the GIL-backed CPython interpreter (see the + thread-safety contract on `Promise.done()`). Single-attribute reads/writes + are assumed atomic; tests should still cover the *ordering* guarantees, not + the per-attribute atomicity. +- All "from any thread" invariants must be checked from at least three + vantage points: the Promise's own event-loop thread, a thread-pool worker + belonging to the same loop, and an unrelated (foreign) thread. + +--- + +## 1. Promise state machine + +### 1.1 Monotonicity +Once `_state` advances past `_PENDING`, it never moves backwards. Allowed +transitions only: + +- `_PENDING → _UNPACKED_ONCE` +- `_PENDING → _FINISHED` +- `_PENDING → _CANCELLED_BEFORE_UNPACKED_ONCE` +- `_UNPACKED_ONCE → _FINISHED` +- `_UNPACKED_ONCE → _CANCELLED_AFTER_UNPACKED_ONCE` + +Test: spin N threads that repeatedly snapshot `_state`. Across the entire run +no thread observes a regression, and the recorded transition (sorted by wall +time) matches one of the allowed pairs. + +### 1.2 Single terminal state +A Promise reaches exactly one terminal state (`_FINISHED`, +`_CANCELLED_BEFORE_UNPACKED_ONCE`, or `_CANCELLED_AFTER_UNPACKED_ONCE`) — and +reaches it at most once. Repeated calls to `_set_result_from_loop` / +`_set_exception_from_loop` / `_set_intermediate_promise_from_loop` after a +terminal state never re-advance it. + +### 1.3 Writer/reader ordering (the contract behind `done()`) +For every state advance, the matching attribute is observable to readers +*before* the state flip: + +- `_state == _UNPACKED_ONCE` ⇒ `_intermediate_promise is not None`. +- `_state == _FINISHED` and `_exception is None` ⇒ `_result is not UNCHANGED`. +- `_state in (_FINISHED, _CANCELLED_*)` and `_exception is not None` ⇒ + `_exception` is fully populated (not a partial assignment). + +Test: thread A drives the Promise to completion; thread B busy-loops calling +`done()` and, the *instant* it sees `True`, calls `result()` / +`exception()` / `intermediate_promise()` without yielding. Those calls must +never raise `PromiseNotDoneError`, `PromiseNotUnpackedError`, nor the +`RuntimeError("Promise result is UNCHANGED…")` fallback in `result()`. + +### 1.4 Predicate consistency under concurrent reads +At any instant, the predicate triple +`(done(), unpacked_once(), unpacked_once_or_done())` is consistent with one +single underlying `_state` value. No reader ever sees a combination such as +`done()==True and unpacked_once_or_done()==False`. + +### 1.5 Result/exception caching is one-shot +For any awaitable handed to `Promise(...)`, the awaitable's `__await__` (or +equivalent) is driven exactly once across all concurrent consumers +(`await`, `sync`, `unpack_once`, `unpack_once_sync`, and any mix of them +fired in parallel). A counter inside the awaitable must read `1` after the +storm. + +### 1.6 Consumers all observe the same cached value +N consumers from M threads — `await`, `sync()`, `unpack_once_sync()` — that +finish without exception receive `is`-identical results (when the result is +a non-primitive object). When the Promise finished with an exception, every +consumer sees the *same* exception instance (`is`-identical). + +### 1.7 No "task created twice" +`_full_unpacking_task` is assigned exactly once; same for +`_single_unpacking_task`. Concurrent calls to +`_ensure_from_loop_full_unpacking_scheduled` / +`_ensure_from_loop_single_unpacking_scheduled` from rapid-fire await/sync +storms never produce two Tasks for the same role. (This invariant relies on +those methods only ever running on the Promise's own loop — that itself is +checked by invariant 4.1.) + +--- + +## 2. Parent–child hierarchy (`_unsettled_children`) + +### 2.1 No lost child +For every `Promise` / `PromisingContext` created with a parent, between +construction and the parent's eventual settling, the child appears in the +parent's `_unsettled_children` at least once. A test that spawns K children +concurrently while another thread snapshots +`parent._unsettled_children` must, in aggregate, witness every child id. + +### 2.2 No stuck child +After the entire subtree is done, every ancestor's `_unsettled_children` is +empty. In particular, +`root.collect_unsettled_children(whole_subtree=True)` returns `set()` once +all leaves have settled, regardless of the order in which threads drove +them. + +### 2.3 No double-register / double-unregister +The child appears in `parent._unsettled_children` at most once at any +moment. After unregister, it does not reappear. A concurrent stream of +`_register_children_threadsafe` / `_unregister_children_threadsafe` calls +must keep the set's element count consistent with a serial schedule +(check by counting `add`/`remove` operations against final size). + +### 2.4 Iteration safety +Iterating `collect_unsettled_children` from one thread while another thread +register/unregisters children must never raise (`RuntimeError: Set changed +size during iteration`), and the returned snapshot must be a *consistent* +subset of `add`s up to some serializable point (no torn reads). The lock +inside `collect_unsettled_children` enforces this; the test is to flood the +set and assert no exception is raised in 1e5 iterations. + +### 2.5 `closed()` after `close_context_threadsafe()` from another thread +After `close_context_threadsafe()` returns on thread A, any thread B's +subsequent call to `register_children_threadsafe` raises +`ContextAlreadyClosedError`. There must be no window in which `closed()` +returns `True` on one thread while another thread's `register` call slips +through and adds a child. + +### 2.6 Late child cannot enter a closed context +A child whose `__init__` overlaps with the parent's +`close_context_threadsafe()` either (a) registers successfully and is later +properly drained, or (b) raises `ContextAlreadyClosedError`. There is no +third outcome — in particular the child must never be silently dropped while +its constructor still considered the parent alive. + +### 2.7 Unregister ordering +`_unregister_from_parent_if_time` only unregisters when +`done() and not _unsettled_children`. Under concurrency, no parent ever sees +its child unregister while the child still has unsettled descendants. Test +by polling `parent._unsettled_children` and, for each child observed there, +asserting that either the child is not yet done, or it itself has unsettled +descendants. The set difference must always be empty. + +### 2.8 No registration on a torn parent +A child's `_parent` pointer is set in `__init__` before the registration +call. Snapshotted in any reader thread, `child._parent` is always either +the originally captured value or `None` (when explicitly created as a root) +— never an unrelated context. + +--- + +## 3. Cancellation + +### 3.1 `cancel()` is thread-safe and never deadlocks +`Promise.cancel()` invoked simultaneously from K foreign threads against +the same Promise completes in bounded time, returns a deterministic result +(at most one `True`, rest `False` once the Promise is terminal), and never +deadlocks regardless of which thread the event loop runs on. + +### 3.2 Cancellation always yields a terminal state +Every successful `cancel()` (`returned True` for at least one caller) +results in `done() == True and cancelled() == True` eventually (bounded by +a generous timeout). The Promise never lingers in a non-terminal state when +the loop is alive. This includes the corner case where +`task.cancel()` lands between `create_task` and the first `__step` — the +`_unpacking_task_done_callback` synthesize-path must drive the Promise to +terminal. + +### 3.3 Result vs cancel race +If a Promise's underlying awaitable resolves in the same time window as a +foreign-thread `cancel()`, the outcome is *one of*: + +- terminal `_FINISHED` with the produced result/exception, `cancel()` did + not flip the state; or +- terminal `_CANCELLED_BEFORE_UNPACKED_ONCE` / `_CANCELLED_AFTER_UNPACKED_ONCE`, + with a `CancelledError` stored as `_exception`. + +The result must never be: torn state, two terminal states recorded, or a +mix of "result stored" + "exception stored". + +### 3.4 Idempotent cancellation +Repeated `cancel()` calls on an already-cancelled Promise return `False` +and do not raise. A `CancelledError` arriving on an already-terminal +Promise via `_set_exception_from_loop` is silently dropped (per docstring). +A non-CancelledError arriving on a terminal Promise raises `RuntimeError` +(framework bug detector) — tests should assert this never happens under +normal user-driven races. + +### 3.5 Wake-up of waiters +After `cancel()` from a foreign thread, every consumer blocked on `await`, +`.sync()`, or `unpack_once_sync()` is unblocked within a bounded time and +sees `CancelledError` (or its `__cause__` chain) — none hang indefinitely. + +### 3.6 Cancellation propagates only as documented +Cancelling a *parent* Promise does **not** cancel a nested (returned-from) +Promise that the parent is currently awaiting (per the inline TODO in +`_fully_unpack_from_loop`). Tests should pin this current behaviour so any +future change is intentional. + +### 3.7 Awaitable cleanup on synthesize-cancel +When `cancel()` synthesizes a `CancelledError` for a never-started Promise, +the wrapped coroutine is `close()`d exactly once (no "coroutine was never +awaited" warning across many runs). + +--- + +## 4. Event-loop discipline + +### 4.1 "From loop only" methods stay on the loop +Methods documented as "can only be used from the event loop of the Promise" +(`_ensure_from_loop_*`, `_unpack_once_from_loop`, +`_fully_unpack_from_loop`, `_set_*_from_loop`, `_cancel_from_loop`, +`_synthesize_cancellation_from_loop`) must never be invoked from a foreign +thread. Test: install a thread-id assertion at the top of each (a +test-only monkeypatch is fine) and run the full race suite — the assertion +must never trigger. + +### 4.2 `SyncUsageError` is raised, not deadlock +Calling `promise.sync()`, `promise.unpack_once_sync()`, or +`await_children_sync()` from the Promise's own event-loop thread raises +`SyncUsageError` *immediately*, even when the call lands in the same +microsecond window as a foreign-thread `cancel()` or `await`. No deadlock, +no spurious success. + +### 4.3 `start_soon=True` scheduling +A Promise created with `start_soon=True` from a non-loop thread eventually +schedules its full-unpacking task on the correct loop — even when the +constructing thread immediately drops its reference. Test: construct 10k +Promises in a thread pool, all targeting the same loop, then assert all +reach a terminal state. + +### 4.4 No leaked task references on prefilled / never-awaited Promises +A prefilled Promise (`prefilled_result=…` or `prefilled_exception=…`) never +constructs `_full_unpacking_task` or `_single_unpacking_task`. A Promise +constructed with `start_soon=False` that is then cancelled before any +`await` also leaves both task attributes as `None`. Holds across concurrent +constructor/cancel races. + +### 4.5 Loop-mismatch detection is race-free +`await promise` from a different running loop than `promise.loop` raises +`EventLoopMismatchError` synchronously. Holds even when the Promise's own +loop is concurrently mutating the Promise's state. + +--- + +## 5. `await_children()` under churn + +### 5.1 Eventual quiescence +`await_children(whole_subtree=True)` returns once all descendants are +done, even if children spawn new grand-children during the wait. Tests +should fan out a tree where leaves themselves spawn new leaves on +resolution, and assert the call still returns. + +### 5.2 No surprise hang from non-awaitable contexts +A non-awaitable `PromisingContext` child does not stall its parent's +`await_children()` — they are filtered out by `awaitables_only=True`. +Tests should mix `promising.context` siblings with `Promise` siblings and +assert quiescence. + +### 5.3 Exceptions in children do not interrupt the wait +`await_children` uses `return_exceptions=True`. If half of N concurrent +children fail and half succeed, the call still completes and the parent +sees its `_unsettled_children` drained. + +### 5.4 Sync counterpart cannot deadlock +`await_children_sync()` from the event-loop thread raises +`SyncUsageError`. From a foreign thread it completes within a bounded +timeout for any subtree that would have completed under `await_children`. + +### 5.5 `unpack_promises_fully=False` lets parents return early +With `unpack_promises_fully=False`, the call returns as soon as every +direct child has reached `unpacked_once_or_done()` — even if their full +unpacking is still in flight. Subsequent reads of those children must +still show monotonic state progression (invariant 1.1) without crashing. + +--- + +## 6. `ContextVar` activation (`__active_context`) + +### 6.1 Per-task isolation +Two coroutines running on the same loop, each inside their own +`with ctx:` block, see their own context as active when they call +`get_active_context()`. The `ContextVar` must not leak across tasks. + +### 6.2 Activation is non-reentrant +Entering the same `PromisingContext` instance twice (concurrently or +sequentially) raises `ContextAlreadyActiveError`. Under concurrent +attempts, exactly one `__enter__` succeeds and the rest raise; the +successful one's `__exit__` correctly restores the previous token. + +### 6.3 Cross-thread context inheritance +A sync promising function (with `use_thread_pool=True`) launched in a +worker thread sees its parent Promise as the active context (per the +`contextvars.copy_context()` call in `PromisingFunction._call_wrapped`). +Holds even when the parent Promise's own state is concurrently changing. + +### 6.4 No active-context bleed across worker invocations +A thread-pool worker that finishes one sync promising function and is +reused for an unrelated callable observes no leftover `__active_context` +from the previous job. + +--- + +## 7. Exception attachment (`try_to_link_exception`) + +### 7.1 Deepest context wins, exactly once +`exception.__promising_context__` is set by the deepest context whose +`with` block sees the exception, and not overwritten by ancestors. Holds +under concurrent re-raise paths (e.g. multiple sibling Promises failing +simultaneously, each running on its own thread-pool worker). + +### 7.2 No torn attribute writes +A reader thread that observes `__promising_context__` on an exception +also observes a matching `__promising_collapse_traceback__` boolean. The +two attributes never appear in a half-set state. + +--- + +## 8. Settings frozen at creation + +### 8.1 Defaults snapshot +A Promise's resolved settings (`_start_soon`, `_start_soon_default`, +`_children_start_soon`, `_collapse_tracebacks`, `_thread_pool`) are +captured during `__init__` and never change afterwards. Mutating +`promising.Defaults.*` from another thread during construction either +takes effect for that specific Promise (because `__init__` read the +default before the mutation) or does not — but the Promise's snapshot is +internally consistent: every getter on it returns the same value +throughout its lifetime. + +### 8.2 No cross-promise leak via parent inheritance +When child Promise A inherits a setting from parent P at construction +time, and concurrently child Promise B is constructed from P, mutating +P's setting between the two (where API allows) does not retroactively +change A's resolved value. + +--- + +## 9. Thread-pool dispatch + +### 9.1 Correct executor used +A sync promising function with `use_thread_pool=True` runs on the +executor returned by `get_thread_pool_executor()` of its active Promise, +regardless of which thread called the wrapper. Test by tagging worker +threads per executor and asserting the function body ran on the +expected pool. + +### 9.2 No starvation deadlock from sibling sync calls +Multiple sibling sync promising functions submitted to the same +bounded-size `ThreadPoolExecutor`, each performing `.sync()` on another +*non-overlapping* sibling, all complete. (Mutual `.sync()` between two +siblings that contends on the same pool is a user-side deadlock — that +is documented behaviour, not an invariant; tests should isolate it.) + +--- + +## 10. `wrap_awaitable` / construction races + +### 10.1 Bare-coroutine wrapping is concurrency-safe +`wrap_awaitable(coro)` invoked from K threads with K different coroutines +never produces a Promise that is associated with the wrong coroutine, +nor a Promise whose `_awaitable is None` despite an awaitable being +passed. + +### 10.2 Validation runs before parent registration +`Promise.__init__` validates arguments before calling `super().__init__`. +Therefore, on validation failure, the parent's `_unsettled_children` +must not contain the would-be child. Test: feed bad arguments +concurrently with valid sibling construction; the parent's set never +contains a `Promise` instance that later raised in `__init__`. + +--- + +## 11. `Defaults` mutation under load + +### 11.1 No torn Promise from a flipping `START_SOON` +A test thread flips `Defaults.START_SOON` between `True` and `False` in a +tight loop while another thread constructs Promises. Every Promise either +has `_start_soon == True` (and was scheduled) or `_start_soon == False` +(and was not). No Promise is left in an in-between state where it was +scheduled but `_start_soon` reads `False`, or vice versa. + +--- + +## 12. Sentinel safety + +### 12.1 `Sentinel.__bool__` is unreachable from internal code +Under any of the races above, no internal code path produces a +`SentinelUsageError` (which would indicate the framework itself +truthiness-tested a sentinel). Tests should set up exception capture and +assert zero `SentinelUsageError` instances across the run. + +--- + +## Suggested test harness primitives + +Useful building blocks for the `tests/race_conditions/` suite: + +- **`spin_until(predicate, timeout)`** — a tight `while not predicate()` + loop with a generous timeout, used as the "observe the moment the flag + flips" probe. +- **`run_on_many_threads(callable, n, *args)`** — fork N threads, all + blocked on a `threading.Barrier`, then released simultaneously. +- **`assert_monotonic(samples, allowed_transitions)`** — given a list of + observed states (with timestamps), assert that the sorted sequence is + a valid walk through the documented state graph. +- **`exception_aggregator()`** — capture `SystemExit` / + `BaseException` from all threads so a thread-only failure cannot be + silently lost by the test driver. +- **stress repeats** — wrap each invariant in a loop of, say, 200 + iterations to surface low-probability windows. Combine with + `pytest-repeat` or hypothesis stateful machines for schedule fuzzing. +- **dual-loop fixture** — one event loop in the main thread, another in + a background thread, so cross-loop invariants (4.5, 6.3) get + exercised. diff --git a/RACE_CONDITION_INVARIANTS.md b/RACE_CONDITION_INVARIANTS_B.md similarity index 100% rename from RACE_CONDITION_INVARIANTS.md rename to RACE_CONDITION_INVARIANTS_B.md