From 3b174b6467b9d36d0433349fe0942cd0cfa9a46c Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 11:52:07 -0700 Subject: [PATCH 01/21] New pitch for testing: Polling Expectations --- .../testing/NNNN-polling-expectations.md | 278 ++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 proposals/testing/NNNN-polling-expectations.md diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md new file mode 100644 index 0000000000..4d7816bacf --- /dev/null +++ b/proposals/testing/NNNN-polling-expectations.md @@ -0,0 +1,278 @@ +# Polling Expectations + +* Proposal: [ST-NNNN](NNNN-polling-expectations.md) +* Authors: [Rachel Brindle](https://github.com/younata) +* Review Manager: TBD +* Status: **Awaiting implementation** or **Awaiting review** +* Implementation: (Working on it) +* Review: (Working on it) + +## Introduction + +Test authors frequently need to wait for some background activity to complete +or reach an expected state before continuing. This proposal introduces a new API +to enable polling for an expected state. + +## Motivation + +Test authors can currently utilize the existing [`confirmation`](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)-5mqz2) +APIs or awaiting on an `async` callable in order to block test execution until +a callback is called, or an async callable returns. However, this requires the +code being tested to support callbacks or return a status as an async callable. + +This proposal adds another avenue for waiting for code to update to a specified +value, by proactively polling the test closure until it passes or a timeout is +reached. + +More concretely, we can imagine a type that updates its status over an +indefinite timeframe: + +```swift +actor Aquarium { + var dolphins: [Dolphin] + + func raiseDolphins() async { + // over a very long timeframe + dolphins.append(Dolphin()) + } +} +``` + +## Proposed solution + +This proposal introduces new overloads of the `#expect()` and `#require()` +macros that take, as arguments, a closure and a timeout value. When called, +these macros will continuously evaluate the closure until either the specific +condition passes, or the timeout has passed. The timeout period will default +to 1 second. + +There are 2 Polling Behaviors that we will add: Passes Once and Passes Always. +Passes Once will continuously evaluate the expression until the expression +returns true. If the timeout passes without the expression ever returning true, +then a failure will be reported. Passes Always will continuously execute the +expression until the first time expression returns false or the timeout passes. +If the expression ever returns false, then a failure will be reported. + +Tests will now be able to poll code updating in the background using +either of the new overloads: + +```swift +let subject = Aquarium() +Task { + await subject.raiseDolphins() +} +await #expect(until: .passesOnce) { + subject.dolphins.count() == 1 +} +``` + +## Detailed design + +### New expectations + +We will introduce the following new overloads of `#expect()` and `#require()` to +the testing library: + +```swift +/// Continuously check an expression until it matches the given PollingBehavior +/// +/// - Parameters: +/// - until: The desired PollingBehavior to check for. +/// - timeout: How long to run poll the expression until stopping. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which the recorded expectations +/// and issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// Use this overload of `#expect()` when you wish to poll whether a value +/// changes as the result of activity in another task/queue/thread. +@freestanding(expression) public macro expect( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") + +/// Continuously check an expression until it matches the given PollingBehavior +/// +/// - Parameters: +/// - until: The desired PollingBehavior to check for. +/// - timeout: How long to run poll the expression until stopping. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which the recorded expectations +/// and issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// Use this overload of `#expect()` when you wish to poll whether a value +/// changes as the result of activity in another task/queue/thread, and you +/// expect the expression to throw an error as part of succeeding +@freestanding(expression) public macro expect( + until pollingBehavior: PollingBehavior, + throws error: E, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws(E) -> Bool +) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") +where E: Error & Equatable + +@freestanding(expression) public macro expect( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @Sendable () async throws(E) -> Bool, + throws errorMatcher: (E) async throws -> Bool +) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") +where E: Error + +/// Continuously check an expression until it matches the given PollingBehavior +/// +/// - Parameters: +/// - until: The desired PollingBehavior to check for. +/// - timeout: How long to run poll the expression until stopping. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which the recorded expectations +/// and issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// Use this overload of `#require()` when you wish to poll whether a value +/// changes as the result of activity in another task/queue/thread. +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") + +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws -> R? +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +where R: Sendable + +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + throws error: E, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws(E) -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +where E: Error & Equatable + +@freestanding(expression) public macro require( + until pollingBehavior: PollingBehavior, + timeout: Duration = .seconds(1), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + expression: @Sendable () async throws(E) -> Bool, + throwing errorMatcher: (E) async throws -> Bool, +) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +where E: Error +``` + +### Polling Behavior + +A new type, `PollingBehavior`, to represent the behavior of a polling +expectation: + +```swift +public enum PollingBehavior { + /// Continuously evaluate the expression until the first time it returns + /// true. + /// If it does not pass once by the time the timeout is reached, then a + /// failure will be reported. + case passesOnce + + /// Continuously evaluate the expression until the first time it returns + /// false. + /// If the expression returns false, then a failure will be reported. + /// If the expression only returns true before the timeout is reached, then + /// no failure will be reported. + /// If the expression does not finish evaluating before the timeout is + /// reached, then a failure will be reported. + case passesAlways +} +``` + +### Usage + +These macros can be used with an async test function: + +```swift +@Test func `The aquarium's dolphin nursery works`() async { + let subject = Aquarium() + Task { + await subject.raiseDolphins() + } + await #expect(until: .passesOnce) { + subject.dolphins.count() == 1 + } +} +``` + +With the definition of `Aquarium` above, the closure will only need to be +evaluated a few times before it starts returning true. At which point the macro +will end, and no failure will be reported. + +If the expression never returns a value within the timeout period, then a +failure will be reported, noting that the expression was unable to be evaluated +within the timeout period: + +```swift +await #expect(until: .passesOnce, timeout: .seconds(1)) { + // Failure: The expression timed out before evaluation could finish. + try await Task.sleep(for: .seconds(10)) +} +``` + +In the case of `#require` where the expression returns an optional value, under +`PollingBehavior.passesOnce`, the expectation is considered to have passed the +first time the expression returns a non-nil value, and that value will be +returned by the expectation. Under `PollingBehavior.passesAlways`, the +expectation is considered to have passed if the expression always returns a +non-nil value. If it passes, the value returned by the last time the +expression is evaluated will be returned by the expectation. + +When no error is expected, then the first time the expression throws any error +will cause the polling expectation to stop & report the error as a failure. + +When an error is expected, then the expression is not considered to pass +unless it throws an error that equals the expected error or returns true when +evaluated by the `errorMatcher`. After which the polling continues under the +specified PollingBehavior. + +## Source compatibility + +This is a new interface that is unlikely to collide with any existing +client-provided interfaces. The typical Swift disambiguation tools can be used +if needed. + +## Integration with supporting tools + +We will expose the polling mechanism under ForToolsIntegrationOnly spi so that +tools may integrate with them. + +## Future directions + +The timeout default could be configured as a Suite or Test trait. Additionally, +it could be configured in some future global configuration tool. + +## Alternatives considered + +Instead of creating the `PollingBehavior` type, we could have introduced more +macros to cover that situation: `#expect(until:)` and `#expect(always:)`. +However, this would have resulted in confusion for the compiler and test authors +when trailing closure syntax is used. + +## Acknowledgments + +This proposal is heavily inspired by Nimble's [Polling Expectations](https://quick.github.io/Nimble/documentation/nimble/pollingexpectations/). +In particular, thanks to [Jeff Hui](https://github.com/jeffh) for writing the +original implementation of Nimble's Polling Expectations. From 81ce3cf8ac9119b89915d592df21f98f6f2c0d99 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 12:24:29 -0700 Subject: [PATCH 02/21] Add an alternatives considered section on having passesOnce continue to evaluate the expression after it passes --- .../testing/NNNN-polling-expectations.md | 37 ++++++++++++++++++- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 4d7816bacf..7c1860f352 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -62,7 +62,7 @@ Task { await subject.raiseDolphins() } await #expect(until: .passesOnce) { - subject.dolphins.count() == 1 + await subject.dolphins.count == 1 } ``` @@ -212,7 +212,7 @@ These macros can be used with an async test function: await subject.raiseDolphins() } await #expect(until: .passesOnce) { - subject.dolphins.count() == 1 + await subject.dolphins.count == 1 } } ``` @@ -266,11 +266,44 @@ it could be configured in some future global configuration tool. ## Alternatives considered +### Remove `PollingBehavior` in favor of more macros + Instead of creating the `PollingBehavior` type, we could have introduced more macros to cover that situation: `#expect(until:)` and `#expect(always:)`. However, this would have resulted in confusion for the compiler and test authors when trailing closure syntax is used. +### `PollingBehavior.passesOnce` continues to evaluate expression after passing + +Under `PollingBehavior.passesOnce`, we thought about requiring the expression +to continue to pass after it starts passing. The idea is to prevent test +flakiness caused by an expectation that initially passes, but stops passing as +a result of (intended) background activity. For example: + +```swift +@Test func `The aquarium's dolphin nursery works`() async { + let subject = Aquarium() + await subject.raiseDolphins() + Task { + await subject.raiseDolphins() + } + await #expect(until: .passesOnce) { + await subject.dolphins.count == 1 + } +} +``` + +This test is flaky, but will pass more often than not. However, it is still +incorrect. If we were to change `PollingBehavior.passesOnce` to instead check +that the expression continues to pass after the first time it succeeds until the +timeout is reached, then this test would correctly be flagged as failing each +time it's ran. + +We chose to address this by using the name `passesOnce` instead of changing the +behavior. `passesOnce` makes it clear the exact behavior that will happen: the +expression will be evaluated until the first time it passes, and no more. We +hope that this will help test authors to better recognize these situations. + ## Acknowledgments This proposal is heavily inspired by Nimble's [Polling Expectations](https://quick.github.io/Nimble/documentation/nimble/pollingexpectations/). From 019586c9de0ba6073ea9ec949ee40a37c62d621e Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 12:26:25 -0700 Subject: [PATCH 03/21] Add a link to the pitch thread --- proposals/testing/NNNN-polling-expectations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 7c1860f352..f8c98fa384 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -5,7 +5,7 @@ * Review Manager: TBD * Status: **Awaiting implementation** or **Awaiting review** * Implementation: (Working on it) -* Review: (Working on it) +* Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) ## Introduction From c94d324f86ff9d97ee67deb9c64111b4d950f242 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 13 May 2025 12:27:22 -0700 Subject: [PATCH 04/21] Set polling expectation's status to awaiting implementation I'm working on it, but macros are hard. --- proposals/testing/NNNN-polling-expectations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index f8c98fa384..16e54ad972 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -3,7 +3,7 @@ * Proposal: [ST-NNNN](NNNN-polling-expectations.md) * Authors: [Rachel Brindle](https://github.com/younata) * Review Manager: TBD -* Status: **Awaiting implementation** or **Awaiting review** +* Status: **Awaiting implementation** * Implementation: (Working on it) * Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) From 18ac34cb6b506fcf7c0737819e52a34bf902843e Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Wed, 14 May 2025 23:10:45 -0700 Subject: [PATCH 05/21] Swift Testing Polling Expectations: - Change the default timeout. - Change how unexpected thrown errors are treated. - Add future direction to add change monitoring via Observation - Add alternative considered of Just Use A While Loop - Add alternative considered for shorter default timeouts --- .../testing/NNNN-polling-expectations.md | 105 ++++++++++-------- 1 file changed, 57 insertions(+), 48 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 16e54ad972..7dda74e45f 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -44,7 +44,7 @@ This proposal introduces new overloads of the `#expect()` and `#require()` macros that take, as arguments, a closure and a timeout value. When called, these macros will continuously evaluate the closure until either the specific condition passes, or the timeout has passed. The timeout period will default -to 1 second. +to 1 minute. There are 2 Polling Behaviors that we will add: Passes Once and Passes Always. Passes Once will continuously evaluate the expression until the expression @@ -88,60 +88,34 @@ the testing library: /// changes as the result of activity in another task/queue/thread. @freestanding(expression) public macro expect( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, expression: @Sendable () async throws -> Bool ) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -/// Continuously check an expression until it matches the given PollingBehavior -/// -/// - Parameters: -/// - until: The desired PollingBehavior to check for. -/// - timeout: How long to run poll the expression until stopping. -/// - comment: A comment describing the expectation. -/// - sourceLocation: The source location to which the recorded expectations -/// and issues should be attributed. -/// - expression: The expression to be evaluated. -/// -/// Use this overload of `#expect()` when you wish to poll whether a value -/// changes as the result of activity in another task/queue/thread, and you -/// expect the expression to throw an error as part of succeeding @freestanding(expression) public macro expect( until pollingBehavior: PollingBehavior, throws error: E, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws(E) -> Bool -) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") + expression: @Sendable () async throws -> Bool +) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") where E: Error & Equatable -@freestanding(expression) public macro expect( +@freestanding(expression) public macro expect( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - performing expression: @Sendable () async throws(E) -> Bool, - throws errorMatcher: (E) async throws -> Bool -) -> E = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -where E: Error + performing expression: @Sendable () async throws -> Bool, + throws errorMatcher: (any Error) async throws -> Bool +) -> Error = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -/// Continuously check an expression until it matches the given PollingBehavior -/// -/// - Parameters: -/// - until: The desired PollingBehavior to check for. -/// - timeout: How long to run poll the expression until stopping. -/// - comment: A comment describing the expectation. -/// - sourceLocation: The source location to which the recorded expectations -/// and issues should be attributed. -/// - expression: The expression to be evaluated. -/// -/// Use this overload of `#require()` when you wish to poll whether a value -/// changes as the result of activity in another task/queue/thread. @freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, expression: @Sendable () async throws -> Bool @@ -149,7 +123,7 @@ where E: Error @freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, expression: @Sendable () async throws -> R? @@ -159,22 +133,21 @@ where R: Sendable @freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, throws error: E, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws(E) -> Bool + expression: @Sendable () async throws -> Bool ) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") where E: Error & Equatable -@freestanding(expression) public macro require( +@freestanding(expression) public macro require( until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(1), + timeout: Duration = .seconds(60), _ comment: @autoclosure () -> Comment? = nil, sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws(E) -> Bool, - throwing errorMatcher: (E) async throws -> Bool, + expression: @Sendable () async throws -> Bool, + throwing errorMatcher: (any Error) async throws -> Bool, ) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") -where E: Error ``` ### Polling Behavior @@ -201,6 +174,11 @@ public enum PollingBehavior { } ``` +### Platform Availability + +Polling expectations will not be available on platforms that do not support +Swift Concurrency, nor on platforms that do not support multiple threads. + ### Usage These macros can be used with an async test function: @@ -240,14 +218,14 @@ expectation is considered to have passed if the expression always returns a non-nil value. If it passes, the value returned by the last time the expression is evaluated will be returned by the expectation. -When no error is expected, then the first time the expression throws any error -will cause the polling expectation to stop & report the error as a failure. - When an error is expected, then the expression is not considered to pass unless it throws an error that equals the expected error or returns true when evaluated by the `errorMatcher`. After which the polling continues under the specified PollingBehavior. +When no error is expected, then this is treated as if the expression returned +false. This is specifically to invert the case when an error is expected. + ## Source compatibility This is a new interface that is unlikely to collide with any existing @@ -264,8 +242,39 @@ tools may integrate with them. The timeout default could be configured as a Suite or Test trait. Additionally, it could be configured in some future global configuration tool. +On the topic of monitoring for changes, we could add a tool integrating with the +Observation module which monitors changes to `@Observable` objects during some +lifetime. + ## Alternatives considered +### Just use a while loop + +Polling could be written as a simple while loop that continuously executes the +expression until it returns, something like: + +```swift +func poll(timeout: Duration, expression: () -> Bool) -> Bool { + let clock: Clock = // ... + let endTimestamp = clock.now + timeout + while clock.now < endTimestamp { + if expression() { return true } + } + return false +} +``` + +Which works in most naive cases, but is not robust. Notably, This approach does +not handle the case when the expression never returns, or does not return within +the timeout period. + +### Shorter default timeout + +Due to the nature of Swift Concurrency scheduling, using short default +timeouts will result in high rates of test flakiness. This is why the default +timeout is 1 minute. We do not recommend that test authors use timeouts any +shorter than this. + ### Remove `PollingBehavior` in favor of more macros Instead of creating the `PollingBehavior` type, we could have introduced more From 0865f54a83a15a028ecb9ade716ac4dfda171caf Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Wed, 14 May 2025 23:21:09 -0700 Subject: [PATCH 06/21] Add a link to the PR containing polling expectations --- proposals/testing/NNNN-polling-expectations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 7dda74e45f..85d87f51eb 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -3,8 +3,8 @@ * Proposal: [ST-NNNN](NNNN-polling-expectations.md) * Authors: [Rachel Brindle](https://github.com/younata) * Review Manager: TBD -* Status: **Awaiting implementation** -* Implementation: (Working on it) +* Status: **Awaiting review** +* Implementation: [swiftlang/swift-testing#1115](https://github.com/swiftlang/swift-testing/pull/1115) * Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) ## Introduction From b7961ff7c11a62654b30be5b1b879954b566fd77 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Mon, 26 May 2025 16:05:39 -0700 Subject: [PATCH 07/21] Update polling proposal to refer to new confirmation functions --- .../testing/NNNN-polling-expectations.md | 325 +++++++++--------- 1 file changed, 155 insertions(+), 170 deletions(-) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-expectations.md index 85d87f51eb..ae86be4f08 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-expectations.md @@ -40,18 +40,14 @@ actor Aquarium { ## Proposed solution -This proposal introduces new overloads of the `#expect()` and `#require()` -macros that take, as arguments, a closure and a timeout value. When called, -these macros will continuously evaluate the closure until either the specific -condition passes, or the timeout has passed. The timeout period will default -to 1 minute. - -There are 2 Polling Behaviors that we will add: Passes Once and Passes Always. -Passes Once will continuously evaluate the expression until the expression -returns true. If the timeout passes without the expression ever returning true, -then a failure will be reported. Passes Always will continuously execute the -expression until the first time expression returns false or the timeout passes. -If the expression ever returns false, then a failure will be reported. +This proposal introduces new members of the `confirmation` family of functions: +`confirmPassesEventually` and `confirmAlwaysPasses`. These functions take in +a closure to be continuously evaluated until the specific condition passes. + +`confirmPassesEventually` will evaluate the closure until the first time it +returns true or a non-nil value. `confirmAlwaysPasses` will evaluate the +closure until it returns false or nil. If neither case happens, evaluation will +continue until the closure has been called some amount of times. Tests will now be able to poll code updating in the background using either of the new overloads: @@ -61,127 +57,141 @@ let subject = Aquarium() Task { await subject.raiseDolphins() } -await #expect(until: .passesOnce) { - await subject.dolphins.count == 1 +await confirmPassesEventually { + subject.dolphins.count == 1 } ``` ## Detailed design -### New expectations +### New confirmation functions -We will introduce the following new overloads of `#expect()` and `#require()` to -the testing library: +We will introduce 4 new members of the confirmation family of functions to the +testing library: ```swift -/// Continuously check an expression until it matches the given PollingBehavior +/// Confirm that some expression eventually returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmPassesEventually( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async + +/// Confirm that some expression eventually returns a non-nil value /// /// - Parameters: -/// - until: The desired PollingBehavior to check for. -/// - timeout: How long to run poll the expression until stopping. -/// - comment: A comment describing the expectation. -/// - sourceLocation: The source location to which the recorded expectations -/// and issues should be attributed. -/// - expression: The expression to be evaluated. +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. /// -/// Use this overload of `#expect()` when you wish to poll whether a value -/// changes as the result of activity in another task/queue/thread. -@freestanding(expression) public macro expect( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") - -@freestanding(expression) public macro expect( - until pollingBehavior: PollingBehavior, - throws error: E, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") -where E: Error & Equatable - -@freestanding(expression) public macro expect( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - performing expression: @Sendable () async throws -> Bool, - throws errorMatcher: (any Error) async throws -> Bool -) -> Error = #externalMacro(module: "TestingMacros", type: "PollingExpectMacro") - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> R? -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") -where R: Sendable - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - throws error: E, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") -where E: Error & Equatable - -@freestanding(expression) public macro require( - until pollingBehavior: PollingBehavior, - timeout: Duration = .seconds(60), - _ comment: @autoclosure () -> Comment? = nil, - sourceLocation: SourceLocation = #_sourceLocation, - expression: @Sendable () async throws -> Bool, - throwing errorMatcher: (any Error) async throws -> Bool, -) = #externalMacro(module: "TestingMacros", type: "PollingRequireMacro") +/// - Returns: The first non-nil value returned by `body`. +/// +/// - Throws: A `PollingFailedError` will be thrown if `body` never returns a +/// non-optional value +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmPassesEventually( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> R? +) async throws -> R where R: Sendable + +/// Confirm that some expression always returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, confirming that some state does not change. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmAlwaysPasses( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async + +/// Confirm that some expression always returns a non-optional value +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Returns: The value from the last time `body` was invoked. +/// +/// - Throws: A `PollingFailedError` will be thrown if `body` ever returns a +/// non-optional value +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, confirming that some state does not change. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func confirmAlwaysPasses( + _ comment: Comment? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> R? +) async throws -> R where R: Sendable ``` -### Polling Behavior +### New Error Type -A new type, `PollingBehavior`, to represent the behavior of a polling -expectation: +A new error type, `PollingFailedError` to be thrown when polling doesn't return +a non-nil value: ```swift -public enum PollingBehavior { - /// Continuously evaluate the expression until the first time it returns - /// true. - /// If it does not pass once by the time the timeout is reached, then a - /// failure will be reported. - case passesOnce - - /// Continuously evaluate the expression until the first time it returns - /// false. - /// If the expression returns false, then a failure will be reported. - /// If the expression only returns true before the timeout is reached, then - /// no failure will be reported. - /// If the expression does not finish evaluating before the timeout is - /// reached, then a failure will be reported. - case passesAlways -} +/// A type describing an error thrown when polling fails to return a non-nil +/// value +public struct PollingFailedError: Error {} ``` +### New Issue Kind + +A new Issue.Kind, `confirmationPollingFailed` will be added to represent the +case here confirmation polling fails. This issue kind will be recorded when +polling fails. + ### Platform Availability Polling expectations will not be available on platforms that do not support -Swift Concurrency, nor on platforms that do not support multiple threads. +Swift Concurrency. ### Usage -These macros can be used with an async test function: +These functions can be used with an async test function: ```swift @Test func `The aquarium's dolphin nursery works`() async { @@ -189,42 +199,24 @@ These macros can be used with an async test function: Task { await subject.raiseDolphins() } - await #expect(until: .passesOnce) { + await confirmPassesEventually { await subject.dolphins.count == 1 } } ``` With the definition of `Aquarium` above, the closure will only need to be -evaluated a few times before it starts returning true. At which point the macro +evaluated a few times before it starts returning true. At which point polling will end, and no failure will be reported. -If the expression never returns a value within the timeout period, then a -failure will be reported, noting that the expression was unable to be evaluated -within the timeout period: - -```swift -await #expect(until: .passesOnce, timeout: .seconds(1)) { - // Failure: The expression timed out before evaluation could finish. - try await Task.sleep(for: .seconds(10)) -} -``` - -In the case of `#require` where the expression returns an optional value, under -`PollingBehavior.passesOnce`, the expectation is considered to have passed the -first time the expression returns a non-nil value, and that value will be -returned by the expectation. Under `PollingBehavior.passesAlways`, the -expectation is considered to have passed if the expression always returns a -non-nil value. If it passes, the value returned by the last time the -expression is evaluated will be returned by the expectation. - -When an error is expected, then the expression is not considered to pass -unless it throws an error that equals the expected error or returns true when -evaluated by the `errorMatcher`. After which the polling continues under the -specified PollingBehavior. +Polling will be stopped in the following cases: -When no error is expected, then this is treated as if the expression returned -false. This is specifically to invert the case when an error is expected. +- After the expression has been evaluated 1 million times. +- If the task that started the polling is cancelled. +- For `confirmPassesEventually`: The first time the closure returns true or a + non-nil value +- For `confirmAlwaysPasses`: The first time the closure returns false or nil. +- The first time the closure throws an error. ## Source compatibility @@ -232,26 +224,18 @@ This is a new interface that is unlikely to collide with any existing client-provided interfaces. The typical Swift disambiguation tools can be used if needed. -## Integration with supporting tools - -We will expose the polling mechanism under ForToolsIntegrationOnly spi so that -tools may integrate with them. - ## Future directions -The timeout default could be configured as a Suite or Test trait. Additionally, -it could be configured in some future global configuration tool. - -On the topic of monitoring for changes, we could add a tool integrating with the -Observation module which monitors changes to `@Observable` objects during some +We plan to add support for more push-based monitoring, such as integrating with +the Observation module to monitor changes to `@Observable` objects during some lifetime. ## Alternatives considered -### Just use a while loop +### Use timeouts -Polling could be written as a simple while loop that continuously executes the -expression until it returns, something like: +Polling could be written in such a way that it stops after some amount of time +has passed. Naively, this could be written as: ```swift func poll(timeout: Duration, expression: () -> Bool) -> Bool { @@ -264,27 +248,27 @@ func poll(timeout: Duration, expression: () -> Bool) -> Bool { } ``` -Which works in most naive cases, but is not robust. Notably, This approach does -not handle the case when the expression never returns, or does not return within -the timeout period. +Unfortunately, while this could work reasonably well in an environment where +tests are executed serially, the concurrent test runner the testing library uses +means that timeouts are inherently unreliable. Importantly, timeouts become more +unreliable the more tests in the test suite. -### Shorter default timeout +### Use macros instead of functions -Due to the nature of Swift Concurrency scheduling, using short default -timeouts will result in high rates of test flakiness. This is why the default -timeout is 1 minute. We do not recommend that test authors use timeouts any -shorter than this. +Instead of adding new bare functions, polling could be written as additional +macros, something like: -### Remove `PollingBehavior` in favor of more macros +```swift +#expectUntil { ... } +#expectAlways { ... } +``` -Instead of creating the `PollingBehavior` type, we could have introduced more -macros to cover that situation: `#expect(until:)` and `#expect(always:)`. -However, this would have resulted in confusion for the compiler and test authors -when trailing closure syntax is used. +However, there's no additional benefit to doing this, and it may even lead test +authors to use polling when other mechanisms would be more appropriate. -### `PollingBehavior.passesOnce` continues to evaluate expression after passing +### `confirmPassesEventually` continues to evaluate expression after passing -Under `PollingBehavior.passesOnce`, we thought about requiring the expression +For `confirmPassesEventually`, we thought about requiring the expression to continue to pass after it starts passing. The idea is to prevent test flakiness caused by an expectation that initially passes, but stops passing as a result of (intended) background activity. For example: @@ -296,22 +280,23 @@ a result of (intended) background activity. For example: Task { await subject.raiseDolphins() } - await #expect(until: .passesOnce) { + await confirmPassesEventually { await subject.dolphins.count == 1 } } ``` This test is flaky, but will pass more often than not. However, it is still -incorrect. If we were to change `PollingBehavior.passesOnce` to instead check +incorrect. If we were to change `confirmPassesEventually` to instead check that the expression continues to pass after the first time it succeeds until the timeout is reached, then this test would correctly be flagged as failing each time it's ran. -We chose to address this by using the name `passesOnce` instead of changing the -behavior. `passesOnce` makes it clear the exact behavior that will happen: the -expression will be evaluated until the first time it passes, and no more. We -hope that this will help test authors to better recognize these situations. +We chose to address this by using the name `confirmPassesEventually` instead of +changing the behavior. `confirmPassesEventually` makes it clear the exact +behavior that will happen: the expression will be evaluated until the first time +it passes, and no more. We hope that this will help test authors to better +recognize these situations. ## Acknowledgments From 818372597f6a2fb07e2d7c852811f4821f36bdb2 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Sat, 7 Jun 2025 21:15:05 -0700 Subject: [PATCH 08/21] Polling Confirmations: Renamed from Polling Expectations - Include commenting on why not pure counts (i.e. why counts + interval) - Specify the configuration traits - Specify callsite configuration of counts + interval --- ...tions.md => NNNN-polling-confirmations.md} | 202 +++++++++++++++--- 1 file changed, 171 insertions(+), 31 deletions(-) rename proposals/testing/{NNNN-polling-expectations.md => NNNN-polling-confirmations.md} (54%) diff --git a/proposals/testing/NNNN-polling-expectations.md b/proposals/testing/NNNN-polling-confirmations.md similarity index 54% rename from proposals/testing/NNNN-polling-expectations.md rename to proposals/testing/NNNN-polling-confirmations.md index ae86be4f08..8c587545a4 100644 --- a/proposals/testing/NNNN-polling-expectations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -1,6 +1,6 @@ -# Polling Expectations +# Polling Confirmations -* Proposal: [ST-NNNN](NNNN-polling-expectations.md) +* Proposal: [ST-NNNN](NNNN-polling-confirmations.md) * Authors: [Rachel Brindle](https://github.com/younata) * Review Manager: TBD * Status: **Awaiting review** @@ -42,7 +42,9 @@ actor Aquarium { This proposal introduces new members of the `confirmation` family of functions: `confirmPassesEventually` and `confirmAlwaysPasses`. These functions take in -a closure to be continuously evaluated until the specific condition passes. +a closure to be repeatedly evaluated until the specific condition passes, +waiting at least some amount of time - specified by `pollingInterval` and +defaulting to 1 millisecond - before evaluating the closure again. `confirmPassesEventually` will evaluate the closure until the first time it returns true or a non-nil value. `confirmAlwaysPasses` will evaluate the @@ -75,6 +77,21 @@ testing library: /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. @@ -87,6 +104,8 @@ testing library: @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) public func confirmPassesEventually( _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool @@ -97,6 +116,20 @@ public func confirmPassesEventually( /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. @@ -114,6 +147,8 @@ public func confirmPassesEventually( @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) public func confirmPassesEventually( _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> R? @@ -124,6 +159,19 @@ public func confirmPassesEventually( /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. @@ -135,36 +183,12 @@ public func confirmPassesEventually( @available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) public func confirmAlwaysPasses( _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool ) async - -/// Confirm that some expression always returns a non-optional value -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// - Returns: The value from the last time `body` was invoked. -/// -/// - Throws: A `PollingFailedError` will be thrown if `body` ever returns a -/// non-optional value -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, confirming that some state does not change. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmAlwaysPasses( - _ comment: Comment? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> R? -) async throws -> R where R: Sendable ``` ### New Error Type @@ -184,9 +208,99 @@ A new Issue.Kind, `confirmationPollingFailed` will be added to represent the case here confirmation polling fails. This issue kind will be recorded when polling fails. +### New Traits + +Two new traits will be added to change the default values for the +`maxPollingIterations` and `pollingInterval` arguments. Test authors often +want to poll for the `passesEventually` behavior more than they poll for the +`alwaysPasses` behavior, which is why there are separate traits for configuring +defaults for these functions. + +```swift +/// A trait to provide a default polling configuration to all usages of +/// ``confirmPassesEventually`` within a test or suite. +/// +/// To add this trait to a test, use the +/// ``Trait/confirmPassesEventuallyDefaults`` function. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public struct ConfirmPassesEventuallyConfigurationTrait: TestTrait, SuiteTrait { + public var maxPollingIterations: Int? + public var pollingInterval: Duration? + + public var isRecursive: Bool { true } + + public init(maxPollingIterations: Int?, pollingInterval: Duration?) +} + +/// A trait to provide a default polling configuration to all usages of +/// ``confirmPassesAlways`` within a test or suite. +/// +/// To add this trait to a test, use the ``Trait/confirmAlwaysPassesDefaults`` +/// function. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public struct ConfirmAlwaysPassesConfigurationTrait: TestTrait, SuiteTrait { + public var maxPollingIterations: Int? + public var pollingInterval: Duration? + + public var isRecursive: Bool { true } + + public init(maxPollingIterations: Int?, pollingInterval: Duration?) +} + +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +extension Trait where Self == ConfirmPassesEventuallyConfigurationTrait { + /// Specifies defaults for ``confirmPassesEventually`` in the test or suite. + /// + /// - Parameters: + /// - maxPollingIterations: The maximum amount of times to attempt polling. + /// If nil, polling will be attempted up to 1000 times. + /// `maxPollingIterations` must be greater than 0. + /// - pollingInterval: The minimum amount of time to wait between polling + /// attempts. + /// If nil, polling will wait at least 1 millisecond between polling + /// attempts. + /// `pollingInterval` must be greater than 0. + public static func confirmPassesEventuallyDefaults( + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil + ) -> Self +} + +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +extension Trait where Self == ConfirmPassesAlwaysConfigurationTrait { + /// Specifies defaults for ``confirmAlwaysPasses`` in the test or suite. + /// + /// - Parameters: + /// - maxPollingIterations: The maximum amount of times to attempt polling. + /// If nil, polling will be attempted up to 1000 times. + /// `maxPollingIterations` must be greater than 0. + /// - pollingInterval: The minimum amount of time to wait between polling + /// attempts. + /// If nil, polling will wait at least 1 millisecond between polling + /// attempts. + /// `pollingInterval` must be greater than 0. + public static func confirmAlwaysPassesDefaults( + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil + ) -> Self +} +``` + +Specifying `maxPollingIterations` or `pollingInterval` directly on either +`confirmPassesEventually` or `confirmAlwaysPasses` will override any value +provided by the trait. + +### Default Polling Configuration + +For both `confirmPassesEventually` and `confirmsAlwaysPasses`, the Testing +library will default `maxPollingIterations` to 1000, and `pollingInterval` to +1 millisecond. This allows for tests on lightly-loaded systems such as developer +workstations to run in a little over 1 second wall-clock time, while still +being able to gracefully handle running on large loads. + ### Platform Availability -Polling expectations will not be available on platforms that do not support +Polling confirmations will not be available on platforms that do not support Swift Concurrency. ### Usage @@ -211,7 +325,8 @@ will end, and no failure will be reported. Polling will be stopped in the following cases: -- After the expression has been evaluated 1 million times. +- After the expression has been evaluated up to the count specified in + `maxPollingInterations`. - If the task that started the polling is cancelled. - For `confirmPassesEventually`: The first time the closure returns true or a non-nil value @@ -253,6 +368,31 @@ tests are executed serially, the concurrent test runner the testing library uses means that timeouts are inherently unreliable. Importantly, timeouts become more unreliable the more tests in the test suite. +### Use only polling iterations + +Another option considered was only using polling iterations. Naively, this +would write the main polling loop as: + +```swift +func poll(iterations: Int, expression: () -> Bool) async -> Bool { + for _ in 0.. Date: Sat, 7 Jun 2025 21:42:31 -0700 Subject: [PATCH 09/21] Polling Confirmations: Add a couple sentences on why pollingInterval must be positive --- proposals/testing/NNNN-polling-confirmations.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 8c587545a4..0e83f78737 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -393,6 +393,14 @@ well under a millisecond. Because of this, we decided to add on the polling interval argument: a minimum duration to wait between polling, to make it much easier for test authors to predict a good-enough guess for when to stop polling. +### Allow `pollingInterval` to be `.zero` + +We could allow test authors to set `pollingInterval` as `Duration.zero`, making +polling behave as if only polling iterations is counted. +We chose not to allow this for the same reason we chose to add a wait between +polling: this makes it much easier for test authors to predict when to stop +polling. + ### Use macros instead of functions Instead of adding new bare functions, polling could be written as additional From 5b9067b652857f66b805cc9f47426b1250fe7aa6 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Sat, 7 Jun 2025 22:10:02 -0700 Subject: [PATCH 10/21] Polling Confirmations: Add requirePassesEventually and requireAlwaysPasses --- .../testing/NNNN-polling-confirmations.md | 94 +++++++++++++++++-- 1 file changed, 87 insertions(+), 7 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 0e83f78737..0a95c475bb 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -68,7 +68,7 @@ await confirmPassesEventually { ### New confirmation functions -We will introduce 4 new members of the confirmation family of functions to the +We will introduce 5 new members of the confirmation family of functions to the testing library: ```swift @@ -111,6 +111,47 @@ public func confirmPassesEventually( _ body: @escaping () async throws -> Bool ) async +/// Require that some expression eventually returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Throws: A `PollingFailedError` will be thrown if the expression never +/// returns true. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, waiting on some state to change that cannot be easily confirmed +/// through other forms of `confirmation`. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func requirePassesEventually( + _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async throws + /// Confirm that some expression eventually returns a non-nil value /// /// - Parameters: @@ -138,7 +179,7 @@ public func confirmPassesEventually( /// - Returns: The first non-nil value returned by `body`. /// /// - Throws: A `PollingFailedError` will be thrown if `body` never returns a -/// non-optional value +/// non-optional value. /// /// Use polling confirmations to check that an event while a test is running in /// complex scenarios where other forms of confirmation are insufficient. For @@ -189,6 +230,45 @@ public func confirmAlwaysPasses( sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool ) async + +/// Require that some expression always returns true +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will be attempted 1000 times before recording an issue. +/// `maxPollingIterations` must be greater than 0. +/// - pollingInterval: The minimum amount of time to wait between polling +/// attempts. +/// If nil, this uses whatever value is specified under the last +/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. +/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then +/// polling will wait at least 1 millisecond between polling attempts. +/// `pollingInterval` must be greater than 0. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to whych any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Throws: A `PollingFailedError` will be thrown if the expression ever +/// returns false. +/// +/// Use polling confirmations to check that an event while a test is running in +/// complex scenarios where other forms of confirmation are insufficient. For +/// example, confirming that some state does not change. +@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) +public func requireAlwaysPasses( + _ comment: Comment? = nil, + maxPollingIterations: Int? = nil, + pollingInterval: Duration? = nil, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: @escaping () async throws -> Bool +) async throws ``` ### New Error Type @@ -197,16 +277,16 @@ A new error type, `PollingFailedError` to be thrown when polling doesn't return a non-nil value: ```swift -/// A type describing an error thrown when polling fails to return a non-nil -/// value -public struct PollingFailedError: Error {} +/// A type describing an error thrown when polling fails. +public struct PollingFailedError: Error, Equatable {} ``` ### New Issue Kind A new Issue.Kind, `confirmationPollingFailed` will be added to represent the -case here confirmation polling fails. This issue kind will be recorded when -polling fails. +case where a polling confirmation failed. This issue kind will be recorded when +`confirmPassesEventually` and `confirmAlwaysPasses` fail, but not when +`requirePassesEventually` or `requireAlwaysPasses` fail. ### New Traits From 08fb9aebdc716f5ee9e930641dbb2f049759949d Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Mon, 21 Jul 2025 00:09:54 -0700 Subject: [PATCH 11/21] Polling Confirmations: rewrite, again, for new API design --- .../testing/NNNN-polling-confirmations.md | 530 ++++++++---------- 1 file changed, 240 insertions(+), 290 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 0a95c475bb..0a223d3084 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -41,25 +41,36 @@ actor Aquarium { ## Proposed solution This proposal introduces new members of the `confirmation` family of functions: -`confirmPassesEventually` and `confirmAlwaysPasses`. These functions take in -a closure to be repeatedly evaluated until the specific condition passes, -waiting at least some amount of time - specified by `pollingInterval` and -defaulting to 1 millisecond - before evaluating the closure again. - -`confirmPassesEventually` will evaluate the closure until the first time it -returns true or a non-nil value. `confirmAlwaysPasses` will evaluate the -closure until it returns false or nil. If neither case happens, evaluation will -continue until the closure has been called some amount of times. - -Tests will now be able to poll code updating in the background using -either of the new overloads: +`confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)`. These +functions take in a closure to be repeatedly evaluated until the specific +condition passes, waiting at least some amount of time - specified by +`pollingEvery`/`interval` and defaulting to 1 millisecond - before evaluating +the closure again. + +Both of these use the new `PollingStopCondition` enum to determine when to end +polling: `PollingStopCondition.firstPass` configures polling to stop as soon +as the `body` closure returns `true` or a non-`nil` value. At this point, +the confirmation will be marked as passing. +`PollingStopCondition.stopsPassing` configures polling to stop once the `body` +closure returns `false` or a `nil` value. At this point, the confirmation will +be marked as failing: an error will be thrown, and an issue will be recorded. + +Under both `PollingStopCondition` cases, when the early stop condition isn't +reached, polling will continue up until approximately the `within`/`duration` +value has elapsed. When `PollingStopCondition.firstPass` is specified, reaching +the duration stop point will mark the confirmation as failing. +When `PollingStopCondition.stopsPassing` is specified, reaching the duration +stop point will mark the confirmation as passing. + +Tests will now be able to poll code updating in the background using either of +the stop conditions: ```swift let subject = Aquarium() Task { await subject.raiseDolphins() } -await confirmPassesEventually { +await confirmation(until: .firstPass) { subject.dolphins.count == 1 } ``` @@ -68,85 +79,52 @@ await confirmPassesEventually { ### New confirmation functions -We will introduce 5 new members of the confirmation family of functions to the +We will introduce 2 new members of the confirmation family of functions to the testing library: ```swift -/// Confirm that some expression eventually returns true +/// Poll expression within the duration based on the given stop condition /// /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or -/// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. +/// - stopCondition: When to stop polling. +/// - duration: The expected length of time to continue polling for. +/// This value may not correspond to the wall-clock time that polling lasts +/// for, especially on highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or /// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, waiting on some state to change that cannot be easily confirmed -/// through other forms of `confirmation`. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmPassesEventually( - _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> Bool -) async - -/// Require that some expression eventually returns true -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// If no such trait has been added, then polling will be attempted for +/// about 1 second before recording an issue. +/// `duration` must be greater than 0. +/// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or /// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. +/// If no such trait has been added, then polling will wait at least +/// 1 millisecond between polling attempts. +/// `interval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. /// - body: The function to invoke. /// -/// - Throws: A `PollingFailedError` will be thrown if the expression never -/// returns true. +/// - Throws: A `PollingFailedError` if the `body` does not return true within +/// the polling duration. /// /// Use polling confirmations to check that an event while a test is running in /// complex scenarios where other forms of confirmation are insufficient. For /// example, waiting on some state to change that cannot be easily confirmed /// through other forms of `confirmation`. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func requirePassesEventually( +@available(_clockAPI, *) +public func confirmation( _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, + until stopCondition: PollingStopCondition, + within duration: Duration? = nil, + pollingEvery interval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, _ body: @escaping () async throws -> Bool @@ -157,232 +135,219 @@ public func requirePassesEventually( /// - Parameters: /// - comment: An optional comment to apply to any issues generated by this /// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. +/// - stopCondition: When to stop polling. +/// - duration: The expected length of time to continue polling for. +/// This value may not correspond to the wall-clock time that polling lasts +/// for, especially on highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or /// suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. +/// If no such trait has been added, then polling will be attempted for +/// about 1 second before recording an issue. +/// `duration` must be greater than 0. +/// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last -/// ``ConfirmPassesEventuallyConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmPassesEventuallyConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. +/// ``PollingUntilFirstPassConfigurationTrait`` or +/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or +/// suite. +/// If no such trait has been added, then polling will wait at least +/// 1 millisecond between polling attempts. +/// `interval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. /// - sourceLocation: The source location to whych any recorded issues should /// be attributed. /// - body: The function to invoke. /// -/// - Returns: The first non-nil value returned by `body`. +/// - Throws: A `PollingFailedError` if the `body` does not return true within +/// the polling duration. /// -/// - Throws: A `PollingFailedError` will be thrown if `body` never returns a -/// non-optional value. +/// - Returns: The last non-nil value returned by `body`. /// /// Use polling confirmations to check that an event while a test is running in /// complex scenarios where other forms of confirmation are insufficient. For /// example, waiting on some state to change that cannot be easily confirmed /// through other forms of `confirmation`. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmPassesEventually( +@available(_clockAPI, *) +@discardableResult +public func confirmation( _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, + until stopCondition: PollingStopCondition, + within duration: Duration? = nil, + pollingEvery interval: Duration? = nil, isolation: isolated (any Actor)? = #isolation, sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> R? -) async throws -> R where R: Sendable + _ body: @escaping () async throws -> sending R? +) async throws -> R +``` -/// Confirm that some expression always returns true -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, confirming that some state does not change. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func confirmAlwaysPasses( - _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> Bool -) async +### New `PollingStopCondition` enum -/// Require that some expression always returns true -/// -/// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. -/// - maxPollingIterations: The maximum amount of times to attempt polling. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will be attempted 1000 times before recording an issue. -/// `maxPollingIterations` must be greater than 0. -/// - pollingInterval: The minimum amount of time to wait between polling -/// attempts. -/// If nil, this uses whatever value is specified under the last -/// ``ConfirmAlwaysPassesConfigurationTrait`` added to the test or suite. -/// If no ``ConfirmAlwaysPassesConfigurationTrait`` has been added, then -/// polling will wait at least 1 millisecond between polling attempts. -/// `pollingInterval` must be greater than 0. -/// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. -/// -/// - Throws: A `PollingFailedError` will be thrown if the expression ever -/// returns false. -/// -/// Use polling confirmations to check that an event while a test is running in -/// complex scenarios where other forms of confirmation are insufficient. For -/// example, confirming that some state does not change. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public func requireAlwaysPasses( - _ comment: Comment? = nil, - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil, - isolation: isolated (any Actor)? = #isolation, - sourceLocation: SourceLocation = #_sourceLocation, - _ body: @escaping () async throws -> Bool -) async throws +A new enum type, `PollingStopCondition` will be defined, specifying when to stop +polling before the duration has elapsed. Additionally, if the early stop +condition isn't fulfilled before the duration elapses, then this also defines +how the confirmation should be handled. + +```swift +/// A type defining when to stop polling early. +/// This also determines what happens if the duration elapses during polling. +public enum PollingStopCondition: Sendable { + /// Evaluates the expression until the first time it returns true. + /// If it does not pass once by the time the timeout is reached, then a + /// failure will be reported. + case firstPass + + /// Evaluates the expression until the first time it returns false. + /// If the expression returns false, then a failure will be reported. + /// If the expression only returns true before the timeout is reached, then + /// no failure will be reported. + /// If the expression does not finish evaluating before the timeout is + /// reached, then a failure will be reported. + case stopsPassing +} ``` ### New Error Type -A new error type, `PollingFailedError` to be thrown when polling doesn't return -a non-nil value: +A new error type, `PollingFailedError` to be thrown when the polling +confirmation doesn't pass: ```swift /// A type describing an error thrown when polling fails. -public struct PollingFailedError: Error, Equatable {} +public struct PollingFailedError: Error, Sendable, CustomIssueRepresentable {} ``` -### New Issue Kind - -A new Issue.Kind, `confirmationPollingFailed` will be added to represent the -case where a polling confirmation failed. This issue kind will be recorded when -`confirmPassesEventually` and `confirmAlwaysPasses` fail, but not when -`requirePassesEventually` or `requireAlwaysPasses` fail. - ### New Traits Two new traits will be added to change the default values for the -`maxPollingIterations` and `pollingInterval` arguments. Test authors often -want to poll for the `passesEventually` behavior more than they poll for the -`alwaysPasses` behavior, which is why there are separate traits for configuring -defaults for these functions. +`duration` and `interval` arguments. Test authors will often want to poll for +the `firstPass` stop condition for longer than they poll for the +`stopsPassing` stop condition, which is why there are separate traits for +configuring defaults for these functions. ```swift /// A trait to provide a default polling configuration to all usages of -/// ``confirmPassesEventually`` within a test or suite. +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` +/// and +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` +/// within a test or suite for the ``PollingStopCondition.firstPass`` +/// stop condition. /// /// To add this trait to a test, use the -/// ``Trait/confirmPassesEventuallyDefaults`` function. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public struct ConfirmPassesEventuallyConfigurationTrait: TestTrait, SuiteTrait { - public var maxPollingIterations: Int? - public var pollingInterval: Duration? +/// ``Trait/pollingUntilFirstPassDefaults`` function. +@available(_clockAPI, *) +public struct PollingUntilFirstPassConfigurationTrait: TestTrait, SuiteTrait { + /// How long to continue polling for + public var duration: Duration? + /// The minimum amount of time to wait between polling attempts + public var interval: Duration? public var isRecursive: Bool { true } - - public init(maxPollingIterations: Int?, pollingInterval: Duration?) } /// A trait to provide a default polling configuration to all usages of -/// ``confirmPassesAlways`` within a test or suite. +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` +/// and +/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` +/// within a test or suite for the ``PollingStopCondition.stopsPassing`` +/// stop condition. /// -/// To add this trait to a test, use the ``Trait/confirmAlwaysPassesDefaults`` +/// To add this trait to a test, use the ``Trait/pollingUntilStopsPassingDefaults`` /// function. -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -public struct ConfirmAlwaysPassesConfigurationTrait: TestTrait, SuiteTrait { - public var maxPollingIterations: Int? - public var pollingInterval: Duration? +@available(_clockAPI, *) +public struct PollingUntilStopsPassingConfigurationTrait: TestTrait, SuiteTrait { + /// How long to continue polling for + public var duration: Duration? + /// The minimum amount of time to wait between polling attempts + public var interval: Duration? public var isRecursive: Bool { true } - - public init(maxPollingIterations: Int?, pollingInterval: Duration?) } -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -extension Trait where Self == ConfirmPassesEventuallyConfigurationTrait { +@available(_clockAPI, *) +extension Trait where Self == PollingUntilFirstPassConfigurationTrait { /// Specifies defaults for ``confirmPassesEventually`` in the test or suite. /// /// - Parameters: - /// - maxPollingIterations: The maximum amount of times to attempt polling. - /// If nil, polling will be attempted up to 1000 times. - /// `maxPollingIterations` must be greater than 0. - /// - pollingInterval: The minimum amount of time to wait between polling + /// - duration: The expected length of time to continue polling for. + /// This value may not correspond to the wall-clock time that polling + /// lasts for, especially on highly-loaded systems with a lot of tests + /// running. + /// if nil, polling will be attempted for approximately 1 second. + /// `duration` must be greater than 0. + /// - interval: The minimum amount of time to wait between polling /// attempts. /// If nil, polling will wait at least 1 millisecond between polling /// attempts. - /// `pollingInterval` must be greater than 0. - public static func confirmPassesEventuallyDefaults( - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil + /// `interval` must be greater than 0. + public static func pollingUntilFirstPassDefaults( + until duration: Duration? = nil, + pollingEvery interval: Duration? = nil ) -> Self } -@available(macOS 13, iOS 17, watchOS 9, tvOS 17, visionOS 1, *) -extension Trait where Self == ConfirmPassesAlwaysConfigurationTrait { - /// Specifies defaults for ``confirmAlwaysPasses`` in the test or suite. +@available(_clockAPI, *) +extension Trait where Self == PollingUntilStopsPassingConfigurationTrait { + /// Specifies defaults for ``confirmPassesAlways`` in the test or suite. /// /// - Parameters: - /// - maxPollingIterations: The maximum amount of times to attempt polling. - /// If nil, polling will be attempted up to 1000 times. - /// `maxPollingIterations` must be greater than 0. - /// - pollingInterval: The minimum amount of time to wait between polling + /// - duration: The expected length of time to continue polling for. + /// This value may not correspond to the wall-clock time that polling + /// lasts for, especially on highly-loaded systems with a lot of tests + /// running. + /// if nil, polling will be attempted for approximately 1 second. + /// `duration` must be greater than 0. + /// - interval: The minimum amount of time to wait between polling /// attempts. /// If nil, polling will wait at least 1 millisecond between polling /// attempts. - /// `pollingInterval` must be greater than 0. - public static func confirmAlwaysPassesDefaults( - maxPollingIterations: Int? = nil, - pollingInterval: Duration? = nil + /// `interval` must be greater than 0. + public static func pollingUntilStopsPassingDefaults( + until duration: Duration? = nil, + pollingEvery interval: Duration? = nil ) -> Self } ``` -Specifying `maxPollingIterations` or `pollingInterval` directly on either -`confirmPassesEventually` or `confirmAlwaysPasses` will override any value -provided by the trait. +Specifying `duration` or `interval` directly on either new `confirmation` +function will override any value provided by the relevant trait. Additionally, +when multiple of these configuration traits are specified, the innermost or +last trait will be applied. ### Default Polling Configuration -For both `confirmPassesEventually` and `confirmsAlwaysPasses`, the Testing -library will default `maxPollingIterations` to 1000, and `pollingInterval` to -1 millisecond. This allows for tests on lightly-loaded systems such as developer -workstations to run in a little over 1 second wall-clock time, while still -being able to gracefully handle running on large loads. +For all polling confirmations, the Testing library will default `duration` to +1 second, and `interval` to 1 millisecond. ### Platform Availability Polling confirmations will not be available on platforms that do not support Swift Concurrency. +### Duration and Concurrent Execution + +It is an unfortunate side effect that directly using the `duration` to determine +when to stop polling (i.e. `while duration has not elapsed { poll() }`) is +unreliable in a parallel execution environment. Especially on systems that are +under-resourced, under very high load, or both - such as CI systems. This is +especially the case for the Testing library, which, at time of writing, submits +every test at once to the concurrency system for scheduling. Under this +environment, with heavily-burdened machines running test suites with a very +large amount of tests, there is a very real case that a polling confirmation's +`duration` might elapse before the `body` has had a chance to return even once. + +To prevent this, the Testing library will calculate how many times to poll the +`body`. This is done by dividing the `duration` by the `interval`. For example, +with the default 1 second duration and 1 millisecond interval, the Testing +library will poll 1000 times, waiting 1 millisecond between polling attempts. +This works and is immune to the issues posed by concurrent execution on +heavily-burdened systems. +This is also very easy for test authors to understand and predict, even if it is +not fully accurate - each poll attempt takes some amount of time, even for very +fast `body` closures. Which means that the real-time duration of a polling +confirmation will always be longer than the value specified in the `duration` +argument. + ### Usage These functions can be used with an async test function: @@ -393,7 +358,7 @@ These functions can be used with an async test function: Task { await subject.raiseDolphins() } - await confirmPassesEventually { + await confirmation(until: .firstPass) { await subject.dolphins.count == 1 } } @@ -405,12 +370,12 @@ will end, and no failure will be reported. Polling will be stopped in the following cases: -- After the expression has been evaluated up to the count specified in - `maxPollingInterations`. +- The specified `duration` has elapsed. - If the task that started the polling is cancelled. -- For `confirmPassesEventually`: The first time the closure returns true or a - non-nil value -- For `confirmAlwaysPasses`: The first time the closure returns false or nil. +- For `PollingStopCondition.firstPass`: The first time the closure returns true + or a non-nil value +- For `PollingStopCondition.stopsPassing`: The first time the closure returns + false or nil. - The first time the closure throws an error. ## Source compatibility @@ -421,13 +386,49 @@ if needed. ## Future directions +### More `confirmation` types + We plan to add support for more push-based monitoring, such as integrating with the Observation module to monitor changes to `@Observable` objects during some lifetime. +These are out of scope for this proposal, and may be part of future proposals. + +### Adding timeouts to existing `confirmation` APIs + +One common request for the existing `confirmation` APIs is a timeout: wait +either until the condition is met, or some amount of time has passed. Adding +that would require additional consideration outside of the context of this +proposal. As such, adding timeouts to the existing (or future) `confirmation` +APIs may be part of a future proposal. + +### More Stop Conditions + +One possible future direction is adding additional stop conditions. For example, +a stop condition where we expect the body closure to initially be false, but to +continue passing once it starts passing. Or a `custom` stop condition, allowing +test authors to define their own stop conditions. + +In order to keep this proposal focused, I chose not to add them yet. They may +be added as part of future proposals. + ## Alternatives considered -### Use timeouts +### Use separate functions instead of the `PollingStopCondition` enum + +Instead of the `PollingStopCondition` enum, we could have created different +functions for each stop condition. This would double the number new confirmation +functions being added, and require additional `confirmation` functions to be +added as we define new stop conditions. In addition to ballooning the number +of `confirmation` functions, this would also harm usability: to differentiate +polling confirmations from the other `confirmation` functions, there needs to be +at least one named argument without a default which isn't the `body` closure. +I was unwilling to compromise on the `duration` and `interval` arguments, +because being able to fall back to defaults is important to usability. +Instead, I created the `stopCondition` argument as the one named argument +without a default. + +### Directly use timeouts Polling could be written in such a way that it stops after some amount of time has passed. Naively, this could be written as: @@ -448,38 +449,16 @@ tests are executed serially, the concurrent test runner the testing library uses means that timeouts are inherently unreliable. Importantly, timeouts become more unreliable the more tests in the test suite. -### Use only polling iterations +### Use polling iterations -Another option considered was only using polling iterations. Naively, this -would write the main polling loop as: - -```swift -func poll(iterations: Int, expression: () -> Bool) async -> Bool { - for _ in 0.. Date: Mon, 28 Jul 2025 14:25:27 -0700 Subject: [PATCH 12/21] Mention the new Issue.Kind case, as well as why polling confirmations do not have a configurable clock --- .../testing/NNNN-polling-confirmations.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 0a223d3084..31fcc2be45 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -217,6 +217,25 @@ confirmation doesn't pass: public struct PollingFailedError: Error, Sendable, CustomIssueRepresentable {} ``` +### New `Issue.Kind` case + +A new issue kind will be added to report specifically when a test fails due to +a failed polling confirmation. + +```swift +public struct Issue { + public enum Kind { + /// An issue due to a polling confirmation having failed. + /// + /// This issue can occur when calling ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` + /// or + /// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` + /// whenever the polling fails, as described in ``PollingStopCondition``. + case pollingConfirmationFailed + } +} +``` + ### New Traits Two new traits will be added to change the default values for the @@ -460,6 +479,16 @@ to predict a good-enough polling iterations value. Most test authors will think in terms of a duration, and we would expect nearly all test authors to add helpers to compute a polling iteration for them. +### Take in a `Clock` instance + +Polling confirmations could take in and use an custom Clock by test authors. +This is not supported because Polling is often used to wait out other delays, +which may or may not use the specified Clock. By staying with the default +continuous clock, Polling confirmations will continue to work even if a test +author otherwise uses a non-standard clock, such as one that skips all sleep +calls, or a clock that allows test authors to specifically control how it +advances. + ### Use macros instead of functions Instead of adding new bare functions, polling could be written as additional From 49119a54e29e48b7f02093f7246127ed230b46d9 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Tue, 19 Aug 2025 14:14:53 -0700 Subject: [PATCH 13/21] PR Feedback --- .../testing/NNNN-polling-confirmations.md | 188 ++++++++---------- 1 file changed, 88 insertions(+), 100 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 31fcc2be45..83585eb4ec 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -93,17 +93,15 @@ testing library: /// This value may not correspond to the wall-clock time that polling lasts /// for, especially on highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last -/// ``PollingUntilFirstPassConfigurationTrait`` or -/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or -/// suite. +/// ``PollingConfirmationConfigurationTrait`` added to the test or suite +/// with a matching stopCondition. /// If no such trait has been added, then polling will be attempted for /// about 1 second before recording an issue. /// `duration` must be greater than 0. /// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last -/// ``PollingUntilFirstPassConfigurationTrait`` or -/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or -/// suite. +/// ``PollingConfirmationConfigurationTrait`` added to the test or suite +/// with a matching stopCondition. /// If no such trait has been added, then polling will wait at least /// 1 millisecond between polling attempts. /// `interval` must be greater than 0. @@ -119,7 +117,7 @@ testing library: /// complex scenarios where other forms of confirmation are insufficient. For /// example, waiting on some state to change that cannot be easily confirmed /// through other forms of `confirmation`. -@available(_clockAPI, *) +@available(macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0, *) public func confirmation( _ comment: Comment? = nil, until stopCondition: PollingStopCondition, @@ -140,17 +138,15 @@ public func confirmation( /// This value may not correspond to the wall-clock time that polling lasts /// for, especially on highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last -/// ``PollingUntilFirstPassConfigurationTrait`` or -/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or -/// suite. +/// ``PollingConfirmationConfigurationTrait`` added to the test or suite +/// with a matching stopCondition. /// If no such trait has been added, then polling will be attempted for /// about 1 second before recording an issue. /// `duration` must be greater than 0. /// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last -/// ``PollingUntilFirstPassConfigurationTrait`` or -/// ``PollingUntilStopsPassingConfigurationTrait`` added to the test or -/// suite. +/// ``PollingConfirmationConfigurationTrait`` added to the test or suite +/// with a matching stopCondition. /// If no such trait has been added, then polling will wait at least /// 1 millisecond between polling attempts. /// `interval` must be greater than 0. @@ -168,7 +164,7 @@ public func confirmation( /// complex scenarios where other forms of confirmation are insufficient. For /// example, waiting on some state to change that cannot be easily confirmed /// through other forms of `confirmation`. -@available(_clockAPI, *) +@available(macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0, *) @discardableResult public func confirmation( _ comment: Comment? = nil, @@ -191,7 +187,7 @@ how the confirmation should be handled. ```swift /// A type defining when to stop polling early. /// This also determines what happens if the duration elapses during polling. -public enum PollingStopCondition: Sendable { +public enum PollingStopCondition: Sendable, Equatable { /// Evaluates the expression until the first time it returns true. /// If it does not pass once by the time the timeout is reached, then a /// failure will be reported. @@ -214,7 +210,7 @@ confirmation doesn't pass: ```swift /// A type describing an error thrown when polling fails. -public struct PollingFailedError: Error, Sendable, CustomIssueRepresentable {} +public struct PollingFailedError: Error, Sendable {} ``` ### New `Issue.Kind` case @@ -224,7 +220,10 @@ a failed polling confirmation. ```swift public struct Issue { + // ... public enum Kind { + // ... + /// An issue due to a polling confirmation having failed. /// /// This issue can occur when calling ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` @@ -232,62 +231,56 @@ public struct Issue { /// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` /// whenever the polling fails, as described in ``PollingStopCondition``. case pollingConfirmationFailed + + // ... } + + // ... } ``` -### New Traits +### New Trait -Two new traits will be added to change the default values for the -`duration` and `interval` arguments. Test authors will often want to poll for -the `firstPass` stop condition for longer than they poll for the -`stopsPassing` stop condition, which is why there are separate traits for -configuring defaults for these functions. +A new trait will be added to change the default values for the +`duration` and `interval` arguments for matching `PollingStopCondition`s. +Test authors will often want to poll for the `firstPass` stop condition for +longer than they poll for the `stopsPassing` stop condition, which is why there +are separate traits for configuring defaults for these functions. ```swift /// A trait to provide a default polling configuration to all usages of /// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` /// and /// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` -/// within a test or suite for the ``PollingStopCondition.firstPass`` -/// stop condition. -/// -/// To add this trait to a test, use the -/// ``Trait/pollingUntilFirstPassDefaults`` function. -@available(_clockAPI, *) -public struct PollingUntilFirstPassConfigurationTrait: TestTrait, SuiteTrait { - /// How long to continue polling for - public var duration: Duration? - /// The minimum amount of time to wait between polling attempts - public var interval: Duration? - - public var isRecursive: Bool { true } -} - -/// A trait to provide a default polling configuration to all usages of -/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` -/// and -/// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` -/// within a test or suite for the ``PollingStopCondition.stopsPassing`` -/// stop condition. +/// within a test or suite using the specified stop condition. /// -/// To add this trait to a test, use the ``Trait/pollingUntilStopsPassingDefaults`` +/// To add this trait to a test, use the ``Trait/pollingConfirmationDefaults`` /// function. -@available(_clockAPI, *) -public struct PollingUntilStopsPassingConfigurationTrait: TestTrait, SuiteTrait { - /// How long to continue polling for - public var duration: Duration? - /// The minimum amount of time to wait between polling attempts - public var interval: Duration? - - public var isRecursive: Bool { true } +@available(macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0, *) +public struct PollingConfirmationConfigurationTrait: TestTrait, SuiteTrait { + /// The stop condition to this configuration is valid for + public var stopCondition: PollingStopCondition { get } + + /// How long to continue polling for. If nil, this will fall back to the next + /// inner-most `PollingUntilStopsPassingConfigurationTrait.duration` value. + /// If no non-nil values are found, then it will use 1 second. + public var duration: Duration? { get } + + /// The minimum amount of time to wait between polling attempts. If nil, this + /// will fall back to earlier `PollingUntilStopsPassingConfigurationTrait.interval` + /// values. If no non-nil values are found, then it will use 1 millisecond. + public var interval: Duration? { get } + + /// This trait will be recursively applied to all children. + public var isRecursive: Bool { get } } -@available(_clockAPI, *) -extension Trait where Self == PollingUntilFirstPassConfigurationTrait { - /// Specifies defaults for ``confirmPassesEventually`` in the test or suite. +@available(macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0, *) +extension Trait where Self == PollingConfirmationConfigurationTrait { + /// Specifies defaults for polling confirmations in the test or suite. /// /// - Parameters: + /// - stopCondition: The `PollingStopCondition` this trait applies to. /// - duration: The expected length of time to continue polling for. /// This value may not correspond to the wall-clock time that polling /// lasts for, especially on highly-loaded systems with a lot of tests @@ -299,30 +292,9 @@ extension Trait where Self == PollingUntilFirstPassConfigurationTrait { /// If nil, polling will wait at least 1 millisecond between polling /// attempts. /// `interval` must be greater than 0. - public static func pollingUntilFirstPassDefaults( - until duration: Duration? = nil, - pollingEvery interval: Duration? = nil - ) -> Self -} - -@available(_clockAPI, *) -extension Trait where Self == PollingUntilStopsPassingConfigurationTrait { - /// Specifies defaults for ``confirmPassesAlways`` in the test or suite. - /// - /// - Parameters: - /// - duration: The expected length of time to continue polling for. - /// This value may not correspond to the wall-clock time that polling - /// lasts for, especially on highly-loaded systems with a lot of tests - /// running. - /// if nil, polling will be attempted for approximately 1 second. - /// `duration` must be greater than 0. - /// - interval: The minimum amount of time to wait between polling - /// attempts. - /// If nil, polling will wait at least 1 millisecond between polling - /// attempts. - /// `interval` must be greater than 0. - public static func pollingUntilStopsPassingDefaults( - until duration: Duration? = nil, + public static func pollingConfirmationDefaults( + until stopCondition: PollingStopCondition, + within duration: Duration? = nil, pollingEvery interval: Duration? = nil ) -> Self } @@ -330,19 +302,21 @@ extension Trait where Self == PollingUntilStopsPassingConfigurationTrait { Specifying `duration` or `interval` directly on either new `confirmation` function will override any value provided by the relevant trait. Additionally, -when multiple of these configuration traits are specified, the innermost or -last trait will be applied. - -### Default Polling Configuration - -For all polling confirmations, the Testing library will default `duration` to -1 second, and `interval` to 1 millisecond. +when multiple of these configuration traits with matching stop conditions are +specified, the innermost or last trait will be applied. When no trait with a +matching stop condition is found and no `duration` or `interval` values are +specified at the callsite, then the Testing library will use some default +values. ### Platform Availability Polling confirmations will not be available on platforms that do not support Swift Concurrency. +Polling confirmations will also not be available on platforms that do not +have the `Clock`, `Duration`, and related types. For Apple platforms, this +requires macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0 and visionOS 1.0. + ### Duration and Concurrent Execution It is an unfortunate side effect that directly using the `duration` to determine @@ -383,19 +357,16 @@ These functions can be used with an async test function: } ``` -With the definition of `Aquarium` above, the closure will only need to be +With the definition of `Aquarium` above, the closure may only need to be evaluated a few times before it starts returning true. At which point polling will end, and no failure will be reported. -Polling will be stopped in the following cases: +Polling will be stopped when either: -- The specified `duration` has elapsed. -- If the task that started the polling is cancelled. -- For `PollingStopCondition.firstPass`: The first time the closure returns true - or a non-nil value -- For `PollingStopCondition.stopsPassing`: The first time the closure returns - false or nil. -- The first time the closure throws an error. +- the specified `duration` has elapsed, +- the task that started the polling is cancelled, +- the closure returns a value that satisfies the stopping condition, or +- the closure throws an error. ## Source compatibility @@ -431,6 +402,21 @@ test authors to define their own stop conditions. In order to keep this proposal focused, I chose not to add them yet. They may be added as part of future proposals. +### Curved polling rates + +As initially specified, the polling rate is flat: poll, sleep for the +specified polling interval, repeat until the stop condition or timeout is +reached. + +Instead, polling could be implemented as a curve. For example, poll very +frequently at first, but progressively wait longer and longer between poll +attempts. Or the opposite: poll sporadically at first, increasing in frequency +as polling continues. We could even offer custom curve options. + +For this initial implementation, I wanted to keep this simple. As such, while +a curve is promising, I think it is better considered on its own as a separate +proposal. + ## Alternatives considered ### Use separate functions instead of the `PollingStopCondition` enum @@ -473,11 +459,13 @@ unreliable the more tests in the test suite. Another option considered was using polling iterations, either solely or combined with the interval value. -However, while this works and is resistant to many of the issues timeouts face -in concurrent testing environments, it is extremely difficult for test authors -to predict a good-enough polling iterations value. Most test authors will think -in terms of a duration, and we would expect nearly all test authors to -add helpers to compute a polling iteration for them. +While this works and is resistant to many of the issues timeouts face +in concurrent testing environments - which is why polling is implemented using +iterations & sleep intervals - it is extremely difficult for test authors to +predict a good-enough polling iterations value, reducing the utility of this +feature. Most test authors think in terms of a duration, and I would expect test +authors to either not use this feature, or to add helpers to compute a polling +iteration count from a duration value anyway. ### Take in a `Clock` instance From 342f6409f6458767757a658d88672ca1e16784c6 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Sat, 30 Aug 2025 15:17:45 -0700 Subject: [PATCH 14/21] Polling confirmations: expand the motivation section, adding a section on when polling should not be used. --- .../testing/NNNN-polling-confirmations.md | 122 +++++++++++++++--- 1 file changed, 104 insertions(+), 18 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 83585eb4ec..461e09c1da 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -20,24 +20,66 @@ APIs or awaiting on an `async` callable in order to block test execution until a callback is called, or an async callable returns. However, this requires the code being tested to support callbacks or return a status as an async callable. -This proposal adds another avenue for waiting for code to update to a specified -value, by proactively polling the test closure until it passes or a timeout is -reached. - -More concretely, we can imagine a type that updates its status over an -indefinite timeframe: +Consider the following class, `Aquarium`, modeling raising dolphins: ```swift -actor Aquarium { - var dolphins: [Dolphin] - - func raiseDolphins() async { - // over a very long timeframe - dolphins.append(Dolphin()) +@MainActor +final class Aquarium { + private(set) var isRaising = false + var hasFunding = true + + func raiseDolphins() { + Task { + if hasFunding { + isRaising = true + + // Long running work that I'm not qualified to describe. + // ... + + isRaising = false + } + } } } ``` +As is, it is extremely difficult to check that `isRaising` is correctly set to +true once `raiseDolphins` is called. The system offers test authors no +control for when the created task runs, leaving test authors add arbitrary sleep +calls. Like this example: + +```swift +@Test func `raiseDolphins if hasFunding sets isRaising to true`() async throws { + let subject = Aquarium() + subject.hasFunding = true + + subject.raiseDolphins() + + try await Task.sleep(for: .seconds(1)) + + #expect(subject.isRaising == true) +} +``` + +This requires test authors to have to figure out how long to wait so that +`isRaising` will reliably be set to true, while not waiting too long, such that +the test suite is not unnecessarily delayed or task itself finishes. + +As another example, imagine a test author wants to verify that no dolphins are +raised when there isn't any funding. There isn't and can't be a mechanism for +verifying that `isRaising` is never set to `true`, but if we constrain the +problem to within a given timeframe, then we can have a reasonable assumption +that `isRaising` remains set to false. Again, without some other mechanism to +notify the test when to check `isRaising`, test authors are left to add +arbitrary sleep calls, when having the ability to fail fast would save a not +insignificant amount of time in the event that `isRaising` is mistakenly set to +true. + +This proposal introduces polling to help test authors address these cases. In +this and other similar cases, polling makes these tests practical or even +possible, as well as speeding up the execution of individual tests as well as +the entire test suite. + ## Proposed solution This proposal introduces new members of the `confirmation` family of functions: @@ -63,15 +105,26 @@ When `PollingStopCondition.stopsPassing` is specified, reaching the duration stop point will mark the confirmation as passing. Tests will now be able to poll code updating in the background using either of -the stop conditions: +the stop conditions. For the example of `Aquarium.raiseDolphins`, valid tests +might look like: ```swift -let subject = Aquarium() -Task { - await subject.raiseDolphins() +@Test func `raiseDolphins if hasFunding sets isRaising to true`() async throws { + let subject = Aquarium() + subject.hasFunding = true + + subject.raiseDolphins() + + try await confirmation(until: .firstPass) { subject.isRaising == true } } -await confirmation(until: .firstPass) { - subject.dolphins.count == 1 + +@Test func `raiseDolphins if no funding keeps isRaising false`() async throws { + let subject = Aquarium() + subject.hasFunding = false + + subject.raiseDolphins() + + try await confirmation(until: .stopsPassing) { subject.isRaising == false } } ``` @@ -368,6 +421,39 @@ Polling will be stopped when either: - the closure returns a value that satisfies the stopping condition, or - the closure throws an error. +### When Polling should not be used + +Polling is not a silver bullet, and should not be abused. In many cases, the +problems that polling solves can be solved through other, better means. Such as +the observability system, using Async sequences, callbacks, or delegates. When +possible, implementation code which requires polling to be tested should be +refactored to support other means. Polling exists for the case where such +refactors are either not possible or require a large amount of overhead. + +Polling introduces a small amount of instability to the tests - in the example +of waiting for `Aquarium.isRaising` to be set to true, it is entirely possible +that, unless the code covered by +`// Long running work that I'm not qualified to describe` has a test-controlled +means to block further execution, the created `Task` could finish between +polling attempts - resulting `Aquarium.isRaising` to always be read as false, +and failing the test despite the code having done the right thing. + +Polling also only offers a snapshot in time of the state. When +`PollingStopCondition.firstPass` is used, polling will stop and return a pass +after the first time the `body` returns true, even if any subsequent calls +would've returned false. + +Furthermore, polling introduces delays to the running code. This isn't that +much of a concern for `PollingStopCondition.firstPass`, where the passing +case minimizes test execution time. However, the +passing case when using `PollingStopCondition.stopsPassing` utilizes the full +duration specified. If the test author specifies the polling duration to be +10 minutes, then the test will poll for approximately that long, so long as the +polling body keeps returning true. + +Despite all this, we think that polling is an extremely valuable tool, and is +worth adding to the Testing library. + ## Source compatibility This is a new interface that is unlikely to collide with any existing From dee97928e88f578175b39a0dbb51efc433acc1c2 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Mon, 8 Sep 2025 15:16:01 -0700 Subject: [PATCH 15/21] Polling confirmations: Expand on why directly using duration doesn't work. --- .../testing/NNNN-polling-confirmations.md | 56 +++++++++++++------ 1 file changed, 39 insertions(+), 17 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 461e09c1da..42c5be46af 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -372,27 +372,49 @@ requires macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0 and visionOS 1.0. ### Duration and Concurrent Execution -It is an unfortunate side effect that directly using the `duration` to determine -when to stop polling (i.e. `while duration has not elapsed { poll() }`) is -unreliable in a parallel execution environment. Especially on systems that are -under-resourced, under very high load, or both - such as CI systems. This is -especially the case for the Testing library, which, at time of writing, submits -every test at once to the concurrency system for scheduling. Under this -environment, with heavily-burdened machines running test suites with a very -large amount of tests, there is a very real case that a polling confirmation's -`duration` might elapse before the `body` has had a chance to return even once. +Directly using the `duration` to determine when to stop polling is incredibly +unreliable in a +parallel execution environment, like most platforms Swift Testing runs on. The +fundamental issue is that if polling were to directly use a timeout to determine +when to stop execution, such as: + +```swift +let end = ContinuousClock.now + timeout +while ContinuousClock.now < end { + if await runPollAndCheckIfShouldStop() { + // alert the user! + } + await Task.yield +} +``` + +With enough system load, the polling check might only run a handful of times, or +even once, before the timeout is triggered. In this case, the component being +polled might not have had time to update its status such that polling could +pass. Using the `Aquarium.raiseDolphins` example from earlier: On the first time +that `runPollAndCheckIfShouldStop` executes the background task created by +`raiseDolphins` might not have started executing its closure, leading the +polling to continue. If the system is under sufficiently high load, which can +be caused by having a very large amount of tests in the test suite, then once +the `Task.yield` finishes and the while condition is checked again, then it +might now be past the timeout. Or the task created by `Aquarium.runDolphins` +might have started and the closure run to completion before the next time +`runPollAndCheckIfShouldStop()` is executed. Or both. This approach of using +a clock to check when to stop is inherently unreliable, and becomes increasingly +unreliable as the load on the system increases and as the size of the test suite +increases. To prevent this, the Testing library will calculate how many times to poll the -`body`. This is done by dividing the `duration` by the `interval`. For example, +`body`. This can be done by dividing the `duration` by the `interval`. For example, with the default 1 second duration and 1 millisecond interval, the Testing -library will poll 1000 times, waiting 1 millisecond between polling attempts. -This works and is immune to the issues posed by concurrent execution on -heavily-burdened systems. +library could poll 1000 times, waiting 1 millisecond between polling attempts. +This is immune to the issues posed by concurrent execution, allowing it to +scale with system load and test suite size. This is also very easy for test authors to understand and predict, even if it is -not fully accurate - each poll attempt takes some amount of time, even for very -fast `body` closures. Which means that the real-time duration of a polling -confirmation will always be longer than the value specified in the `duration` -argument. +not fully accurate to wall-clock time - each poll attempt takes some amount of +time, even for very fast `body` closures. Which means that the real-time +duration of a polling confirmation will always be longer than the value +specified in the `duration` argument. ### Usage From 15d4ece0fe7070b4ba71686609875b16625f58cc Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Thu, 11 Sep 2025 10:51:39 -0700 Subject: [PATCH 16/21] Update proposals/testing/NNNN-polling-confirmations.md Co-authored-by: Jonathan Grynspan --- proposals/testing/NNNN-polling-confirmations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 42c5be46af..1d6aa77c2b 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -384,7 +384,7 @@ while ContinuousClock.now < end { if await runPollAndCheckIfShouldStop() { // alert the user! } - await Task.yield + await Task.yield() } ``` From 34c30c88b99d612149b123787f68ff3f85771942 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Sun, 21 Sep 2025 20:53:00 -0700 Subject: [PATCH 17/21] Testing/Polling Confirmations: Add a failure reason API --- .../testing/NNNN-polling-confirmations.md | 31 +++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 1d6aa77c2b..9d5b1821d6 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -256,6 +256,23 @@ public enum PollingStopCondition: Sendable, Equatable { } ``` +### New `PollingFailureReason` enum + +There are 2 reasons why polling confirmations can fail: the stop condition +failed, or the confirmation was cancelled during the run. To help express this, +we will be adding a new `PollingFailureReason` enum. + +```swift +/// A type describing why polling failed +public enum PollingFailureReason: Sendable, Codable { + /// The polling failed because it was cancelled using `Task.cancel`. + case cancelled + + /// The polling failed because the stop condition failed. + case stopConditionFailed(PollingStopCondition) +} +``` + ### New Error Type A new error type, `PollingFailedError` to be thrown when the polling @@ -263,7 +280,13 @@ confirmation doesn't pass: ```swift /// A type describing an error thrown when polling fails. -public struct PollingFailedError: Error, Sendable {} +public struct PollingFailedError: Error, Sendable { + /// A user-specified comment describing this confirmation + public var comment: Comment? { get } + + /// Why polling failed, either cancelled, or because the stop condition failed. + public var reason: PollingFailureReason { get } +} ``` ### New `Issue.Kind` case @@ -279,11 +302,15 @@ public struct Issue { /// An issue due to a polling confirmation having failed. /// + /// - Parameters: + /// - reason: The ``PollingFailureReason`` behind why the polling + /// confirmation failed. + /// /// This issue can occur when calling ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-455gr`` /// or /// ``confirmation(_:until:within:pollingEvery:isolation:sourceLocation:_:)-5tnlk`` /// whenever the polling fails, as described in ``PollingStopCondition``. - case pollingConfirmationFailed + case pollingConfirmationFailed(reason: PollingFailureReason) // ... } From c939b48c7306c3048fc80b1b8f145364a86545f2 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Wed, 22 Oct 2025 21:31:19 -0700 Subject: [PATCH 18/21] PollingFailureReason is now nested inside of PollingFailedError --- .../testing/NNNN-polling-confirmations.md | 34 ++++++++----------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 9d5b1821d6..f8a7558b59 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -256,36 +256,30 @@ public enum PollingStopCondition: Sendable, Equatable { } ``` -### New `PollingFailureReason` enum - -There are 2 reasons why polling confirmations can fail: the stop condition -failed, or the confirmation was cancelled during the run. To help express this, -we will be adding a new `PollingFailureReason` enum. - -```swift -/// A type describing why polling failed -public enum PollingFailureReason: Sendable, Codable { - /// The polling failed because it was cancelled using `Task.cancel`. - case cancelled - - /// The polling failed because the stop condition failed. - case stopConditionFailed(PollingStopCondition) -} -``` - -### New Error Type +### New `PollingFailedError` Error Type and `PollingFailedError.Reason` enum A new error type, `PollingFailedError` to be thrown when the polling -confirmation doesn't pass: +confirmation doesn't pass. This contains a nested enum expressing the 2 reasons +why polling confirmations can fail: the stop condition failed, or the +confirmation was cancelled during the run: ```swift /// A type describing an error thrown when polling fails. public struct PollingFailedError: Error, Sendable { + /// A type describing why polling failed + public enum Reason: Sendable, Codable { + /// The polling failed because it was cancelled using `Task.cancel`. + case cancelled + + /// The polling failed because the stop condition failed. + case stopConditionFailed(PollingStopCondition) + } + /// A user-specified comment describing this confirmation public var comment: Comment? { get } /// Why polling failed, either cancelled, or because the stop condition failed. - public var reason: PollingFailureReason { get } + public var reason: Reason { get } } ``` From a417230ff9a22873fdf6b5e386f15528d68d5a15 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Fri, 24 Oct 2025 21:46:30 -0700 Subject: [PATCH 19/21] Update documentation comments in the code samples --- .../testing/NNNN-polling-confirmations.md | 50 ++++++++++--------- 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index f8a7558b59..a156a8efb2 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -139,18 +139,18 @@ testing library: /// Poll expression within the duration based on the given stop condition /// /// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. +/// - comment: A user-specified comment describing this confirmation. /// - stopCondition: When to stop polling. /// - duration: The expected length of time to continue polling for. -/// This value may not correspond to the wall-clock time that polling lasts -/// for, especially on highly-loaded systems with a lot of tests running. +/// This value does not incorporate the time to run `body`, and may not +/// correspond to the wall-clock time that polling lasts for, especially on +/// highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last /// ``PollingConfirmationConfigurationTrait`` added to the test or suite /// with a matching stopCondition. /// If no such trait has been added, then polling will be attempted for /// about 1 second before recording an issue. -/// `duration` must be greater than 0. +/// `duration` must be greater than or equal to `interval`. /// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last /// ``PollingConfirmationConfigurationTrait`` added to the test or suite @@ -159,9 +159,10 @@ testing library: /// 1 millisecond between polling attempts. /// `interval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. +/// - sourceLocation: The location in source where the confirmation was called. +/// - body: The function to invoke. The expression is considered to pass if +/// the `body` returns true. Similarly, the expression is considered to fail +/// if `body` returns false. /// /// - Throws: A `PollingFailedError` if the `body` does not return true within /// the polling duration. @@ -184,18 +185,18 @@ public func confirmation( /// Confirm that some expression eventually returns a non-nil value /// /// - Parameters: -/// - comment: An optional comment to apply to any issues generated by this -/// function. +/// - comment: A user-specified comment describing this confirmation. /// - stopCondition: When to stop polling. /// - duration: The expected length of time to continue polling for. -/// This value may not correspond to the wall-clock time that polling lasts -/// for, especially on highly-loaded systems with a lot of tests running. +/// This value does not incorporate the time to run `body`, and may not +/// correspond to the wall-clock time that polling lasts for, especially on +/// highly-loaded systems with a lot of tests running. /// If nil, this uses whatever value is specified under the last /// ``PollingConfirmationConfigurationTrait`` added to the test or suite /// with a matching stopCondition. /// If no such trait has been added, then polling will be attempted for /// about 1 second before recording an issue. -/// `duration` must be greater than 0. +/// `duration` must be greater than or equal to `interval`. /// - interval: The minimum amount of time to wait between polling attempts. /// If nil, this uses whatever value is specified under the last /// ``PollingConfirmationConfigurationTrait`` added to the test or suite @@ -204,9 +205,10 @@ public func confirmation( /// 1 millisecond between polling attempts. /// `interval` must be greater than 0. /// - isolation: The actor to which `body` is isolated, if any. -/// - sourceLocation: The source location to whych any recorded issues should -/// be attributed. -/// - body: The function to invoke. +/// - sourceLocation: The location in source where the confirmation was called. +/// - body: The function to invoke. The expression is considered to pass if +/// the `body` returns a non-nil value. Similarly, the expression is +/// considered to fail if `body` returns nil. /// /// - Throws: A `PollingFailedError` if the `body` does not return true within /// the polling duration. @@ -240,17 +242,17 @@ how the confirmation should be handled. ```swift /// A type defining when to stop polling early. /// This also determines what happens if the duration elapses during polling. -public enum PollingStopCondition: Sendable, Equatable { - /// Evaluates the expression until the first time it returns true. - /// If it does not pass once by the time the timeout is reached, then a +public enum PollingStopCondition: Sendable, Equatable, Codable { + /// Evaluates the expression until the first time it passes + /// If it does not pass once by the time the duration is reached, then a /// failure will be reported. case firstPass - /// Evaluates the expression until the first time it returns false. - /// If the expression returns false, then a failure will be reported. - /// If the expression only returns true before the timeout is reached, then + /// Evaluates the expression until the first time it returns fails. + /// If the expression fails, then a failure will be reported. + /// If the expression only passes before the duration is reached, then /// no failure will be reported. - /// If the expression does not finish evaluating before the timeout is + /// If the expression does not finish evaluating before the duration is /// reached, then a failure will be reported. case stopsPassing } @@ -360,7 +362,7 @@ extension Trait where Self == PollingConfirmationConfigurationTrait { /// lasts for, especially on highly-loaded systems with a lot of tests /// running. /// if nil, polling will be attempted for approximately 1 second. - /// `duration` must be greater than 0. + /// `duration` must be greater than `interval`. /// - interval: The minimum amount of time to wait between polling /// attempts. /// If nil, polling will wait at least 1 millisecond between polling From 46624a539896430b816e886b24e18d487eb85d95 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Fri, 24 Oct 2025 21:48:44 -0700 Subject: [PATCH 20/21] Testing: Link the second pitch thread for polling confirmations --- proposals/testing/NNNN-polling-confirmations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index a156a8efb2..9519fc849f 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -5,7 +5,7 @@ * Review Manager: TBD * Status: **Awaiting review** * Implementation: [swiftlang/swift-testing#1115](https://github.com/swiftlang/swift-testing/pull/1115) -* Review: ([Pitch](https://forums.swift.org/t/pitch-polling-expectations/79866)) +* Review: ([Pitch 1](https://forums.swift.org/t/pitch-polling-expectations/79866), [Pitch 2](https://forums.swift.org/t/pitch-2-polling-confirmations-in-the-testing-library/81711)) ## Introduction From 3fcb85120a4d6728e0c3ede262d6f373cf633298 Mon Sep 17 00:00:00 2001 From: Rachel Brindle Date: Sun, 26 Oct 2025 12:10:12 -0700 Subject: [PATCH 21/21] Update documentation comment for PollingConfirmationConfigurationTrait to mention that duration may be greater than or equal to interval --- proposals/testing/NNNN-polling-confirmations.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/testing/NNNN-polling-confirmations.md b/proposals/testing/NNNN-polling-confirmations.md index 9519fc849f..26c2d24845 100644 --- a/proposals/testing/NNNN-polling-confirmations.md +++ b/proposals/testing/NNNN-polling-confirmations.md @@ -362,12 +362,12 @@ extension Trait where Self == PollingConfirmationConfigurationTrait { /// lasts for, especially on highly-loaded systems with a lot of tests /// running. /// if nil, polling will be attempted for approximately 1 second. - /// `duration` must be greater than `interval`. + /// If specified, `duration` must be greater than or equal to `interval`. /// - interval: The minimum amount of time to wait between polling /// attempts. /// If nil, polling will wait at least 1 millisecond between polling /// attempts. - /// `interval` must be greater than 0. + /// If specified, `interval` must be greater than 0. public static func pollingConfirmationDefaults( until stopCondition: PollingStopCondition, within duration: Duration? = nil,