1212Many projects want to migrate from XCTest to Swift Testing, and may be in an
1313intermediate state where test helpers written using XCTest API are called from
1414Swift Testing. Today, the Swift Testing and XCTest libraries stand mostly
15- independently, which means an ` XCTAssert ` failure in a Swift Testing test is
16- silently ignored. To address this, we formally declare a set of interoperability
17- principles and propose changes to the handling of specific APIs that will enable
18- users to migrate with confidence.
15+ independently, which means an [ ` XCTAssert ` ] [ XCTest assertions ] failure in a
16+ Swift Testing test or an [ ` #expect ` ] [ Swift Testing expectations ] failure in an
17+ XCTest test is silently ignored. To address this, we formally declare a set of
18+ interoperability principles and propose changes to the handling of specific APIs
19+ that will enable users to migrate with confidence.
1920
2021## Motivation
2122
@@ -44,19 +45,21 @@ class FooTests: XCTestCase {
4445}
4546```
4647
47- Generally, we get into trouble today when ALL the following conditions are met:
48+ Generally, you encounter the above limitation with testing APIs when _ all_ the
49+ following conditions are met:
4850
4951- You call XCTest API in a Swift Testing test, or call Swift Testing API in a
5052 XCTest test,
5153- The API doesn't function as expected in some or all cases, and
5254- You get no notice at build time or runtime about the malfunction
5355
5456For the remainder of this proposal, we’ll describe test APIs which exhibit this
55- problem as ** lossy without interop** .
57+ limitation as ** lossy without interop** .
5658
57- This problem risks regressing test coverage for projects which migrate to Swift
58- Testing. Furthermore, projects that have switched completely to Swift Testing
59- may want to go and ensure they don't inadvertently add XCTest API.
59+ You could regress test coverage if you migrate to Swift Testing without
60+ replacing usage of lossy without interop test APIs. Furthermore, you may want to
61+ ensure you don't inadvertently introduce new XCTest API after completing your
62+ Swift Testing migration.
6063
6164## Proposed solution
6265
@@ -106,7 +109,7 @@ We also propose highlighting usage of above XCTest APIs in Swift Testing:
106109
107110- ** Report [ runtime warning issues] [ ] ** for XCTest API usage in Swift Testing.
108111 This ** applies to both assertion failures _ and successes_ ** ! This notifies you
109- about opportunities to modernise even if your tests currently pass.
112+ about opportunities to modernize even if your tests currently pass.
110113
111114- Opt-in ** strict interop mode** , where XCTest API usage will result in
112115 ` fatalError("Usage of XCTest API in a Swift Testing context is unsupported") ` .
@@ -123,7 +126,8 @@ Here are some concrete examples:
123126
124127We propose supporting the following Swift Testing APIs in XCTest:
125128
126- - ` #expect ` and ` #require `
129+ - [ ` #expect ` and ` #require ` ] [ Swift Testing expectations ]
130+ - Includes [ ` #expect(throws:) ` ] [ testing for errors ]
127131 - Includes [ exit testing] [ ]
128132- ` withKnownIssue ` : marking an XCTest issue in this way will generate a runtime
129133 warning issue. In strict interop mode, this becomes a ` fatalError ` .
@@ -144,7 +148,7 @@ is analogous to `XCTSkip`. If that pitch were accepted, this proposal would
144148support interop of the new API with XCTest.
145149
146150On the other hand, [ traits] [ ] are a powerful Swift Testing feature which is not
147- related to any functionality in XCTest. Therefore, there would be
151+ related to any functionality in XCTest. Therefore, there would not be
148152interoperability for traits under this proposal.
149153
150154Here are some concrete examples:
@@ -174,21 +178,18 @@ Here are some concrete examples:
174178Configure the interoperability mode when running tests using the
175179` SWIFT_TESTING_XCTEST_INTEROP_MODE ` environment variable:
176180
177- | Interop Mode | Issue behaviour across framework boundary | ` SWIFT_TESTING_XCTEST_INTEROP_MODE ` |
181+ | Interop Mode | Issue behavior across framework boundary | ` SWIFT_TESTING_XCTEST_INTEROP_MODE ` |
178182| ------------ | -------------------------------------------------------------------------- | ---------------------------------------------- |
179- | Advisory | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ⚠️ Runtime Warning Issue | ` warning-only ` |
183+ | Advisory | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ⚠️ Runtime Warning Issue | ` advisory ` |
180184| Permissive | XCTest API: ⚠️ Runtime Warning Issue. All Issues: ❌ Test Failure | ` permissive ` , or empty value, or invalid value |
181185| Strict | XCTest API: 💥 ` fatalError ` . Swift Testing API: ❌ Test Failure | ` strict ` |
182186
183- ### Phased Rollout
187+ ## Source compatibility
184188
185189When interoperability is first available, "permissive" will be the default
186- interop mode enabled for new projects. In a future release, "strict" will become
187- the default interop mode.
188-
189- ## Source compatibility
190+ interop mode enabled for new projects.
190191
191- As the main goal of interoperability is to change behaviour , this proposal will
192+ As the main goal of interoperability is to change behavior , this proposal will
192193lead to situations where previously "passing" test code now starts showing
193194failures. We believe this should be a net positive if it can highlight actual
194195bugs you would have missed previously.
@@ -199,31 +200,31 @@ interoperability.
199200
200201## Integration with supporting tools
201202
202- Interoperability will be first available in future toolchain version,
203- hypothetically named ` 6.X ` , where permissive interop mode will be enabled for
204- projects. After that, a ` 7.Y ` release would make strict interop mode the
205- default.
206-
207203- Swift packages: ` swift-tools-version ` declared in Package.swift will be used
208204 to determine interop mode, regardless of the toolchain used to run tests.
205+ Specifically, it will use the default interop mode associated with that
206+ toolchain version ("permissive" for the initial release version).
209207
210- - Otherwise, installed toolchain version will be used to determine interop mode.
208+ - Otherwise, the default interop mode associated with the installed toolchain
209+ version will be used to determine interop mode.
211210
212211- Any project can use ` SWIFT_TESTING_XCTEST_INTEROP_MODE ` to override interop
213- mode at runtime, provided they are on toolchain version ` 6.X ` or newer
212+ mode at runtime.
214213
215214## Future directions
216215
217216There's still more we can do to make it easier to migrate from XCTest to Swift
218217Testing:
219218
219+ - In a future release, we would consider making strict interop mode the default.
220+
220221- Provide fixups at compile-time to replace usage of XCTest API with the
221222 corresponding Swift Testing API, e.g. replace ` XCTAssert ` with ` #expect ` .
222223 However, this would require introspection of the test body to look for XCTest
223224 API usage, which would be challenging to do completely and find usages of this
224225 API within helper methods.
225226
226- - After new API is added to Swift Testing in future, will need to evaluate for
227+ - When new API is added to Swift Testing, we will need to evaluate it for
227228 interoperability with XCTest.
228229
229230## Alternatives considered
@@ -263,16 +264,16 @@ that we want users to migrate to Swift Testing.
263264However, we are especially sensitive to use cases that depend upon the currently
264265lossy without interop APIs. With strict interop mode, the test process will
265266crash on the first instance of XCTest API usage in Swift Testing, completely
266- halting testing. In this same scenario, the default permissive interop mode
267- would record a runtime warning issue and continue the remaining test, which we
268- believe strikes a better balance between notifying users yet not being totally
269- disruptive to the testing flow.
267+ halting testing. In this same scenario, the proposed default permissive interop
268+ mode would record a runtime warning issue and continue the remaining test, which
269+ we believe strikes a better balance between notifying users yet not being
270+ totally disruptive to the testing flow.
270271
271272### Alternative methods to control interop mode
272273
273274- ** Build setting:** e.g. a new ` SwiftSetting ` that can be included in
274- Package.swift or an Xcode project . A project could then configure their test
275- targets to have a non-default interop mode.
275+ Package.swift. A project could then configure their test targets to have a
276+ non-default interop mode.
276277
277278 However, interop is a runtime concept, and would be difficult or at least
278279 non-idiomatic to modify with a build setting.
@@ -291,12 +292,17 @@ disruptive to the testing flow.
291292Thanks to Stuart Montgomery, Jonathan Grynspan, and Brian Croom for feedback on
292293the proposal.
293294
295+ <!-- XCTest -->
294296[ XCTest assertions ] : https://developer.apple.com/documentation/xctest/equality-and-inequality-assertions
295297[ XCTest attachments ] : https://developer.apple.com/documentation/xctest/adding-attachments-to-tests-activities-and-issues
296298[ unconditional failure ] : https://developer.apple.com/documentation/xctest/unconditional-test-failures
297- [ runtime warning issues ] : https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md
298299[ expected failures ] : https://developer.apple.com/documentation/xctest/expected-failures
300+ <!-- Swift Testing -->
301+ [ Swift Testing expectations ] : https://developer.apple.com/documentation/testing/expectations
302+ [ Testing for errors ] : https://developer.apple.com/documentation/testing/testing-for-errors-in-swift-code
303+ [ exit testing ] : https://developer.apple.com/documentation/testing/exit-testing
299304[ issue handling traits ] : https://developer.apple.com/documentation/testing/issuehandlingtrait
305+ [ traits ] : https://developer.apple.com/documentation/testing/traits
306+ <!-- Misc -->
307+ [ runtime warning issues ] : https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0013-issue-severity-warning.md
300308[ test cancellation ] : https://forums.swift.org/t/pitch-test-cancellation/81847
301- [ traits ] : https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/traits
302- [ exit testing ] : https://developer.apple.com/documentation/testing/exit-testing
0 commit comments