Add configuration option for test executor arguments

.github/workflows/test.yml

@@ -79,9 +79,16 @@ jobs:
         - os: macos-15
           python_version: '3.13'
           test_select: ios
+          # Github Actions macOS-15 runner has disk performance issues; so it
+          # has problems using any simulator that isn't pre-warmed.
+          # Unfortunately, the simulator that it picks by default isn't
+          # pre-warmed. So - we need to explicitly select a simulator.
+          test_execution_args: '--simulator "iPhone 16e,OS=18.5"'
         - os: macos-15-intel
           python_version: '3.13'
           test_select: android
+          # Exercise Android on a non-default simulator
+          test_execution_args: '--managed minVersion'
         - os: macos-15
           python_version: '3.13'
           test_select: android
@@ -154,6 +161,7 @@ jobs:
         CIBW_ARCHS_MACOS: x86_64 universal2 arm64
         CIBW_BUILD_FRONTEND: ${{ matrix.test_select && 'build' || 'build[uv]' }}
         CIBW_PLATFORM: ${{ matrix.test_select }}
+        CIBW_TEST_EXECUTION_ARGS: ${{ matrix.test_execution_args }}
 
     - name: Run a sample build (GitHub Action, only)
       uses: ./
@@ -179,6 +187,7 @@ jobs:
       uses: ./
       env:
         CIBW_PLATFORM: ${{ matrix.test_select }}
+        CIBW_TEST_EXECUTION_ARGS: ${{ matrix.test_execution_args }}
       with:
         package-dir: sample_proj
         output-dir: wheelhouse_config_file
@@ -202,6 +211,8 @@ jobs:
         path: wheelhouse/*.whl
 
     - name: Test cibuildwheel
+      env:
+        CIBW_TEST_EXECUTION_ARGS: ${{ matrix.test_execution_args }}
       run: |
         uv run --no-sync bin/run_tests.py --test-select=${{ matrix.test_select || 'native' }} ${{ (runner.os == 'Linux' && runner.arch == 'X64') && '--run-podman' || '' }}
 

README.md

@@ -59,8 +59,8 @@ Usage
 
 |                 | Linux | macOS | Windows | Linux ARM | macOS ARM | Windows ARM | Android | iOS |
 |-----------------|-------|-------|---------|-----------|-----------|-------------|---------|-----|
-| GitHub Actions  | ✅    | ✅    | ✅       | ✅        | ✅        | ✅²         | ✅⁴      | ✅³⁵ |
-| Azure Pipelines | ✅    | ✅    | ✅       |           | ✅        | ✅²         | ✅⁴      | ✅³⁵ |
+| GitHub Actions  | ✅    | ✅    | ✅       | ✅        | ✅        | ✅²         | ✅⁴      | ✅³  |
+| Azure Pipelines | ✅    | ✅    | ✅       |           | ✅        | ✅²         | ✅⁴      | ✅³  |
 | Travis CI       | ✅    |       | ✅      | ✅        |           |             | ✅⁴      |     |
 | CircleCI        | ✅    | ✅    |         | ✅        | ✅        |             | ✅⁴      | ✅³  |
 | Gitlab CI       | ✅    | ✅    | ✅      | ✅¹       | ✅        |             | ✅⁴      | ✅³  |
@@ -70,7 +70,6 @@ Usage
 <sup>² [Uses cross-compilation](https://cibuildwheel.pypa.io/en/stable/faq/#windows-arm64). It is not possible to test `arm64` on this CI platform.</sup><br>
 <sup>³ Requires a macOS runner; runs tests on the simulator for the runner's architecture. </sup><br>
 <sup>⁴ Building for Android requires the runner to be Linux x86_64, macOS ARM64 or macOS x86_64. Testing has [additional requirements](https://cibuildwheel.pypa.io/en/stable/platforms/#android).</sup><br>
-<sup>⁵ The `macos-15` and `macos-latest` images are [incompatible with cibuildwheel at this time](platforms/#ios-system-requirements)</sup><br> when building iOS wheels.
 
 <!--intro-end-->
 
@@ -160,12 +159,13 @@ The following diagram summarises the steps that cibuildwheel takes on each platf
 |  | [`test-groups`](https://cibuildwheel.pypa.io/en/stable/options/#test-groups) | Specify test dependencies from your project's `dependency-groups` |
 |  | [`test-skip`](https://cibuildwheel.pypa.io/en/stable/options/#test-skip) | Skip running tests on some builds |
 |  | [`test-environment`](https://cibuildwheel.pypa.io/en/stable/options/#test-environment) | Set environment variables for the test environment |
+|  | [`test-execution-args`](https://cibuildwheel.pypa.io/en/stable/options/#test-execution-args) | Define additional arguments that will be passed to the command that runs the tests. |
 | **Debugging** | [`debug-keep-container`](https://cibuildwheel.pypa.io/en/stable/options/#debug-keep-container) | Keep the container after running for debugging. |
 |  | [`debug-traceback`](https://cibuildwheel.pypa.io/en/stable/options/#debug-traceback) | Print full traceback when errors occur. |
 |  | [`build-verbosity`](https://cibuildwheel.pypa.io/en/stable/options/#build-verbosity) | Increase/decrease the output of the build |
 
 
-<!--[[[end]]] (sum: FxE3nIgFiY) -->
+<!--[[[end]]] (sum: gNUxwlNyTE) -->
 
 These options can be specified in a pyproject.toml file, or as environment variables, see [configuration docs](https://cibuildwheel.pypa.io/en/latest/configuration/).
 

bin/generate_schema.py

@@ -224,6 +224,9 @@
   test-environment:
     description: Set environment variables for the test environment
     type: string_table
+  test-execution-args:
+    description: Additional arguments for the test runner to use when configuring the test environment
+    type: string_array
 """
 
 schema = yaml.safe_load(starter)
@@ -304,6 +307,7 @@
         test-sources: {"$ref": "#/$defs/inherit"}
         test-requires: {"$ref": "#/$defs/inherit"}
         test-environment: {"$ref": "#/$defs/inherit"}
+        test-execution-args: {"$ref": "#/$defs/inherit"}
 """
 )
 

cibuildwheel/options.py

@@ -110,6 +110,7 @@ class BuildOptions:
     test_extras: str
     test_groups: list[str]
     test_environment: ParsedEnvironment
+    test_execution_args: list[str] | None
     build_verbosity: int
     build_frontend: BuildFrontendConfig
     config_settings: str
@@ -761,6 +762,12 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
                 msg = f"Malformed environment option {test_environment_config!r}"
                 raise errors.ConfigurationError(msg) from e
 
+            test_execution_args = shlex.split(
+                self.reader.get(
+                    "test-execution-args", option_format=ListFormat(sep=" ", quote=shlex.quote)
+                )
+            )
+
             test_requires = self.reader.get(
                 "test-requires", option_format=ListFormat(sep=" ")
             ).split()
@@ -868,6 +875,7 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
                 test_command=test_command,
                 test_sources=test_sources,
                 test_environment=test_environment,
+                test_execution_args=test_execution_args,
                 test_requires=[*test_requires, *test_requirements_from_groups],
                 test_extras=test_extras,
                 test_groups=test_groups,

cibuildwheel/platforms/android.py

@@ -635,17 +635,28 @@ def test_wheel(state: BuildState, wheel: Path) -> None:
         )
         raise errors.FatalError(msg)
 
+    # By default, run on a testbed managed emulator running the newest supported
+    # Android version. However, if the user specifies a --managed or --connected
+    # test execution argument, that argument takes precedence.
+    if state.options.test_execution_args and (
+        "--managed" in state.options.test_execution_args
+        or "--connected" in state.options.test_execution_args
+    ):
+        simulator_args = []
+    else:
+        simulator_args = ["--managed", "maxVersion"]
+
     # Run the test app.
     call(
         state.python_dir / "android.py",
         "test",
-        "--managed",
-        "maxVersion",
         "--site-packages",
         site_packages_dir,
         "--cwd",
         cwd_dir,
+        *simulator_args,
         *(["-v"] if state.options.build_verbosity > 0 else []),
+        *(state.options.test_execution_args if state.options.test_execution_args else []),
         *test_args,
         env=state.build_env,
     )

cibuildwheel/platforms/ios.py

@@ -658,6 +658,11 @@ def build(options: Options, tmp_path: Path) -> None:
                                 testbed_path,
                                 "run",
                                 *(["--verbose"] if build_options.build_verbosity > 0 else []),
+                                *(
+                                    build_options.test_execution_args
+                                    if build_options.test_execution_args
+                                    else []
+                                ),
                                 "--",
                                 *final_command,
                                 env=test_env,

cibuildwheel/resources/cibuildwheel.schema.json

@@ -569,6 +569,21 @@
       ],
       "title": "CIBW_TEST_ENVIRONMENT"
     },
+    "test-execution-args": {
+      "description": "Additional arguments for the test runner to use when configuring the test environment",
+      "oneOf": [
+        {
+          "type": "string"
+        },
+        {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        }
+      ],
+      "title": "CIBW_TEST_EXECUTION_ARGS"
+    },
     "overrides": {
       "type": "array",
       "description": "An overrides array",
@@ -638,6 +653,9 @@
               },
               "test-environment": {
                 "$ref": "#/$defs/inherit"
+              },
+              "test-execution-args": {
+                "$ref": "#/$defs/inherit"
               }
             }
           },
@@ -748,6 +766,9 @@
           },
           "test-environment": {
             "$ref": "#/properties/test-environment"
+          },
+          "test-execution-args": {
+            "$ref": "#/properties/test-execution-args"
           }
         }
       }
@@ -876,6 +897,9 @@
         },
         "test-environment": {
           "$ref": "#/properties/test-environment"
+        },
+        "test-execution-args": {
+          "$ref": "#/properties/test-execution-args"
         }
       }
     },
@@ -936,6 +960,9 @@
         },
         "test-environment": {
           "$ref": "#/properties/test-environment"
+        },
+        "test-execution-args": {
+          "$ref": "#/properties/test-execution-args"
         }
       }
     },
@@ -1009,6 +1036,9 @@
         },
         "test-environment": {
           "$ref": "#/properties/test-environment"
+        },
+        "test-execution-args": {
+          "$ref": "#/properties/test-execution-args"
         }
       }
     },
@@ -1069,6 +1099,9 @@
         },
         "test-environment": {
           "$ref": "#/properties/test-environment"
+        },
+        "test-execution-args": {
+          "$ref": "#/properties/test-execution-args"
         }
       }
     },
@@ -1129,6 +1162,9 @@
         },
         "test-environment": {
           "$ref": "#/properties/test-environment"
+        },
+        "test-execution-args": {
+          "$ref": "#/properties/test-execution-args"
         }
       }
     },
@@ -1189,6 +1225,9 @@
         },
         "test-environment": {
           "$ref": "#/properties/test-environment"
+        },
+        "test-execution-args": {
+          "$ref": "#/properties/test-execution-args"
         }
       }
     }

cibuildwheel/resources/defaults.toml

@@ -25,6 +25,7 @@ test-requires = []
 test-extras = []
 test-groups = []
 test-environment = {}
+test-execution-args = []
 
 container-engine = "docker"
 

docs/options.md

@@ -1669,6 +1669,41 @@ Platform-specific environment variables are also available:<br/>
     CIBW_TEST_ENVIRONMENT: PYTHONSAFEPATH=1
     ```
 
+### `test-execution-args` {: #test-execution-args toml env-var }
+
+> Define additional arguments that will be passed to the command that runs the tests.
+
+A list of arguments that will be used by the test runner when running tests. This is used by environments where the execution environment can be customized. For example, mobile platforms will use these arguments to control the execution of the testbed application that is used to run tests.
+
+This option will be ignored on platforms that do not have a separate test runner.
+
+Platform-specific environment variables are also available:<br/>
+`CIBW_TEST_EXECUTION_ARGS_MACOS` | `CIBW_TEST_EXECUTION_ARGS_WINDOWS` | `CIBW_TEST_EXECUTION_ARGS_LINUX` | `CIBW_TEST_EXECUTION_ARGS_ANDROID` |`CIBW_TEST_EXECUTION_ARGS_IOS` | `CIBW_TEST_EXECUTION_ARGS_PYODIDE`
+
+#### Examples
+
+!!! tab examples "pyproject.toml"
+
+    ```toml
+    [tool.cibuildwheel.ios]
+    # Run the tests on an iPhone 16e simulator running iOS 18.5.
+    test-execution-args = ["--simulator='iPhone 16e,OS=18.5'"]
+
+    [tool.cibuildwheel.android]
+    # Run the Android tests on the minimum supported Android version.
+    test-execution-args = ["--managed", "minVersion"]
+    ```
+
+!!! tab examples "Environment variables"
+
+    ```yaml
+    # Run the tests on an iPhone 16e simulator running iOS 18.5.
+    CIBW_EXECUTION_ARGS_IOS: --simulator='iPhone 16e,OS=18.5'
+
+    # Run the Android tests on the minimum supported Android version.
+    CIBW_EXECUTION_ARGS_ANDROID: --managed minVersion
+    ```
+
 
 ## Debugging
 

docs/platforms.md

@@ -233,6 +233,8 @@ machine – for example, if you're building on an ARM64 machine, then you can te
 ARM64 wheel. Wheels of other architectures can still be built, but testing will
 automatically be skipped.
 
+Any arguments specified using [`test-execution-args`](options.md#test-execution-args) will be passed as arguments to the Python script that starts the testbed project. The most common arguments to use will be `--managed minVersion` or `--managed maxVersion`, specifying the use of a managed Android emulator with the minimum or maximum supported Android version; or `--connected <emulator ID>`, specifying the use of an existing booted Android emulator or device.
+
 Running an emulator requires the build machine to either be bare-metal or support
 nested virtualization. CI platforms known to meet this requirement are:
 
@@ -321,3 +323,7 @@ If tests have been configured, the test suite will be executed on the simulator
 The iOS test environment can't support running shell scripts, so the [`test-command`](options.md#test-command) value must be specified as if it were a command line being passed to `python -m ...`.
 
 The test process uses the same testbed used by CPython itself to run the CPython test suite. It is an Xcode project that has been configured to have a single Xcode "XCUnit" test - the result of which reports the success or failure of running `python -m <test-command>`.
+Any arguments specified using [`test-execution-args`](options.md#test-execution-args) will be passed as arguments to the Python script that starts the testbed project. The most common argument to use will be `--simulator`, which allows the specification of a specific device or iOS version for the test simulator. For example, `test_execution_args = ["--simulator", "iPhone 16e,OS=18.5"]` would specify the use of an iPhone 16e simulator running OS 18.5.
+
+!!! note
+    The `macos-15` image on GitHub Actions and Azure has a [known performance issue](https://github.com/actions/runner-images/issues/12777) that can lead to iOS simulators failing to start if they have not been pre-warmed. At this time, the workaround is to explicitly specify a simulator that is pre-warmed by default on the image; the iPhone 16e simulator running iOS 18.5 is one such image. This workaround is only needed on the `macos-15`; the `macos-14` image is not affected.