Allow users to mark platforms as "required" for wheel coverage #10067

charliermarsh · 2024-12-20T22:00:55Z

Summary

This PR revives #10017, which might be viable now that we don't enforce any platforms by default.

The basic idea here is that users can mark certain platforms as required (empty, by default). When resolving, we ensure that the specified platforms have wheel coverage, backtracking if not.

For example, to require that we include a version of PyTorch that supports Intel macOS:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = ["torch>1.13"]

[tool.uv]
required-platforms = [
    "sys_platform == 'darwin' and platform_machine == 'x86_64'"
]

Other than that, the forking is identical to past iterations of this PR.

This would give users a way to resolve the tail of issues in #9711, but with manual opt-in to supporting specific platforms.

Uhh trying Fork on... Add heuristic Gate to local versions Remove KnownMarkers Add required platform support

BurntSushi

I like this!

BurntSushi · 2024-12-21T13:04:59Z

crates/uv-distribution-types/src/prioritized_distribution.rs

@@ -196,6 +199,15 @@ impl Display for IncompatibleDist {
                IncompatibleWheel::RequiresPython(python, _) => {
                    write!(f, "Python {python}")
                }
+                IncompatibleWheel::MissingPlatform(marker) => {
+                    if let Some(platform) = KnownPlatform::from_marker(*marker) {
+                        write!(f, "no {platform}-compatible wheels")


BurntSushi · 2024-12-21T13:22:17Z

docs/reference/settings.md

@@ -273,6 +273,36 @@ package = false

 ---

+### [`required-platforms`](#required-platforms) {: #required-platforms }


I think it might help if the docs specifically "compared and contrasted" required-platforms with environments. They do different things of course, but they take the same inputs and I could see end users being confused by them.

(What I tend to do in docs is to mention the other in the docs and contrast them. e.g., mention environments in this section and required-platforms in the environments section. That way, users see the contrasting doc regardless of where they look.)

I think doing this would be helpful for naming. (as well as very helpful for users)

charliermarsh · 2024-12-21T23:59:14Z

Another case that this could solve (but would require user intervention): #10085. Ideally, I think we should be able to give users a hint when the sync fails -- like "There are no available wheels for this platform, add required-platforms".

konstin

The feature is definitely needed and i like the approach, though the find_environments if i haven't missed something fails to uphold the required platforms in certain conditions.

konstin · 2025-01-10T13:17:18Z

crates/uv-workspace/src/pyproject.rs

+            ]
+        "#
+    )]
+    pub required_platforms: Option<SupportedEnvironments>,


Should this be required_environments or required_markers? The other markers list is environments, platforms lacks symmetry

It was somewhat intentional but I can change it.

We might want to change the other one? I feel like environments is a bit overloaded in the longterm.

konstin · 2025-01-10T13:25:15Z

crates/uv-resolver/src/resolver/mod.rs

+
+    // Find all dependencies on the current package.
+    let mut marker = MarkerTree::FALSE;
+    for index in incompatibilities {


Don't we add additional incompatibilities that target id when we find a new package foo with foo --[marker]--> id? Could that lead to packages missing platform support because we could have a case like the following:

Required platform linux

Choose A 2, with A -> B; windows, A -> C
Choose B 2. B 2 does have only a mac wheel but that's fine because it's only required on windows, so our required linux is coverd.
Choose C 2, with C -> B

Try to install on linux. We try to install A 2, C 2 and B 2, but B 2 fails since it has no wheel.

@konstin -- I think this would actually work as-is, though for "bad" reasons. Basically, when we visit B, we won't know that it's "only required for Windows", because we don't do "aggressive forking" today, so we'd still try to solve for Linux. It might get filtered out later? I can't quite remember.

Could you add a test case covering this scenario? So it doesn't fail with future resolver changes.

zanieb · 2025-01-10T17:50:07Z

Since this is specific to wheels should we name this differently? Like, if I encounter a build failure why would it be obvious that setting a required_platform would solve that? Is this sort of no_build_platforms?

charliermarsh · 2025-01-11T14:13:17Z

@zanieb -- That's true, although I'd read that setting as "platforms on which building should be disabled", which isn't quite the same. We're not enforcing that all packages have wheels on these platforms. We're enforcing that all packages have wheels or source distributions on these platforms (and so, building is considered ok).

charliermarsh · 2025-01-11T14:13:42Z

I sort of prefer something that doesn't get into the source vs. binary distinction, and instead gets at the intent of the flag: to ensure that the lockfile supports the platforms you care about.

zanieb · 2025-01-11T19:53:21Z

Oh I guess I'm getting confused by the pull request title and body which say...

When resolving, we ensure that the specified platforms have wheel coverage

This sounds like we require wheels for the platforms?

charliermarsh · 2025-01-11T19:54:39Z

We require that there are compatible distributions for the platform. That could be a source distribution, or (for packages that lack source distributions) a wheel.

zanieb · 2025-01-11T20:26:04Z

That makes sense, it'd help to change the way this is presented in the title / body then. With this context, no_build_platforms definitely does not make sense.

Respect all platforms

eeb249d

Uhh trying Fork on... Add heuristic Gate to local versions Remove KnownMarkers Add required platform support

charliermarsh force-pushed the charlie/required-platforms branch from 59b6690 to eeb249d Compare December 21, 2024 00:21

charliermarsh marked this pull request as ready for review December 21, 2024 04:14

charliermarsh requested review from BurntSushi, zanieb and konstin December 21, 2024 04:14

BurntSushi approved these changes Dec 21, 2024

View reviewed changes

konstin mentioned this pull request Jan 10, 2025

error: Distribution kaleido==0.2.1.post1 @ registry+https://pypi.org/simple can't be installed because it doesn't have a source distribution or wheel for the current platform #10464

Open

konstin reviewed Jan 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow users to mark platforms as "required" for wheel coverage #10067

Allow users to mark platforms as "required" for wheel coverage #10067

charliermarsh commented Dec 20, 2024 •

edited

Loading

BurntSushi left a comment

BurntSushi Dec 21, 2024

BurntSushi Dec 21, 2024

zanieb Jan 11, 2025

charliermarsh commented Dec 21, 2024

konstin left a comment

konstin Jan 10, 2025

charliermarsh Jan 10, 2025

zanieb Jan 10, 2025

konstin Jan 10, 2025

charliermarsh Jan 10, 2025

konstin Jan 13, 2025

zanieb commented Jan 10, 2025

charliermarsh commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

		@@ -273,6 +273,36 @@ package = false

		---

		### [`required-platforms`](#required-platforms) {: #required-platforms }

Allow users to mark platforms as "required" for wheel coverage #10067

Are you sure you want to change the base?

Allow users to mark platforms as "required" for wheel coverage #10067

Conversation

charliermarsh commented Dec 20, 2024 • edited Loading

Summary

BurntSushi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charliermarsh commented Dec 21, 2024

konstin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb commented Jan 10, 2025

charliermarsh commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

charliermarsh commented Dec 20, 2024 •

edited

Loading