feat: add host-based upload support to `rattler_upload` #1580

magentaqin · 2025-08-04T14:14:14Z

Description

This PR is to support uploading by host. It's based on my previous PR: #1575

Why we need this host support?

upload function in pixi and rattler_build is different. In pixi, it only supports upload by url, but in rattler_build, it only supports upload by subcommands "prefix", "anaconda"...etc. We need to align these two.

Code changes I made

1.support parsing url, match it to specific server type and extract base_url and channel from the host
2.align progress bar style: migrate pixi_progress and create rattler_progress that serves the uploading part.

wolfv · 2025-08-05T08:12:26Z

crates/rattler_upload/src/utils/server_util.rs

+impl ServerPatterns {
+    fn new() -> Self {
+        Self {
+            // Prefix.dev patterns


I wonder if using regex is the best choice. We could probably also do this with regular URL parsing and some contains queries and similar, e.g.:

url.host_str.contains(prefix.dev) url.path().ends_with(/conda-forge) url.protocol() == "s3" ...

Good point! Indeed, Regex is fragile and has some maintainability issues...I'll switch to the URL parsing.

wolfv · 2025-08-05T08:14:19Z

crates/rattler_upload/src/utils/server_util.rs

+pub fn extract_artifactory_info(url: &Url) -> Result<(Url, String), Box<dyn std::error::Error>> {
+    let url_str = url.as_str();
+    let artifactory_pattern =
+        Regex::new(r"^(https?://[^/]+)/artifactory/([^/]+)(?:/(.+)/([^/]+))?").unwrap();


I also think here just using regular URL parsing and then skipping components will be more robust than regex.

For example:

let path_segments: Vec<&str> = url.path_segments() .ok_or("Invalid URL: cannot extract path segments")? .collect(); if path_segments.len() >= 2 && path_segments[0] == "artifactory" { // 1. Create base URL by removing path components let mut base_url = url.clone(); base_url.set_path(""); base_url.set_query(None); base_url.set_fragment(None); // 2. Extract channel let channel = path_segments[1].to_string(); Ok((base_url, channel)) } else { Err("Invalid Artifactory URL format".into()) }

baszalmstra

Idk if a "generic" rattler_progress makes sense. The implementation is very specific. Id much rather have a slimmed down version of progress reporting in the rattler_upload code itself that can be enabled with a feature (we do the same for rattler install code).

magentaqin · 2025-08-12T09:10:38Z

Idk if a "generic" rattler_progress makes sense. The implementation is very specific. Id much rather have a slimmed down version of progress reporting in the rattler_upload code itself that can be enabled with a feature (we do the same for rattler install code).

Thanks @baszalmstra for your advice! My original thought is not only for rattler_upload, but to align "progress" in pixi and rattler-build, so I directly moved the progress bar from pixi to rattler. Do you think it's necessary to have this rattler_progress crate?

wolfv · 2025-09-04T09:19:31Z

crates/rattler_upload/src/lib.rs

+            let host_url = args.host.as_ref().unwrap();
+            let (base_url, channel) =
+                extract_prefix_info(host_url).expect("Failed to parse Prefix URL");
+            ServerType::Prefix(PrefixOpts {


Shouldn't we fill in e.g.. API key from the args here?

Hi, Wolf. There're three things I need to explain.

According to pixi doc, upload doesn't have api_key arg. https://pixi.sh/latest/reference/cli/pixi/upload/#description

The common args was designed like this(not having api_key) option.

There're two ways to upload packages: URL and subcommand.

Only when you specify the subcommand as "prefix", can you pass the arg api_key. But in our case, we upload package through URL(prefix pattern), api_key is not allowed to pass through the args. Users must pass the token by setting them locally.

… duplicates

…to feature/upload-url

baszalmstra · 2025-10-03T12:16:15Z

crates/rattler_s3/src/lib.rs


    /// The region of the S3 backend
-    pub region: String,
+    pub region: Option<String>,


Why is this now optional?

Because it supports upload by URL now, and the region can be written in the URL. For example, the user can upload like this " https://.s3..amazonaws.com/my-channel"

baszalmstra · 2025-10-03T12:17:37Z

crates/rattler_upload/src/upload/opt.rs

 /// Options for uploading to S3
 #[cfg(feature = "s3")]
 #[derive(Clone, Debug, PartialEq, Parser)]
 pub struct S3Opts {


You can reuse the S3CredentialsOpts from crates/rattler_s3/src/clap.rs here to get a consistent CLI.

will do it! thanks for reminding!

baszalmstra · 2025-10-03T12:18:19Z

crates/rattler_upload/src/upload/opt.rs

 #[derive(Parser, Debug)]
 pub struct UploadOpts {
+    /// The host + channel (optional if the server type is provided)
+    pub host: Option<Url>,


There are clap arguments here to make sure that if host is provide that channel must also be provide. Or that it conflicts with server_type. Use those.

Hi, Bas! Nice catch! I mean the hostURL should consider "host" + "channel", and in check_server_type(https://github.com/magentaqin/rattler/blob/14a0e826791da57dbe5e02db24597b6cf5b8873a/crates/rattler_upload/src/utils/server_util.rs#L25), I will check whether the URL contains channel.
But you're right, I should use clap arguments to make "host" and "server_type" mutually exclusive.

baszalmstra · 2025-10-03T12:19:07Z

crates/rattler_upload/src/upload/opt.rs

+#[cfg(feature = "s3")]
+#[derive(Debug)]
+#[allow(missing_docs)]
+pub struct S3Data {


What is the purpose of this struct? The name is quite ambigous. Also, doesnt the rattler_s3 crate provide similar functionality?

You're right, Bas. S3Data was migrated before we had rattler_s3 crate. Let me see how to reuse the functionality!

baszalmstra · 2025-10-03T12:20:47Z

crates/rattler_upload/src/utils/server_util.rs

+/// * `SimpleServerType` - The detected server type or Unknown
+///
+/// ```
+pub fn check_server_type(host_url: &Url) -> SimpleServerType {


Implement From or SimpleServerType::from_url.

baszalmstra · 2025-10-03T12:21:04Z

crates/rattler_upload/src/utils/server_util.rs

+    #[cfg(feature = "s3")]
+    S3,
+    CondaForge,
+    Unknown,


Cant we represent this as an Option<SimpleServerType>?

baszalmstra · 2025-10-03T12:22:19Z

crates/rattler_upload/src/utils/server_util.rs

+
+    // 3. Check Conda-forge (GitHub)
+    if host.contains("conda-forge") {
+        return SimpleServerType::CondaForge;


Is conda-forge a server?

It's originally from here: https://github.com/prefix-dev/rattler-build/blob/v0.44.0/src/upload/conda_forge.rs I just migrated it to rattler! And for the URL part, I add the corresponding part.

baszalmstra · 2025-10-03T12:24:35Z

crates/rattler_upload/src/utils/server_util.rs

This file needs a lot of tests to verify that the SimpleServerType is correct for a number of URLs.

baszalmstra · 2025-10-03T12:24:55Z

crates/rattler_upload/src/utils/server_util.rs

+    Anaconda,
+    #[cfg(feature = "s3")]
+    S3,
+    CondaForge,


What kind of server is this?

It's originally from here: https://github.com/prefix-dev/rattler-build/blob/v0.44.0/src/upload/conda_forge.rs I just migrated it to rattler!

baszalmstra · 2025-10-03T12:26:57Z

crates/rattler_upload/Cargo.toml

  "rt-multi-thread",
  "process",
 ] }
+regex = "1.10"


Is this used?

I'll remove it. It was used before refactor.

wolfv · 2025-10-13T10:00:27Z

Hey @magentaqin I'll take this PR over and get it merged! I'll remove some changes from the S3 upload because IMO progress bar shoudl go to a different PR.

magentaqin · 2025-10-14T08:49:18Z

Hey @magentaqin I'll take this PR over and get it merged! I'll remove some changes from the S3 upload because IMO progress bar shoudl go to a different PR.

Sounds good, thanks for taking it over! 🙌

magentaqin added 18 commits July 31, 2025 11:06

feat: add auth_store for UploadOpts and update get_auth_store

c2ecece

fix: fix linting

7796ddd

feat: add regex package

d05406c

feat: check server type by host

a490759

feat: extract Quetz and Artifactory info from host

9e1a01a

feat: extract prefix base_url and channel from host

676a6f7

feat: move pixi_progress to rattler

63f33ee

feat: extract s3 info

29a1f5c

feat: make host and server_type optional in UploadOpts

fc4a790

feat: prefix upload supports rattler_progress

96ea212

fix: update rattler_progress name

7718b02

feat: detect server type and extract components from host

cfc2690

feat: update Cargo.toml

9f3b4bb

fix: fix Cargo.toml

05e9c12

fix: fix s3 type

00d0a45

fix: fix import error

0bf614c

chore: cargo fmt

af31a6a

fix: fix linting error

9c8c525

wolfv reviewed Aug 5, 2025

View reviewed changes

magentaqin added 3 commits August 7, 2025 16:23

fix: fix 'auth_store' type

1c0fdd8

fix: fix linting issue

09777b0

Merge branch 'feature/upload-auth-store' into feature/upload-url

4573261

baszalmstra reviewed Aug 11, 2025

View reviewed changes

magentaqin added 2 commits August 12, 2025 11:01

chore: sync cargo.lock

1ead736

Merge branch 'main' into feature/upload-url

5f71202

magentaqin added 3 commits August 12, 2025 16:14

refactor: replace RegExp with URL parsing

d23039e

chore: update default region

0958d9f

Merge main

db59292

magentaqin added 3 commits August 21, 2025 16:02

fmt

8b5e6e8

Merge main

99dd9e7

refactor: remove unecessary 'rattler_progress'

9eadb87

wolfv reviewed Sep 4, 2025

View reviewed changes

magentaqin added 11 commits September 9, 2025 10:14

Merge main and update s3 api

bb464fb

fmt

c709b62

fix: fix the region not working

5a824b4

Merge main and solve conflicts

3ae4aef

feat: optimize progressbar for S3 uploading and responds when package…

d18ba47

… duplicates

fix: fix lint err

09ddbb4

Merge branch 'feature/upload-url' of github.com:magentaqin/rattler in…

059db57

…to feature/upload-url

fix: fix S3 force option

16d1fd0

feat: add addressing_style for 'S3Opts'

9d62564

Merge main

e3fa364

Merge main

54c5c18

baszalmstra reviewed Oct 3, 2025

View reviewed changes

baszalmstra requested changes Oct 3, 2025

View reviewed changes

magentaqin added 3 commits October 6, 2025 12:34

Merge main

9bf9bb2

refactor: reuse the S3CredentialsOpts

5fc6195

chore: remove crate 'regex'

14a0e82

remove S3 related changes for now

7650fa2

feat: add host-based upload support to rattler_upload #1580

Are you sure you want to change the base?

feat: add host-based upload support to rattler_upload #1580

Uh oh!

Conversation

magentaqin commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Why we need this host support?

Code changes I made

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

baszalmstra left a comment

Choose a reason for hiding this comment

Uh oh!

magentaqin commented Aug 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wolfv commented Oct 13, 2025

Uh oh!

magentaqin commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: add host-based upload support to `rattler_upload` #1580

feat: add host-based upload support to `rattler_upload` #1580

magentaqin commented Aug 4, 2025 •

edited

Loading