Add --max-concurrent-downloads flag for parallel layer downloads #716

sbhavani · 2025-10-04T05:12:38Z

Adds --max-concurrent-downloads flag to container image pull for configurable concurrent layer downloads.

Fixes #715
Depends on apple/containerization#311

Usage:

container image pull nginx:latest --max-concurrent-downloads 6

Changes:

Add CLI flag (default: 3)
Thread parameter through XPC stack
Update to use forked containerization with configurable concurrency

Performance: ~1.2-1.3x faster pulls for multi-layer images with higher concurrency

Tests: Included standalone tests verify concurrency behavior and parameter flow

jglogan · 2025-10-07T19:19:04Z

testConcurrency.swift

@@ -0,0 +1,109 @@
+#!/usr/bin/env swift


Why is this and test_parameter_flow.swift in the top level directory?

should they belong in Tests/CLITests/Subcommands/Images/?

Yes, this might be a good place for the tests.

This is still unresolved.

Package.swift

Sources/ContainerClient/Core/ClientImage.swift

Implements concurrent layer downloads to improve image pull performance by fetching multiple layers simultaneously instead of sequentially.

sbhavani · 2025-10-17T15:46:19Z

@dcantah anything else needed to merge this?

dcantah · 2025-10-21T05:17:52Z

@sbhavani Yes. We need a tag of Containerization that has the changes. I'm making some changes over there but should have a tag out within the next day or so and then we can bump to this. We should not have our dependency on Containerization be to a branch, and especially not main 😄

jglogan

@sbhavani Just a couple of nits if you still have time to move this forward. Thank you for contributing!

Package.swift

jglogan · 2025-11-13T19:04:04Z

Sources/ContainerClient/Flags.swift


+        public init(disableProgressUpdates: Bool, maxConcurrentDownloads: Int) {
+            self.disableProgressUpdates = disableProgressUpdates
+            self.maxConcurrentDownloads = maxConcurrentDownloads


What's the reason behind putting this in the progress group?

The problem I see here is that any command that wants to present the --progress flag (--disable-progress-updates is gone, apologies for letting this PR go so long so as to force a rebase here) will also see --max-concurrent-downloads whether downloads are relevant to the other command or not. Is there a better home for this? Or do we need to create another reusable group especially for download-related options?

I don't see any harm in copy/pasting this option into each command that needs it either, unless there's simply too many of them.

Good point! I think image pull definitely needs it as a user might change this for large images.

container run and container create could use it but its probably not needed as an option.

Is there anything similar to Docker daemon.json? It would be convenient to set a config so all images are pulled with the same maxConcurrentDownloads

This is still unresolved. Consider:

Removing maxConcurrentDownloads from Flags.Progress

Creating new Flags.ImageFetch group specifically for image fetch options

Adding @OptionGroup var imageFetchFlags: Flags.ImageFetch to:

container image pull

container run

container create

sbhavani · 2025-11-13T21:29:09Z

@sbhavani Just a couple of nits if you still have time to move this forward. Thank you for contributing!

no problem! I'll take a look and make the changes

dkovba

Thank you for the contribution 👍
The default value 3 seems reasonable.

testConcurrency.swift

Package.swift

Sources/TerminalProgress/ProgressTaskCoordinator.swift

Sources/ContainerClient/Core/ClientImage.swift

dkovba · 2025-11-14T02:07:16Z

testConcurrency.swift

@@ -0,0 +1,109 @@
+#!/usr/bin/env swift


Yes, this might be a good place for the tests.

Sources/TerminalProgress/ProgressTaskCoordinator.swift

dkovba

Please run make fmt to pass the PR checks.

dkovba · 2025-11-19T21:16:59Z

Package.swift

 let gitCommit = ProcessInfo.processInfo.environment["GIT_COMMIT"] ?? "unspecified"
 let builderShimVersion = "0.7.0"
-let scVersion = "0.13.0"
+let scVersion = "0.11.0"


Originally, we had to upgrade to 0.11.0. At this moment, we don't have to downgrade to it. Please revert this change.

dkovba · 2025-11-19T21:53:16Z

Sources/Services/ContainerSandboxService/SandboxService.swift

        czConfig.process.workingDirectory = process.workingDirectory
        czConfig.process.rlimits = process.rlimits.map {
-            .init(type: $0.limit, hard: $0.hard, soft: $0.soft)
+            POSIXRlimit(type: $0.limit, hard: $0.hard, soft: $0.soft)


This changes seems to be unnecessary (repeats 2 times).

dkovba · 2025-11-19T21:53:27Z

Sources/Services/ContainerSandboxService/SandboxService.swift

        switch process.user {
        case .raw(let name):
-            czConfig.process.user = .init(
+            czConfig.process.user = ContainerizationOCI.User(


This changes seems to be unnecessary (repeats 4 times).

dkovba · 2025-11-19T21:54:53Z

testConcurrency.swift

@@ -0,0 +1,109 @@
+#!/usr/bin/env swift


This is still unresolved.

dkovba · 2025-11-19T21:57:58Z

Sources/ContainerClient/Core/ClientImage.swift

-    public static func pull(reference: String, platform: Platform? = nil, scheme: RequestScheme = .auto, progressUpdate: ProgressUpdateHandler? = nil) async throws -> ClientImage {
+    public static func pull(reference: String, platform: Platform? = nil, scheme: RequestScheme = .auto, progressUpdate: ProgressUpdateHandler? = nil, maxConcurrentDownloads: Int = 3) async throws -> ClientImage {
+        guard maxConcurrentDownloads > 0 else {
+            throw ContainerizationError(.invalidArgument, message: "maxConcurrentDownloads must be greater than 0, got \(maxConcurrentDownloads)")


The error message should be user-facing. Consider something like:

--max-concurrent-downloads must be greater than 0, got
(maxConcurrentDownloads)

invalid value for --max-concurrent-downloads: must be greater
than 0, got (maxConcurrentDownloads)

dkovba · 2025-11-19T22:04:14Z

Sources/ContainerClient/Flags.swift

        @Option(name: .long, help: ArgumentHelp("Progress type (format: none|ansi)", valueName: "type"))
        public var progress: ProgressType = .ansi
+
+        @Option(name: .long, help: "Maximum number of concurrent layer downloads (default: 3)")


@adityaramani Should we use the word "layer" or "blob" here?

kmk142789 approved these changes Oct 6, 2025

View reviewed changes

jglogan reviewed Oct 7, 2025

View reviewed changes

dcantah marked this pull request as draft October 10, 2025 00:54

dcantah reviewed Oct 10, 2025

View reviewed changes

Package.swift Outdated Show resolved Hide resolved

dcantah reviewed Oct 10, 2025

View reviewed changes

Sources/ContainerClient/Core/ClientImage.swift Show resolved Hide resolved

sbhavani force-pushed the feature/parallel-layer-downloads branch from c843892 to cc15745 Compare October 14, 2025 06:21

sbhavani marked this pull request as ready for review October 14, 2025 06:23

sbhavani force-pushed the feature/parallel-layer-downloads branch from cc15745 to 26ebe63 Compare October 14, 2025 06:25

Add parallel layer downloads support for image pull operations

53e6c45

Implements concurrent layer downloads to improve image pull performance by fetching multiple layers simultaneously instead of sequentially.

sbhavani force-pushed the feature/parallel-layer-downloads branch from 26ebe63 to 53e6c45 Compare October 14, 2025 06:27

Merge branch 'main' into feature/parallel-layer-downloads

903beb8

jglogan requested changes Nov 13, 2025

View reviewed changes

dkovba requested changes Nov 14, 2025

View reviewed changes

dkovba reviewed Nov 14, 2025

View reviewed changes

Sources/TerminalProgress/ProgressTaskCoordinator.swift Outdated Show resolved Hide resolved

dkovba reviewed Nov 14, 2025

View reviewed changes

Sources/TerminalProgress/ProgressTaskCoordinator.swift Outdated Show resolved Hide resolved

sbhavani added 2 commits November 14, 2025 19:47

chore: upgrade to containerization 0.11.0

2c97436

refactor: remove unused concurrent progress methods

21ac59f

sbhavani force-pushed the feature/parallel-layer-downloads branch from e02e2ff to 21ac59f Compare November 15, 2025 03:50

sbhavani added 4 commits November 14, 2025 19:52

refactor: rename test files to camelCase

9659c53

fix: validate maxConcurrentDownloads > 0

b2e13d9

chore: trigger CI

efa7b87

Merge upstream/main into feature/parallel-layer-downloads

341c918

sbhavani requested a review from dkovba November 15, 2025 04:06

dkovba requested changes Nov 19, 2025

View reviewed changes

dkovba reviewed Nov 19, 2025

View reviewed changes

apple deleted a comment from 0777888212 Nov 19, 2025

Add --max-concurrent-downloads flag for parallel layer downloads #716

Are you sure you want to change the base?

Add --max-concurrent-downloads flag for parallel layer downloads #716

Uh oh!

Conversation

sbhavani commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sbhavani commented Oct 17, 2025

Uh oh!

dcantah commented Oct 21, 2025

Uh oh!

jglogan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbhavani commented Nov 13, 2025

Uh oh!

dkovba left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dkovba left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sbhavani commented Oct 4, 2025 •

edited

Loading

dkovba left a comment •

edited

Loading