Parallelize workload pack downloads Fixes #43870

[Forgind]

Fixes #43870

There are multiple possible reasons AzDO might be much slower than nuget.org. My untested hypothesis is that it may be some authentication work, which would be substantially mitigated by parallelizing the workload pack downloads (not the installs). That's what I tried to do here.

@AArnott, any chance you could try this out and let me know if it helps?

[marcpopMSFT]

You may need to build a local copy that he can install and try. Now that sdk and installer are merged, you might be able to just run build -pack locally in your branch and create zips for private testing.

[baronfel]

Could also just drop-in replace dotnet.dll, no? Because this command is entirely part of the CLI, not touching anything in the SDK props/targets/etc.

[AArnott]

any chance you could try this out and let me know if it helps?

I'd be happy to. But yes, a binary would help as I haven't practiced building this repo.

[AArnott]

I've been testing this locally rather than in Azure Pipelines. With your change, I see a steady 100Mbps download rate (on a 1Gbps connection) vs. a ~30-50Mbps download rate when using the 9.0.100 SDK. So bandwidth wise it looks like a good improvement.

The 9.0.100 SDK completed successfully in 3 minutes 43 seconds.
However, the SDK with your change failed after 76 seconds with an error:

Workload installation failed: One or more errors occurred. (An item with the same key has already been added. Key: PrivateConsumption [https://pkgs.dev.azure.com/andrewarnott/Git/_packaging/PrivateConsumption/nuget/v3/index.json]) (An item with the same key has already been added. Key: PrivateConsumption [https://pkgs.dev.azure.com/andrewarnott/Git/_packaging/PrivateConsumption/nuget/v3/index.json]) (An item with the same key has already been added. Key: PrivateConsumption [https://pkgs.dev.azure.com/andrewarnott/Git/_packaging/PrivateConsumption/nuget/v3/index.json])

I wonder if the parallelism you added ended up trying to add the package feed source multiple times somewhere. It certainly does not appear multiple times in my nuget.config or anywhere else environmental on the docker container I tested it on.

Full console log

Skipping NuGet package signature verification.
Updated advertising manifest microsoft.net.workload.emscripten.net8.
Updated advertising manifest microsoft.net.workload.mono.toolchain.net8.
Updated advertising manifest microsoft.net.sdk.aspire.
Updated advertising manifest microsoft.net.sdk.android.
Updated advertising manifest microsoft.net.sdk.maui.
Updated advertising manifest microsoft.net.workload.mono.toolchain.net6.
Updated advertising manifest microsoft.net.workload.mono.toolchain.net7.
Updated advertising manifest microsoft.net.workload.mono.toolchain.current.
Updated advertising manifest microsoft.net.sdk.ios.
Updated advertising manifest microsoft.net.sdk.tvos.
Updated advertising manifest microsoft.net.sdk.macos.
Updated advertising manifest microsoft.net.sdk.maccatalyst.
Updated advertising manifest microsoft.net.workload.emscripten.current.
Updated advertising manifest microsoft.net.workload.emscripten.net6.
Updated advertising manifest microsoft.net.workload.emscripten.net7.
Installing workload manifest microsoft.net.sdk.android version 35.0.24...
Installing workload manifest microsoft.net.sdk.ios version 18.2.9173...
Installing workload manifest microsoft.net.sdk.maccatalyst version 18.2.9173...
Installing workload manifest microsoft.net.sdk.macos version 15.2.9173...
Installing workload manifest microsoft.net.sdk.maui version 9.0.14...
Installing workload manifest microsoft.net.sdk.tvos version 18.2.9173...
Installing workload manifest microsoft.net.workload.mono.toolchain.current version 9.0.1...
Installing workload manifest microsoft.net.workload.emscripten.current version 9.0.1...
Installing workload manifest microsoft.net.workload.emscripten.net6 version 9.0.1...
Installing workload manifest microsoft.net.workload.emscripten.net7 version 9.0.1...
Installing workload manifest microsoft.net.workload.emscripten.net8 version 9.0.1...
Installing workload manifest microsoft.net.workload.mono.toolchain.net6 version 9.0.1...
Installing workload manifest microsoft.net.workload.mono.toolchain.net7 version 9.0.1...
Installing workload manifest microsoft.net.workload.mono.toolchain.net8 version 9.0.1...
No workloads installed for this feature band. To update workloads installed with earlier SDK versions, include the --from-previous-sdk option.
Garbage collecting for SDK feature band(s) 9.0.200-preview.0...

Successfully updated workload(s): .

Installing workloads: android wasm-tools

Skipping NuGet package signature verification.
Workload installation failed. Rolling back installed packs...
Workload installation failed: One or more errors occurred. (An item with the same key has already been added. Key: PrivateConsumption [https://pkgs.dev.azure.com/andrewarnott/Git/_packaging/PrivateConsumption/nuget/v3/index.json]) (An item with the same key has already been added. Key: PrivateConsumption [https://pkgs.dev.azure.com/andrewarnott/Git/_packaging/PrivateConsumption/nuget/v3/index.json]) (An item with the same key has already been added. Key: PrivateConsumption [https://pkgs.dev.azure.com/andrewarnott/Git/_packaging/PrivateConsumption/nuget/v3/index.json])

[AArnott]

With your concurrency bug fix, I can say that your fix drops the elapsed time on a linux docker container to 2:28 (from 3:43). A nice improvement!