Use Windows_SDK.json from UE to determine what VS components to install

Author: slonopotamus

.github/workflows/ci.yml

@@ -34,17 +34,10 @@ jobs:
         # All supported Ubuntu LTS with system python
         # Be careful when removing EOL versions so that we still test our oldest supported python at least somewhere
         - { os: ubuntu-22.04, python: "3.10" }
-        - { os: ubuntu-24.04, python: "3.12" }
+        - { os: ubuntu-24.04, python: "3.13" }
 
-          # All supported Visual Studio on Windows Server 2022
-        - { os: windows-2022, python: "3.12", visual-studio: 2017 }
-        - { os: windows-2022, python: "3.12", visual-studio: 2019 }
-        - { os: windows-2022, python: "3.12", visual-studio: 2022 }
-
-        # All supported Visual Studio on Windows Server 2025
-        - { os: windows-2025, python: "3.12", visual-studio: 2017 }
-        - { os: windows-2025, python: "3.12", visual-studio: 2019 }
-        - { os: windows-2025, python: "3.12", visual-studio: 2022 }
+        - { os: windows-2022, python: "3.13" }
+        - { os: windows-2025, python: "3.13" }
     runs-on: ${{ matrix.os }}
     steps:
     - name: Checkout
@@ -56,6 +49,6 @@ jobs:
     - name: Install
       run: pip install . --user
     - name: Build prerequisites
-      run: ue4-docker build --target=build-prerequisites --visual-studio ${{ matrix.visual-studio || '2017' }}
+      run: ue4-docker build --target=build-prerequisites
     - name: Run diagnostics
       run: ue4-docker diagnostics all

README.md

@@ -7,15 +7,42 @@
 
 The ue4-docker Python package contains a set of Dockerfiles and accompanying build infrastructure that allows you to build Docker images for Epic Games' [Unreal Engine](https://www.unrealengine.com/). The images also incorporate the infrastructure from [ue4cli](https://github.com/adamrehn/ue4cli), [conan-ue4cli](https://github.com/adamrehn/conan-ue4cli), and [ue4-ci-helpers](https://github.com/adamrehn/ue4-ci-helpers) to facilitate a wide variety of use cases.
 
-Key features include:
+## Key features
 
-- The six most recent versions of the Unreal Engine are supported (currently Unreal Engine 4.27 and newer).
 - Both Windows containers and Linux containers are supported.
 - Building and packaging Unreal Engine projects is supported.
 - Running automation tests is supported.
 - Running built Unreal Engine projects with offscreen rendering is supported via the NVIDIA Container Toolkit under Linux.
 
-Resources:
+## Supported engine versions
+
+Unreal Engine 4.27 and five most recent versions of Unreal Engine 5 are supported. 
+
+### 4.27
+
+No known issues.
+
+### 5.2
+
+You need to apply [this fix](https://forums.unrealengine.com/t/need-help-building-5-2-0-from-github/1441996/4) to your engine fork in order to build on Windows.
+
+### 5.3
+
+No known issues.
+
+### 5.4
+
+No known issues.
+
+### 5.5
+
+You need to apply [this patch](https://github.com/EpicGames/UnrealEngine/commit/b86e244161df5bb32030818a48ffeccd3ed310ef) to your engine fork in order to build on Windows.
+
+### 5.6
+
+You need to apply [this](https://github.com/EpicGames/UnrealEngine/commit/467f8b31124c89b61c00ba73c26229897cafd7bd) and [this](https://github.com/EpicGames/UnrealEngine/commit/cdda65cecacd9a1278020925c357bf4cc0b17e8c) patches to your engine fork in order to build on Windows.
+
+## Resources
 
 - **Documentation:** <https://adamrehn.github.io/ue4-docker>
 - **GitHub repository:** <https://github.com/adamrehn/ue4-docker>

docs/ue4-docker-build.adoc

@@ -147,14 +147,6 @@ Use Linux containers under Windows hosts (useful when testing Docker Desktop or
 *--random-memory*::
 Use a random memory limit for Windows containers
 
-*--visual-studio {2017,2019,2022}*::
-Specify Visual Studio Build Tools version.
-+
-By default, ue4-docker uses Visual Studio Build Tools 2017 to build Unreal Engine.
-Starting with Unreal Engine 4.25, you may choose to use Visual Studio Build Tools 2019 instead.
-+
-Unreal Engine 5.0 adds support for VS2022 but removes support for VS2017.
-
 == Environment
 
 This section describes several environment variables that affect how `ue4-docker build` operates.

src/ue4docker/build.py

@@ -200,9 +200,6 @@ def build():
                 "Detected max image size:      {:.0f}GB".format(DockerUtils.maxsize()),
                 False,
             )
-            logger.info(
-                "Visual Studio:                {}".format(config.visualStudio), False
-            )
 
             # Verify that the host OS is not a release that is blacklisted due to critical bugs
             if (
@@ -405,9 +402,6 @@ def build():
                     prereqsArgs = prereqsArgs + [
                         "--build-arg",
                         "DLLSRCIMAGE=" + config.dllSrcImage,
-                        "--build-arg",
-                        "VISUAL_STUDIO_BUILD_NUMBER="
-                        + config.visualStudio.build_number,
                     ]
 
                 custom_prerequisites_dockerfile = config.args.prerequisites_dockerfile

src/ue4docker/dockerfiles/ue4-build-prerequisites/windows/Dockerfile

@@ -56,9 +56,8 @@ LABEL com.adamrehn.ue4-docker.sentinel="1"
 RUN reg add HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem /v LongPathsEnabled /t REG_DWORD /d 1 /f
 
 # Install Chocolatey
-RUN powershell -NoProfile -ExecutionPolicy Bypass -Command "$env:chocolateyVersion = '1.4.0'; Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
+RUN powershell -NoProfile -ExecutionPolicy Bypass -Command "Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))"
 
 # Install the rest of our build prerequisites and clean up afterwards to minimise image size
 COPY install-prerequisites.ps1 C:\
-ARG VISUAL_STUDIO_BUILD_NUMBER
-RUN powershell -ExecutionPolicy Bypass -File C:\install-prerequisites.ps1 %VISUAL_STUDIO_BUILD_NUMBER%
+RUN powershell -ExecutionPolicy Bypass -File C:\install-prerequisites.ps1

src/ue4docker/dockerfiles/ue4-build-prerequisites/windows/install-prerequisites.ps1

@@ -54,69 +54,12 @@ Invoke-WebRequest -Uri "https://sdk.lunarg.com/sdk/download/1.4.304.0/windows/Vu
 Expand-Archive -Path "$env:TEMP\vulkan-runtime-components.zip" -DestinationPath "$env:TEMP"
 Copy-Item -Path "*\x64\vulkan-1.dll" -Destination C:\Windows\System32\
 
-$visual_studio_build = $args[0]
-
-if ($visual_studio_build -eq "15")
-{
-    $windows_sdk_version = 18362
-}
-else
-{
-    $windows_sdk_version = 20348
-}
-
-# NOTE: We use the Visual Studio 2022 installer even for Visual Studio 2019 and 2017 here because the old (2017) installer now breaks
-Invoke-WebRequest -Uri "https://aka.ms/vs/17/release/vs_buildtools.exe" -OutFile "$env:TEMP\vs_buildtools.exe"
-
-# NOTE: Microsoft.NetCore.Component.SDK only exists for VS2019+. And it is actually *needed* only for UE5
-# NOTE: .NET 4.5 is required for some programs even in UE5, for example https://github.com/EpicGames/UnrealEngine/blob/5.0.1-release/Engine/Source/Programs/UnrealSwarm/SwarmCoordinator/SwarmCoordinator.csproj#L26
-# NOTE: Microsoft.NetCore.Component.Runtime.3.1 is required by the AutomationTool tool and does not come installed with VS2022 so it needs targetting here.
-$vs_args = @(
-    "--quiet",
-    "--wait",
-    "--norestart",
-    "--nocache",
-    "--installPath", "C:\BuildTools",
-    "--channelUri", "https://aka.ms/vs/$visual_studio_build/release/channel",
-    "--installChannelUri", "https://aka.ms/vs/$visual_studio_build/release/channel",
-    "--channelId", "VisualStudio.$visual_studio_build.Release",
-    "--productId", "Microsoft.VisualStudio.Product.BuildTools",
-    "--locale", "en-US",
-    "--add", "Microsoft.VisualStudio.Workload.VCTools",
-    "--add", "Microsoft.VisualStudio.Workload.MSBuildTools",
-    "--add", "Microsoft.VisualStudio.Component.NuGet",
-    "--add", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
-    "--add", "Microsoft.VisualStudio.Component.Windows10SDK.$windows_sdk_version",
-    "--add", "Microsoft.Net.Component.4.5.TargetingPack",
-    "--add", "Microsoft.Net.Component.4.6.2.TargetingPack",
-    "--add", "Microsoft.Net.ComponentGroup.DevelopmentPrerequisites",
-    "--add", "Microsoft.NetCore.Component.SDK",
-    "--add", "Microsoft.NetCore.Component.Runtime.3.1"
-)
-
-# Install the Visual Studio Build Tools workloads and components we need
-RunProcessChecked "$env:TEMP\vs_buildtools.exe" $vs_args
-
-# NOTE: Install the .Net 4.5 Framework Pack via NuGet as it is no longer available via Visual Studio, but still needed
-# NOTE: some programs even in UE5, for example https://github.com/EpicGames/UnrealEngine/blob/5.0.1-release/Engine/Source/Programs/UnrealSwarm/SwarmCoordinator/SwarmCoordinator.csproj#L26
-Invoke-WebRequest -Uri "https://www.nuget.org/api/v2/package/Microsoft.NETFramework.ReferenceAssemblies.net45/1.0.3" -OutFile "$env:TEMP\DotNet45.zip"
-Expand-Archive -Path "$env:TEMP\DotNet45.zip" -DestinationPath "$env:TEMP"
-Copy-Item -Path "$env:TEMP\build\.NETFramework\v4.5\*" -Destination "C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\" -Recurse -Force
-
 # Clean up any temp files generated during prerequisite installation
 Remove-Item -LiteralPath "$env:TEMP" -Recurse -Force
 New-Item -Type directory -Path "$env:TEMP"
 
 # This shaves off ~300MB as of 2021-08-31
 RunProcessChecked "choco-cleaner" @("--dummy")
 
-# Something that gets installed in ue4-build-prerequisites creates a bogus NuGet config file
-# Just remove it, so a proper one will be generated on next NuGet run
-# See https://github.com/adamrehn/ue4-docker/issues/171#issuecomment-852136034
-if (Test-Path "$env:APPDATA\NuGet")
-{
-    Remove-Item -LiteralPath "$env:APPDATA\NuGet" -Recurse -Force
-}
-
 # Display a human-readable completion message
 Write-Output "Finished installing build prerequisites and cleaning up temporary files."

src/ue4docker/dockerfiles/ue4-source/windows/Dockerfile

@@ -1,10 +1,10 @@
 # escape=`
 {% if combine %}
-FROM prerequisites as source
+FROM prerequisites as source-prep
 {% else %}
 ARG NAMESPACE
 ARG PREREQS_TAG
-FROM ${NAMESPACE}/ue4-build-prerequisites:${PREREQS_TAG}
+FROM ${NAMESPACE}/ue4-build-prerequisites:${PREREQS_TAG} as source-prep
 {% endif %}
 
 {% if source_mode == "copy" %}
@@ -51,6 +51,24 @@ RUN mkdir C:\UnrealEngine && `
 
 {% endif %}
 
+{% if combine %}
+FROM prerequisites as source
+{% else %}
+ARG NAMESPACE
+ARG PREREQS_TAG
+FROM ${NAMESPACE}/ue4-build-prerequisites:${PREREQS_TAG} as source
+{% endif %}
+
+RUN pip install json-with-comments
+# TODO: Rework this. If Windows_SDK.json is missing, generate it on previous stage. So the only file that VS installation depends on is Windows_SDK.json and we don't reinstall VS when other files change
+# Only copy files required to determine Visual Studio version to avoid reinstalling it when minor engine changes happen
+COPY --from=source-prep C:\UnrealEngine\Engine\Config\Windows C:\install-vs\
+COPY --from=source-prep C:\UnrealEngine\Engine\Build\Build.version C:\install-vs\
+COPY install-vs.py C:\install-vs\
+RUN python C:\install-vs\install-vs.py C:\install-vs\Windows_SDK.json C:\install-vs\Build.Version
+
+COPY --from=source-prep C:\UnrealEngine C:\UnrealEngine
+
 {% if (not disable_all_patches) and (not disable_windows_setup_patch) %}
 # Since the UE4 prerequisites installer appears to break when newer versions
 # of the VC++ runtime are present, patch out the prereqs call in Setup.bat

src/ue4docker/dockerfiles/ue4-source/windows/install-vs.py

@@ -0,0 +1,144 @@
+import json
+import os.path
+import typing
+
+import jsonc
+import subprocess
+import sys
+from typing import Any
+from urllib import request
+
+common_components = [
+    "Microsoft.Net.ComponentGroup.DevelopmentPrerequisites",
+    "Microsoft.VisualStudio.Component.NuGet",
+    "Microsoft.VisualStudio.Workload.VCTools",
+    "Microsoft.VisualStudio.Workload.MSBuildTools",
+]
+
+vs2022_components = common_components + [
+    "Microsoft.NetCore.Component.SDK",
+]
+
+ue5_vs_fallback_version = "17.8"
+
+
+def build_vs_channel_url(vs_version: str) -> str:
+    version_major = vs_version.split(".", 1)[0]
+    return f"https://aka.ms/vs/{version_major}/release.ltsc.{vs_version}/channel"
+
+
+class VisualStudio:
+    def __init__(self, version: str, channel_url: str, components: typing.List[str]):
+        self.version = version
+        self.channel_url = channel_url
+        self.components = sorted(set(components))
+
+    @staticmethod
+    def from_json(json_config: Any):
+        # TODO: Add option to request VS2025
+
+        try:
+            vs_version = json_config["MinimumVisualStudio2022Version"]
+        except KeyError:
+            vs_version = ue5_vs_fallback_version
+
+        components = (
+            vs2022_components
+            + json_config["VisualStudioSuggestedComponents"]
+            + json_config["VisualStudio2022SuggestedComponents"]
+        )
+
+        # UE-5.4 has buggy component version
+        if "Microsoft.VisualStudio.Component.Windows10SDK.22621" in components:
+            components.remove("Microsoft.VisualStudio.Component.Windows10SDK.22621")
+            components.append("Microsoft.VisualStudio.Component.Windows11SDK.22621")
+
+        return VisualStudio(
+            vs_version,
+            build_vs_channel_url(vs_version),
+            components,
+        )
+
+    def download_installer(self, installer_path: str) -> None:
+        print(f"Downloading Visual Studio {self.version}...")
+        # NOTE: We use the Visual Studio 2022 installer even for older Visual Studio here because the old (2017) installer now breaks
+        request.urlretrieve(
+            "https://aka.ms/vs/17/release/vs_buildtools.exe", installer_path
+        )
+
+    def install(self, installer_path: str) -> None:
+        argv = [
+            installer_path,
+            "--quiet",
+            "--wait",
+            "--channelUri",
+            self.channel_url,
+            "--productId",
+            "Microsoft.VisualStudio.Product.BuildTools",
+            "--norestart",
+            "--nocache",
+            "--installPath",
+            "C:/BuildTools",
+            "--locale",
+            "en-US",
+        ]
+
+        print(f"Installing Visual Studio {self.version}...")
+        print("Components:")
+        for component in self.components:
+            argv.append("--add")
+            argv.append(component)
+            print(f" * {component}")
+
+        sys.stdout.flush()
+
+        subprocess.run(argv, check=True)
+
+
+if __name__ == "__main__":
+    windows_sdk_path = sys.argv[1]
+    build_version_path = sys.argv[2]
+
+    if os.path.exists(windows_sdk_path):
+        print(f"Using Windows_SDK.json to determine VS components")
+        # 5.4 or newer
+        with open(sys.argv[1]) as windows_sdk_file:
+            windows_sdk_json = jsonc.load(windows_sdk_file)
+        vs = VisualStudio.from_json(windows_sdk_json)
+    else:
+        print(f"Using fallback to determine VS components")
+        with open(build_version_path) as build_version_file:
+            engine_version = json.load(build_version_file)
+        engine_major = engine_version["MajorVersion"]
+        engine_minor = engine_version["MinorVersion"]
+
+        if engine_major == 5:
+            # Assuming 5.3 or older
+            vs = VisualStudio(
+                ue5_vs_fallback_version,
+                build_vs_channel_url(ue5_vs_fallback_version),
+                vs2022_components
+                + [
+                    "Microsoft.Net.Component.4.6.2.TargetingPack",
+                    "Microsoft.NetCore.Component.Runtime.3.1",
+                    f"Microsoft.VisualStudio.Component.VC.14.38.{ue5_vs_fallback_version}.x86.x64",
+                    "Microsoft.VisualStudio.Component.Windows10SDK.20348",
+                ],
+            )
+        else:
+            # Assuming 4.27
+            vs = VisualStudio(
+                "15",
+                f"https://aka.ms/vs/15/release/channel",
+                common_components
+                + [
+                    "Microsoft.Net.Component.4.5.TargetingPack",
+                    "Microsoft.Net.Component.4.6.2.TargetingPack",
+                    "Microsoft.VisualStudio.Component.VC.Tools.x86.x64"
+                    "Microsoft.VisualStudio.Component.Windows10SDK.17763",
+                ],
+            )
+
+    vs_installer_path = "C:/vs_buildtools.exe"
+    vs.download_installer(vs_installer_path)
+    vs.install(vs_installer_path)