feat: package on Node.js 24 instead of 20 MONGOSH-3062 (#2622)

* feat: package on Node.js 24 instead of 20

* chore: Update the aws-sdk to the latest 2x (unmaintained)

The depepdency to the aws-sdk 2 will be removed once we
start using the MongoDB Node.js driver 7.0.0

* chore: Use --legacy-peer-deps with npm@11 and upgrade Lerna to 9.0.3

npm@11 handles peer dependencies differently, so when bootstrapping
mongosh with Node.js 24 it breaks the package-lock.json for Node.js
20. This is an issue if we want to keep backwards compatiblity for
some time, so we have to use --legacy-peer-deps. However, lerna 8x
uses nx in a way it's incompatible with --legacy-peer-deps, which
required upgrading to 9.0.3.

* chore: Remove patches already merged upstream

* chore: Fix run-if-package-requested.js for Node.js v24

It seems a change broke it and because Node.js 24 pipelines
do not run by default it wasn't catched.

* chore: Drop support for building in macos-11

Node.js 24 drops support for macos older than 13, as
documented in:
https://github.com/nodejs/node/blob/v24.12.0/BUILDING.md#platform-list

* chore: Update windows distro to use windows-2022-large

Windows-vsCurrent-small does not have the clang module for
msbuild, so use a more modern one that does have it. This
change mimicks what we've done in boxednode to have the
Windows build working.

* chore: Force GYP to use the correct Python installation

It has to use 3.11 to avoid issues with _ctypes.

* chore: Upgrade clang in osx machines

* chore: Run tests also in Node.js 24 now

We should be able to run all the tests in Node.js 24 and detect
any outstanding issue that has to be fixed in code.

Eventually, we will want to get rid of running the tests in
Node.js 20

* chore: Update mongodb-crypt-library-version to a Node.js 24 compatible version

* chore: Upgrade Amazon2 to the latest version with an updated toolchain

* chore: Add memfd_create patch as we've done in our toolchain

* chore: Fix tests where the Node.js repl prompt uses '|'

* chore: revert unnecessary changes on distros and disable n20

* chore: Add the aws-sdk as a peer dependency

As it's necessary for the build and the driver

* chore: Add explicit dependency to aws-sdk/credential-providers

* chore: Hoist buffer in package-lock.json

With the change to npm@11, npm seems to hvae decided unhost buffer,
which makes some tests fail. Here we are forcing buffer to be
hoisted by doing:

`npm install buffer && git reset -- package.json && npm install`

In case in the future we see a similar issue, hoisting it manually
seems to solve it.

* chore: Fix issue with the change from '...' to '|' for unterminated outputs

* chore: Update devtools-connect to the latest version

* chore: Update macos-export-certificate-and-key for Node.js 24 support

* chore: Compile in vcdev env for Windows

* chore: Hoist macos-export-certificate-and-key

* chore: Update system-ca and transitive deps to Node.js 24

* chore: Force updating mongodb-client-encryption to 7.0.0

It's the first version that supports Node.js 24

* chore: Update devtools-connect and proxy support

* chore: Patch kerberos to make private fields non linkable

* chore: Support delayed-template-parsing due to clang constrains

* chore: Use latest 2022 version

* chore: Remove patch for older compilers

* chore: Do not enable ltcg by default on release builds

* chore: Try to use the inherited vcdev

* chore: Fix hoisting problem on mongodb-client-encryption

We need a few overrides because we still depend on the
6.x branch of the driver, but we need to upgrade
the client encryption to 7.x to support Node.js 24.

* fix: Update oidc-plugin to latest version

The latest version changes how we use spawn and avoids
deprecation warnings.

* chore: Allow runNpm to return the stdout on failures for specific commands

Right now npm outdated exits with a non-zero status. We have some
additional checks where, if the stderr is empty, we consider the
command to be successful. However, npm prints warnings to stderr,
so a successful command with a warning fails.

For cases where we really don't care about the status code, like
npm ls or npm outdated, we pass the lenient flag, that will tell
runNpm to ignore the exitCode and return stdout either way.

* chore: Use a smaller instance for signature verification

* fix: Fix tests related to the Node.js 24 update.

There is a bug in the Node.js REPL that we will need to fix upstream
or with a patch whenever we compile Node.js, so unless it runs on
a patched Node.js version, some tests will still fail.

* chore: Remove error handling for history loading

In Node.js 24, the REPL itself will shallow the error
and print the error itself, so we don't need to catch
the error anymore.

* chore: Monkey-patch Node.js 24 if we detect the history rewrite bug

In Node.js 24 there is a bug in the history handling where, in the
event that an input fails, and the next works fine, the failed
input is rewritten. This monkey-patch detects the issue and rewrites
the internal history handler to ignore errors, as really don't care
about them at this point.

* chore: Typing error

* chore: Fix tests that depend on Node.js 24 REPL formatting and behaviour

* chore: Fix the rest of the tests that depend on formatting

* chore: Fix dependencies with oktokit (< n24)

* chore: Fix package-lock inconsistencies after the merge with main

* chore: Remove duplicate keys

* chore: Ensure that we test on windows pre-2022

* chore: Disable RHEL10 tests

Enabling them depends on DEVPROD-27117

* fix: Drop support for macos-11 hosts

* chore: Run pkg_test_ssh on pre-2022 machines

* chore: Compile in Sequoia but target Ventura

Also, add e2e tests in Sequoia.

* chore: Use GUI variants as we have a bigger pool

* chore: Add macos-15 for distro uploadlist regex

Author: kmruiz

.evergreen.yml

@@ -67,20 +67,20 @@ exports.COMPILE_BUILD_VARIANTS = [
     executableOsId: 'linux-s390x',
   },
   {
-    displayName: 'MacOS Big Sur',
+    displayName: 'MacOS 15 Sequoia (amd64)',
     id: 'darwin',
-    runOn: 'macos-11',
+    runOn: 'macos-15-amd64-gui',
     executableOsId: 'darwin-x64',
   },
   {
-    displayName: 'MacOS Big Sur arm64',
-    runOn: 'macos-11-arm64',
+    displayName: 'MacOS 15 Sequoia (arm64)',
+    runOn: 'macos-15-arm64-gui',
     executableOsId: 'darwin-arm64',
   },
   {
     id: 'win32',
     displayName: 'Windows VS 2022',
-    runOn: 'windows-vsCurrent-large',
+    runOn: 'windows-2022-latest-small',
     executableOsId: 'win32',
   },
 ].map((buildVariant) => {

.evergreen/build-variants/compile-build-variants.js

@@ -92,6 +92,7 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
     tags: ['nightly-driver'],
     executableOsId: 'linux-x64',
     mVersion: 'stable',
+    disabled: true,
   },
   {
     displayName: 'RHEL 10 x64',
@@ -100,6 +101,7 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
     sharedOpenSsl: 'openssl3',
     executableOsId: 'linux-x64-openssl3',
     mVersion: 'stable',
+    disabled: true,
   },
   {
     displayName: 'Ubuntu 18.04 x64',
@@ -319,6 +321,7 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
     tags: ['nightly-driver'],
     executableOsId: 'linux-arm64',
     mVersion: 'stable',
+    disabled: true,
   },
   {
     displayName: 'RHEL 10 arm64',
@@ -327,6 +330,7 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
     sharedOpenSsl: 'openssl3',
     executableOsId: 'linux-arm64-openssl3',
     mVersion: 'stable',
+    disabled: true,
   },
   {
     displayName: 'RHEL 8 PPC',
@@ -370,6 +374,24 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
     executableOsId: 'linux-s390x',
     mVersion: '8.2.x',
   },
+  {
+    displayName: 'MacOS 15 Sequoia (amd64)',
+    runOn: 'macos-15-amd64-gui',
+    executableOsId: 'darwin-x64',
+    mVersion: 'stable',
+  },
+  {
+    displayName: 'MacOS 15 Sequoia (arm64)',
+    runOn: 'macos-15-arm64-gui',
+    executableOsId: 'darwin-arm64',
+    mVersion: 'stable',
+  },
+  {
+    displayName: 'MacOS 15 Sequoia (arm64)',
+    runOn: 'macos-15-arm64-gui',
+    executableOsId: 'darwin-arm64',
+    mVersion: '8.2.x',
+  },
   {
     displayName: 'MacOS 14 x64',
     runOn: 'macos-14',
@@ -391,13 +413,13 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
   {
     displayName: 'MacOS Big Sur',
     id: 'darwin',
-    runOn: 'macos-11',
+    runOn: 'macos-13',
     executableOsId: 'darwin-x64',
     mVersion: '8.0.5',
   },
   {
     displayName: 'MacOS Big Sur arm64',
-    runOn: 'macos-11-arm64',
+    runOn: 'macos-13-arm64',
     id: 'darwin_arm64',
     executableOsId: 'darwin-arm64',
     mVersion: '8.0.5',
@@ -409,7 +431,7 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
     ],
   },
   {
-    displayName: 'Windows VS 2022',
+    displayName: 'Windows VS PRE-2022',
     runOn: 'windows-vsCurrent-small',
     executableOsId: 'win32',
     mVersion: 'stable',
@@ -419,12 +441,29 @@ exports.E2E_TESTS_BUILD_VARIANTS = [
     ],
   },
   {
-    displayName: 'Windows VS 2022',
+    displayName: 'Windows VS PRE-2022',
     runOn: 'windows-vsCurrent-small',
     executableOsId: 'win32',
     mVersion: '8.2.x'
   },
-].map((buildVariant) => {
+  {
+    displayName: 'Windows VS 2022',
+    runOn: 'windows-2022-latest-small',
+    executableOsId: 'win32',
+    mVersion: 'stable',
+    additionalTasks: [
+      'package_artifact_win32_x64',
+      'package_artifact_win32msi_x64',
+    ],
+  },
+  {
+    displayName: 'Windows VS 2022',
+    runOn: 'windows-2022-latest-small',
+    executableOsId: 'win32',
+    mVersion: '8.2.x'
+  },
+].filter(({ disabled }) => disabled !== true)
+ .map((buildVariant) => {
   const { displayName, fips, sharedOpenSsl, mVersion, runOn, executableOsId } =
     buildVariant;
   let id = (buildVariant.id ?? runOn ?? executableOsId).replaceAll('-', '_');

.evergreen/build-variants/e2e-tests-build-variants.js

@@ -1,5 +1,6 @@
-export DISTRO_UPLOADLIST_REGEX="^(rhel70|win|rhel7-zseries|rhel8-power|amazon2-arm64|macos)"
+export DISTRO_UPLOADLIST_REGEX="^(rhel70|win|rhel7-zseries|rhel8-power|amazon2-arm64|macos-12|macos-13|macos-14|macos-15)"
 export COMPILING_CONTEXT_FILE="$(pwd)/../tmp/compiling-context.yml"
+mkdir -p $(dirname $COMPILING_CONTEXT_FILE)
 
 if ! [[ "$DISTRO_ID" =~ $DISTRO_UPLOADLIST_REGEX ]]; then
   echo "'$DISTRO_ID' will be disabled as it is not part of the DISTRO_UPLOADLIST_REGEX '$DISTRO_UPLOADLIST_REGEX'."

.evergreen/compilation-context-expansions.sh

@@ -20,8 +20,8 @@ amazon2-arm64-large=true
 amazon2-large=false
 debian10-small=false
 debian11-small=false
-macos-11=true
-macos-11-arm64=true
+macos-11=false
+macos-11-arm64=false
 macos-13-arm64=true
 rhel70-build=true
 rhel70-large=true

.evergreen/compilation-context-expansions.test.sh

@@ -23,10 +23,10 @@ export TMP=/tmp/m
 export TMPDIR=/tmp/m
 
 if [ $(uname) = Darwin ]; then
-  # match what Node.js 20 does on their own builder machines
-  export CFLAGS='-mmacosx-version-min=10.15'
-  export CXXFLAGS='-mmacosx-version-min=10.15'
-  export MACOSX_DEPLOYMENT_TARGET=10.15
+  # match what Node.js 24 does on their own builder machines
+  export CFLAGS='-mmacosx-version-min=13.5'
+  export CXXFLAGS='-mmacosx-version-min=13.5'
+  export MACOSX_DEPLOYMENT_TARGET=13.5
 fi
 
 # The CI machines we have for Windows and x64 macOS are not

.evergreen/compile-artifact.sh

@@ -16,7 +16,7 @@ exports.platformToDetails = {
   },
   win32: {
     displayName: 'Windows',
-    runOn: 'windows-vsCurrent-small',
+    runOn: 'windows-2022-small',
     executableOsId: 'win32',
   },
 };
@@ -50,12 +50,12 @@ exports.NODE_VERSIONS = [
     shortName: '20',
     versionSpec: NODE_JS_VERSION_20,
     skipNodeVersionCheck: '',
-    optional: false,
+    optional: true,
   },
   {
     shortName: '24',
     versionSpec: NODE_JS_VERSION_24,
     skipNodeVersionCheck: '',
-    optional: true,
+    optional: false,
   },
 ];