Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel node-gyp builds failing due to concurrent file access (Windows only) #3095

Open
MRayermannMSFT opened this issue Nov 21, 2024 · 4 comments
Open

Parallel node-gyp builds failing due to concurrent file access (Windows only) #3095

MRayermannMSFT opened this issue Nov 21, 2024 · 4 comments
Labels
Windows

Comments

@MRayermannMSFT
Copy link

MRayermannMSFT commented Nov 21, 2024 •
edited
Loading

Hi! I'm looking for help solving an issue we're starting to see more often on Windows, when npm install builds our native node modules.

The issue is that because npm install runs native module builds in parallel, and each needs exclusive access to some files, they end up failing due to not being able to get that exclusive access. We believe we're seeing this issue more often because we recently standardized on using node-addon-api for all five of our native modules. Thus, we have five parallel builds, all trying to access a lot of the same files. I've provided a series of example errors below (if someone wants full logs, I can provide those as well).

To be clear, I don't think this is necessarialy node-gyp's fault, however it may be part of the solution. For now, I am here to see if anyone in the node-gyp community has any ideas/thoughts/suggestions on how to proceed next. So far here's some ideas I've had:

  1. Use --install-strategy=nested to de-deduplicate node-addon-api, thus reducing (maybe even eliminating) the chance of conflicting file access. Our project is ultimately bundled, so this will probably work out ok, but there are likely going to be other complications. Would be nice if we could only de-deduplicate node-addon-api.
  2. Combine some of the five native modules, thus reducing the total number. This won't fix the problem, but it'll reduce the chance of it happening.
  3. Solicit npm to not do parallel native node module installs.
  4. Add a retry mechanism to node-gyp.
  5. Prebuild the binaries for as many of our native node_modules as we can, thus avoiding having to rebuild them for the main project.
  6. Write our own build-time code to de-deduplicate node-addon-api.
  7. Use --ignore-scripts, and then use npm rebuild for any package which has an installScript: true in package-lock. Currently using this.
  8. Use --foreground-scripts. Need to test this.

The specified task executable "CL.exe" could not be run. System.IO.IOException: The process cannot access the file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.command.1.tlog' because it is being used by another process. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamReader..ctor(String path) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.OpenText(String path) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.MapSourcesToCommandLines() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.GenerateSourcesOutOfDateDueToCommandLine() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.ComputeOutOfDateSources() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.SkipTaskExecution() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.ToolTask.Execute() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]


node-gyp\src\win_delay_load_hook.cc(1,1): error C1083: Cannot open compiler generated file: 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\win_delay_load_hook.obj': Permission denied [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]


npm error LINK : fatal error LNK1181: cannot open input file 'Release\obj\nothing\\node-addon-api\nothing.obj' [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]


The specified task executable "CL.exe" could not be run. System.IO.IOException: The process cannot access the file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.write.1.tlog' because it is being used by another process. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.InternalWriteAllText(String path, String contents, Encoding encoding, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.WriteAllText(String path, String contents, Encoding encoding) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.CanonicalTrackedOutputFiles.SaveTlog(DependencyFilter includeInTLog) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.PostExecuteTool(Int32 exitCode) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.ToolTask.Execute() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]


npm error LINK : fatal error LNK1181: cannot open input file 'Release\obj\nothing\win_delay_load_hook.obj' [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]


npm error C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(386,5): error MSB3374: The last access/last write time on file "Release\obj\nothing\nothing.tlog\unsuccessfulbuild" cannot be set. Could not find file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\unsuccessfulbuild'.  [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]


npm error FileTracker : error FTK1011: could not create the new file tracking log file: PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.write.1.tlog. The file exists. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

@lukekarrys
Copy link
Member

@MRayermannMSFT I think 3 is likely the best option since npm is in charge of how and when those scripts get run.

There is also a hack/workaround that uses the npm flag --foreground-scripts which forces npm scripts to run sequentially: npm/cli#3034 (comment). Can you try and see if this flag does the same to native module builds?

@MRayermannMSFT
Copy link
Author

There is also a hack/workaround that uses the npm flag --foreground-scripts which forces npm scripts to run sequentially:

Oh nice! I had actually just implemented a workaround using --ignore-scripts, where we use that flag, and then we use npm rebuild for any package which has an installScript: true in package-lock. I'll give --foreground-scripts a try as well though.

@tmm1
Copy link

tmm1 commented Dec 29, 2024 •
edited
Loading

Does this manifest as follows?

node-addon-api\Release\obj\node_addon_api_except\node_add.508A2FFA.tlog\node_addon_api_except.lastbuildstate' because it is being used by another process.

Does the --foreground-scripts workaround help?

EDIT: Missed the detailed logs above. I can confirm --foreground-scripts runs in serial and works around the issue.

tmm1 added a commit to tmm1/node-gyp that referenced this issue Dec 29, 2024
@tmm1
fix: disable msbuild.exe nodeReuse
5d9beef 
On windows, one may experience errors due to concurrent node-gyp calls (see nodejs#3095)

When retrying after such a failure, retries are observed to fail repeatedly with strange
EBUSY and EPERM errors. This turns out to be due to a caching feature where msbuild.exe
will continue to run in the background (for either 15s or 15m depending on VS version),
and keep open file handles to files and directories that the user may be trying to delete
(for example with a new call to `npm ci`).

This behavior is well documented, as is the recommended workaround implemented here.
@tmm1 tmm1 mentioned this issue Dec 29, 2024
Open
4 tasks
@tmm1
Copy link

tmm1 commented Dec 30, 2024

An alternative to --foreground-scripts is to use a simple retry loop around npm ci. However, this requires a taskkill /im vctip.exe /f within the loop to work (or the patch from #3112)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@tmm1 @lukekarrys @cclauss @MRayermannMSFT