feat(core): lock graph creation when running in another process

[AgentEnder]

Current Behavior

Running Nx in multiple processes at the same time with the daemon disabled can cripple a system due to excess memory usage when creating the graph. This is due to plugin workers being started per-parent process when there is no daemon. This change enables a file lock to prevent the simultaneous processing, and read from the cache when the first run completes.

Currently, running nx show projects 30 times in parallel looks something like this:

30 processes exited within 37535ms

Expected Behavior

30 processes exited within 6435ms

Test Script

//@ts-check

const { spawn } = require('child_process');

let alive = new Set();

let start = Date.now();
let iterations = 30;

for (let i = 0; i < iterations; i++) {
  const cp = spawn('npx nx show projects', [], {
    shell: true,
    env: {
      ...process.env,
      NX_DAEMON: 'false',
      NX_VERBOSE_LOGGING: 'true',
    },
  });
  alive.add(i);
  //   cp.stdout.on('data', (data) => {
  //     console.log(`stdout [${i}]: ${data}`);
  //   });
  cp.stderr.on('data', (data) => {
    console.error(`stderr [${i}]: ${data}`);
  });
  cp.on('exit', (code) => {
    console.log(`child process ${i} exited with code ${code}`);
    alive.delete(i);
  });
}

const i = setInterval(() => {
  if (alive.size > 0) {
  } else {
    clearInterval(i);
    console.log(
      `${iterations} processes exited within ${Date.now() - start}ms`
    );
  }
}, 1);

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
nx-dev	✅ Ready (Inspect)	Visit Preview	Jan 17, 2025 10:09pm

[nx-cloud]

View your CI Pipeline Execution ↗ for commit 78836c3.

Command	Status	Duration	Result
nx affected --targets=lint,test,build,e2e,e2e-c...	❌ Failed	56m 1s	View ↗
nx run-many -t check-imports check-commit check...	✅ Succeeded	16s	View ↗
nx-cloud record -- nx-cloud conformance:check	✅ Succeeded	2s	View ↗
nx-cloud record -- nx format:check --base=ee135...	✅ Succeeded	2s	View ↗
nx-cloud record -- nx sync:check	✅ Succeeded	<1s	View ↗
nx documentation --no-dte	✅ Succeeded	<1s	View ↗

---

☁️ Nx Cloud last updated this comment at 2025-01-18 00:04:06 UTC

[FrozenPandaz]

This should be in a finally at least

[FrozenPandaz]

Use polling

[FrozenPandaz]

Don't unwrap this. Use ? to return the Err

[FrozenPandaz]

rename

[FrozenPandaz]

Get the current time sooner before Nx even starts waiting. The graph will be computing... and should have been written after it has started waiting... This guarantees that it is not old.

Also handle when there is no graph after waiting.