New crash @ HEAD

[keith]

Description of the bug:

INFO: Caught InterruptedException from ExecutionException for remote branch of external/rules_pycross++lock_file+modular_pip_lock_file_repo/deps/msgspec@0.19.0, which may cause a crash.
INFO: Caught InterruptedException from ExecutionException for remote branch of external/rules_pycross++lock_file+modular_pip_lock_file_repo/deps/regex@2024.11.6, which may cause a crash.
INFO: Caught InterruptedException from ExecutionException for remote branch of external/rules_pycross++lock_file+modular_pip_lock_file_repo/deps/markdown-it-py@3.0.0, which may cause a crash.
INFO: Caught InterruptedException from ExecutionException for remote branch of external/rules_pycross++lock_file+modular_pip_lock_file_repo/deps/charset-normalizer@3.3.2, which may cause a crash.
Analyzing: 7000 targets (3605 packages loaded, 186060 targets and 4943 aspects configured)
[4,068 / 5,899] 2 / 332 tests; checking cached actions; last test: //bazel/pip/requirements:update_requirements_0_test


[4,068 / 5,899] 2 / 332 tests; checking cached actions; last test: //bazel/pip/requirements:update_requirements_0_test
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.AssertionError: Neither branch of utils/benchmarking/mammoth/cluster-configurations/built-overlays.tar completed. Local was cancelled and done and remote was not cancelled and done.
        at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy.waitBranches(DynamicSpawnStrategy.java:555)
        at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy.exec(DynamicSpawnStrategy.java:244)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:46)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:282)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1236)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1141)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:652)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:780)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:345)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:202)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:471)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

Not sure if the exceptions above are related to the final trace, but I believe this is the first time I've seen this

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

9df37fd

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[keith]

cc @fmeum

[fmeum]

This is most likely a consequence of 62df880 combined with this handling of InterruptedException in this case:

bazel/src/main/java/com/google/devtools/build/lib/dynamic/DynamicSpawnStrategy.java

Lines 604 to 618 in 3e1ce33

	} else if (cause instanceof InterruptedException) {
	// If the branch was interrupted, it might be due to a user interrupt or due to our request
	// for cancellation. Assume the latter here because if this was actually a user interrupt,
	// our own get() would have been interrupted as well. It makes no sense to propagate the
	// interrupt status across threads.
	context
	.getEventHandler()
	.handle(
	Event.info(
	String.format(
	"Caught InterruptedException from ExecutionException for %s branch of %s,"
	+ " which may cause a crash.",
	mode, getSpawnReadableId(branch.getSpawn()))));
	return null;
	} else {

The messages at the top make it seem like their local branches won and cancelled the remote branches, which results in an InterruptedException wrapped in an ExecutionException since 62df880. Catching CancellationException was necessary to avoid this crash:

Caused by: java.util.concurrent.CancellationException: Task was cancelled.
        at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1021)
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:288)
        at com.google.common.util.concurrent.AbstractFutureState.blockingGet(AbstractFutureState.java:255)
        at com.google.common.util.concurrent.Platform.get(Platform.java:54)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:253)
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
        at com.google.devtools.build.lib.remote.merkletree.MerkleTreeComputer.getFromFuture(MerkleTreeComputer.java:929)
        at com.google.devtools.build.lib.remote.merkletree.MerkleTreeComputer.buildForSpawn(MerkleTreeComputer.java:309)
        at com.google.devtools.build.lib.remote.RemoteExecutionService.uploadInputsIfNotPresent(RemoteExecutionService.java:1867)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$1(RemoteSpawnRunner.java:280)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:268)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:111)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:271)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:148)
        at com.google.devtools.build.lib.dynamic.RemoteBranch.runRemotely(RemoteBranch.java:89)
        at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy.maybeExecuteNonDynamically(DynamicSpawnStrategy.java:432)
        at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy.exec(DynamicSpawnStrategy.java:207)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:46)
        at com.google.devtools.build.lib.analysis.actions.SpawnAction.execute(SpawnAction.java:282)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1237)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1142)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:653)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:780)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:345)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:202)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:471)
        ... 8 more

The remote branch would then be considered not cancelled and done, which matches the exception below. But for this to actually crash the local branch would also need to be cancelled and I'm not yet sure how that can happen.

[fmeum]

@keith Does this happen before 62df880 (both the crash an the extra log messages that precede it)? How often do you see this? And did you interrupt this build?