Add maxWorkflowThreadHeapMiB to prevent OOM crashes from workflow cache - #2302
Open
sachinsharma3191 wants to merge 1 commit into
Open
Add maxWorkflowThreadHeapMiB to prevent OOM crashes from workflow cache#2302sachinsharma3191 wants to merge 1 commit into
sachinsharma3191 wants to merge 1 commit into
Conversation
Add a new WorkerOptions.maxWorkflowThreadHeapMiB option that sets resourceLimits.maxOldGenerationSizeMb on workflow worker threads. This converts a fatal process-level OOM crash into a per-thread error that the worker handles through existing error paths. Without this, when the workflow cache grows large enough (due to heavy per-workflow module state like zod schemas), the V8 heap exhaustion kills the entire process with ERR_WORKER_OUT_OF_MEMORY, which cannot be caught or recovered from. Also improves the error message on OOM to suggest actionable fixes. Ref temporalio#2227
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
WorkerOptions.maxWorkflowThreadHeapMiBoption that setsresourceLimits.maxOldGenerationSizeMbon workflow worker threads created inthreaded-vm.ts.Problem: The workflow cache is count-based LRU (
maxCachedWorkflows) with no memory awareness. When per-workflow memory footprint is large (e.g. heavyzodschemas or top-level module state), the cache can exhaust the V8 heap, causingERR_WORKER_OUT_OF_MEMORY— a fatal, unrecoverable crash that kills the entire worker process.Solution: By setting
resourceLimitson thenode:worker_threadsWorker constructor, V8 heap exhaustion in a workflow thread becomes a catchable error instead of a process crash. The existingUnexpectedErrorhandling path already reports the failure and initiates graceful shutdown.Changes
worker-options.ts: NewmaxWorkflowThreadHeapMiB?: numberoption with documentation. Validation rejects non-positive values.worker.ts: Passes the option through toThreadedVMWorkflowCreator.create().threaded-vm.ts:maxOldGenerationSizeMband passes it asresourceLimitstonew NodeWorker()maxCachedWorkflowsormaxWorkflowThreadHeapMiBtuningUsage
Design note
This is the minimal, self-contained fix that prevents the fatal crash. A full memory-aware cache (feeding V8 heap stats back to the Rust core's eviction decisions) would require cross-layer changes and is tracked separately. This PR gives users an immediate escape hatch.
Ref #2227
Test plan
resourceLimitsis only set whenmaxWorkflowThreadHeapMiBis provided (default behavior unchanged)maxWorkflowThreadHeapMiB <= 0throwsTypeErrormaxWorkflowThreadHeapMiBset