Skip to content

Add maxWorkflowThreadHeapMiB to prevent OOM crashes from workflow cache - #2302

Open
sachinsharma3191 wants to merge 1 commit into
temporalio:mainfrom
sachinsharma3191:fix/2227-workflow-cache-oom
Open

Add maxWorkflowThreadHeapMiB to prevent OOM crashes from workflow cache#2302
sachinsharma3191 wants to merge 1 commit into
temporalio:mainfrom
sachinsharma3191:fix/2227-workflow-cache-oom

Conversation

@sachinsharma3191

Copy link
Copy Markdown

Summary

Adds a new WorkerOptions.maxWorkflowThreadHeapMiB option that sets resourceLimits.maxOldGenerationSizeMb on workflow worker threads created in threaded-vm.ts.

Problem: The workflow cache is count-based LRU (maxCachedWorkflows) with no memory awareness. When per-workflow memory footprint is large (e.g. heavy zod schemas or top-level module state), the cache can exhaust the V8 heap, causing ERR_WORKER_OUT_OF_MEMORY — a fatal, unrecoverable crash that kills the entire worker process.

Solution: By setting resourceLimits on the node:worker_threads Worker constructor, V8 heap exhaustion in a workflow thread becomes a catchable error instead of a process crash. The existing UnexpectedError handling path already reports the failure and initiates graceful shutdown.

Changes

  • worker-options.ts: New maxWorkflowThreadHeapMiB?: number option with documentation. Validation rejects non-positive values.
  • worker.ts: Passes the option through to ThreadedVMWorkflowCreator.create().
  • threaded-vm.ts:
    • Accepts maxOldGenerationSizeMb and passes it as resourceLimits to new NodeWorker()
    • Detects OOM errors specifically and logs an actionable message suggesting maxCachedWorkflows or maxWorkflowThreadHeapMiB tuning

Usage

const worker = await Worker.create({
  taskQueue: 'my-task-queue',
  workflowsPath: require.resolve('./workflows'),
  maxWorkflowThreadHeapMiB: 512, // cap each thread to 512 MiB
});

Design note

This is the minimal, self-contained fix that prevents the fatal crash. A full memory-aware cache (feeding V8 heap stats back to the Rust core's eviction decisions) would require cross-layer changes and is tracked separately. This PR gives users an immediate escape hatch.

Ref #2227

Test plan

  • Syntax and structural correctness verified (TypeScript parses clean)
  • resourceLimits is only set when maxWorkflowThreadHeapMiB is provided (default behavior unchanged)
  • Validation: maxWorkflowThreadHeapMiB <= 0 throws TypeError
  • OOM error detection: specific log message with tuning guidance
  • Manual: Reproduce OOM with millerick's repro, verify worker survives with maxWorkflowThreadHeapMiB set

Add a new WorkerOptions.maxWorkflowThreadHeapMiB option that sets
resourceLimits.maxOldGenerationSizeMb on workflow worker threads.
This converts a fatal process-level OOM crash into a per-thread error
that the worker handles through existing error paths.

Without this, when the workflow cache grows large enough (due to heavy
per-workflow module state like zod schemas), the V8 heap exhaustion
kills the entire process with ERR_WORKER_OUT_OF_MEMORY, which cannot
be caught or recovered from.

Also improves the error message on OOM to suggest actionable fixes.

Ref temporalio#2227
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant