Container orchestrator ignores CONTAINER_ORCHESTRATOR_MEMORY_LIMIT and REPLICATION_ORCHESTRATOR_MEMORY_LIMIT - always uses 1Gi

[clementmuffatjoly]

Helm Chart Version

2.0.19

What step the error happened?

During the Sync

Relevant information

Describe the bug

The container orchestrator in replication jobs ignores all memory configuration (Helm values, environment variables) and always runs with 1Gi memory limit, causing OOM crashes on large CDC syncs.

Environment

• Airbyte Version: 2.0.1
• Helm Chart Version: 2.0.19
• Kubernetes: GKE Standard 1.31
• Deployment: Self-hosted (Kubernetes)

Configuration

We have configured orchestrator memory in multiple places:

1. Helm values (via Terraform):

global:
  workloads:
    resources:
      containerOrchestrator:
        memory:
          request: 512Mi
          limit: 2Gi

2. Environment variables in ConfigMap:

REPLICATION_ORCHESTRATOR_MEMORY_LIMIT: 4Gi
REPLICATION_ORCHESTRATOR_MEMORY_REQUEST: 1Gi

3. Environment variables in workload-launcher deployment:

CONTAINER_ORCHESTRATOR_MEMORY_REQUEST: 512Mi
CONTAINER_ORCHESTRATOR_MEMORY_LIMIT: 2Gi

Actual Behavior

Despite all configurations, the orchestrator container in replication pods always has:

orchestrator    1Gi limit    1Gi request

Verified with:

kubectl -n airbyte get pod replication-job-XXX-attempt-0 \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.limits.memory}{"\n"}{end}'

# Output:
orchestrator    1Gi      # Should be 2Gi or 4Gi
source          32Gi     # Correctly applied from connection resource_requirements
destination     32Gi     # Correctly applied from connection resource_requirements

Expected Behavior

The orchestrator container should respect the configured memory limits (2Gi or 4Gi).

Impact

This causes OOM crashes on CDC syncs with large binlog gaps:

java.lang.OutOfMemoryError: Java heap space
Terminating due to java.lang.OutOfMemoryError: Java heap space

The sync fails after processing only 4 records, even though source/destination have 32Gi available.

Workarounds Attempted

1. ✅ connection.resource_requirements in database → Works for source/destination, NOT for orchestrator
2. ❌ Helm values containerOrchestrator.memory → Ignored
3. ❌ CONTAINER_ORCHESTRATOR_MEMORY_LIMIT env var → Ignored
4. ❌ REPLICATION_ORCHESTRATOR_MEMORY_LIMIT env var → Ignored

Related Issues

• workload resource requests and limits not applied for replication jobs #68162 - workload resource requests and limits not applied for replication jobs
• Sync jobs do not appear to respect CPU / memory requests and limits #42921 - Sync jobs do not appear to respect CPU / memory requests and limits

Logs

INFO debezium-engine BaseSourceTask(logStatistics):354 4 records sent during previous 00:00:16.555,
last recorded offset of {server=gestcom} partition is {ts_sec=1770182352, file=mysql-bin.004401, pos=272435933}

# java.lang.OutOfMemoryError: Java heap space
Terminating due to java.lang.OutOfMemoryError: Java heap space

Request

Please provide a way to configure the orchestrator container memory, either:

1. Fix the existing configuration options to be respected
2. Add support in connection.resource_requirements for orchestrator
3. Document the correct way to configure orchestrator resources

This is blocking production CDC pipelines for critical business data.

Relevant log output

---

Internal Tracking: https://github.com/airbytehq/oncall/issues/11158

[octavia-bot]

AI Triage Starting...

Thank you for opening this issue! I'm an AI assistant that will help with initial triage.

I'll review your issue and:

• Ask clarifying questions if needed
• Provide immediate guidance if available
• Escalate to the appropriate team if this requires engineering attention

View playbook

• 🔗 Session URL: https://app.devin.ai/sessions/25e1ede0f70b434a883577f1d624763c

✅ Devin AI session created successfully!

[devin-ai-integration]

Hi @clementmuffatjoly, thank you for reporting this issue with such detailed information!

I can confirm this is a known issue affecting multiple users. The container orchestrator memory configuration is not being respected, and this appears to be related to how resource requirements are resolved in the platform.

Root Cause Analysis

Based on investigation, the issue stems from how the workload-launcher resolves resource requirements. The Airbyte database contains actor-specific resource requirements that can override global workloads.resources settings. Even with useConnectorResourceDefaults: false, persisted actor definitions in the database may take precedence over Helm values and environment variables.

Temporary Workaround

Until this is fixed, you can set resource_requirements directly in the database at the connection level. This has been confirmed to work by other users experiencing the same issue:

-- Update specific connection(s) with orchestrator-appropriate resources
-- Note: This affects source/destination containers, not the orchestrator directly
UPDATE connection
SET resource_requirements = jsonb_build_object(
  'cpu_request', '0.5',
  'cpu_limit', '2',
  'memory_request', '1Gi',
  'memory_limit', '4Gi'
)
WHERE id = '<your-connection-id>';

Important Note: The connection.resource_requirements setting applies to source/destination containers. Unfortunately, there is currently no direct database-level override specifically for the orchestrator container resources.

Related Issues

This issue is related to:

• workload resource requests and limits not applied for replication jobs #68162 - workload resource requests and limits not applied for replication jobs
• Sync jobs do not appear to respect CPU / memory requests and limits #42921 - Sync jobs do not appear to respect CPU / memory requests and limits

I'm escalating this to the internal engineering team for investigation and will update this issue with a tracking link shortly.

---

This message is generated from Devin.ai (internal session link).

[devin-ai-integration]

This issue has been escalated to the internal engineering team for investigation.

Internal Tracking: https://github.com/airbytehq/oncall/issues/11158

The team will investigate why the orchestrator container resource configuration is not being respected and work on a fix. I'll update this issue when there's progress.

---

This message is generated from Devin.ai (internal session link).

[mako042]

Привет! У меня такая же проблема, поэтому я установил ресурсы через SQL команду (что крайне неудобно..)
Хотелось бы видеть в веб интерфейсе возможность указания ресурсов для коннектора

[clementmuffatjoly]

Temporary workaround: Kubernetes Mutating Webhook

We implemented a workaround using a Kubernetes Mutating Webhook that automatically patches the orchestrator container memory from 1Gi to 4Gi for all replication job pods.

How it works

1. A MutatingWebhookConfiguration intercepts pod CREATE requests in the Airbyte namespace
2. Pods with label airbyte=job-pod are sent to our webhook
3. The webhook inspects the pod spec, finds the orchestrator container, and applies a JSONPatch to set memory limits/requests to 4Gi
4. The pod is created with the corrected memory allocation

Key characteristics

• Transparent: No modification to Airbyte itself
• Fail-open: failurePolicy: Ignore ensures pods are created normally if the webhook is unavailable
• Easy rollback: Delete the MutatingWebhookConfiguration to disable instantly

Validation

We tested with CDC syncs that were previously failing with OOMKilled. The orchestrator memory peaked at 2.1Gi and 2.9Gi respectively - both would have crashed with the hardcoded 1Gi limit but succeeded with our 4Gi patch.

Implementation

• Simple Python Flask webhook (~100 lines)
• Self-signed TLS certificates
• Deployed as a single-replica Deployment in the Airbyte namespace

This is obviously a temporary workaround until the upstream fix is available. Happy to share the code if anyone needs it.