[source-github] Commits stream sync fails on empty repositories with 409 Conflict error

[jjose1508]

Connector Name

source-github

Connector Version

1.8.6

What step the error happened?

None

Relevant information

Description

The GitHub source connector fails during sync when one or more repositories in the configured repository list are empty (have no commits or branches). The entire sync job crashes instead of gracefully skipping the empty repository and continuing with others.

Error Message

{
  "failureOrigin": "source",
  "failureType": "system_error",
  "internalMessage": "Conflict.",
  "externalMessage": "Git Repository is empty."
}

Environment

• Airbyte version: 0.64 (Self-hosted, Docker)
• GitHub connector version: 1.8.6

Steps to Reproduce

1. Create a new GitHub repository but don't add any files/commits (leave it empty)
2. Configure the GitHub source connector with a repository list that includes the empty repo
3. Enable the commits stream in the sync
4. Run a sync job
5. The sync fails with the "Git Repository is empty" error

Expected Behavior

The connector should detect that a repository is empty (has no branches), log a warning, skip it gracefully, and continue syncing commits from other non-empty repositories.

Actual Behavior

The entire sync job fails when it encounters an empty repository.

Root Cause

The issue is in Commits._validate_branches_to_pull() method in streams.py (lines ~736-758). When iterating repositories, it tries to access default_branches[repo] but for empty repos this key doesn't exist, causing a KeyError.

Related

• PR 🐛 Source Github: handling empty repos, check method using RepositoryStats stream #5788 (Sept 2021) fixed empty repo handling for connection check but not for commits stream sync
• Community forum report: https://discuss.airbyte.io/t/issue-with-github-source-logs/7962

Impact

Organizations using org/* wildcards or managing many repos cannot sync if any repo is empty.

Relevant log output

{
  "failureOrigin": "source",
  "failureType": "system_error", 
  "internalMessage": "Conflict.",
  "externalMessage": "Git Repository is empty."
}

Contribute

• Yes, I want to contribute

---

Internal Tracking: https://github.com/airbytehq/oncall/issues/11171

[octavia-bot]

AI Triage Starting...

Thank you for opening this issue! I'm an AI assistant that will help with initial triage.

I'll review your issue and:

• Ask clarifying questions if needed
• Provide immediate guidance if available
• Escalate to the appropriate team if this requires engineering attention

View playbook

• 🔗 Session URL: https://app.devin.ai/sessions/84f1c17fbb2248b69249776d32818592

✅ Devin AI session created successfully!

[devin-ai-integration]

Hi @jjose1508, thank you for reporting this issue with such detailed information!

I've analyzed the source-github connector code and can confirm this is a bug. Your root cause analysis is accurate.

Technical Analysis

The issue is in the Commits stream's _validate_branches_to_pull() method in streams.py (lines 736-758):

1. The method iterates through repositories and builds a mapping of branches to pull commits from
2. For empty repositories, the Branches stream returns no branches (because empty repos have no branches)
3. At line 756-757, when no branches match for a repo, the code falls back to default_branches[repo]
4. While the RepositoryStats stream returns a default_branch value even for empty repos, attempting to fetch commits from this branch triggers the 409 Conflict error

The error handler in errors_handlers.py (lines 98-104) does handle 409 Conflict with "Git Repository is empty" by returning ResponseAction.IGNORE, but this only applies during record reading, not during the stream_slices() phase where _validate_branches_to_pull() is called.

Suggested Fix:
The _validate_branches_to_pull() method should check if a repository has any branches before adding it to branches_to_repos. If a repo has no branches (empty repo), it should be skipped with a warning log message.

I'm escalating this to our internal team for further investigation and a fix. I'll update this issue with a tracking link shortly.

I also noticed you've indicated willingness to contribute - that's great! If you'd like to work on a fix, the approach would be to modify _validate_branches_to_pull() to skip repositories that have no branches (indicating they're empty).

---

This message is generated from Devin.ai (internal session link).

[devin-ai-integration]

I've escalated this issue to our internal team for tracking and prioritization: https://github.com/airbytehq/oncall/issues/11171

The team will review and work on a fix. In the meantime, as a workaround, you can exclude empty repositories from your repository list configuration.

---

This message is generated from Devin.ai (internal session link).

[devin-ai-integration]

I've created a Draft PR to fix this issue: #72892

The fix adds a check in _validate_branches_to_pull() to identify repositories that have no branches (indicating they are empty) and skip them with a warning log message. This prevents the 409 error from occurring in the first place.

The PR includes a unit test to verify the fix works correctly.

---

This message is generated from Devin.ai (internal session link).