feat: add wms_oracle community connector by fivetran-clgritton · Pull Request #26 · fivetran/community_connectors

fivetran-clgritton · 2026-06-11T19:34:51Z

Summary

Adds wms_oracle/ — a generalized Oracle WMS Cloud REST API connector syncing 26 warehouse entities (orders, inventory, containers, purchasing documents, and more)
Adds wms_oracle/README.md with full documentation following the repo template
Updates root README.md to list wms_oracle under SaaS & APIs
debug run:

Connector highlights

Two-phase incremental sync per entity: mod_ts cursor-advancement (Phase 1) + create_ts catch-up for backdated records (Phase 2)
Descending historical backfill across rolling 30-day windows, newest data first
Pre-cursor hourly drift check: probes the 24 clock-aligned hours before each entity's cursor and re-pulls any window whose count increased
Parallel entity processing via ThreadPoolExecutor for backfill and mod_ts capability discovery
Adaptive page sizing: halves page_size on timeout and recalculates offset to resume without data loss
Two monitoring tables (counts_by_day, pre_cursor_hourly_counts) written each sync for observability

Test plan

Run fivetran debug --configuration=configuration.json with a real Oracle WMS instance and confirm warehouse.db is populated
Confirm no customer-specific strings in any file (grep -ri "simfoods\|simmons")
Confirm flake8 passes (W503 only, pre-existing repo-wide)
Confirm black --check passes

🤖 Generated with Claude Code

Copilot

Pull request overview

Adds a new wms_oracle/ community connector example that syncs Oracle WMS Cloud REST API entities into Fivetran, including incremental sync + historical backfill logic and pre-sync drift monitoring, and wires it into the repository documentation.

Changes:

Added a full Oracle WMS connector implementation split across connector.py and helper modules (api.py, incremental.py, backfill.py, pre_sync_drift_check.py, utils.py).
Added connector documentation (wms_oracle/README.md) and example config (wms_oracle/configuration.json).
Updated repository-level docs/linting (README.md, .flake8).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
wms_oracle/connector.py	Main connector entrypoint, schema, and sync orchestration (incremental/backfill/drift-check).
wms_oracle/api.py	Oracle WMS REST API request + pagination utilities and capability probing.
wms_oracle/incremental.py	Two-phase incremental sync implementation (mod_ts + create_ts catch-up).
wms_oracle/backfill.py	Historical backfill implementation using rolling windows and DESC pagination.
wms_oracle/pre_sync_drift_check.py	Pre-sync hourly drift detection and monitoring table writes.
wms_oracle/utils.py	Constants, entity list, configuration validation, timestamp utilities.
wms_oracle/configuration.json	Example connector configuration template.
wms_oracle/README.md	Connector example documentation.
README.md	Adds `wms_oracle` to the catalog list.
.flake8	Updates global flake8 ignore list.

fivetran-JenasVimal

Please Address all of the copilot comments

fivetran-clgritton

I have made the requested changes

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.

fivetran-sahilkhirwal

Please check the copilot comments
Logically code looks good

fivetran-JenasVimal

LGTM

fivetran-JenasVimal · 2026-07-03T07:59:05Z

Hi @fivetran-clgritton

Could you please rebase your PR so that the latest checks can run and pass

Thank you!

fivetran-JenasVimal

LGTM

Adds a generalized Oracle WMS REST API connector syncing 26 warehouse entities with two-phase incremental sync, descending historical backfill across rolling 30-day windows, pre-cursor hourly drift detection, and parallel entity processing via ThreadPoolExecutor. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Require https:// in validate_configuration (was accepting http://) - Remove dead code: apply_lookback(), INCREMENTAL_LOOKBACK_MINUTES, timedelta import - Remove lag_minutes throughout (config, code, docs) — default was 0, adds confusion - Fix page_size default inconsistency: configuration.json now shows 1000 to match utils.py and README - Delete requirements.txt — requests is pre-installed in the SDK runtime - Fix INCREMENTAL_CHECKPOINT_INTERVAL_SECONDS formatting (Black parens artifact) - README: expand Data handling with mod_ts detection mechanics and adding-entity instructions - README: add Pre-cursor drift check section with full mechanics explanation - README: expand monitoring tables with column-level documentation and usage notes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Remove unused datetime imports (timezone, timedelta) from connector.py introduced when lag_minutes was removed - Wrap all E501 log message f-strings across api.py, backfill.py, connector.py, incremental.py, and pre_sync_drift_check.py - Apply black reformatting to utils.py (INCREMENTAL_CHECKPOINT_INTERVAL_SECONDS) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Black always generates W503-style line breaks (operator at start of continuation line). Without this, any connector using multi-line boolean conditions will fail the linting check. W503 and W504 are mutually exclusive — suppressing W503 is the standard fix for Black + flake8 coexistence. Also removes noqa: W503 comments added as a temporary workaround. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The comments were adding enough length to trigger Black reformatting. W503 is now globally ignored via .flake8 so the comments are redundant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

- Add template-style import comments to connector.py and api.py - Replace update() docstring with verbatim template wording - Add log.warning("Example: Oracle WMS : wms_oracle") as first log in update() - Replace single-line upsert comment with full template comment block - Add template checkpoint comment block before op.checkpoint() - Add template __main__ block comments - Add full Description/Args/Returns docstrings to check_entity_has_mod_ts and probe_entity_count - Add retry logic (408, 429, 500-504, Timeout) to check_entity_has_mod_ts - Replace literal defaults in configuration.json and README with angle-bracket placeholders - Add numbered credential-setup steps to README Authentication section Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Add full Description/Args/Returns/Raises to make_api_request and fetch_entity_data - Refactor multi-line boolean conditions into intermediate variables to eliminate W503 (line-break-before-binary-operator) violations without flake8 suppression Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace chained variable assignments with clearer named variables and explicit break conditions, eliminating W503 violations without flake8 suppression. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…nt log - Refactor multi-line boolean conditions in backfill.py and incremental.py into named variables to eliminate W503 without flake8 suppression - Simplify string concatenation in connector.py log statement - Remove redundant log.info after log.warning example banner in update() Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ckfill Replace two-line ts_boundary_cleared assignment with a single readable line using short-circuit evaluation, and rename to crossed_ts_boundary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… by Pass 1 All incremental_only entities have an incremental cursor (required by classification), so they are always included in entities_with_cursor and processed in Pass 1. Pass 3 could never execute. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Worker threads in the backfill pass now do API fetching only, putting upsert/truncate/checkpoint messages on a queue.Queue. The main thread drains the queue and performs all op.* calls, complying with the SDK's single-threaded output stream requirement. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… error ("checkpoint", state) and ("truncate", entity) were 2-tuples but the drain loop always unpacked 3 values, causing ValueError at runtime. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Guard against any unexpected queue message format with a warning log before unpacking. If a non-3-tuple arrives it is skipped and logged rather than crashing the sync, so the next run reveals the exact message that caused the original ValueError. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…_has_mod_ts Returning False after all retries fail was silently caching an incorrect mod_ts capability flag in state, causing repeated unnecessary full scans and op.truncate() calls. Now raises on transient failures (408, 429, 5xx, Timeout) so the sync fails cleanly instead of poisoning the cache. Permanent errors (non-transient status codes) still return False. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Permanent errors (auth failures, connection errors, non-transient HTTP status codes) now also raise instead of returning False, so no describe failure can silently cache an incorrect mod_ts capability in state. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ConnectionError has no HTTP status code (status=None), so it bypassed the transient_status_codes check and raised immediately without retrying. Now any RequestException without a status code is treated as transient and retried like 408/429/5xx. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…ed repull callback - Add full 6-line checkpoint comment block above final op.checkpoint() in update() - Replace lambda list comprehension in pre_sync_drift_check with named repull_records() function so the required op.upsert() comment block can be included - Add standard 3-line op.upsert() comment block to counts_by_day upsert Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ichandrasekar - Add Oracle WMS Cloud link in Requirements - Remove duplicated overview paragraph (already covered by Features section) - Remove configuration.json submission note - Delete crash-note paragraph (op.truncate/op.upsert edge case) - Capitalize all destination table names to ALL_CAPS per repo convention Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fivetran-chinmayichandrasekar

LGTM. Thanks

github-actions Bot added size/XXL ai-assisted/unknown risk/unknown labels Jun 11, 2026

fivetran-clgritton requested review from fivetran-rishabhghosh and fivetran-sahilkhirwal June 12, 2026 18:25

fivetran-rishabhghosh requested review from fivetran-JenasVimal and removed request for fivetran-rishabhghosh June 14, 2026 13:30

fivetran-JenasVimal assigned fivetran-clgritton Jun 15, 2026

fivetran-JenasVimal added ai-assisted/yes risk/low and removed risk/unknown ai-assisted/unknown labels Jun 15, 2026

fivetran-sahilkhirwal requested a review from Copilot June 16, 2026 07:22

Copilot started reviewing on behalf of fivetran-sahilkhirwal June 16, 2026 07:23 View session

Copilot AI reviewed Jun 16, 2026

View reviewed changes

fivetran-JenasVimal requested changes Jun 22, 2026

View reviewed changes

fivetran-clgritton requested a review from fivetran-JenasVimal June 24, 2026 20:35

fivetran-clgritton commented Jun 26, 2026

View reviewed changes

fivetran-sahilkhirwal requested a review from Copilot June 28, 2026 19:04

Copilot started reviewing on behalf of fivetran-sahilkhirwal June 28, 2026 19:04 View session

Copilot AI reviewed Jun 28, 2026

View reviewed changes

fivetran-sahilkhirwal reviewed Jun 28, 2026

View reviewed changes

fivetran-JenasVimal previously approved these changes Jun 29, 2026

View reviewed changes

fivetran-clgritton dismissed fivetran-JenasVimal’s stale review via 6bebd67 June 29, 2026 19:18

fivetran-clgritton requested a review from fivetran-sahilkhirwal June 30, 2026 19:47

fivetran-clgritton force-pushed the feature/wms_oracle branch from 942bd29 to b4cfcc1 Compare July 6, 2026 14:22

fivetran-JenasVimal self-requested a review July 6, 2026 20:31

fivetran-JenasVimal previously approved these changes Jul 6, 2026

View reviewed changes

fivetran-JenasVimal requested a review from fivetran-chinmayichandrasekar July 8, 2026 11:30

fivetran-clgritton and others added 23 commits July 22, 2026 17:00

fix(wms_oracle): remove noqa W503 comments now covered by .flake8 config

d84c680

The comments were adding enough length to trigger Black reformatting. W503 is now globally ignored via .flake8 so the comments are redundant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Apply suggestions from code review

efcde64

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

refactor(wms_oracle): simplify boolean conditions in fetch_entity_data

41972a2

Replace chained variable assignments with clearer named variables and explicit break conditions, eliminating W503 violations without flake8 suppression. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor(wms_oracle): rename and simplify ts boundary condition in ba…

373bb50

…ckfill Replace two-line ts_boundary_cleared assignment with a single readable line using short-circuit evaluation, and rename to crossed_ts_boundary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(wms_oracle): align __main__ block with template connector exactly

04197f8

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(wms_oracle): pad queue messages to uniform 3-tuples to fix unpack…

2a1540c

… error ("checkpoint", state) and ("truncate", entity) were 2-tuples but the drain loop always unpacked 3 values, causing ValueError at runtime. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

style(wms_oracle): wrap long warning log line to satisfy Black

66d8cce

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Apply suggestions from code review

b2f3fd4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

fivetran-clgritton dismissed fivetran-JenasVimal’s stale review via b453ef5 July 22, 2026 21:00

fivetran-clgritton force-pushed the feature/wms_oracle branch from b4cfcc1 to b453ef5 Compare July 22, 2026 21:00

fivetran-clgritton requested a review from fivetran-chinmayichandrasekar July 22, 2026 21:09

fivetran-sahilkhirwal approved these changes Jul 23, 2026

View reviewed changes

fivetran-chinmayichandrasekar approved these changes Jul 24, 2026

View reviewed changes

fivetran-clgritton merged commit 95948a1 into fivetran:main Jul 24, 2026
3 checks passed

fivetran-clgritton deleted the feature/wms_oracle branch July 24, 2026 15:35

Uh oh!

Conversation

fivetran-clgritton commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Connector highlights

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fivetran-JenasVimal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fivetran-clgritton left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fivetran-sahilkhirwal left a comment

Choose a reason for hiding this comment

Uh oh!

fivetran-JenasVimal left a comment

Choose a reason for hiding this comment

Uh oh!

fivetran-JenasVimal commented Jul 3, 2026

Uh oh!

fivetran-JenasVimal left a comment

Choose a reason for hiding this comment

Uh oh!

fivetran-chinmayichandrasekar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fivetran-clgritton commented Jun 11, 2026 •

edited

Loading