feat: add schema isolation infrastructure for parallel Jest workers #3425
+354
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🍹 The Update (preview) for dailydotdev/api/prod (at 692be83) was successful. Resource Changes Name Type Operation
~ vpc-native-update-current-streak-cron kubernetes:batch/v1:CronJob update
~ vpc-native-post-analytics-history-day-clickhouse-cron kubernetes:batch/v1:CronJob update
~ vpc-native-calculate-top-readers-cron kubernetes:batch/v1:CronJob update
~ vpc-native-post-analytics-clickhouse-cron kubernetes:batch/v1:CronJob update
~ vpc-native-ws-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-update-trending-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-tags-str-cron kubernetes:batch/v1:CronJob update
~ vpc-native-validate-active-users-cron kubernetes:batch/v1:CronJob update
~ vpc-native-user-profile-updated-sync-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-source-tag-view-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-zombie-images-cron kubernetes:batch/v1:CronJob update
~ vpc-native-generic-referral-reminder-cron kubernetes:batch/v1:CronJob update
~ vpc-native-sync-subscription-with-cio-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-gifted-plus-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-zombie-users-cron kubernetes:batch/v1:CronJob update
+ vpc-native-api-clickhouse-migration-d6a6de39 kubernetes:batch/v1:Job create
~ vpc-native-daily-digest-cron kubernetes:batch/v1:CronJob update
~ vpc-native-hourly-notification-cron kubernetes:batch/v1:CronJob update
~ vpc-native-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-update-tag-recommendations-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-zombie-opportunities-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-highlighted-views-cron kubernetes:batch/v1:CronJob update
~ vpc-native-generate-search-invites-cron kubernetes:batch/v1:CronJob update
~ vpc-native-clean-stale-user-transactions-cron kubernetes:batch/v1:CronJob update
~ vpc-native-bg-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-temporal-deployment kubernetes:apps/v1:Deployment update
+ vpc-native-api-db-migration-d6a6de39 kubernetes:batch/v1:Job create
- vpc-native-api-db-migration-b54763f8 kubernetes:batch/v1:Job delete
~ vpc-native-check-analytics-report-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-source-public-threshold-cron kubernetes:batch/v1:CronJob update
~ vpc-native-update-views-cron kubernetes:batch/v1:CronJob update
~ vpc-native-private-deployment kubernetes:apps/v1:Deployment update
~ vpc-native-personalized-digest-deployment kubernetes:apps/v1:Deployment update
- vpc-native-api-clickhouse-migration-b54763f8 kubernetes:batch/v1:Job delete
~ vpc-native-clean-zombie-user-companies-cron kubernetes:batch/v1:CronJob update
~ vpc-native-personalized-digest-cron kubernetes:batch/v1:CronJob update
|
Add infrastructure for PostgreSQL schema isolation to enable parallel Jest workers within CI jobs. Each worker gets its own schema to prevent data conflicts between tests. Changes: - Add TYPEORM_SCHEMA env var support and auto-schema selection based on JEST_WORKER_ID when ENABLE_SCHEMA_ISOLATION=true - Set PostgreSQL search_path at connection level for raw SQL queries - Add createWorkerSchema() to copy table structures, views, and migrations data from public schema to worker schemas - Use pg_get_serial_sequence() for sequence resets to handle different sequence naming conventions Known limitation: Database triggers are not copied as they reference functions in the public schema. Schema isolation is opt-in via ENABLE_SCHEMA_ISOLATION=true environment variable. Addresses ENG-283
idoshamun
force-pushed
the
eng-283-schema-isolation-infrastructure
branch
from
82b36ef to
fe26d6a
Compare
Enable parallel test execution within CI jobs by giving each Jest worker its own PostgreSQL schema. This significantly improves test throughput. Changes: - Update CircleCI to use --maxWorkers=4 with ENABLE_SCHEMA_ISOLATION=true - Add test:parallel npm script for local parallel test execution - Enhance createWorkerSchema() to copy: - Table structures (LIKE ... INCLUDING ALL) - Views with schema references updated - Materialized views with schema references updated - All user-defined functions with schema references updated - Triggers with schema and function references updated The schema isolation copies all database objects from public schema to worker-specific schemas (test_worker_1, test_worker_2, etc.), allowing tests to run in parallel without data conflicts. Addresses ENG-284
Fixes several issues with PostgreSQL schema isolation for parallel Jest workers: 1. FK constraint copying: Tables copied with INCLUDING ALL don't include FK constraints. Now explicitly copy FK constraints with correct schema references so CASCADE and SET NULL actions work properly. 2. Seed data copying: Copy critical seed data (ghost user '404', system user, system sources, etc.) to worker schemas so tests don't fail when expecting these records. 3. Trigger function search_path: Add SET search_path clause to plpgsql functions so unqualified table names in trigger bodies resolve to the correct worker schema instead of defaulting to public. 4. Hardcoded schema references: Remove explicit 'public.' references from cron jobs (updateViews, updateDiscussionScore, checkReferralReminder) so they work with schema isolation. 5. Increased beforeAll timeout to 60s to accommodate FK constraint copying. Test results with schema isolation: 180/198 test suites pass (3785/3916 tests).
Prevent deletion of predefined seed/reference data tables during test cleanup to maintain test stability and ensure critical data remains intact.
When CREATE TABLE ... LIKE ... INCLUDING ALL copies tables, column
defaults still reference the original public schema sequences. This
caused FK constraint violations when tests used TypeORM's save() with
@PrimaryGeneratedColumn('increment') - the database used the wrong
sequence position instead of starting at 1.
Changes:
- Create new sequences in worker schemas and update column defaults
- Remove seed data copying for tables where tests create own fixtures
(advanced_settings, source_category, prompt)
- Use schema-qualified table names in sequence reset logic
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
TYPEORM_SCHEMAandENABLE_SCHEMA_ISOLATIONenvironment variable support indata-source.tscreateWorkerSchema()to create isolated schemas by copying table structures and viewscleanDatabase()to use schema-aware queriesHow it works
When
ENABLE_SCHEMA_ISOLATION=trueis set along withJEST_WORKER_ID, each Jest worker gets its own PostgreSQL schema (test_worker_1,test_worker_2, etc.). The setup:LIKE ... INCLUDING ALLCurrent state
The infrastructure is in place but not enabled in CI. Enabling it requires setting
ENABLE_SCHEMA_ISOLATION=trueand changing--runInBandto--maxWorkers=4.Known limitation
68/73 boot tests pass with schema isolation, but 5 tests fail due to raw SQL queries in auth/boot code that don't use the schema prefix. Full enablement (ENG-284) is blocked until these are addressed.
Test plan
NODE_ENV=test npx jest --runInBand)ENG-283