EXPERIMENT: Windows shardCount=4 (es94x) - #1295
Closed
sscarduzio wants to merge 30 commits into
Closed
Conversation
Move CI to Ubicloud (Linux, ubicloud-standard-4 = 16GB) and Blacksmith (Windows, blacksmith-4vcpu-windows-2025), reusing ci/run-pipeline.sh unchanged. Adds .github/workflows/ci.yml mirroring all 14 Azure stages with full ES-version parity (34 Linux IT + 33 Windows IT legs), plus a pre-analysis of every secret to port (ci/github/SECRETS.md) and a placeholder registration script (set-secrets.sh). Validated with actionlint. Azure azure-pipelines.yml is left in place for parallel-run cutover. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hosted; disable Azure
Re-audit of the port against azure-pipelines.yml found and fixed:
- BLOCKER: manual release_without_testing path could never run (skipped
needs propagate; release jobs now use !cancelled() + explicit
needs.<job>.result two-arm conditions mirroring Azure)
- BLOCKER: cancel-in-progress applied to develop/master pushes and could
kill an in-flight release; now PR-only (Azure batch semantics)
- Secret leak: docker-hub-auth.sh echoed the Azure ##vso isSecret line,
which on GH Actions PRINTS base64(user:token); now CI-aware (::add-mask::)
- Missing contents:write for release tag push (resolves the deploy-key
question: GITHUB_TOKEN suffices, GH_DEPLOY_KEY_B64 dropped from docs)
- it_windows now gates build_ror and the release path (Azure
succeeded('TEST') parity); es_s3_up now ordered before all check/test
jobs (newes/* artifact race)
- Added missing Windows unit-tests job (manual run_all_tests_on_windows)
- Dynamic IT matrices from setup (no VM boots for skipped legs), monthly
CVE cache key, **/epic/** + *newes/* pattern fixes, TRAVIS_BUILD_NUMBER
mapped to run_number, job-scope GRADLE_USER_HOME clear on Windows
Per review decisions: Windows runs on free GH-hosted windows-2025
(Blacksmith dropped); Azure triggers disabled (trigger:none, pr:none,
schedules removed) leaving Azure manually runnable as fallback.
PORT.md now carries a per-trigger scenario table as the regression spec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Pin the weekly toolchains rebuild to develop (GH schedules fire on the default branch; ref guard protects against a default-branch change) - Add an image-tag drift guard to build_toolchains_image: all ror-ci-toolchains tags in ci.yml must match TOOLCHAINS_IMAGE - Gate it_windows on toolchains_verify (Azure TEST-stage parity: a broken image fails fast instead of running doomed Windows legs) - Unset empty CVE credentials on fork PRs: GitHub passes empty (not absent) secrets, and a set-but-empty NVD/OSS-Index key makes dependency-check attempt authenticated calls (401) instead of anonymous mode — Azure only exported these when not a fork - PORT.md: document the deliberately-broader md paths-ignore and that probe-host-disk.sh stays for the Azure manual fallback Remaining review findings were already fixed (toolchains gating of the manual release path) or confirmed non-issues by the reviewer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sm, sh gave exit 127) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…PshardIndex The parent shardedTest -PshardCount=K invocation also configures the test task; on a fresh CI workspace the shard filter's class-tree walk ran at configuration time before testClasses compiled anything, matched 0 suites, and the empty-shard guard killed the parent build. Children always receive -PshardIndex (ShardedGradlewTest), so gating on it skips the parent cleanly. Never seen before because CI always ran IT_PARALLELISM=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-shard log artifacts k=2 on standard-4 (16GB): ~1.4x faster when stable, but 6/10 legs lost a shard to memory death (truncated TEST xmls, no assertion failures) — 8GB per shard is under the floor. k=1 remains the stable point on 16GB. Also re-adds the per-shard log upload so shard deaths are diagnosable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…output dirs
All K shards ran integration-tests:test in the same workspace sharing
build/test-results/test — concurrent shards corrupted/cleaned each
other's result files ('Could not write XML test results', truncated
TEST-*.xml), killing otherwise-green shards. Root cause of both the
k=2 (6/10) and k=3 (10/10) leg failures; not memory pressure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ran it) The child's prebuildEsImage dependency is not a cache hit: it launches a nested Tooling-API build (Assembling ROR) and K concurrent nested builds in one workspace race each other — shards died in 31s-1m15s. The parent shardedTest task's dependsOn already prebuilds the image once before any shard spawns, so children exclude the task with -x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…16GB runner Third shard race: suites creating non-singleton ES containers call RorPluginGradleProject.assemble at TEST RUNTIME — a nested gradle build. K shard processes' nested builds race Gradle's project locks and die with BuildException (initializationError across random suites). Fix at the choke point: cross-process file lock in runTask, so every nested build (prebuild or runtime) serializes; up-to-date builds hold it briefly. Sweep: it_linux back on ubicloud-standard-4 (16GB, 5.3GB/shard) at k=3 — the k>1 failures were all software races, not RAM, so the cheap box may hold 3 shards. k=3@32GB measured ~2.1x vs k=1 (5 green legs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ine) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sweep: k=1 51.4min avg/leg, k=3 ~24min, k=4 ~22min (best), k=6 kills the VM (host OOM, runner shutdown signal, exit 137). Serial floor ~10min makes k=5 pointless next to the cliff. 2.3x faster IT legs vs k=1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…6 (VM death) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
xpack.ml.enabled: false moves to the universal base config (baseEsConfigBuilder): no suite exercises ML, and the ML native processes cost ~200-400MB RSS per container plus startup time. The key is supported across the whole matrix (6.3->9.x) unlike xpack.monitoring.enabled (removed in 8.0), so no version guard. XpackSecurityPlugin's explicit 'true' is removed (a re-add would duplicate the elasticsearch.yml key, which is fatal). k=5 probe verdict: host OOM on 5/6 heavier modules (same exit-137 VM death as k=6); only the lightest module survived. The RAM cliff is at k=5, so IT_PARALLELISM returns to the k=4 optimum (one notch of margin). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-mode matrix Optimization iteration 1 (measured baseline: es94x leg 24.7 min at k=4): - Shard packing: replace duration-blind hashCode-mod with LPT bin-packing weighted by committed suite-timings.json (measured wall seconds per suite INCLUDING container boots, from run 29167332908 sharded-logs; junit-XML time= excludes boots and would mis-balance). Measured hash split was 20.1/11.6/17.8/13.4 min; planned LPT split is 17.0 x4. Unknown suites default to 60s; packing is deterministic across shard JVMs. TEST-xml upload flipped to always() so green runs refresh data. - Child shard daemons get -Xmx1024m (were inheriting gradle.properties' -Xmx6144m: ~6GB x K reserved on a 16GB box — the actual ceiling behind the k=5/k=6 host-OOM deaths, more than Elasticsearch). - DEV MODE (temporary, restore before merge): it_linux matrix = es94x only, it_windows skipped — one cheap leg per optimization iteration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d sleeps in admin suite
- it_linux -> ubicloud-premium-4 (same 4vCPU/16GB shape, ~2x CPU at 1.6x
price). Iteration-1 rerun showed LPT packing works (all shards level at
~17min planned / 19-22 actual) but one suite died on a 100s HTTP read
timeout: level shards = sustained full utilization, so container-heavy
inits lost the CPU slack that lopsided packing accidentally provided.
More per-core speed is the structural fix; revert if gain < 20%.
- BaseAdminApiSuite: the two fixed 15s "wait for 2nd node reload" sleeps
become eventually{} polls on the second instance's expected state
(patienceConfig already bounds them at the old sleep duration).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… 19.3-22 min) at 1.6x price Tests are wait-bound (ES boots, network), not CPU-bound, so faster cores barely help. Balanced shards + all fixes stay. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…starts Balanced shards keep all 4 workers at full load the whole run; the 16GB standard box died 3/3 times since (2 host-OOM, 1 timeout) while premium passed. Two memory-pressure cuts to keep the cheap box: - workers reuse the ROR plugin zip the parent prebuild already built (ROR_REUSE_ASSEMBLED=1, opt-in via ParallelProcessRunner only, so a plain local test run still rebuilds fresh) — removes one nested gradle build per worker at the busiest moment - 30s stagger between worker starts de-synchronizes the heaviest multi-container suite boots (costs ~1.5min wall worst-case) If this still dies on standard-4: keep hash sharding on standard, or pay for premium. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-PbalancedShards Balanced shards host-OOMed the 16GB standard runner 4/4 times (stagger and zip-reuse did not save it) while the same box was stable for two full runs with hash packing: the hash split's idle gaps double as memory slack. Balanced packing stays available for 32GB+ machines. The 30s start stagger is removed with it. Kept: worker zip reuse, 1GB worker daemons, poll-instead-of-sleep — none implicated, all pure wins. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
toolchains_verify, required_checks, cve_check, unit_tests move to ubuntu-latest (free for public repos, same 4cpu/16GB shape). Heavy IT legs and builds stay on Ubicloud. Temporary it_linux_ghfree probe runs the same leg on the free runner to measure the speed gap head-to-head. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p answered free-Linux probe Free-vs-paid Linux verdict: the free GitHub 16GB box host-OOMs under the 4-worker IT load (exit 137) while Ubicloud passes in 21.3 min — IT legs stay paid, light jobs stay free. Probe job removed. Windows: ShardedGradlewTest is now Windows-capable (cmd.exe wrapper for gradlew.bat; ProcessHandle reaping is already cross-platform). The dev-mode Windows leg (es94x) runs shardedTest with 3 workers on the free GH box (baseline ~70-78 min serial), plus the same leg on blacksmith-4vcpu-windows-2025 as a paid probe (continue-on-error). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… shard isolation - HeavySuiteGate: machine-wide file-lock permits (default off; CI sets 2) cap how many multi-cluster suites boot containers concurrently across shard JVMs. Converts the balanced-packing OOM deaths into short waits; self-healing (a crashed worker's lock dies with it). One permit per SUITE, taken before any of its clusters start — no hold-and-wait, no deadlock. Wraps run() via SuiteMixin (containers start inside run()). - Balanced packing re-enabled via ROR_BALANCED_SHARDS env (Linux+Windows IT jobs), with ROR_HEAVY_SUITE_PERMITS=2. - Windows sharding round 2: launch worked (3 shards ran 10-32min) but everything name-keyed collided — ES ports (fixed per node name) and install dirs (es_<cluster>_<node>). Both now offset per shard via the ror.shard.index system property (1000-port windows, per-shard dirs). - Blacksmith probe removed: Blacksmith requires a GitHub organization; personal-account jobs queue forever. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…wnload cache The HeavySuiteGate decouples worker count from peak memory, unlocking the k=5 experiment that host-OOMed before the gate existed: 5 workers, max 3 concurrent heavy suites (below the ~4 that used to kill the box). Windows keeps 2 permits; its ES zip download cache is shared across shards again (the per-shard split made every shard re-download the distribution). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Windows (leg was 52.8 min; ~16.5 min setup + 36 min shards):
- ror-tools:test removed from per-leg runs (10 min/leg; every node
install already exercises the patcher; suite coverage moves to the
manual unit_tests_windows job)
- Windows Defender exclusions for workspace/temp/java.exe — real-time
scanning taxes every file write and our tests write hundreds of
thousands of files
- ES distribution unzipped ONCE per version into a lock-guarded shared
template, then robocopy /MT per node (was: single-threaded Java unzip
of ~600MB per node, through the AV filter)
Linux k=5 verdict so far: memory SOLVED by the gate (machine alive,
4/5 shards green 15.7-17.6 min, permit waits <=102s); the one failure
was a pre-existing race in IndexLifecycleManagementApiSuite — ILM
assigns the initial step asynchronously and the fixed-order move call
intermittently gets 400 under load. Wrapped the three step-moves in
eventually{}. k=5 stays for one more measured run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All optimizations banked: k=5 sharding + HeavySuiteGate(3) + balanced packing on Linux (es94x 20.6 min, was 24.7); Windows sharded k=3 with Defender exclusions, template-copy ES installs, self-test moved to unit_tests_windows (es94x 40.8 min, was ~70-78). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J7ZwM6WNAkeHKqSjxnqgPa
Full-matrix run 29198717346 showed k=5 host-OOMs 4/10 Linux modules (es80x/810x/818x/90x, exit 137) — they run suites es94x excludes, so their memory peaks exceed the es94x-tuned envelope. Back to k=4 with the HeavySuiteGate kept. Windows es818x exposed a fixed-port race: WireMock binds 8080/8081 on the shared host, so two shards starting wiremock-backed suites together collided. Same per-shard +k*1000 window as WindowsEsPortProvider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J7ZwM6WNAkeHKqSjxnqgPa
es90x still host-OOM'd at k=4/permits=3: three 3-node heavies + a single-node suite + four gradle JVMs graze 16GB, and balanced packing front-loads the heaviest suites so the peak hits at minute one. Two permits cap the worst case ~12GB. Also disable machinery no suite exercises, version-guarded per key: watcher (6.3->9.x), built-in stack templates + ILM policies (8.0+), apm-data templates (8.15+), deprecation indexing (7.16+), SLM history (7.5+). Cuts per-node boot work and cluster-state bloat, biggest on the 8.x/9.x legs that were OOMing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J7ZwM6WNAkeHKqSjxnqgPa
…arena CI packs 4-5 ES nodes on a 4-vCPU box; each node sizing its ~20 thread pools for 4 cores wastes ~1MB stack per surplus thread. node.processors (or 'processors' pre-7.4) tells each node it has 2. Test nodes never fill the default 240m JIT code cache either, and one Netty direct-memory arena covers test traffic. Applied to both docker and native-Windows paths; Windows shares esConfigFile so the yaml keys flow automatically. Deliberately NOT swapping the GC: ES pins G1/CMS per version+JDK in its own jvm.options and conflicting or removed GC flags are boot-fatal on several bundled JDKs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J7ZwM6WNAkeHKqSjxnqgPa
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J7ZwM6WNAkeHKqSjxnqgPa
Owner
Author
|
Experiment concluded — results recorded in PR #1292 discussion/session. Config verdict: k=4 + 2 heavy-suite permits is the optimum for 4-vCPU/16GB runners. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Isolated probe: 4 shards on the 4-vCPU Windows box vs current 3. Do not merge.
🤖 Generated with Claude Code
https://claude.ai/code/session_01J7ZwM6WNAkeHKqSjxnqgPa