Skip to content

feat(apply): add --concurrency limit for deploy/destroy#504

Open
sam-goodwin wants to merge 1 commit into
mainfrom
claude/parallelism-deploy-destroy-EYZPn
Open

feat(apply): add --concurrency limit for deploy/destroy#504
sam-goodwin wants to merge 1 commit into
mainfrom
claude/parallelism-deploy-destroy-EYZPn

Conversation

@sam-goodwin

@sam-goodwin sam-goodwin commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Cap how many resources apply at once. Defaults to unbounded.

# no more than 64 resources mid-lifecycle at any time
alchemy deploy --concurrency 64
alchemy destroy --concurrency 64

# omit the flag → unbounded (every ready resource applies at once)
alchemy deploy

Programmatic:

yield* apply(plan, { concurrency: 64 });   // bounded
yield* apply(plan);                         // { concurrency: "unbounded" } (default)

The limit is a shared semaphore taken only around the provider call, never around the dependency wait — so a resource blocked on its upstreams holds no permit and a tight limit (even --concurrency 1) can't deadlock the DAG.

Bound how many resources run a provider lifecycle operation at once.
Defaults to unbounded; defers to Effect's concurrency model via a shared
semaphore that gates only the actual provider call, never the dependency
wait, so a tight limit can never deadlock the DAG.

- `apply(plan, { concurrency })` accepts `"unbounded" | number` (default
  `"unbounded"`). The limit is enforced at `instrumentLifecycle`, the single
  dispatch chokepoint, so it covers create/update/replace/precreate/delete
  across executePlan, converge, and collectGarbage with one shared permit.
- `--concurrency <n>` flag on `deploy` and `destroy` (e.g. `--concurrency 64`).
- Test harness `deploy`/`destroy` accept `ApplyOptions` so the limit is
  testable.
- apply.test.ts: unbounded runs all at once; a limit caps the high-water
  mark; concurrency 1 serializes; a dependency chain doesn't deadlock under a
  tight limit; destroy honors the limit.
@alchemy-version-bot

Copy link
Copy Markdown
Contributor

Install the packages built from this commit:

alchemy

bun add alchemy@https://pkg.ing/alchemy/a85fd1e

@alchemy.run/better-auth

bun add @alchemy.run/better-auth@https://pkg.ing/@alchemy.run/better-auth/a85fd1e

@alchemy.run/pr-package

bun add @alchemy.run/pr-package@https://pkg.ing/@alchemy.run/pr-package/a85fd1e

@alchemy-version-bot

Copy link
Copy Markdown
Contributor

Website Preview Deployed

URL: https://alchemyeffectwebsite-worker-pr-504-67p2sklcbinzz3cl.testing-2b2.workers.dev

Built from commit a85fd1e.


This comment updates automatically with each push.

@sam-goodwin

Copy link
Copy Markdown
Contributor Author

@claude please merge latest main into this PR, fix any merge conflicts and push to this pr

@claude

claude Bot commented Jul 7, 2026

Copy link
Copy Markdown

Claude finished @sam-goodwin's task in 26m 37s —— View job


Unable to push the merge from this runner — but here is everything needed to complete it, including the exact conflict resolutions.

  • Fetch latest origin/main and analyze divergence
  • Identify merge conflicts (2 textual + 1 semantic)
  • Work out exact resolutions (below)
  • Create merge commit and push — blocked by the Action's tool allowlist (details below)

Why I couldn't push the merge

This workflow runs claude-code-action@v1 with the default tool allowlist and fetch-depth: 1. The allowlist permits git add/commit/rm/status/diff/log/show and a hardened git push origin <ref> wrapper — but not git merge, fetch, pull, checkout, reset, restore, read-tree, apply, clone, worktree, or any scripting runtime, and all writes under .git/ (e.g. MERGE_HEAD) are blocked as sensitive files. I exhausted every composition of the allowed primitives; none can produce a two-parent merge commit, and main has moved 2,452 files since this branch was cut, so a tree reconstruction by hand isn't viable either. Nothing was changed or pushed; the branch is untouched.

To let @claude do this next time, .github/workflows/claude.yml needs (I can't edit workflows myself — GitHub App permissions):

- uses: actions/checkout@v4
  with:
    fetch-depth: 0          # currently 1 — merge-base with main is unreachable
- uses: anthropics/claude-code-action@v1
  with:
    anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
    claude_args: |
      --model claude-fable-5
      --allowedTools Bash(git fetch:*),Bash(git merge:*),Bash(git checkout:*)

Merge analysis

I compared both sides against the branch point f1b6620. Three of the five PR files merge cleanly (Apply.ts textually, _shared.ts, apply.test.ts). Two conflict, and one clean merge hides a compile error:

1. packages/alchemy/src/Cli/commands/deploy.ts — textual conflict. Both sides rewrote const outputs = yield* apply(updatePlan); — this PR threads { concurrency } through, main wrapped it in the dev keep-alive catchCause (#455 follow-up). Resolution combining both:

const applyPlan = dev
  ? apply(updatePlan, { concurrency: concurrency ?? "unbounded" }).pipe(
      Effect.catchCause((cause) =>
        Cause.hasInterruptsOnly(cause)
          ? Effect.failCause(cause)
          : Console.error(
              `alchemy dev: apply failed; keeping dev alive so healthy resources keep serving.\n${Cause.pretty(cause)}`,
            ).pipe(Effect.as(undefined)),
      ),
    )
  : apply(updatePlan, { concurrency: concurrency ?? "unbounded" });
const outputs = yield* applyPlan;

2. packages/alchemy/src/Test/Core.ts — textual conflict. Main inserted the new plan() method exactly where this PR changed the deploy/destroy signatures (both in the ScratchStack interface and the scratchStack return object). Resolution — keep all three members:

return {
  name: stackName,
  state: stateLayer,
  deploy: ((
    effect: Effect.Effect<any, any, any>,
    applyOptions?: ApplyOptions,
  ) => buildAndApply(effect, applyOptions)) as ScratchStack<ROut>["deploy"],
  plan: ((effect: Effect.Effect<any, any, any>) =>
    buildPlan(effect)) as ScratchStack<ROut>["plan"],
  destroy: (applyOptions?: ApplyOptions) =>
    ...

(and in the interface, deploy gains options?: ApplyOptions, plan stays as on main, destroy becomes destroy(options?: ApplyOptions).)

3. packages/alchemy/src/Apply.ts — semantic conflict (merges clean, then fails tsc). Main's new delete-first replacement path added an instrumentLifecycle(...) call inside deleteOldGenerations using the old 5-arg signature; this PR made limit the first parameter. After merging, add limit:

.pipe(
  instrumentLifecycle(
    limit,          // ← add
    "delete",
    fqn,
    node.resource.Type,
    logicalId,
    old.instanceId,
  ),
);

To finish locally

git fetch origin
git checkout claude/parallelism-deploy-destroy-EYZPn
git merge origin/main   # conflicts in deploy.ts + Test/Core.ts → resolve as above
# then apply fix 3 in Apply.ts (it will NOT show up as a conflict)
bun tsc -b
git commit --no-edit && git push

One thing to watch: main bumped effect from >=4.0.0-beta.74 to >=4.0.0-beta.93 — if Semaphore.makeUnsafe shifted in that range, bun tsc -b will surface it in Apply.ts.

Fix this →

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants