Skip to content

Fix AWS account create retry for transient service activation errors - #144

Merged
bshutterkion merged 2 commits into
mainfrom
fix/aws-account-create-retry-transient-errors
Mar 30, 2026
Merged

Fix AWS account create retry for transient service activation errors#144
bshutterkion merged 2 commits into
mainfrom
fix/aws-account-create-retry-transient-errors

Conversation

@bshutterkion

@bshutterkion bshutterkion commented Mar 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Newly created AWS accounts may not have all services (e.g. CloudFormation) fully activated when the provider attempts to move them from the account cache to a project
  • Account creation now has two wait phases:
    1. Poll /v3/account-cache/{id} until the account number is assigned
    2. Poll /v3/account-cache/{id}/status until the account is actually accessible in AWS (verifies the service role can be assumed and APIs called)
  • The cache-to-project conversion now retries all errors until the Terraform create timeout (default 20 min) expires, rather than failing after 3 fixed attempts
  • Added AccountCacheStatusResponse model for the status endpoint
  • Version bumped to 0.3.33

Context

A customer creating a new AWS account and project simultaneously via Terraform was hitting:

OptInRequired: The AWS Access Key Id needs a subscription for the service

The account was created successfully in the cache, but the move to the project failed because CloudFormation wasn't yet available in the brand-new account. Manual moves from the cache worked fine since enough time had passed.

The root cause is that AWS assigns an account number before all services are fully activated. The previous code only waited for the account number, then immediately attempted the convert. Now we also verify the account is accessible via the existing /v3/account-cache/{id}/status endpoint before proceeding.

Test plan

  • Verify build, vet, staticcheck, and unit tests pass
  • Test with customer environment where OptInRequired error was occurring
  • Verify accounts that don't hit activation delays still create and move without unnecessary wait
  • Verify the status polling respects the Terraform create timeout

Newly created AWS accounts may not have all services (e.g. CloudFormation)
fully activated when the provider attempts to move them from the cache to
a project. The convert call now retries on transient AWS errors including
OptInRequired, SubscriptionRequiredException, InvalidClientTokenId,
ServiceUnavailable, ThrottlingException, and InternalFailure. Retry
attempts increased from 3 to 6 (~3 min total) to accommodate propagation
delays.
Account creation now has two phases: first wait for the account number
to be assigned, then poll /v3/account-cache/{id}/status to verify the
account is actually accessible in AWS before attempting the move.  This
prevents failures caused by AWS service activation delays (e.g.
CloudFormation not yet available) on newly created accounts.

The cache-to-project conversion also now retries all errors until the
Terraform create timeout expires rather than failing after 3 fixed
attempts.
@bshutterkion
bshutterkion merged commit b76c0f0 into main Mar 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants