-
Notifications
You must be signed in to change notification settings - Fork 119
Aws step functions #87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
projeffpalmer
wants to merge
13
commits into
kirodotdev:main
Choose a base branch
from
projeffpalmer:aws-step-functions
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 11 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
910ded9
First commit for aws-step-functions-jsonata
projeffpalmer 4806582
Added steering files for converting from JSONPath to JSONata and vali…
projeffpalmer a613549
Merge branch 'kirodotdev:main' into aws-step-functions
projeffpalmer 5bb175e
Update aws-step-functions/POWER.md
projeffpalmer 3fb8bbb
Update aws-step-functions/POWER.md
projeffpalmer c858ab9
Update aws-step-functions/POWER.md
projeffpalmer 80e70a5
Merge branch 'kirodotdev:main' into aws-step-functions
projeffpalmer 54a99ec
Removed old folder
projeffpalmer ac87772
Merge branch 'kirodotdev:main' into aws-step-functions
projeffpalmer 609f5fa
Made changes based on peer feedback
projeffpalmer 17aa9a6
Merge branch 'kirodotdev:main' into aws-step-functions
projeffpalmer 4dfe3e9
Added more map state details
projeffpalmer 8aaaa14
Added more map state details
projeffpalmer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| --- | ||
| name: "aws-step-functions" | ||
| displayName: "AWS Step Functions" | ||
| description: "Build workflows with AWS Step Functions state machines using the JSONata query language. Covers Amazon States Language (ASL) structure, state types, variables, data transformation, error handling, AWS service integration, and migrating from the JSONPath to the JSONata query language." | ||
| keywords: ["step functions", "state machine", "serverless", "jsonata", "asl", "amazon states language", "workflow", "orchestration"] | ||
| author: "AWS" | ||
| --- | ||
|
|
||
| # AWS Step Functions | ||
|
|
||
| ## Overview | ||
|
|
||
| AWS Step Functions uses Amazon States Language (ASL) to define state machines as JSON. With AWS Step Functions, you can create workflows, also called State machines, to build distributed applications, automate processes, orchestrate microservices, and create data and machine learning pipelines. | ||
|
|
||
| This power provides comprehensive guidance for writing state machines in ASL, covering: | ||
| - ASL structure and JSONata expression syntax | ||
| - Details on the eight available workflow states | ||
| - The `$states` reserved variable | ||
| - Workflow variables with `Assign` | ||
| - Error handling | ||
| - AWS Service integration patterns | ||
| - Data transformation and architecture examples | ||
| - Validation and testing of state machines | ||
| - How to migrate from JSONPath to JSONata | ||
|
|
||
| ## When to Load Steering Files | ||
|
|
||
| Load the appropriate steering file based on what the user is working on: | ||
projeffpalmer marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - **ASL structure**, **state types**, **Task**, **Pass**, **Choice**, **Wait**, **Succeed**, **Fail**, **Parallel**, **Map** → see `asl-state-types.md` | ||
| - **Variables**, **Assign**, **data passing**, **scope**, **$states**, **input**, **output**, **Arguments**, **Output**, **data transformation**, **QueryEvaluationError** → see `variables-and-data.md` | ||
| - **Error handling**, **troubleshooting**, **Retry**, **Catch**, **fallback**, **error codes**, **States.Timeout**, **States.ALL** → see `error-handling.md` | ||
| - **Service integrations**, **Lambda invoke**, **DynamoDB**, **SNS**, **SQS**, **SDK integrations**, **Resource ARN**, **sync**, **async** → see `service-integrations.md` | ||
| - **Migrating from JSONPath to JSONata**, **migration**, **JSONPath to JSONata**, **InputPath**, **Parameters**, **ResultSelector**, **ResultPath**, **OutputPath**, **intrinsic functions**, **Iterator**, **payload template** → see `migrating-from-jsonpath-to-jsonata.md` | ||
| - **Validation**, **linting**, **testing**, **TestState**, **test state**, **mock**, **mocking**, **unit test**, **inspection level**, **DEBUG**, **TRACE**, **validate state**, **test in isolation** → see `validation-and-testing.md` | ||
|
|
||
| ## Quick Reference | ||
|
|
||
| ### Standard vs Express Workflows | ||
|
|
||
| | | Standard | Express | | ||
| | --------------------------------- | ------------------------------------ | ------------------------------------------- | | ||
| | **Max duration** | 1 year | 5 minutes | | ||
| | **Execution semantics** | Exactly-once | At-least-once (async) / At-most-once (sync) | | ||
| | **Execution history** | Retained 90 days, queryable via API | CloudWatch Logs only | | ||
| | **Max throughput** | 2,000 exec/sec | 100,000 exec/sec | | ||
| | **Pricing model** | Per state transition | Per execution count + duration | | ||
| | **`.sync` / `.waitForTaskToken`** | Supported | Not supported | | ||
| | **Best for** | Auditable, non-idempotent operations | High-volume, idempotent event processing | | ||
|
|
||
| **Choose Standard** for: payment processing, order fulfillment, compliance workflows, anything that must never execute twice. | ||
|
|
||
| **Choose Express** for: IoT data ingestion, streaming transformations, mobile backends, high-throughput short-lived processing. | ||
|
|
||
| ### Key State Types | ||
|
|
||
| | State | Purpose | | ||
| | ------------------ | ------------------------------------------------------------------------------------ | | ||
| | `Task` | Execute work — invoke Lambda, call any AWS service via SDK integration | | ||
| | `Choice` | Branch based on input data conditions (no `Next` required on branches) | | ||
| | `Parallel` | Execute multiple branches concurrently; waits for all branches to complete | | ||
| | `Map` | Iterate over an array; use Distributed Map mode for up to 10M items from S3/DynamoDB | | ||
| | `Wait` | Pause for a fixed duration or until a specific timestamp | | ||
| | `Pass` | Pass input to output, optionally injecting or transforming data | | ||
| | `Succeed` / `Fail` | End execution successfully or with an error and cause | | ||
projeffpalmer marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ### Setting the State Machine Query Language | ||
projeffpalmer marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| JSONata is the modern, preferred way to reference and transform data in ASL. It replaces the five JSONPath I/O fields (`InputPath`, `Parameters`, `ResultSelector`, `ResultPath`, `OutputPath`) with just two: `Arguments` (inputs) and `Output`. | ||
|
|
||
| **Enable at the top level** to apply to all states: | ||
|
|
||
| ```json | ||
| { "QueryLanguage": "JSONata", "StartAt": "...", "States": {...} } | ||
| ``` | ||
|
|
||
| **Or per-state** to migrate from JSONPath incrementally: | ||
|
|
||
| ```json | ||
| { "Type": "Task", "QueryLanguage": "JSONata", ... } | ||
| ``` | ||
|
|
||
| **JSONPath is still supported** and is the default if `QueryLanguage` is omitted — existing state machines do not need to be migrated. | ||
|
|
||
| ### The `$states` Reserved Variable (JSONata only) | ||
projeffpalmer marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ``` | ||
| $states.input → Original state input | ||
| $states.result → Task/Parallel/Map result (on success) | ||
| $states.errorOutput → Error output (only in Catch) | ||
| $states.context → Execution context object | ||
| ``` | ||
|
|
||
| ## Best Practices | ||
|
|
||
| - Set `"QueryLanguage": "JSONata"` at the top level for new state machines unless JSONPath is mandatory | ||
projeffpalmer marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - Keep `Output` minimal — only include what the state immediately after the current state needs | ||
| - Use `Assign` to store variables needed in later states instead of threading it through Output | ||
| - Use `$states.input` to reference original state input | ||
| - Remember: `Assign` and `Output` are evaluated in parallel — variable assignments in `Assign` are NOT available in `Output` of the same state | ||
| - All JSONata expressions must produce a defined value — `$data.nonExistentField` throws `States.QueryEvaluationError` | ||
| - Use `$states.context.Execution.Input` to access the original workflow input from any state | ||
| - Save state machine definitions with `.asl.json` extension when working outside the console | ||
| - Prefer the optimized Lambda integration (`arn:aws:states:::lambda:invoke`) over the SDK integration | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Common Errors | ||
|
|
||
| - `States.QueryEvaluationError` — JSONata expression failed. Check for type errors, undefined fields, or out-of-range values. | ||
| - Mixing JSONPath fields with JSONata fields in the same state. | ||
| - Using `$` or `$$` at the top level of a JSONata expression — use `$states.input` instead. | ||
| - Forgetting `{% %}` delimiters around JSONata expressions — the string will be treated as a literal. | ||
| - Assigning variables in `Assign` and expecting them in `Output` of the same state — new values only take effect in the next state. | ||
| - Reference validation-and-testing.md and error-handling.md for detailed troubleshooting information. | ||
|
|
||
| ## Resources | ||
|
|
||
| - [ASL Specification](https://states-language.net/spec.html) | ||
| - [Transforming data with JSONata in Step Functions](https://docs.aws.amazon.com/step-functions/latest/dg/transforming-data.html) | ||
| - [Passing data between states with variables](https://docs.aws.amazon.com/step-functions/latest/dg/workflow-variables.html) | ||
| - [JSONata documentation](https://docs.jsonata.org/overview.html) | ||
| - [Step Functions Developer Guide](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.