Skip to content

Commit 91242c1

Browse files
c-warrenc-warren
authored
Add documentation to the simulation tests (#7271)
<!-- Describe what has changed in this PR --> **What changed?** Updated the documentation in the simulation folder. Updated each `run.sh` script to: - support named parameters, allowing users to only specify the fields they need - support a `dockerfile-suffix`, allowing users to run the matching/history sims locally - require a scenario file <!-- Tell your future self why have you made these changes --> **Why?** My hope for the documentation is that it will increase usage of the simulation folder by decreasing the barrier to running simulations. <!-- How have you verified this change? Tested locally? Added a unit test? Checked in staging env? --> **How did you test it?** Manually - needs to also be tested in CI/CD still. <!-- Assuming the worst case, what can be broken when deploying this change to production? --> **Potential risks** CI/CD could stop working if the simulations can't be run in Canary. <!-- Is it notable for release? e.g. schema updates, configuration or data migration required? If so, please mention it, and also update CHANGELOG.md --> **Release notes** N/A <!-- Is there any documentation updates should be made for config, https://cadenceworkflow.io/docs/operation-guide/setup/ ? If so, please open an PR in https://github.com/cadence-workflow/cadence-docs --> **Documentation Changes** Not sure - is it worth writing something about how to use simulations?
1 parent 70c5df0 commit 91242c1

19 files changed

+481
-70
lines changed

‎.github/workflows/replication-simulation.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ jobs:
4343
max_attempts: 2
4444
timeout_minutes: 20
4545
command: |
46-
./simulation/replication/run.sh ${{ matrix.scenario }}
46+
./simulation/replication/run.sh --scenario ${{ matrix.scenario }}
4747
4848
- name: Upload test logs
4949
uses: actions/upload-artifact@v4

‎simulation/README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Simulation Tests
2+
3+
Simulation tests are black box tests that validate complex multi-component scenarios against a Cadence cluster. They enable systematic testing of workflows that would otherwise require manual sanity checks.
4+
5+
## Overview
6+
7+
Each suite of simulation tests:
8+
- instantiates a local docker cluster
9+
- makes requests to the local cluster, based on the test configuration
10+
- analyses the results
11+
12+
### Test Types
13+
14+
- **replication** - Tests the edge-case behaviour of workflows replicated to multiple clusters, e.g during failover, replication lag, or in active-active domains.
15+
- **matching** - Tests the performance of the cadence-matching service.
16+
- **history** - Tests the behaviour of the history service.
17+
18+
See individual subdirectory READMEs for specific test details.
19+
20+
## Running Simulations
21+
22+
Running the tests requires that the developer has
23+
- [setup their development environment](../CONTRIBUTING.md#development-environment)
24+
- has docker running
25+
26+
### Quick Start
27+
28+
```bash
29+
# Run a basic replication test
30+
./simulation/replication/run.sh --scenario default
31+
32+
# Run a matching performance test
33+
./simulation/matching/run.sh --scenario throughput
34+
35+
# Run a history analysis test
36+
./simulation/history/run.sh --scenario default
37+
```
38+
39+
### Local Development
40+
41+
Some contributors may require a custom dockerfile to be passed to allow the simulation tests to build locally.
42+
Add your dockerfile to `docker/github_actions/` with a custom suffix, and then pass the `--dockerfile-suffix` paramater to use it:
43+
44+
```bash
45+
# For a file named Dockerfile.local in docker/github_actions:
46+
./simulation/replication/run.sh --scenario default --dockerfile-suffix .local
47+
```
48+
49+
## Adding New Simulations
50+
51+
Any time you are writing code that is complex, touches multiple components, or requires manual testing to sanity check its behaviour it is a candidate for a simulation test. To add a new simulation:
52+
53+
1. Create scenario file: `{type}/testdata/{type}_simulation_{name}.yaml`
54+
2. Add config overrides: `config/dynamicconfig/{type}_simulation_{name}.yml` (if needed)
55+
3. Test locally: `./simulation/{type}/run.sh --scenario {name}`
56+
4. Add to CI matrix in `.github/workflows/{type}-simulation.yml`
57+
58+
### CI/CD
59+
60+
Note: only replication simulations run in GitHub Actions currently. To add a new replication scenario to CI, update the matrix in `.github/workflows/replication-simulation.yml`.

‎simulation/history/run.sh‎

Lines changed: 110 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,126 @@
11
#!/bin/bash
22

3-
# This script can be used to run history simulator and check the critical flow via logs
3+
# Cadence History Simulation Test Script
44
#
5+
# Usage:
6+
# ./simulation/history/run.sh [OPTIONS]
7+
#
8+
# Examples:
9+
# # Run default scenario
10+
# ./simulation/history/run.sh --scenario default
11+
#
12+
# # Run specific scenario
13+
# ./simulation/history/run.sh --scenario queuev2
14+
#
15+
# # Run with custom timestamp
16+
# ./simulation/history/run.sh --scenario default --timestamp 2024-01-15-10-30-00
17+
#
18+
# # Run with custom dockerfile
19+
# ./simulation/history/run.sh --scenario queuev2 --dockerfile-suffix .local
20+
#
21+
# TODO: Add simulation/history/README.md.
522

623
set -eo pipefail
724

8-
testCase="${1:-default}"
25+
show_help() {
26+
cat << EOF
27+
Cadence History Simulation Test Script
28+
29+
USAGE:
30+
$0 [OPTIONS]
31+
32+
OPTIONS:
33+
-s, --scenario SCENARIO Test scenario to run (required)
34+
Corresponds to testdata/history_simulation_SCENARIO.yaml
35+
36+
-t, --timestamp TIMESTAMP Custom timestamp for test naming (default: current time)
37+
Format: YYYY-MM-DD-HH-MM-SS
38+
39+
-d, --dockerfile-suffix SUFFIX Dockerfile suffix for custom builds (default: empty)
40+
Example: .local for Dockerfile.local
41+
42+
-h, --help Show this help message
43+
44+
EXAMPLES:
45+
# Run default scenario
46+
$0 --scenario default
47+
48+
# Run specific scenario
49+
$0 --scenario queuev2
50+
51+
# Run with custom timestamp
52+
$0 --scenario default --timestamp 2024-01-15-10-30-00
53+
54+
# Run with custom dockerfile
55+
$0 --scenario queuev2 --dockerfile-suffix .local
56+
57+
EOF
58+
}
59+
60+
# Default values
61+
testCase=""
62+
timestamp=""
63+
dockerFileSuffix=""
64+
65+
# Parse command line arguments
66+
while [[ $# -gt 0 ]]; do
67+
case $1 in
68+
-s|--scenario)
69+
testCase="$2"
70+
shift 2
71+
;;
72+
-t|--timestamp)
73+
timestamp="$2"
74+
shift 2
75+
;;
76+
-d|--dockerfile-suffix)
77+
dockerFileSuffix="$2"
78+
shift 2
79+
;;
80+
-h|--help)
81+
show_help
82+
exit 0
83+
;;
84+
--)
85+
shift
86+
break
87+
;;
88+
-*)
89+
echo "Unknown option: $1" >&2
90+
echo "Use --help for usage information." >&2
91+
exit 1
92+
;;
93+
*)
94+
echo "Unexpected positional argument: $1" >&2
95+
echo "Use --scenario to specify the test scenario." >&2
96+
echo "Use --help for usage information." >&2
97+
exit 1
98+
;;
99+
esac
100+
done
101+
102+
# Require scenario parameter
103+
if [[ -z "$testCase" ]]; then
104+
echo "Error: --scenario parameter is required" >&2
105+
echo "" >&2
106+
show_help
107+
exit 1
108+
fi
109+
110+
# Set default timestamp if not provided
111+
if [[ -z "$timestamp" ]]; then
112+
timestamp="$(date '+%Y-%m-%d-%H-%M-%S')"
113+
fi
114+
9115
testCfg="testdata/history_simulation_$testCase.yaml"
10-
now="$(date '+%Y-%m-%d-%H-%M-%S')"
11-
timestamp="${2:-$now}"
12116
testName="test-$testCase-$timestamp"
13117
resultFolder="history-simulator-output"
14118
mkdir -p "$resultFolder"
15119
eventLogsFile="$resultFolder/$testName-events.json"
16120
testSummaryFile="$resultFolder/$testName-summary.txt"
17121

18122
echo "Building test image"
19-
DOCKERFILE_SUFFIX=$DOCKERFILE_SUFFIX docker compose -f docker/github_actions/docker-compose-local-history-simulation.yml \
123+
DOCKERFILE_SUFFIX=$dockerFileSuffix docker compose -f docker/github_actions/docker-compose-local-history-simulation.yml \
20124
build history-simulator
21125

22126
function check_test_failure()
@@ -37,7 +141,7 @@ function check_test_failure()
37141
trap check_test_failure EXIT
38142

39143
echo "Running the test $testCase"
40-
DOCKERFILE_SUFFIX=$DOCKERFILE_SUFFIX docker compose \
144+
DOCKERFILE_SUFFIX=$dockerFileSuffix docker compose \
41145
-f docker/github_actions/docker-compose-local-history-simulation.yml \
42146
run -e HISTORY_SIMULATION_CONFIG=$testCfg --rm --remove-orphans --service-ports --use-aliases \
43147
history-simulator \

‎simulation/matching/comparison/main.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,7 @@ func mustRunScenario(root, scenario, ts string) {
233233

234234
fmt.Printf("Running scenario: %s\n", scenario)
235235
start := time.Now()
236-
cmd := exec.Command("bash", path.Join(root, "simulation/matching/run.sh"), scenario, ts)
236+
cmd := exec.Command("bash", path.Join(root, "simulation/matching/run.sh"), "--scenario", scenario, "--timestamp", ts)
237237
var stdout, stderr bytes.Buffer
238238
cmd.Stdout = &stdout
239239
cmd.Stderr = &stderr

‎simulation/matching/run.sh‎

Lines changed: 110 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,126 @@
11
#!/bin/bash
22

3-
# This script can be used to run matching simulator and check the critical flow via logs
3+
# Cadence Matching Simulation Test Script
44
#
5+
# Usage:
6+
# ./simulation/matching/run.sh [OPTIONS]
7+
#
8+
# Examples:
9+
# # Run default scenario
10+
# ./simulation/matching/run.sh --scenario default
11+
#
12+
# # Run specific scenario
13+
# ./simulation/matching/run.sh --scenario throughput
14+
#
15+
# # Run with custom timestamp
16+
# ./simulation/matching/run.sh --scenario default --timestamp 2024-01-15-10-30-00
17+
#
18+
# # Run with custom dockerfile
19+
# ./simulation/matching/run.sh --scenario throughput --dockerfile-suffix .local
20+
#
21+
# TODO: Add README with more information on how to analyse the test results, as well as more detailed information on the comparison tests.
522

623
set -eo pipefail
724

8-
testCase="${1:-default}"
25+
show_help() {
26+
cat << EOF
27+
Cadence Matching Simulation Test Script
28+
29+
USAGE:
30+
$0 [OPTIONS]
31+
32+
OPTIONS:
33+
-s, --scenario SCENARIO Test scenario to run (required)
34+
Corresponds to testdata/matching_simulation_SCENARIO.yaml
35+
36+
-t, --timestamp TIMESTAMP Custom timestamp for test naming (default: current time)
37+
Format: YYYY-MM-DD-HH-MM-SS
38+
39+
-d, --dockerfile-suffix SUFFIX Dockerfile suffix for custom builds (default: empty)
40+
Example: .local for Dockerfile.local
41+
42+
-h, --help Show this help message
43+
44+
EXAMPLES:
45+
# Run default scenario
46+
$0 --scenario default
47+
48+
# Run specific scenario
49+
$0 --scenario throughput
50+
51+
# Run with custom timestamp
52+
$0 --scenario default --timestamp 2024-01-15-10-30-00
53+
54+
# Run with custom dockerfile
55+
$0 --scenario throughput --dockerfile-suffix .local
56+
57+
EOF
58+
}
59+
60+
# Default values
61+
testCase=""
62+
timestamp=""
63+
dockerFileSuffix=""
64+
65+
# Parse command line arguments
66+
while [[ $# -gt 0 ]]; do
67+
case $1 in
68+
-s|--scenario)
69+
testCase="$2"
70+
shift 2
71+
;;
72+
-t|--timestamp)
73+
timestamp="$2"
74+
shift 2
75+
;;
76+
-d|--dockerfile-suffix)
77+
dockerFileSuffix="$2"
78+
shift 2
79+
;;
80+
-h|--help)
81+
show_help
82+
exit 0
83+
;;
84+
--)
85+
shift
86+
break
87+
;;
88+
-*)
89+
echo "Unknown option: $1" >&2
90+
echo "Use --help for usage information." >&2
91+
exit 1
92+
;;
93+
*)
94+
echo "Unexpected positional argument: $1" >&2
95+
echo "Use --scenario to specify the test scenario." >&2
96+
echo "Use --help for usage information." >&2
97+
exit 1
98+
;;
99+
esac
100+
done
101+
102+
# Require scenario parameter
103+
if [[ -z "$testCase" ]]; then
104+
echo "Error: --scenario parameter is required" >&2
105+
echo "" >&2
106+
show_help
107+
exit 1
108+
fi
109+
110+
# Set default timestamp if not provided
111+
if [[ -z "$timestamp" ]]; then
112+
timestamp="$(date '+%Y-%m-%d-%H-%M-%S')"
113+
fi
114+
9115
testCfg="testdata/matching_simulation_$testCase.yaml"
10-
now="$(date '+%Y-%m-%d-%H-%M-%S')"
11-
timestamp="${2:-$now}"
12116
testName="test-$testCase-$timestamp"
13117
resultFolder="matching-simulator-output"
14118
mkdir -p "$resultFolder"
15119
eventLogsFile="$resultFolder/$testName-events.json"
16120
testSummaryFile="$resultFolder/$testName-summary.txt"
17121

18122
echo "Building test image"
19-
docker compose -f docker/github_actions/docker-compose-local-matching-simulation.yml \
123+
DOCKERFILE_SUFFIX=$dockerFileSuffix docker compose -f docker/github_actions/docker-compose-local-matching-simulation.yml \
20124
build matching-simulator
21125

22126
function check_test_failure()
@@ -37,7 +141,7 @@ function check_test_failure()
37141
trap check_test_failure EXIT
38142

39143
echo "Running the test $testCase"
40-
docker compose \
144+
DOCKERFILE_SUFFIX=$dockerFileSuffix docker compose \
41145
-f docker/github_actions/docker-compose-local-matching-simulation.yml \
42146
run -e MATCHING_SIMULATION_CONFIG=$testCfg --rm --remove-orphans --service-ports --use-aliases \
43147
matching-simulator \

0 commit comments

Comments
 (0)