Skip to content

feat(ci): implement change detection in CI workflow to optimize job e… #1306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ytallo wants to merge 1 commit into from
Closed

feat(ci): implement change detection in CI workflow to optimize job e… #1306

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 111 additions & 15 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,86 @@ env:
III_TELEMETRY_ENABLED: "false"

jobs:
# ──────────────────────────────────────────────────────────────
# Change Detection — skip jobs when their paths are untouched
# ──────────────────────────────────────────────────────────────

changes:
name: Detect Changes
runs-on: ubuntu-latest
permissions:
pull-requests: read
outputs:
engine: ${{ steps.filter.outputs.engine || 'true' }}
sdk_node: ${{ steps.filter.outputs.sdk_node || 'true' }}
sdk_python: ${{ steps.filter.outputs.sdk_python || 'true' }}
sdk_rust: ${{ steps.filter.outputs.sdk_rust || 'true' }}
motia_js: ${{ steps.filter.outputs.motia_js || 'true' }}
motia_py: ${{ steps.filter.outputs.motia_py || 'true' }}
console: ${{ steps.filter.outputs.console || 'true' }}
needs_engine: ${{ steps.filter.outputs.needs_engine || 'true' }}
steps:
- uses: actions/checkout@v4

- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
engine:
- 'engine/**'
- 'cli/**'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/ci.yml'
sdk_node:
- 'sdk/packages/node/**'
- 'sdk/fixtures/**'
- 'scripts/**'
- 'pnpm-lock.yaml'
- '.github/workflows/ci.yml'
sdk_python:
- 'sdk/packages/python/**'
- 'sdk/fixtures/**'
- 'scripts/**'
- '.github/workflows/ci.yml'
sdk_rust:
- 'sdk/packages/rust/**'
- 'sdk/fixtures/**'
- 'scripts/**'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/ci.yml'
Comment on lines +50 to +67
Copy link
Copy Markdown
Contributor

@sergiofilhowz sergiofilhowz Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's include engine in these, we want to run the tests in all sdk even if we change 1 line in engine

motia_js:
- 'frameworks/motia/motia-js/**'
- 'sdk/packages/node/**'
- 'pnpm-lock.yaml'
- '.github/workflows/ci.yml'
motia_py:
- 'frameworks/motia/motia-py/**'
- 'sdk/packages/python/**'
- '.github/workflows/ci.yml'
Comment on lines +68 to +76
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Include scripts/** in Motia filters to avoid false skips.

motia-js-ci and motia-py-ci both depend on scripts/start-iii.sh / scripts/stop-iii.sh, but motia_js and motia_py filters (Lines 68-76) don’t watch scripts/**. A PR that only changes scripts can skip Motia tests incorrectly.

Proposed patch 
             motia_js:
               - 'frameworks/motia/motia-js/**'
               - 'sdk/packages/node/**'
+              - 'scripts/**'
               - 'pnpm-lock.yaml'
               - '.github/workflows/ci.yml'
             motia_py:
               - 'frameworks/motia/motia-py/**'
               - 'sdk/packages/python/**'
+              - 'scripts/**'
               - '.github/workflows/ci.yml'

Also applies to: 425-439, 484-497

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/ci.yml around lines 68 - 76, The CI path filters for the
Motia jobs (motia_js and motia_py) currently list directories like
'frameworks/motia/motia-js/**', 'sdk/packages/node/**', and
'.github/workflows/ci.yml' but omit 'scripts/**', so changes to
scripts/start-iii.sh or scripts/stop-iii.sh can incorrectly skip motia-js-ci and
motia-py-ci; update the motia_js and motia_py path filter entries to include
'scripts/**' (and apply the same addition to the other similar motia blocks
referenced in the comment) so the Motia jobs run when script files change.

console:
- 'console/**'
- 'pnpm-lock.yaml'
- '.github/workflows/ci.yml'
Comment on lines +77 to +80
Copy link
Copy Markdown
Contributor

@sergiofilhowz sergiofilhowz Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's include

  • sdk_rust
  • engine

needs_engine:
- 'engine/**'
- 'cli/**'
- 'sdk/**'
- 'frameworks/motia/**'
- 'scripts/**'
- 'Cargo.toml'
- 'Cargo.lock'
- '.github/workflows/ci.yml'
Comment on lines +81 to +89
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Mar 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add pnpm-lock.yaml to needs_engine to avoid dependency-induced skips.

needs_engine at Line [81]-Line [89] omits pnpm-lock.yaml, but sdk_node/motia_js include it (Line [54], Line [71]) and both jobs require build-engine (Line [217], Line [391]). A lockfile-only PR can satisfy SDK/Motia filters while build-engine is skipped, causing downstream jobs to skip.

Proposed patch 
             needs_engine:
               - 'engine/**'
               - 'cli/**'
               - 'sdk/**'
               - 'frameworks/motia/**'
               - 'scripts/**'
+              - 'pnpm-lock.yaml'
               - 'Cargo.toml'
               - 'Cargo.lock'
               - '.github/workflows/ci.yml'
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/ci.yml around lines 81 - 89, The needs_engine job filter
is missing pnpm-lock.yaml which allows lockfile-only PRs to match
sdk_node/motia_js and skip build-engine; update the needs_engine entries (the
needs_engine block) to include "pnpm-lock.yaml" so changes to the lockfile
trigger build-engine as intended; verify related jobs build-engine, sdk_node,
and motia_js still reference the same filters after adding pnpm-lock.yaml.


# ──────────────────────────────────────────────────────────────
# Engine (test + build + upload artifact for downstream jobs)
# ──────────────────────────────────────────────────────────────

build-engine:
name: Build Engine
needs: changes
if: github.event_name != 'pull_request' || needs.changes.outputs.needs_engine == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -31,9 +105,14 @@ jobs:
components: rustfmt, clippy, llvm-tools-preview

- uses: Swatinem/rust-cache@v2
with:
save-always: true
Comment on lines +108 to +109
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Mar 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
curl -fsSL https://raw.githubusercontent.com/Swatinem/rust-cache/v2/action.yml \
  | sed -n '/^inputs:/,/^runs:/p'

Repository: iii-hq/iii

Length of output: 2962

🏁 Script executed:

cat -n .github/workflows/ci.yml | sed -n '105,115p; 156,165p; 193,205p; 352,360p; 529,540p'

Repository: iii-hq/iii

Length of output: 1676

Replace invalid rust-cache input save-always with save-if.

save-always is not a valid input for Swatinem/rust-cache@v2. The correct input is save-if, which has the same intended behavior.

Proposed patch 
       - uses: Swatinem/rust-cache@v2
         with:
-          save-always: true
+          save-if: true

Apply to all 5 occurrences: lines 109, 160, 198, 356, 533.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
with:
save-always: true
with:
save-if: true
🧰 Tools
🪛 actionlint (1.7.11)

[error] 109-109: input "save-always" is not defined in action "Swatinem/rust-cache@v2". available inputs are "add-job-id-key", "add-rust-environment-hash-key", "cache-all-crates", "cache-bin", "cache-directories", "cache-on-failure", "cache-provider", "cache-targets", "cache-workspace-crates", "env-vars", "key", "lookup-only", "prefix-key", "save-if", "shared-key", "workspaces"

(action)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/ci.yml around lines 108 - 109, The CI workflow uses the
Swatinem/rust-cache@v2 action with an invalid input key `save-always`; update
each occurrence to the correct input name `save-if` so the action functions as
intended (replace `save-always: true` with `save-if: true` in the
Swatinem/rust-cache@v2 steps referenced in the diff).


- uses: taiki-e/install-action@cargo-llvm-cov

- name: Check formatting
run: cargo fmt --all -- --check

- name: Build and run coverage
run: |
eval "$(cargo llvm-cov show-env --export-prefix)"
Expand All @@ -43,24 +122,21 @@ jobs:
--ignore-filename-regex \
"engine/src/main\.rs|(cron|pubsub|queue|state|stream)/adapters/(redis_adapter|bridge)\.rs|queue/adapters/rabbitmq/|^sdk/"

- name: Check formatting
run: cargo fmt --all -- --check

# - name: Run clippy
# run: cargo clippy -p iii -p function-macros -p iii-sdk --all-targets --all-features -- -D warnings

- uses: actions/upload-artifact@v4
with:
name: iii-binary
path: target/debug/iii
retention-days: 1
compression-level: 0

# ──────────────────────────────────────────────────────────────
# Engine RabbitMQ Integration Tests
# ──────────────────────────────────────────────────────────────

engine-rabbitmq-tests:
name: Engine RabbitMQ Integration Tests
needs: changes
if: github.event_name != 'pull_request' || needs.changes.outputs.engine == 'true'
runs-on: ubuntu-latest
services:
rabbitmq:
Expand All @@ -79,15 +155,23 @@ jobs:
- uses: dtolnay/rust-toolchain@stable

- uses: Swatinem/rust-cache@v2
with:
save-always: true

- name: Run RabbitMQ integration tests
env:
RABBITMQ_URL: amqp://localhost:5672
RUST_LOG: info
run: cargo test -p iii --test rabbitmq_queue_integration -- --nocapture

# ──────────────────────────────────────────────────────────────
# Engine Cross-Platform Builds (main push only)
# ──────────────────────────────────────────────────────────────

engine-build-matrix:
name: Engine Build - ${{ matrix.target }}
needs: changes
if: github.event_name == 'push' && needs.changes.outputs.engine == 'true'
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
Expand All @@ -110,6 +194,7 @@ jobs:
- uses: Swatinem/rust-cache@v2
with:
key: ${{ matrix.target }}
save-always: true

- name: Install musl tools
if: matrix.target == 'x86_64-unknown-linux-musl'
Expand All @@ -129,7 +214,8 @@ jobs:

sdk-node-ci:
name: SDK Node Tests
needs: build-engine
needs: [changes, build-engine]
if: github.event_name != 'pull_request' || needs.changes.outputs.sdk_node == 'true'
Copy link
Copy Markdown
Contributor

@sergiofilhowz sergiofilhowz Mar 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we skipping in pull requests?

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -193,12 +279,13 @@ jobs:

sdk-python-ci:
name: SDK Python (${{ matrix.python-version }})
needs: build-engine
needs: [changes, build-engine]
if: github.event_name != 'pull_request' || needs.changes.outputs.sdk_python == 'true'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ['3.10', '3.11', '3.12']
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["3.12"]') || fromJSON('["3.10", "3.11", "3.12"]') }}

steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -253,7 +340,8 @@ jobs:

sdk-rust-ci:
name: SDK Rust Tests
needs: build-engine
needs: [changes, build-engine]
if: github.event_name != 'pull_request' || needs.changes.outputs.sdk_rust == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -263,6 +351,8 @@ jobs:
components: rustfmt, clippy

- uses: Swatinem/rust-cache@v2
with:
save-always: true

- name: Check formatting
run: cargo fmt -p iii-sdk -- --check
Expand Down Expand Up @@ -293,12 +383,13 @@ jobs:
run: bash scripts/stop-iii.sh /tmp/iii-engine.pid

# ──────────────────────────────────────────────────────────────
# Motia JS (depends on engine + SDK Node)
# Motia JS (runs in parallel with SDK Node)
# ──────────────────────────────────────────────────────────────

motia-js-ci:
name: Motia JS Tests
needs: [build-engine, sdk-node-ci]
needs: [changes, build-engine]
if: github.event_name != 'pull_request' || needs.changes.outputs.motia_js == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -347,17 +438,18 @@ jobs:
run: bash scripts/stop-iii.sh /tmp/iii-engine.pid

# ──────────────────────────────────────────────────────────────
# Motia Python (depends on engine + SDK Python)
# Motia Python (runs in parallel with SDK Python)
# ──────────────────────────────────────────────────────────────

motia-py-ci:
name: Motia Python (${{ matrix.python-version }})
needs: [build-engine, sdk-python-ci]
needs: [changes, build-engine]
if: github.event_name != 'pull_request' || needs.changes.outputs.motia_py == 'true'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ['3.11', '3.12', '3.13']
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["3.12"]') || fromJSON('["3.11", "3.12", "3.13"]') }}

steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -409,6 +501,8 @@ jobs:

console-ci:
name: Console Build
needs: changes
if: github.event_name != 'pull_request' || needs.changes.outputs.console == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
Expand All @@ -434,6 +528,8 @@ jobs:
- uses: dtolnay/rust-toolchain@stable

- uses: Swatinem/rust-cache@v2
with:
save-always: true

- name: Build console binary
run: cargo build -p iii-console --release
Loading