Skip to content

test: Update cross-client testing matrix based on compatibility checkpoints #27233

Open
scottn12 wants to merge 5 commits intomicrosoft:mainfrom
scottn12:updateTestMatrixForPolicy
Open

test: Update cross-client testing matrix based on compatibility checkpoints #27233
scottn12 wants to merge 5 commits intomicrosoft:mainfrom
scottn12:updateTestMatrixForPolicy

Conversation

@scottn12
Copy link
Copy Markdown
Contributor

@scottn12 scottn12 commented May 5, 2026

Description

This PR aligns the e2e cross-client compat matrix with the time-based Compatibility Checkpoint policy from #27064. The changes in this PR ensure we only test the version combinations we explicitly support from the policy.

  • Adds src/checkpoints.ts — designated checkpoint list + window-math helpers
  • genCrossClientCompatConfig now pairs the current build against each in-window prior checkpoint (CC-(N-1), CC-(N-2), CC-(N-3)) in both directions, resolved to the earliest minor of each checkpoint range (~openingVersion) — worst-case target.
  • update-compat-versions resolves the install set from checkpoint data instead of the legacy fast/slow-train deltas.
  • Updated documentation to align with these changes

Updated example test matrix:

compat cross-client - create with 2.101.0 (N) + load with 2.60.0 (CC-3)
  ✔ Example test
compat cross-client - create with 2.101.0 (N) + load with 2.0.9 (CC-2)
  ✔ Example test
compat cross-client - create with 2.101.0 (N) + load with 1.4.0 (CC-1)
  ✔ Example test
compat cross-client - create with 2.60.0 (CC-3) + load with 2.101.0 (N)
  ✔ Example test
compat cross-client - create with 2.0.9 (CC-2) + load with 2.101.0 (N)
  ✔ Example test
compat cross-client - create with 1.4.0 (CC-1) + load with 2.101.0 (N)
  ✔ Example test

Considerations

  • It may be nice to have a single source of truth for checkpoints. With this PR we will need to update both checkpoints.ts and CompatibilityCheckpoints.md.
  • With these changes, we may drop test support for a CC before advancing lowestMinVersionForCollab. This means that customers could still technically use version combinations that are not tested. However, this will only ever apply to version combinations that are explicitly unsupported by our compat policy.

Misc

AB#72039

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

Hi! Thank you for opening this PR. Want me to review it?

Based on the diff (1836 lines, 9 files), I've queued these reviewers:

  • Correctness — logic errors, race conditions, lifecycle issues
  • Security — vulnerabilities, secret exposure, injection
  • API Compatibility — breaking changes, release tags, type design
  • Performance — algorithmic regressions, memory leaks
  • Testing — coverage gaps, hollow tests

How this works

  • Adjust the reviewer set by ticking/unticking boxes above. Reviewer toggles alone don't trigger anything.

  • Tick Start review below to dispatch the review fleet.

  • After review finishes, tick Start review again to request another run — it auto-resets after each dispatch.

  • This comment updates as new commits land; your reviewer selections are preserved.

  • Start review

@scottn12 scottn12 marked this pull request as ready for review May 6, 2026 14:50
Copilot AI review requested due to automatic review settings May 6, 2026 14:50
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

🔗 No broken links found! ✅

Your attention to detail is admirable.

linkcheck output


> fluid-framework-docs-site@0.0.0 ci:check-links /home/runner/work/FluidFramework/FluidFramework/docs
> start-server-and-test "npm run serve -- --no-open" 3000 check-links

1: starting server using command "npm run serve -- --no-open"
and when url "[ 'http://127.0.0.1:3000' ]" is responding with HTTP status code 200
running tests using command "npm run check-links"


> fluid-framework-docs-site@0.0.0 serve
> docusaurus serve --no-open

[SUCCESS] Serving "build" directory at: http://localhost:3000/

> fluid-framework-docs-site@0.0.0 check-links
> linkcheck http://localhost:3000 --skip-file skipped-urls.txt

Crawling...

Stats:
  288641 links
    1922 destination URLs
    2172 URLs ignored
       0 warnings
       0 errors


@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

🔭 PR Review Fleet Report

Note

This report is generated by an experimental AI review fleet and is provided as a beta feature. Findings are a starting point for discussion, not a gate. Use your own judgement.

Verdict: ⚠️ Approve with Suggestions

0 Exterminate, 0 Squash, 1 Investigate

Findings

Sev # Area File What Fix
🐜 Investigate M1 Testing packages/test/test-version-utils/src/checkpoints.ts:170 The non-wildcard semver branch of matchesRange (lines 170–173) is never exercised by any test. All current additionalRanges entries use the wildcard form (2.0.0-internal*, 2.0.0-rc*), so the semver.satisfies(version, range, { includePrerelease: true }) path and the throw for invalid ranges are dead code today. The function is explicitly designed as a future-proofing guard ('authoring mistakes fail loudly'), but without tests, a maintainer could drop { includePrerelease: true } or silently change the throw to a return false and no test would catch it. If a future checkpoint author adds a standard semver range to additionalRanges (e.g., >=2.0.0-rc.1 <2.0.0), the broken path would silently mis-classify versions into the wrong checkpoint, causing the wrong cross-client test matrix to run and hiding compatibility regressions. Add tests for the two untested branches in matchesRange (called indirectly via getCurrentCheckpoint). (1) Semver-range branch: add a synthetic checkpoint with additionalRanges: ['>=1.4.0 <2.0.0'] and verify that getCurrentCheckpoint('1.5.0') returns it, and that a version outside the range does not. Also verify includePrerelease: true matters by checking that '1.5.0-rc.1' is matched. (2) Throw branch: add a synthetic checkpoint with additionalRanges: ['not-a-valid-range'] and assert that getCurrentCheckpoint('1.5.0') throws with a message containing the bad entry.

View workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant