Skip to content

Test the integrations catalog and picker - #46

Merged
nrodd merged 1 commit into
tests/harness-and-cifrom
tests/integrations
Sep 28, 2026
Merged

nrodd merged 1 commit into
tests/harness-and-cifrom
tests/integrations

Conversation

@nrodd

@nrodd nrodd commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Chunk 4 of the test plan. src/integrations.test.ts, 75 tests. No source changes.

Based on #42 (the harness). GitHub will retarget this to main when that merges. Independent of #43, #44 and #45 — different files.

Where the weight went

The picker only steers the agent — anything it misses, the agent's own Step 2 explore still detects. So this doesn't chase prompt coverage. It leans on two things that can actually break a run:

Catalog consistency. The catalog is hand-maintained and feeds three places at once: the picker, package detection, and the linkage examples pasted into the agent prompt. These run per-entry so a failure names the tool:

  • unique ids and labels
  • ids lowercase and spaceless — selectIntegrations lowercases before matching, so a capitalised id in the catalog would be silently unreachable from --integrations
  • every tool detectable by a package or a window global (Appcues has no npm package)
  • every linkageExample non-empty and actually naming the session URL
  • no npm package claimed by two tools

--integrations parsing, which is the path CI and scripted runs take: case-insensitive matching, trimming, unknown names routed to other with their original case preserved, empty segments from a doubled comma dropped, and user order preserved rather than catalog order.

The rest

  • detectInstalledIntegrations against real temp directories — deps and devDeps, package aliases, several tools at once in catalog order, and the four ways it must return [] rather than throw (no file, unparseable file, missing directory, no deps).
  • The collapse. A detection hit confirms the detected set behind a single "add any others?" rather than showing all fifteen options next to one tool. Saying no returns just the detected set; saying yes opens the picker with those pre-checked and hinted detected — but still uncheckable, since a pre-check is a default and not a lock.
  • Cancellation at all three prompts raises CancelledError. This is the one place in the flow where a cancel still aborts the run, which is worth having pinned before chunk 7, where the same gesture must not abort.

One recorded behavior

A repeated id in --integrations isn't deduped, so --integrations posthog,posthog lists PostHog twice in the agent prompt. Harmless, and deduping is a behavior change rather than a test, so there's just a test saying so.

Verification

75 tests passing (128 on this branch: chunk 0 plus this one), typecheck clean. Mutation-checked two assertions — dropping the .toLowerCase() in flag matching, and returning an empty set instead of the detected one — and confirmed exactly those tests fail, then reverted.


Note

Low Risk
Test-only PR with no runtime or CLI behavior changes.

Overview
Adds src/integrations.test.ts (~75 Vitest cases) with no production code changes. Tests pin behavior the CLI and agent rely on: hand-maintained INTEGRATIONS invariants (unique ids/labels, lowercase ids, detectability, linkage examples, no duplicate npm packages), detectInstalledIntegrations against temp package.json trees (including safe empty results on bad/missing input), and selectIntegrations for both --integrations (case/trim, order, other, no prompts) and interactive flows (full picker vs collapsed “any others?” when deps are detected, __other__ free text, catalog ordering, CancelledError on cancel).

Documents current behavior that repeated flag ids are not deduped.

Reviewed by Cursor Bugbot for commit a4adf66. Bugbot is set up for automated code reviews on this repo. Configure here.

Chunk 4. Adds src/integrations.test.ts, 75 tests. No source changes.

The picker only steers the agent — whatever it misses, the agent's own
Step 2 explore still finds — so the weight is on the catalog's internal
consistency and the --integrations parsing that CI runs depend on.

Catalog invariants run per entry, so a failure names the tool: unique
ids and labels, ids lowercase and spaceless (selectIntegrations
lowercases before matching, so a capitalised id would be unreachable
from the flag), every tool detectable by a package or a window global,
every linkage example naming the session URL, and no npm package
claimed by two tools.

detectInstalledIntegrations runs against real temp directories, since
returning [] rather than throwing on a missing or unparseable
package.json is the property that matters.

The interactive flow covers the collapse: a detection confirms the
detected set behind one "any others?" instead of showing all fifteen
options, and saying yes opens the picker with those pre-checked and
hinted but still uncheckable. Cancels at all three prompts raise
CancelledError.

One recorded behavior: a repeated id in --integrations is not deduped,
so the tool is listed twice in the prompt.
@nrodd
nrodd merged commit 6c68db8 into tests/harness-and-ci Sep 28, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant