Skip to content

ci: SQL E2E over subprocess + http + unix transports#1

Merged
rustyconover merged 1 commit into
mainfrom
ci/multi-transport-sql
Jun 24, 2026
Merged

ci: SQL E2E over subprocess + http + unix transports#1
rustyconover merged 1 commit into
mainfrom
ci/multi-transport-sql

Conversation

@rustyconover

Copy link
Copy Markdown
Contributor

Adds multi-transport VGI coverage to the SQL E2E suite: the same test/sql/*.test suite now runs over subprocess, http, and unix transports in real CI (3 transports x 2 OS = 6 integration legs).

What changed

  • ci/run-integration.sh parameterized by a TRANSPORT env var (default subprocess):
    • The canned-response mock news server (importlib-loaded _Handler from scripts/run_sql_e2e.py) stays up for all transports; its VGI_NEWS_*_BASE_URL / VGI_NEWS_TIMEOUT vars are exported, so an out-of-band-booted (http/unix) worker inherits them.
    • http: injects INSTALL httpfs FROM core; LOAD httpfs; into staged tests (the vgi HTTP transport rides DuckDB's httpfs), boots news_worker.py --http --port 0 --port-file <f>, polls the port-file, and ATTACHes http://127.0.0.1:<port>.
    • unix: boots news_worker.py --unix <sock>, polls for the socket, ATTACHes unix://<sock>.
    • One cleanup() trap captures $? first and kills both the mock and the worker.
    • Silent-skip guard (tee + PIPESTATUS): the runner auto-SKIPs any error containing "HTTP" (a green-looking fake pass), so the leg fails unless All tests passed (N>0 assertions) and zero skips.
  • pyproject.toml + news_worker.py PEP 723 header gain an http extra (vgi-python[http], waitress); uv.lock relocked.
  • .github/workflows/ci.yml: integration job is now a transport: [subprocess, http, unix] x os matrix, installing with uv sync --frozen --extra http. The test and lint jobs are unchanged.
  • ci/README.md rewritten for news (transport matrix, discovery, httpfs-on-http, silent-skip guard, externalized cursor).

Streaming over HTTP

news_search is a paging table function, but its cursor is already externalized into the serializable NewsScanState (plain ints/bools/strings; no fetched rows carried in state — each process() tick re-fetches deterministically from the cursor). So it resumes correctly over the stateless HTTP transport across the batch boundary. The http leg runs the full suite including news_scan_state.test (the GDELT window cursor hops to the second window and emits "Older C", reachable only via the resumed cursor) — nothing gated, no worker change needed.

Local validation (all GREEN)

  • subprocess: 39 assertions
  • unix: 39 assertions
  • http: 45 assertions (39 + injected httpfs INSTALL/LOAD across 3 files)

🤖 Generated with Claude Code

Parameterize ci/run-integration.sh by a TRANSPORT env var (subprocess |
http | unix), exercising the same test/sql/*.test suite over all three VGI
transports. The mock news server stays up for all transports; its base_url
env vars are exported so an out-of-band-booted (http/unix) worker inherits
them.

- http: inject httpfs INSTALL/LOAD into staged tests (the vgi HTTP transport
  rides DuckDB's httpfs), boot the worker with --http --port 0 --port-file,
  poll the port-file, ATTACH http://127.0.0.1:<port>.
- unix: boot the worker with --unix <sock>, poll for the socket, ATTACH
  unix://<sock>.
- One cleanup() trap captures $? first and kills both the mock and the worker.
- Silent-skip guard: tee + PIPESTATUS; fail unless 'All tests passed (N>0
  assertions)' and no skips (the runner auto-SKIPs HTTP errors -> fake pass).

Add an http extra (vgi-python[http]) to pyproject.toml and the news_worker.py
PEP 723 header; relock. CI integration job becomes a
transport x os matrix and installs with --extra http.

news_search's pagination cursor is already externalized into the serializable
NewsScanState (no rows carried in state; each tick re-fetches deterministically
from the cursor), so it passes over HTTP unchanged — no worker fix needed.

subprocess=GREEN (39) http=GREEN (45) unix=GREEN (39) locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rustyconover rustyconover merged commit 28e85e1 into main Jun 24, 2026
10 checks passed
@rustyconover rustyconover deleted the ci/multi-transport-sql branch June 24, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant