ci: SQL E2E over subprocess + http + unix transports#1
Merged
Conversation
Parameterize ci/run-integration.sh by a TRANSPORT env var (subprocess | http | unix), exercising the same test/sql/*.test suite over all three VGI transports. The mock news server stays up for all transports; its base_url env vars are exported so an out-of-band-booted (http/unix) worker inherits them. - http: inject httpfs INSTALL/LOAD into staged tests (the vgi HTTP transport rides DuckDB's httpfs), boot the worker with --http --port 0 --port-file, poll the port-file, ATTACH http://127.0.0.1:<port>. - unix: boot the worker with --unix <sock>, poll for the socket, ATTACH unix://<sock>. - One cleanup() trap captures $? first and kills both the mock and the worker. - Silent-skip guard: tee + PIPESTATUS; fail unless 'All tests passed (N>0 assertions)' and no skips (the runner auto-SKIPs HTTP errors -> fake pass). Add an http extra (vgi-python[http]) to pyproject.toml and the news_worker.py PEP 723 header; relock. CI integration job becomes a transport x os matrix and installs with --extra http. news_search's pagination cursor is already externalized into the serializable NewsScanState (no rows carried in state; each tick re-fetches deterministically from the cursor), so it passes over HTTP unchanged — no worker fix needed. subprocess=GREEN (39) http=GREEN (45) unix=GREEN (39) locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds multi-transport VGI coverage to the SQL E2E suite: the same
test/sql/*.testsuite now runs over subprocess, http, and unix transports in real CI (3 transports x 2 OS = 6 integration legs).What changed
ci/run-integration.shparameterized by aTRANSPORTenv var (defaultsubprocess):_Handlerfromscripts/run_sql_e2e.py) stays up for all transports; itsVGI_NEWS_*_BASE_URL/VGI_NEWS_TIMEOUTvars are exported, so an out-of-band-booted (http/unix) worker inherits them.INSTALL httpfs FROM core; LOAD httpfs;into staged tests (the vgi HTTP transport rides DuckDB's httpfs), bootsnews_worker.py --http --port 0 --port-file <f>, polls the port-file, and ATTACHeshttp://127.0.0.1:<port>.news_worker.py --unix <sock>, polls for the socket, ATTACHesunix://<sock>.cleanup()trap captures$?first and kills both the mock and the worker.All tests passed (N>0 assertions)and zero skips.pyproject.toml+news_worker.pyPEP 723 header gain anhttpextra (vgi-python[http], waitress);uv.lockrelocked..github/workflows/ci.yml: integration job is now atransport: [subprocess, http, unix]xosmatrix, installing withuv sync --frozen --extra http. Thetestandlintjobs are unchanged.ci/README.mdrewritten for news (transport matrix, discovery, httpfs-on-http, silent-skip guard, externalized cursor).Streaming over HTTP
news_searchis a paging table function, but its cursor is already externalized into the serializableNewsScanState(plain ints/bools/strings; no fetched rows carried in state — eachprocess()tick re-fetches deterministically from the cursor). So it resumes correctly over the stateless HTTP transport across the batch boundary. The http leg runs the full suite includingnews_scan_state.test(the GDELT window cursor hops to the second window and emits "Older C", reachable only via the resumed cursor) — nothing gated, no worker change needed.Local validation (all GREEN)
🤖 Generated with Claude Code