feat(stack): show gateway requests in Studio's API Gateway logs by avallete · Pull Request #6942 · supabase/cli

avallete · 2026-10-01T11:53:06Z

TL;DR

Studio's Logs → API Gateway page was always empty on the experimental stack, because the stack's API proxy logged nothing per request. The proxy now writes one access line per request, persists it like service logs, and ships it to Analytics in the shape Studio reads, so the page lists every request with its status, method and path. supabase stack logs shows the same lines.

Stacked on #6893.

Before

flowchart LR
  C[Client] --> P["Stack API proxy"]
  P --> S["REST / Auth / Storage / Functions / Realtime"]
  S --> L["Service logs"]
  L --> A[Analytics]
  P -. "no access log" .-> X["Studio API Gateway: empty"]

After

flowchart LR
  C[Client] --> P["Stack API proxy"]
  P --> S["REST / Auth / Storage / Functions / Realtime"]
  P --> G["gateway log stream<br/>one line per request"]
  G --> F["logs/gateway files"]
  F --> CLI["supabase stack logs --service gateway"]
  F --> A["Analytics cloudflare.logs.prod"]
  A --> U["Studio API Gateway"]

Why

The legacy supabase start fills the API Gateway page from Kong's access log (source cloudflare.logs.prod). The experimental stack has no Kong: its shared API port is served by the owner's own HTTP proxy, which had no access log, so the page showed "No data" however much traffic the stack served.

What changed

The shared API proxy records one access line per request and per websocket upgrade, in nginx combined format plus the duration. The status is the one actually sent, including proxy-generated 404/502/503 and requests that wait for a lazy service to wake; a client that leaves before the response is recorded as 499, and body bytes appear only for responses that finished. Logging never delays a response.
apikey, access_token and token query values are replaced by redacted before the line is written; request headers such as Authorization are not logged.
The lines are kept as a gateway log stream under the stack's logs/gateway/ directory with the same rotation and retention as service logs, survive owner restarts, and are removed with the stack.
While Analytics is running and healthy, gateway lines ship to cloudflare.logs.prod with the Kong request/response metadata Studio's API Gateway list, detail panel and status filter read (the path without its query, the query as search).
supabase stack logs includes gateway lines by default when the stack serves the shared API port, and --service gateway selects them; reading works while the stack is stopped.
Studio's per-function logs page stays empty: it filters on a function id that local Studio regenerates on every call, and the local edge runtime output does not carry the function name. The legacy start has the same gap.

Terminal captures

Recorded from this branch's source on macOS (Docker runtime): terminal with vhs, Studio in headless Chrome, at the same time against the same stack. Top: stack logs -f --service gateway (the apikey query value is redacted); bottom: requests to REST, Auth, Storage and Functions; right: Studio's API Gateway list filling in, then the error and warning status filter. Before this change the page showed "No data".

🤖 Generated with Claude Code

Service output lived only in a per-recipe in-memory buffer, so the stack kept no log history, lost it when the owner exited, and native mode had nothing to replay. The owner now writes every instance's output to `logs/<service>/<instanceId>/<generation>.log` under the stack's state directory. Output chunks are tagged with their launch, process part, sequence and publish time at the source, so lines are split once, never glued across launches, and upstream loss is recorded as `lost`. Segments rotate at 5 MiB with byte and count retention, and are removed on instance and stack destroy. A single position-based reader serves history, follow and offline reads. It is exposed as a `readLogs` RPC and `readStackLogs`; the existing `logs` RPC keeps its live-only behaviour for `supabase stack logs`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Studio's Logs pages stayed empty because nothing forwarded the stack's service output to Analytics. The owner now ships persisted Auth, REST, Realtime, Storage, Functions and database lines to their Logflare sources, using the legacy Vector remaps ported to TypeScript. It posts to Analytics' own backend only while the composed Analytics instance is running and healthy, so shipping never wakes it or keeps it awake. Each instance keeps a cursor beside its log files: lines written while Analytics sleeps are shipped after it wakes with their original timestamps. Events carry deterministic ids and Logflare de-duplicates on them, so failed posts are retried without duplicates. Auth and target errors pause shipping and keep the cursor; malformed bodies are skipped. The CLI no longer composes Vector. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`supabase stack logs` now prints retained history and exits by default. `-f/--follow` hands over from history to live output per instance without gaps, `--tail` (default 200), `--since <duration|ISO|start>` and a repeatable `--service` select what is shown, and history is read from the log files when the stack is down. Output keeps the `log-entry` contract with `source: "history" | "live"` and adds `log-marker` events. The legacy live-only `logs` RPC is replaced by `readLogs`, which the Promise client exposes as an async iterable. The experimental stack no longer has a Vector service: saved stacks that contain one are migrated on owner start, and leftover Vector files and containers are removed. `analytics.vector_port` stays in the config schema because the legacy `supabase start` still reads it. PostgREST now logs every request, and request lines reach Analytics with their method, path, protocol and status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The artifacts catalog is also the version table the legacy `supabase start` reads in slim mode, so dropping Vector's entry would silently fall back to the upstream Vector image there. Vector stays in the catalog as an artifact kind that only the legacy start runs; the experimental stack's service kinds, `supabase services` and stack prepare still exclude it. Docker-backed log tests get the same timeout guard as the package's other Docker tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Abort log persistence 5 seconds into close, detaching instances under one deadline; aborted segments are retired and still release their handles. - Flush quiet late partial lines when their grace expires. - Count unwritten records as lost once, and clear lost markers only after they are written. - Read every segment for tails and report interior segment gaps with resumeAt. - Keep the first byte's time for UTF-8 characters split across chunks. - Retry wrapped sharing violations, claim removed instances atomically, and validate the readStackLogs tail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Flush a late partial line only after every chunk published before its grace deadline is processed, comparing publish times. - Report a lost marker when a reader finishes a segment and the next retained generation is not contiguous. - Treat stack logs --since as a value-consuming flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- The shared API proxy records one access line per request and websocket upgrade (nginx combined format plus duration, credentials in the query string redacted), persisted as a gateway log stream. - Gateway lines ship to the cloudflare.logs.prod source with the Kong request and response metadata Studio's API Gateway page reads. - supabase stack logs includes gateway lines and selects them with --service gateway. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Redact apikey, access_token and token in the Referer's query and in fragments, as for the request target. - Settle a forwarded request when the client closes after the response ended but before it finished, so it records once and releases its target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Hold the newline chunk behind a blocked write while the grace flusher wakes past its deadline, so the test fails if the flush skips queued output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…logs

Auth redirect and verify URLs carry PKCE codes, OTP token hashes and refresh, ID and provider tokens; redact them like API keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions

Superseded by a newer AI review

🤖 AI Review

Both independent reviews were available. All eight distinct findings were verified: seven confirmed and one refuted. The credential-redaction gap is critical under the supplied severity definitions. Isolated helper probes reproduced credential leakage and incorrect WebSocket status logging; full tests were not run because dependencies are absent.

Findings

Severity	Location	Category	Sources	Claim
🔴 CRITICAL	`packages/stack/src/HttpProxy.ts:155`	`security`	claude	Credential-bearing query parameters such as token_hash and OAuth code remain in clear text in persisted gateway logs and forwarded Analytics events.
🟡 MINOR	`packages/stack/src/HttpProxy.ts:599`	`observability`	codex	HTTP access durations include target cleanup performed after the response finishes.
🟡 MINOR	`packages/stack/src/HttpProxy.integration.test.ts:1332`	`test-reliability`	codex	The full-body reset test can reset the client before the proxy forwards response headers, making its required status 200 assertion flaky.
🟡 MINOR	`packages/stack/src/Owner.logs.integration.test.ts:158`	`resource-cleanup`	codex	The gateway restart test leaks its first owner scope if setup or an assertion fails before the explicit close.
🟡 MINOR	`packages/stack/src/HttpProxy.integration.test.ts:1034`	`error-handling`	codex	rawClient leaves acquisition pending when its initial socket connection fails.
🟡 MINOR	`packages/stack/src/HttpProxy.ts:512`	`correctness`	codex	WebSocket access logging records an informational HTTP response as the final handshake status.
⚪ NIT	`packages/stack/src/host/GatewayLog.ts:29`	`correctness`	claude	Gateway Analytics timestamps discard the millisecond precision of HttpAccess.time, preventing timestamp-based ordering of requests within the same second.

Findings outside the diff

🟡 MINOR packages/stack/src/HttpProxy.ts:512 — WebSocket access logging records an informational HTTP response as the final handshake status.

Refuted findings (kept for transparency, not posted as review comments)

apps/cli/src/commands/experimental/stack/logs/logs.handler.ts:195 (compatibility): Implicit gateway selection causes stack logs --follow to fail against an older running owner because that owner has no attached gateway stream.
Refuted: The asserted absence of an owner version check is contradicted by HostProcess.ts:292-298. HostProcess.ts:41-50 and internal/release.ts:20-40 derive the release from stack source contents, so an earlier build without gateway logging has a different release and is rejected before log RPCs. Same-release owners unconditionally attach gateway at Owner.ts:200-204. Trusted ADR 0017:163-168 also documents same-release RPC access.

Stats

Claude findings: 3 · Codex findings: 5 · Confirmed: 7 · Refuted: 1 · Uncertain: 0

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

- Measure request duration when the response settles, before target cleanup, and skip interim 1xx answers when recording upgrade status. - Redact parameters whose value carries nested credentials, and URL userinfo in logged URLs. - Make the reset, restart and raw-client test fixtures deterministic and leak-free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Build gateway access records only on the proxy that logs them. - Collapse credential redaction into one pattern and move its cases to a unit table. - Share the month table, rename the log escaper, read upgrade answers in one loop, and trim a forwarder assertion the remap test already pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Keep launch ids increasing per instance across owner restarts so stack logs --since start always selects the latest launch. - Migrate saved Vector stacks without taking the registry lock when there is nothing to migrate, and remove the rendered Vector config. - Fail stack logs --follow when the running stack serves none of the selected instances, and warn about skipped ones. - Detach an instance from log shipping, after its cursor write lands, before its logs are removed. - Remove readStackLogs, the end read option and unused platform options; reuse existing helpers and trim redundant tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…logs # Conflicts: # apps/cli/src/commands/experimental/stack/logs/logs.integration.test.ts # packages/stack/src/effect.ts # packages/stack/src/host/LogForwarder.ts

… one An instance saved before launch ids were persisted resumes after the highest launch id at the end of its newest log segment, so its next launch never reuses an id its logs already hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…avallete/stack-gateway-logs # Conflicts: # packages/stack/src/Owner.logs.integration.test.ts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…igration - Retry failed log reads from the saved cursor with capped backoff instead of pausing shipping until Analytics restarts. - Bound stack logs --since start, history and follow, by each instance's saved launch id. - Keep the PostgREST time when a message contains ": ". - Delete Vector files before committing the migration so a failure is retried, keep reclaiming a session stack when migration fails, and reap stale cursor-write directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…avallete/stack-gateway-logs # Conflicts: # apps/cli/src/commands/experimental/stack/logs/logs.handler.ts # apps/cli/src/commands/experimental/stack/logs/logs.integration.test.ts # packages/stack/src/host/LogForwarder.ts # packages/stack/src/host/LogflareEvents.ts

Develop's native port test now uses this branch's recipe signatures, the catalog error helper both sides stopped using is removed, and the stack log event and marker formatting moves to command-internal so functions serve no longer imports another command's internals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…avallete/stack-gateway-logs

The once-per-launch dropped-output warning now also covers lost markers written after a failed append recovers, and the rejected-body warning says the body is posted again in halves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…avallete/stack-gateway-logs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- Vector migration cleanup only follows safe instance ids and finds leftover Vector files by layout, so a failed cleanup is retried after later state writes and never leaves the stack. - The stored-event client connects with Analytics' database URL, keeping its TLS settings, and looks a source's table up again after a failed query. - Only composition members are shipped to Analytics, so standalone databases do not mix into Studio's database logs. - A launch's queued output is split before its partial line is flushed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…avallete/stack-gateway-logs # Conflicts: # packages/stack/ARCHITECTURE.md # packages/stack/src/host/LogForwarder.ts

Not pushed yet; held while #6893 is the focus. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…eway-logs Brings in the merged log persistence and shipping (#6893). The gateway stream relies on the shared publisher's per-launch ordering instead of its own semaphore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

avallete · 2026-10-05T15:11:47Z

/ai-review

github-actions

Superseded by a newer AI review

🤖 AI Review

Both independent reviews were available. Verified all four findings against the checked-out code and trusted conventions: one critical credential leak, two minor observability/correctness issues, and one documentation nit. All are confirmed; none concern the same substance.

Findings

Severity	Location	Category	Sources	Claim
🔴 CRITICAL	`packages/stack/src/internal/redact-credentials.ts:59`	`security`	codex	Userinfo redaction misses scheme-relative URLs and exposes password suffixes after an unescaped @. These credentials reach persisted gateway logs and Analytics.
🟡 MINOR	`packages/stack/src/internal/redact-credentials.ts:44`	`observability`	claude	The route-independent credential-name filter redacts ordinary PostgREST filters on columns named code or token, losing their values from gateway logs and Studio's API Gateway records.
🟡 MINOR	`packages/stack/src/HttpProxy.ts:491`	`correctness`	claude	When the upgrade status sniffer gives up on an invalid or oversized answer, it leaves handshake completion pending. The access record waits for connection termination, measures connection lifetime instead of handshake duration, and can report 499 despite upstream bytes having been relayed.
⚪ NIT	`apps/cli/src/commands/experimental/stack/start/SIDE_EFFECTS.md:44`	`docs`	claude	The access-log description uses awkward timestamp wording and uneven wrapping, obscuring that each line contains a millisecond-resolution timestamp and trailing duration.

Stats

Claude findings: 3 · Codex findings: 1 · Confirmed: 4 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

…r a torn segment - Redact userinfo through its last `@`, in scheme-relative URLs, and at every decoding level, so a nested password with an encoded `/` is caught. - Redact a credential pair whose `=` is percent-encoded, and pairs after `;` or `&`. - Start a scheme only where a letter run begins, which keeps matching linear on long values. - Log an upgrade answer without an HTTP status line as 502 when it arrives, instead of waiting for the connection to close. - Take the gateway's next launch id from the newest segment that holds a record, so an empty newest segment left by a crash after rotation does not restart at 1. - Move `decodeQuery` back into HttpProxy.ts and document the name-based redaction policy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

avallete · 2026-10-06T11:02:29Z

/ai-review

github-actions

🤖 AI Review

Confirmed Claude's minor test-coverage finding: the gateway fixture reuses a launch ID and therefore asserts behavior that real owner restarts do not produce. Codex reported no findings. Corrected the finding's location to the actual new-side file lines.

Findings

Severity	Location	Category	Sources	Claim
🟡 MINOR	`apps/cli/src/commands/experimental/stack/logs/logs.integration.test.ts:540`	`test-coverage`	claude	The gateway --since start test gives both owner runs launch ID 1 and expects both runs' requests. Real owner restarts increment the launch ID, causing the collector to exclude the first run's requests. The fixture therefore fails to exercise the real restart behavior.

Stats

Claude findings: 1 · Codex findings: 0 · Confirmed: 1 · Refuted: 0 · Uncertain: 0

Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

…ce start test Owner restarts number gateway launches after the retained ones, so `--since start` keeps only the current owner run's requests; the fixture now matches that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

## Summary **TL;DR:** Two `LogForwarder` integration tests occasionally hit the 30 s timeout in CI because each assumed an ordering the forwarder doesn't guarantee. Both tests now cover every ordering without sleeps. The forwarder itself is unchanged. **Why:** CI runs on supabase#6942 timed out in: - `pauses without advancing when Analytics refuses its credentials` - `ships a record persisted while Analytics slept once it wakes on a new port, …` **What changed** - **Credentials refusal:** the test armed its cursor-write gate only after `logflare.next` returned. When the forwarder handled the 401 and removed its cursor first, the gate never fired, and the paused forwarder never slept `pollMillis`, so both race branches hung. The test now holds the 401 answer with `logflare.hold` until the gate is armed. Delaying the test fiber after `logflare.next` reproduces the hang reliably on the old test. - **Retarget to a new port:** when the held post is released, the retired launch-1 session can reach the asleep record before the race interrupts it. It then saves that record as pending to launch 1 and fails `requireCurrent` without posting. The new session waits until the saved post deadline plus the flush window before posting, but the test expected the post without advancing the manual clock. The test now takes that post with `nextPost`, which steps through stored-id polls until a post arrives, so both orderings pass. Delaying the retarget side of the race reproduces the hang reliably on the old test. In production this ordering only delays shipping until that deadline plus the flush window (about 10 s), so the forwarder is left as is. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Coly010

LGTM, approving so you're unblocked. one ask on --since start, the other two are optional - feel free to push back on any of them

…y shape - Persist the gateway's launch marker when the owner starts, so `stack logs --since start` no longer shows the previous owner run until the first request arrives. - Redact query and fragment values shaped like a secret key (`sb_secret_…`) or a JWT under any parameter name. - Escape C1 control characters in access lines like nginx does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

avallete · 2026-10-07T15:01:13Z

/ai-review

…e stack log stream Every owner run now writes a gateway launch marker, so the stack-wide stream no longer starts with the instance's records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…o avallete/stack-gateway-logs

avallete and others added 10 commits October 1, 2026 10:35

test(stack): exercise the late-line grace race deterministically

268e8f1

Hold the newline chunk behind a blocked write while the grace flusher wakes past its deadline, so the test fails if the flush skips queued output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge branch 'avallete/stack-logs-files' into avallete/stack-gateway-…

0cd037d

…logs

avallete marked this pull request as ready for review October 1, 2026 14:26

avallete requested a review from a team as a code owner October 1, 2026 14:26