v1.3.2

[1.3.2] — 2026-07-21

Disclosure and durability. Upstream error text is filtered before it leaves the
process, a crashed MCP subprocess is now noticed instead of being advertised as
healthy, admin config changes and their audit entries are recorded atomically,
and MCP gains the metrics needed to alert on a server that never came up.

Default behaviour is unchanged. No request that was accepted is now refused, no
response shape changes, no status code changes, and no configuration becomes
required. The one opt-in is mcp_servers[].required, which defaults to false.

Fixed

• Provider error text could carry a credential to clients, logs, streams, and
  observability backends. A provider controls the body of its own error
  responses, and some echo the value of the Authorization or api-key header
  they were sent. That text became the error message the gateway reported, so
  the gateway's own credential could reach an end user, a log aggregator, an SSE
  frame, or an exporter. Error messages are now filtered at the points every
  caller passes through — the OpenAI-compatible error writer, the admin error
  writer, the streaming error frame, the request-failure event, and the failure
  log lines. Only the message text changes; status, type, and code are
  written exactly as before. Filtering is best-effort pattern matching and
  covers the key formats that carry a recognisable prefix; a provider whose keys
  have no distinguishing shape cannot be matched, so it reduces exposure rather
  than eliminating it.
• A crashed MCP stdio subprocess was never noticed. A server was marked ready
  once and nothing ever cleared it, so after its subprocess died the gateway kept
  advertising that server's tools to the model, kept resolving them to a dead
  transport, and failed every call. Child exit is now detected two ways — the
  subprocess closing its error stream, and a tool call failing with a closed
  transport or a broken pipe — and the server's tools are withdrawn from the
  model until it comes back. A server closing its error stream is confirmed with
  a ping before anything is withdrawn, so one that does so while still running
  keeps serving, and a server that closes it at startup is still covered later.
  Detection applies to stdio servers; an HTTP server that becomes unreachable
  after a successful handshake is not currently detected. Recovery on the next
  configuration reload is unchanged.
• An MCP server that could not be built was missing from readiness. A server
  whose headers or env referenced an undefined variable was resolved before
  its transport existed, so it never reached the registry that readiness reports
  from. It was absent from mcp_servers rather than listed as down, and one
  marked required left /readyz answering ready while the server it depends on
  had never been attempted. Such a server is now reported unready with its
  required flag intact, and reads 0 on gateway_mcp_server_up rather than
  being absent from /metrics.
• A configuration reload could leave gateway_mcp_server_up reading down for a
  server that was up. The gauge is labelled by server name, so the retiring
  registry and its replacement wrote the same series; a reload with a request
  still in flight let the old one publish 0 after the new one had already
  reported the server ready, and nothing wrote 1 again. Teardown now writes only
  for servers the retiring registry still owns, and drops the series for a server
  that has left the configuration instead of pinning it at 0 forever.
• An MCP tool that reported failure was recorded as a success. Servers signal
  a failed call by returning an error result rather than a transport error. That
  signal was not read, so a failing tool incremented the success counter, was
  audited as successful, and produced a successful span. It is now recorded as an
  error in all three. What the model and the client receive is unchanged.
• Admin config changes and their audit entries could disagree. Applying a
  config and recording its history entry were separate steps under separate
  locks, so two concurrent admin writes could interleave and leave the newest
  history entry describing a config that was not the active one — with a rollback
  then targeting the wrong version. The same gap existed one level lower, between
  persisting a config and applying it, where the disagreement survived a restart.
  Each admin config mutation is now applied and recorded as one serialized
  operation. History reads are unaffected and never wait on a config apply.
• An unknown MCP tool name became an unbounded metric label. The tool name on
  the unknown-tool counter came from the model's output, so an invented name
  minted a new time series. Names that do not resolve to a registered tool now
  collapse to a single bounded label, matching how unroutable model names are
  already handled.

Added

• mcp_servers[].required — marks an MCP server whose availability gates
  /readyz. Defaults to false, so an existing configuration behaves exactly as
  before. See Upgrade notes.
• MCP server state on /readyz. The readiness body now reports each
  configured MCP server's name, readiness, and whether it is required, so MCP
  health is observable without gating on it. The failure reason is deliberately
  omitted — the endpoint is unauthenticated and a failure can quote a server URL,
  an authorization header, or a subprocess command line. It is logged instead.
• gateway_mcp_server_up — per-server gauge, 1 when a server completed its
  handshake and its transport is alive, 0 otherwise.
• gateway_mcp_server_init_failures_total — per-server counter incremented
  when a server fails to initialize. Initialization failures were previously only
  logged, which left a fully dead MCP fleet indistinguishable from an idle one.
  Alert on any non-zero value.
• Tracing for MCP startup. Server initialization and tool discovery now emit
  spans, so the phase where MCP most often fails is visible in a trace rather
  than only in logs.

Changed

• The stored config history query now reads a bounded number of the most recent
  entries instead of the whole table. This is a read bound only; no history row
  is ever deleted.

Upgrade notes

• /readyz behaviour is unchanged unless you opt in. Setting
  required: true on an MCP server means that if the server is unavailable the
  instance reports not-ready and an orchestrator will take it out of rotation —
  stopping all traffic through it, including requests that use no tools. Set it
  only for a server the deployment genuinely cannot serve without.
• Deployments that already configure mcp_servers will see a new mcp_servers
  array in the /readyz response body. It is additive; existing fields are
  unchanged.
• Error messages returned to clients, written to logs, and sent to observability
  exporters are now filtered. Text that matches a credential pattern is masked,
  so tooling that matches on exact upstream error strings may need adjusting.
• MCP tool calls that return an error result now increment
  ferrogw_mcp_tool_calls_total with status="error" rather than status="ok".
  Dashboards that treated the MCP success rate as near-100% will show the real
  rate. Label sets on existing metrics are unchanged.

Contributors

Thanks to everyone who shipped this release:

• fix: disclosure, durability, and MCP readiness (v1.3.2) (#359) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.3.2/CHANGELOG.md

v1.3.1

[1.3.1] — 2026-07-21

Seven defects reported against v1.3.0, spanning request handling, routing, and
runtime reliability. Every one is reachable from ordinary YAML or JSON
configuration. Nothing that worked before stops working: no request the gateway
accepted is now refused, no response shape changes, and authentication is
untouched. Requests that were wrongly refused now succeed.

Much of this is configuration that was quietly not applied. A retry block
worked on chat and was ignored on streaming. A strategy.mode of fallback
fell back on /v1/chat/completions and did not on /v1/embeddings. If you have
been running with settings that looked correct, some of them are now actually in
effect — see Upgrade notes. The rest are plain defects: a request format the
endpoint documented but rejected, a discarded stream_options value, an API-key
update applied after it had been refused, and a panicking provider call that
disabled its target until restart.

Fixed

• /v1/completions rejected documented request shapes. A prompt sent as
  an array — the batch and token-id forms — and a stop sent as a bare string
  both failed to decode, returning 400. They failed before the handler chose
  between pass-through and the chat shim, so they were refused even when the
  request was being forwarded verbatim to an upstream that accepts them.
  gpt-3.5-turbo-instruct is exactly where batch prompts are used.

• Streaming ignored its retry and fallback configuration. RouteStream
  resolved one provider, called it once, and gave up, so a target returning
  503 failed the request outright while the same configuration retried and
  fell back for non-streaming. Only the start of a stream is retried, and only
  before any data reaches the caller, so a stream that has begun is never
  replayed. The start phase is bounded by the configured request timeout; a
  stream that starts successfully is not.

• Retry backoff was zero on non-chat surfaces. Chat applied a default delay
  between attempts where streaming, embeddings and image generation fell through
  to no delay at all, turning three configured attempts into three immediate
  retries against a provider that had just asked for a pause. Backoff now comes
  from one place for every surface.

• Embeddings and image generation ignored the configured strategy. Both
  resolved to the first capable provider, with no fallback, no retry and no
  request timeout. They now follow the configured target order with the same
  behaviour as chat. A model served by a registered provider that is not listed
  under targets remains reachable, as before.

• A content rule matched every promptless request. With
  strategy.mode: content-based, a prompt_not_contains rule treated the
  absence of messages as a match, so one such rule captured every embedding and
  image request regardless of its configured value.

• Provider lookup was not stable between calls. Registration order was not
  retained, so which provider served a model available from several, the default
  target order, and the order of GET /v1/models could all differ from one
  request to the next within a single process.

• A panicking provider call disabled its target permanently. Circuit-breaker
  bookkeeping was not panic-safe: the half-open probe was left held, nothing
  releases it, and the breaker then rejected every later request for that target
  until the process restarted.

• stream_options.include_usage was ignored. The caller's value was
  discarded and a usage chunk always requested. A caller asking not to receive
  one now does not receive one. Usage is still requested upstream and still
  recorded, so cost accounting, metrics and spend limits are unaffected by what
  the caller asks for.

• Updating an API key applied changes it had rejected. The new name and
  scopes were written before the expiration timestamp was validated, leaving the
  key modified by a request that returned 400.

• Errors raised while starting a stream were not redacted before reaching
  observability exporters and event hooks, unlike every other error path.

• A misconfigured provider base URL was returned to the caller. It can carry
  a credential in its query string; it is now logged server-side only.

• The model catalog was never actually downloaded. The fetch was bounded at
  one second, which is below the real cost of retrieving a multi-megabyte asset
  through a redirecting CDN, so the gateway fell back to the snapshot embedded
  at build time on essentially every start — silently, and with pricing that
  goes stale between releases. The budget is now ten seconds.

Added

• Request metrics, cost, tracing spans and lifecycle events for
  /v1/embeddings and /v1/images/generations. These surfaces previously
  reported nothing, so they showed no traffic and no cost regardless of spend.
• FERRO_MODEL_CATALOG_TIMEOUT bounds the catalog fetch (default 10s).
  The fetch runs during startup, before the listener binds, so a deployment with
  blocked or filtered egress would otherwise wait the full budget on every start
  before falling back. Set it to 0 to skip the remote fetch entirely and use
  the embedded catalog immediately.

Changed

• Target ranking no longer holds the gateway lock. Latency and cost lookups,
  and the random draw used for weighted load balancing, previously blocked every
  other request for their duration.

Upgrade notes

No migration runs and no configuration change is required. Three things will
look different, though, all of them the result of configuration now being
applied where it previously was not — each is worth an operational review before
upgrading.

• Which provider serves a request may change. If strategy.mode is anything
  other than single and more than one target can serve a model, embeddings and
  image generation now follow that strategy instead of always using the first
  capable provider. This has cost, latency and data-residency implications worth
  checking before upgrading.
• Requests that used to fail may now succeed, on streaming and on the
  non-chat surfaces, because retry and fallback now apply there.
• Dashboards will show new traffic and cost. Metrics that aggregate across
  providers now include embedding and image requests for the first time, which
  can look like a step change at the upgrade with no change in actual usage.
  Volume- and cost-based alert thresholds are worth reviewing.
• Reported costs may change, because the gateway now reaches the current
  catalog instead of the one embedded at build time. Prices are more accurate;
  models added since the embedded snapshot stop being costed at zero.
• Startup can take up to ten seconds longer where egress to the catalog host
  is blocked or slow, since the fetch precedes the listener binding. If
  readiness probes are tight, either widen them or set
  FERRO_MODEL_CATALOG_TIMEOUT to a shorter value — or to 0 on air-gapped
  deployments, which skips the fetch entirely.

Contributors

Thanks to everyone who shipped this release:

• fix: routing, lifecycle, and request-handling defects (v1.3.1) (#357) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.3.1/CHANGELOG.md

v1.3.0

[1.3.0] - 2026-07-20

MCP servers can now be launched as local subprocesses, so any npx, uvx, or binary MCP server can be used without standing up an HTTP endpoint for it. Alongside the new transport, two long-standing defects in the existing MCP path are fixed — both of which affected the gateway whether or not you used MCP deliberately.

Thanks to @gr3enarr0w for contributing the stdio transport (#121).

Added

• MCP stdio transport. An mcp_servers entry may now set command (with optional args) instead of url, and the gateway launches that process and speaks MCP over its stdin/stdout for the gateway's lifetime. Exactly one of url or command must be set; both or neither is a config error naming the server. The existing Streamable HTTP transport is unchanged, and the two are interchangeable everywhere tools are consumed (#121).
• Subprocess environment isolation. An MCP subprocess does not inherit the gateway's environment. It receives PATH, HOME, LANG, and TMPDIR when set, plus exactly the keys under that server's env — so no gateway credential such as OPENAI_API_KEY or MASTER_KEY reaches an MCP server implicitly. Isolation is from implicit inheritance, not a prohibition: anything a server needs must be listed in its env, and a credential placed there deliberately is passed through as configured. The same applies to HTTPS_PROXY, NODE_PATH, or SSL_CERT_FILE.
• ${VAR} references in a stdio server's env. Resolved when the MCP client is constructed, the same treatment headers already received, so the config keeps the reference and never stores the secret. Since the gateway environment is not inherited, this is the only channel by which a credential reaches an MCP subprocess. GET /admin/config redacts env alongside headers.
• Subprocess diagnostics. A stdio server's stderr is drained into the gateway log at debug level, one record per line. A server that dies on a missing API key or a permission error now says so in the log instead of surfacing as an opaque timeout.

Fixed

• A misconfigured MCP server no longer disables streaming. The agentic-loop redirect keyed off whether any MCP server was registered, which is true before the handshake runs and stays true after one fails. A single unreachable endpoint or typo'd command therefore turned every stream: true request on the gateway into one buffered chunk — for every caller, including those making no use of MCP — with no error and no way to notice beyond the missing token-by-token delivery. Activation now keys off tools actually discovered, matching the non-streaming path.
• Caller-supplied tool calls are no longer intercepted. With MCP active, every tool call in a model response was executed as if the gateway owned it. A client that sent its own tools array — the ordinary OpenAI function-calling pattern, where the client executes the call and posts the result back — had its call swallowed and answered with a fabricated "tool not found in any registered MCP server", never receiving finish_reason: "tool_calls". Tool calls the gateway does not own are now passed through untouched, so enabling MCP no longer breaks existing client-side tool integrations.
• A single model response can no longer trigger unbounded tool execution. max_call_depth bounded how many turns the agentic loop ran but nothing bounded the calls within one turn, so a response carrying thousands of tool_calls produced that many executions and conversation messages, re-sent to the provider on every subsequent turn. Tool calls are capped per turn.
• Stdio servers launched via npx no longer leak processes. npx and uvx exec the real server as a separate process, and terminating the launcher does not terminate what it started, so every gateway shutdown and config reload left a live server behind holding its pipes. Subprocesses are now started in their own process group and the group is swept after the polite shutdown sequence completes. On Windows the previous single-process teardown is retained.
• A stdio server writing heavily to stderr no longer wedges. Its output went to a pipe nobody read, so once the operating system's pipe buffer filled, the server blocked mid-write and stopped answering requests entirely while still appearing to be running.

Changed

• MCP tools are advertised only when the request carries no tools of its own. Previously every MCP tool definition was injected into every chat completion, alongside whatever the caller sent. That let the model answer with one MCP call and one caller call in the same turn — which neither side can complete: the gateway has no implementation for the caller's tool, and the caller never declared the MCP one. Requests that send their own tools array now pass through untouched, streaming included, and are unaffected by MCP entirely. Requests that send none behave exactly as before.
• mcp_servers[].env documentation corrected. The published field documentation described env as merged with the gateway's environment. It is not, and was not: the values listed are combined with a minimal base only. The godoc, README, and example config now describe the actual behavior.
• Environment-reference documentation corrected across the config surface. Several field docs and example comments described $VAR as a usable reference and attributed substitution to LoadConfig. Only the braced ${VAR} form is a reference, a bare $ is literal data, an undefined variable is an error, and resolution happens when the component is constructed — never at config load, which is what keeps secrets out of the config-history store. The behavior is unchanged since v1.2.0; only the documentation was wrong.

Contributors

@gr3enarr0w — Clark Everson authored the MCP stdio transport in #121: the transport itself, its integration and unit test coverage, subprocess environment isolation via a custom CommandFunc, per-call timeouts, and transport-selection validation. That work is the substance of this release.

Those seven commits were cherry-picked onto the release branch with authorship preserved, but #352 was squash-merged, which collapsed them into a single commit and dropped his name from the history on main. This note is the durable record of that attribution.

@MitulShah1 — release hardening on top of that work (#352), plus #349 and #350.

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.3.0/CHANGELOG.md

v1.2.0

[1.2.0] — 2026-07-14

The capability release. The gateway now has a single, declarative record of what each provider can actually express, and enforces it — so a parameter a provider cannot honor is no longer silently dropped on the wire. Alongside it: per-request deadlines, per-target concurrency limits, split liveness/readiness probes, and a cross-provider conformance suite.

This release contains behavior changes for plugin authors, for retry configuration, and for ${VAR} substitution. Read Changed before upgrading.

Added

• Provider capability matrix and GET /v1/capabilities. providers/capabilities declares, from one source, which OpenAI chat parameters each provider forwards, translates, or cannot express. It is served over the API and consumed by enforcement, so what the gateway advertises and what it does are the same thing (#207, #275).
• compatibility.on_unsupported_param. Choose what happens when a request carries a parameter the target provider cannot express: warn (default — log and forward), drop (strip it), or reject (fail with 400 unsupported_parameter). Previously such a parameter was quietly discarded by the adapter with no signal at all.
• request_timeout. Bounds a single non-streaming request end to end — plugin stages, the provider call, and every retry and fallback attempt combined. Omitted means no gateway-imposed deadline; the provider clients' own timeouts still apply. Streaming is exempt, except the MCP agentic path, which returns one complete response and is bounded like any other non-streaming request (#277).
• Per-target concurrency limits. targets[].concurrency bounds in-flight requests to a provider (max_concurrency) with a bounded wait queue (queue_size). A saturated target sheds with 429 provider_saturated rather than piling up. A streaming request holds its slot until the stream ends (#248).
• /livez and /readyz. Liveness and readiness are now separate signals for orchestrator rollout gating: /livez reports process liveness only, /readyz reports whether the gateway can actually serve traffic. /health is retained (#279).
• Cross-provider conformance suite. test/conformance/ builds every provider through its ProviderEntry — the same seam the gateway uses — points it at a stub returning that provider's native payload, and asserts the translated OpenAI-shaped response. It needs no network, Docker, or credentials, and runs with make test. A new provider must add a fixture or declare why it cannot have one (#276).

Changed

• Plugins deny by verdict, not by error (breaking for plugin authors). A plugin that denies a request and a plugin that breaks were reported the same way. They are now distinct: set Context.Reject to deny (unchanged status codes), and return an error only when the plugin itself failed — which now yields 500 plugin_error. The sharpest case this fixes: a rate-limit plugin whose backend was down returned 429 rate_limit_exceeded, which every OpenAI SDK reads as "back off and retry", turning an outage into a retry storm against a gateway that was not busy, just broken. Plugin failures are also counted as errors rather than rejections, so a plugin outage no longer looks like a wave of blocked prompts (#288).
• Context.Reject is honored for every plugin type. It was silently discarded for logging, metrics, and transform plugins, contradicting its documented contract.
• Transform plugins fail closed. They were grouped with logging and metrics as fail-open, so a transform that could not sanitize a payload let it through. Only plugins that observe the request rather than gate it — logging and metrics — fail open.
• Retries are limited to retryable statuses by default. The default retryable set is now 408, 429, and 5xx. A 400 or 401 is a client error that will fail identically on every attempt; retrying it only multiplied the latency. Retry delays use exponential backoff with full jitter, honor a provider's Retry-After (capped at 30s), and stop at the request deadline (#278).
• ${VAR} substitution is stricter and happens later. Only the braced ${VAR} form is a reference; every other dollar sign is data. The previous shell-style expansion consumed bare dollars, so a word-filter blocked word $100 became 00 — silently weakening a guardrail — and a password like pa$$w0rd was mangled. An undefined variable is now an error rather than a silently empty secret. Substitution also moved from config load to component construction, so the config never carries a materialized secret into the config-history store or a rollback, and references pushed through the admin/GitOps config API — which never passed through the file loader — are now resolved too (#283).
• observability.tracing.enabled is a tri-state (source-breaking for Go API users). TracingConfig.Enabled changed from bool to *bool, so an explicit enabled: false is a hard kill switch even when an endpoint is configured. Omitted infers activation from a configured endpoint or exporter, as before (#284).

Fixed

• A hung provider now trips its circuit breaker. The gateway's own request deadline was indistinguishable from a caller cancelling, so it was excused: a provider that stopped responding never tripped its breaker, stayed in rotation indefinitely, and /readyz — whose only provider signal is circuit state — kept reporting the pod ready. The deadline now carries an explicit cause, so a slow provider is attributed to the provider. Errors the gateway raises itself, such as shedding under its own concurrency limit, never blame the provider.
• AI21 no longer has parameters stripped from Jamba requests. AI21 is dual-path: its advertised Jamba models speak the OpenAI-compatible API, while only the deprecated Jurassic endpoint is parameter-limited. Support is now declared where the Jurassic request is built, so drop mode no longer strips tools and response_format from every Jamba request (#207).
• The concurrency limiter no longer misrepresents a provider's capabilities. Wrapping a provider had forced the wrapper to re-declare every optional interface, which made a chat-only provider look like it supported embeddings, images, and discovery. The limiter now decorates at the call site and exposes only the base interface, so capability detection always sees the real provider.
• The capability drift guard can now fail. It asserted a property that held by construction for any input, including a provider ID the matrix had never heard of — precisely the drift it existed to catch.

Contributors

Thanks to everyone who shipped this release:

• Add context7.json with API gateway details — @MitulShah1
• v1.2.0 - Provider Capability Matrix, bounded requests, plugin failure policy (#347) — @MitulShah1

Acknowledgements

The commit list above is squashed, so it does not show everyone whose work is in this release. Two headline features here were contributed by:

• @gr3enarr0w (Clark Everson) — per-provider concurrency limiting: the worker-pool and bounded-queue design behind targets[].concurrency (#248, #249).
• @Rachit-Gandhi (Rachit Gandhi) — the plugin failure policy that separates a plugin denying a request from a plugin breaking (#330), and uniform ${VAR} substitution across config (#329).

Thank you both — this release is better for it.

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.2.0/CHANGELOG.md

v1.1.22

[1.1.22] — 2026-07-11

The closeout of the v1.1.x hardening release line — streaming lifecycle, configuration validation, persistence consolidation, and provider fixes. One behavior change to note: configuration files are now rejected when they contain unrecognized keys (see Changed). The public Go API is otherwise unchanged.

Added

• Config apiVersion field. An optional, advisory apiVersion (currently v1) can be set at the top of a config file. It is informational: an omitted value defaults to the current version, and an unrecognized value is accepted with a warning for forward compatibility.
• Qwen live model discovery. The Qwen provider can now refresh its model list from the provider's /models endpoint, opt-in via FERRO_MODEL_DISCOVERY_INTERVAL, like the other OpenAI-compatible providers.

Changed

• Configuration is now strictly validated: an unrecognized key fails the load instead of being silently ignored, so a typo like strategy: is reported at startup rather than dropping a whole block. Free-form plugin and exporter config maps still accept any key. JSON configs can no longer carry _-prefixed pseudo-comment keys; the field documentation lives in the YAML example instead.
• All persistent stores share one schema-migration path. The API-key, config, and request-log stores now open, bind placeholders, and migrate through a single internal SQL layer, and each store's schema is versioned through the migration runner (baseline-adopted on databases from earlier releases). Behavior is unchanged; the consolidation removes the per-store SQL variations that previously drifted.

Fixed

• Streaming responses no longer leak a goroutine when a client disconnects mid-stream. Every provider's streaming producer now abandons a pending send once the request context is cancelled, so the producer returns and its upstream HTTP body is closed instead of blocking forever.
• Streaming requests probe a recovering provider's circuit breaker exactly once. The streaming path previously consumed the single half-open probe permit at provider-selection time and again at send time, so a recovering provider could never be retried by a streaming request.
• Streaming target selection now matches non-streaming. The load-balance, conditional, and content-based strategies select the same candidate targets on both paths, so a streaming request can no longer be weighted toward or routed to a target that does not serve the requested model.
• Gemini requests no longer emit a doubled models/ prefix in generateContent URLs when a model id already carries it.
• Bedrock Cohere embedding requests now forward the input_type parameter, so embed calls that depend on it are handled correctly upstream.
• Config history is persisted atomically with the active config. A save now writes the active config and its history record in one transaction, so a crash can no longer leave the stored config ahead of its audit trail (#292).
• A concurrent index build that fails transiently is retried on the next start rather than being recorded as applied and skipped permanently.
• Key-store file permissions are applied to the real file even for file: URI DSNs. The SQLite path resolver now decodes percent-escapes and query suffixes the way SQLite opens the file, so the owner-only restriction always targets the actual database file. The API-key schema also migrates through its own dedicated ledger, keeping its version sequence independent of the other stores that may share a database.

Internal

• Consolidated gateway internals: extracted the plugin hook bus, added a shared target-selection seam between the streaming and non-streaming routers, routed plugin and tool spans through the observability seam, and removed redundant locking around math/rand.
• Enabled a stricter linter set and cleared the findings; added fuzz targets and goroutine-leak coverage for the streaming and parser paths; the release build now gates on the test suite and signs its artifacts.

Contributors

Thanks to everyone who shipped this release:

• v1.1.22 - v1.1.x Hardening Closeout (#340) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.1.22/CHANGELOG.md

v1.1.21

[1.1.21] — 2026-07-10

Key hashing and storage hardening — the fourth phase of the v1.1.x hardening release line. The Go API is unchanged; two admin endpoints change their response values (see Changed).

Security

• API keys are now stored hashed at rest. The gateway persists sha256(key) plus a display form, and looks a key up by its hash. The full secret is returned exactly once, from POST /admin/keys and POST /admin/keys/{id}/rotate, and cannot be recovered afterwards. Existing databases are migrated in place on first start.
• The migration rebuilds the key table rather than dropping the plaintext column, so the secrets are removed from the database file itself. Dropping the column would leave them readable on disk: SQLite retains freed pages until VACUUM, and Postgres DROP COLUMN only updates the catalog. On SQLite the migration additionally vacuums and truncates the write-ahead log.
• Operators who need a hard guarantee that no earlier copy of a key survives — in backups, WAL archives, replicas, or unlinked filesystem blocks — should rotate their keys after upgrading. No schema migration can reach those copies.
• Bootstrap credentials now fail closed. ADMIN_BOOTSTRAP_KEY and ADMIN_BOOTSTRAP_READ_ONLY_KEY are accepted only against a key store that is confirmed empty. Previously a key store that could not be read reported zero keys, which re-opened the bootstrap credentials during a database outage.
• SQLite database files are restricted to owner-only access before any data is written to them, rather than after the schema is initialized. SQLite creates files honoring the process umask, so a database could previously be world-readable for the duration of startup.
• An Authorization: Bearer header with an empty value is no longer matched against the key store.
• The request-logger plugin no longer opens its own database from plugin config. It records through the shared request-log store the gateway builds from REQUEST_LOG_STORE_BACKEND / REQUEST_LOG_STORE_DSN, so no request-supplied value reaches the filesystem. This removes a path where a config submitted over POST/PUT /admin/config could create a file at an arbitrary location.

Added

• Versioned schema migrations. Schema changes now run through a schema_migrations ledger, replacing repeated ALTER TABLE statements whose failures were classified by matching the error message text. Databases created by earlier releases are adopted at the baseline version rather than re-initialized. On Postgres the runner holds an advisory lock for its duration, so several gateway instances sharing one database can start at the same time.

Changed

• GET /admin/keys, /admin/keys/{id} and /admin/keys/usage return the key field as fgw_ab12...cd34, keeping both ends of the secret so an operator can match a key they hold against a listed record. It was previously truncated to a leading fragment.
• GET /admin/logs/stats computes its aggregates in the database instead of scanning up to 5,000 rows into memory, so its counts are now exact for any number of matching entries. The truncated, scan_limit, and available_entries summary fields described the old scan cap and have been removed.
• The request-logger plugin's backend and dsn options are obsolete and ignored (with a startup warning). Persistence now targets the shared request-log store; configure it with REQUEST_LOG_STORE_BACKEND and REQUEST_LOG_STORE_DSN. A deployment that previously set only the plugin's dsn should move that value to REQUEST_LOG_STORE_DSN. This also fixes a case where the admin log views read a different database than the plugin wrote to.

Fixed

• The admin dashboard's key table showed every key as fgw_... because it truncated a value the server had already truncated. Keys are now distinguishable from one another.
• request_logs gains an index on created_at, which serves the log listing's ordering, the retention delete, and the stats time filter. On Postgres it is built with CREATE INDEX CONCURRENTLY so an existing table's writers are not blocked during a rolling restart.
• The bootstrap-key check no longer loads every API key on unauthenticated admin requests.

Contributors

Thanks to everyone who shipped this release:

• v1.1.21 - Key Hashing & Storage Hardening (#334) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.1.21/CHANGELOG.md

v1.1.20

[1.1.20] — 2026-07-09

Streaming deadlines and serving robustness — the third phase of the v1.1.x hardening release line. All changes are additive/behavior-preserving for the public API.

Security

• Long-lived streaming responses on pass-through proxy routes and legacy completions streaming are no longer truncated after 120 seconds. Each client-facing write now carries its own short deadline, and a slow client that stops reading is disconnected instead of blocking a request goroutine.
• Upstream idle bound. Because per-write deadlines take the server's overall write timeout out of play, both streaming paths now cancel the upstream request when it produces no data for 5 minutes. A stalled or trickling provider can no longer hold a connection open indefinitely.
• Browser-facing responses now include Content-Security-Policy and Permissions-Policy headers in addition to the existing baseline security headers. The policy sets script-src 'self', so the admin dashboard no longer relies on inline scripts or inline event handlers.
• Build toolchain pinned to Go 1.25.12, picking up the fix for GO-2026-5856 — an Encrypted Client Hello privacy leak in crypto/tls.

Changed

• Provider auto-registration now warns and skips a single configured provider when its factory fails, allowing the gateway to continue serving with other valid providers. Fatal startup behavior is unchanged for invalid config, store initialization, plugin loading, and other process-level failures. Each skip increments the new gateway_provider_init_failures_total{provider} counter — alert on it, since the gateway stays healthy with the provider missing.
• /health now returns HTTP 503 for degraded or no-provider states while preserving its JSON response body, so load balancer and Kubernetes probes can act on it. /admin/health is bearer-authenticated rather than a probe target and continues to return 200, reporting state in the body.
• ferrogw status and ferrogw doctor distinguish a degraded gateway from an unreachable one and print the reported status.

Fixed

• Panic recovery now returns the gateway's JSON error envelope from the outermost HTTP middleware layer, so panics raised inside the tracing and logging middleware are recovered too. Recovered panics are logged with a stack trace and the request's trace ID.
• Request metrics now bucket unknown or arbitrary model names under the bounded unknown model label on both the rejected and error paths. Previously any request naming an unroutable model minted a new Prometheus series and a permanent metric-handle cache entry.

Contributors

Thanks to everyone who shipped this release:

• v1.1.20 — Streaming Deadlines & Serving Robustness (#333) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.1.20/CHANGELOG.md

v1.1.19

[1.1.19] — 2026-07-09

Provider read bounds and typed errors — the second phase of a multi-phase hardening release line. All changes are additive/behavior-preserving.

Security

• Upstream provider responses are now capped at 50 MiB when read into memory, across every provider, closing a gap where a single oversized upstream response could exhaust gateway memory.

Fixed

• DeepSeek streaming responses now report cache-hit token counts (cache_read_tokens) consistently with non-streaming responses; the streaming path was previously dropping this field.

Internal

• Provider HTTP errors now carry their status code as a typed field, recoverable via errors.As, in addition to the existing formatted message, so retry and circuit-breaker classification no longer depends solely on parsing the status out of the error string. Added a cross-provider conformance test covering upstream error-status recovery.

Contributors

Thanks to everyone who shipped this release:

• v1.1.19 — Provider Read Bounds & Typed Errors (#332) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.1.19/CHANGELOG.md

v1.1.18

[1.1.18] — 2026-07-08

Critical fixes and security hardening — the first of a multi-phase hardening release line. All changes are additive/behavior-preserving: no public config key or Go signature is removed or renamed.

Security

• Response cache no longer shares a single cached response object across concurrent requests hitting the same cache key. Each cache hit, and each response being newly cached, now gets its own independent copy, closing a data race that could occur under concurrent load.
• SQLite-backed stores (API keys, config snapshots, request logs) now restrict their on-disk database file to owner-only read/write (0600) instead of the default umask-derived permissions, which could leave the file group- or world-readable.
• Rate limiting is now enabled by default for every deployment: 20 requests/sec with a burst of 40 per client IP, capped at 100,000 tracked IPs. Set RATE_LIMIT_RPS=0 to opt out; RATE_LIMIT_RPS/RATE_LIMIT_BURST override the defaults. Deployments that already set a custom RATE_LIMIT_RPS without RATE_LIMIT_BURST will now get the default burst (40) rather than a burst equal to their custom rate — set RATE_LIMIT_BURST explicitly to preserve the old behavior. Rate limiting keys on the resolved client IP, so TRUSTED_PROXIES must correctly list any reverse proxy in front of the gateway, or all traffic behind that proxy is limited as a single client.

Internal

• The build toolchain is now pinned to Go 1.25.11 (previously 1.25.0), matching the version already used by CI and the published Docker image, closing a gap where a plain source build could pick up an older, unpinned patch version.

Contributors

Thanks to everyone who shipped this release:

• v1.1.18 — Critical fixes and security hardening (#331) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.1.18/CHANGELOG.md

v1.1.17

[1.1.17] — 2026-07-06

Provider readiness closeout — the eighth and final phase of the provider-readiness remediation. A hygiene and quality release: shared validators, dead-code removal, broad test coverage, and one security hardening. It contains a single behavior change (the Ollama Cloud chat surface), noted under Changed.

Security

• Gemini now authenticates native calls with the x-goog-api-key header instead of the ?key= query parameter. The key was previously part of the request URL, where it could be recorded in request-URL span attributes and proxy access logs. Authentication is otherwise unchanged and the key no longer appears in any request URL.

Changed

• Ollama Cloud chat now uses the OpenAI-compatible /v1/chat/completions and /v1/models endpoints instead of the native /api/chat and /api/tags, recovering full sampling-parameter coverage, normalized finish reasons, and real upstream tool-call IDs. Embeddings remain on the native /api/embed endpoint; the base URL (https://ollama.com) and the OLLAMA_API_KEY environment variable are unchanged. Direction credited to community PR #243.
• Transport presets added for Azure AI Foundry, Ollama Cloud, and Perplexity so large-model and long-running first responses are not aborted by the default 30-second header timeout.

Fixed

• Gemini and Vertex AI image generation now surface the returned safety-filter reason when every prediction is filtered, instead of a generic "no images" error.

Internal

• Shared core.NormalizeEmbeddingInput and core.ValidateEmbeddingEncodingFormat replace duplicated per-provider embedding validators (Azure OpenAI and Databricks keep their stricter local validators by design). Extracted chatParams()/headers() helpers (xAI, Moonshot, OpenRouter, Novita) and a Vertex AI doPredict() helper; OpenAI chat request bodies now use the pooled buffer path. Removed dead code (Bedrock section banners and unused image fields, AI21 Jurassic token detail, a Cloudflare field, a duplicate Together model id) and split the Gemini embedding path into its own file to stay under the file-size limit. Added request-shape, error-path, and shared streaming error-path tests across many providers.

Contributors

Thanks to everyone who shipped this release:

• v1.1.17 - Provider Readiness Closeout (#328) — @MitulShah1

---

Full changelog: https://github.com/ferro-labs/ai-gateway/blob/v1.1.17/CHANGELOG.md

Thanks to @gr3enarr0w for the original ollama-cloud /v1 direction (#243).