ferro-labs
diff --git a/‎AGENTS.md‎
Lines changed: 1 addition & 0 deletions b/‎AGENTS.md‎
Lines changed: 1 addition & 0 deletions
diff --git a/‎CHANGELOG.md‎
Lines changed: 32 additions & 0 deletions b/‎CHANGELOG.md‎
Lines changed: 32 additions & 0 deletions
diff --git a/‎config.example.json‎
Lines changed: 1 addition & 3 deletions b/‎config.example.json‎
Lines changed: 1 addition & 3 deletions
diff --git a/‎config.example.yaml‎
Lines changed: 4 additions & 6 deletions b/‎config.example.yaml‎
Lines changed: 4 additions & 6 deletions
diff --git a/‎gateway.go‎
Lines changed: 33 additions & 3 deletions b/‎gateway.go‎
Lines changed: 33 additions & 3 deletions
diff --git a/‎gateway_test.go‎
Lines changed: 85 additions & 0 deletions b/‎gateway_test.go‎
Lines changed: 85 additions & 0 deletions
diff --git a/‎internal/admin/admin_keys_handlers.go‎
Lines changed: 1 addition & 4 deletions b/‎internal/admin/admin_keys_handlers.go‎
Lines changed: 1 addition & 4 deletions
diff --git a/‎internal/admin/admin_logs_handlers.go‎
Lines changed: 20 additions & 72 deletions b/‎internal/admin/admin_logs_handlers.go‎
Lines changed: 20 additions & 72 deletions
@@ -91,6 +91,7 @@ ai-gateway/
 │   ├── discovery/        # Shared OpenAI-compatible model discovery helper
 │   ├── latency/          # Latency tracking for least-latency strategy
 │   ├── metrics/          # Prometheus metrics
+│   ├── migrations/       # Versioned schema-migration runner (schema_migrations ledger)
 │   ├── otel/             # OTel-backed observability.Provider: OTLP exporter, W3C propagation, trace-ID unifying IDGenerator, privacy-aware span errors, HTTP middleware
 │   ├── redact/           # Error-message redaction policies (email / JWT / AWS key)
 │   ├── plugins/          # Built-in plugin implementations
 
@@ -5,6 +5,38 @@ All notable changes to Ferro Labs AI Gateway are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.21] — 2026-07-10
+
+Key hashing and storage hardening — the fourth phase of the v1.1.x hardening release line. The Go API is unchanged; two admin endpoints change their response values (see **Changed**).
+
+### Security
+
+- **API keys are now stored hashed at rest.** The gateway persists `sha256(key)` plus a display form, and looks a key up by its hash. The full secret is returned exactly once, from `POST /admin/keys` and `POST /admin/keys/{id}/rotate`, and cannot be recovered afterwards. Existing databases are migrated in place on first start.
+- The migration **rebuilds the key table** rather than dropping the plaintext column, so the secrets are removed from the database file itself. Dropping the column would leave them readable on disk: SQLite retains freed pages until `VACUUM`, and Postgres `DROP COLUMN` only updates the catalog. On SQLite the migration additionally vacuums and truncates the write-ahead log.
+- Operators who need a hard guarantee that no earlier copy of a key survives — in backups, WAL archives, replicas, or unlinked filesystem blocks — should **rotate their keys after upgrading**. No schema migration can reach those copies.
+- **Bootstrap credentials now fail closed.** `ADMIN_BOOTSTRAP_KEY` and `ADMIN_BOOTSTRAP_READ_ONLY_KEY` are accepted only against a key store that is confirmed empty. Previously a key store that could not be read reported zero keys, which re-opened the bootstrap credentials during a database outage.
+- **SQLite database files are restricted to owner-only access before any data is written to them**, rather than after the schema is initialized. SQLite creates files honoring the process umask, so a database could previously be world-readable for the duration of startup.
+- An `Authorization: Bearer` header with an empty value is no longer matched against the key store.
+- **The request-logger plugin no longer opens its own database from plugin config.** It records through the shared request-log store the gateway builds from `REQUEST_LOG_STORE_BACKEND` / `REQUEST_LOG_STORE_DSN`, so no request-supplied value reaches the filesystem. This removes a path where a config submitted over `POST/PUT /admin/config` could create a file at an arbitrary location.
+
+### Added
+
+- **Versioned schema migrations.** Schema changes now run through a `schema_migrations` ledger, replacing repeated `ALTER TABLE` statements whose failures were classified by matching the error message text. Databases created by earlier releases are adopted at the baseline version rather than re-initialized. On Postgres the runner holds an advisory lock for its duration, so several gateway instances sharing one database can start at the same time.
+
+### Changed
+
+- `GET /admin/keys`, `/admin/keys/{id}` and `/admin/keys/usage` return the `key` field as `fgw_ab12...cd34`, keeping both ends of the secret so an operator can match a key they hold against a listed record. It was previously truncated to a leading fragment.
+- `GET /admin/logs/stats` computes its aggregates in the database instead of scanning up to 5,000 rows into memory, so its counts are now exact for any number of matching entries. The `truncated`, `scan_limit`, and `available_entries` summary fields described the old scan cap and have been removed.
+- **The request-logger plugin's `backend` and `dsn` options are obsolete and ignored** (with a startup warning). Persistence now targets the shared request-log store; configure it with `REQUEST_LOG_STORE_BACKEND` and `REQUEST_LOG_STORE_DSN`. A deployment that previously set only the plugin's `dsn` should move that value to `REQUEST_LOG_STORE_DSN`. This also fixes a case where the admin log views read a different database than the plugin wrote to.
+
+### Fixed
+
+- The admin dashboard's key table showed every key as `fgw_...` because it truncated a value the server had already truncated. Keys are now distinguishable from one another.
+- `request_logs` gains an index on `created_at`, which serves the log listing's ordering, the retention delete, and the stats time filter. On Postgres it is built with `CREATE INDEX CONCURRENTLY` so an existing table's writers are not blocked during a rolling restart.
+- The bootstrap-key check no longer loads every API key on unauthenticated admin requests.
+
+---
+
 ## [1.1.20] — 2026-07-09
 
 Streaming deadlines and serving robustness — the third phase of the v1.1.x hardening release line. All changes are additive/behavior-preserving for the public API.
 
@@ -181,9 +181,7 @@
       "enabled": true,
       "config": {
         "level": "info",
-        "persist": false,
-        "backend": "sqlite",
-        "dsn": "ferrogw-requests.db"
+        "persist": false
       }
     },
     {
 
@@ -138,13 +138,11 @@ plugins:
     enabled: true
     config:
       level: info
-      # Optional persistent request log storage.
-      # When disabled, logs are emitted only to stdout.
+      # Persist each request to the shared request-log store when one is
+      # configured; otherwise logs are emitted only to stdout. The store's
+      # location is process configuration, set with the REQUEST_LOG_STORE_BACKEND
+      # and REQUEST_LOG_STORE_DSN environment variables — not here.
       persist: false
-      # backend: sqlite | postgres
-      backend: sqlite
-      # SQLite file path or Postgres DSN (required for postgres backend).
-      dsn: ferrogw-requests.db
 
   # Advanced guardrails (pii-redact, secret-scan, prompt-shield, schema-guard,
   # regex-guard) are available in FerroCloud. See https://docs.ferrolabs.ai/guardrails
 
@@ -23,6 +23,7 @@ import (
 	"github.com/ferro-labs/ai-gateway/internal/latency"
 	"github.com/ferro-labs/ai-gateway/internal/mcp"
 	"github.com/ferro-labs/ai-gateway/internal/metrics"
+	"github.com/ferro-labs/ai-gateway/internal/requestlog"
 	"github.com/ferro-labs/ai-gateway/internal/strategies"
 	"github.com/ferro-labs/ai-gateway/models"
 	"github.com/ferro-labs/ai-gateway/observability"
@@ -41,6 +42,7 @@ type Gateway struct {
 	strategy           strategies.Strategy
 	streamingContent   []streamingContentCondition
 	plugins            *plugin.Manager
+	requestLogWriter   requestlog.Writer
 	closeOnce          sync.Once
 	hooks              []EventHookFunc
 	hookSnapshot       atomic.Value
@@ -165,6 +167,21 @@ func (g *Gateway) SetObservability(p observability.Provider) {
 	}
 }
 
+// SetRequestLogWriter installs the shared request-log store that logging
+// plugins write through, instead of each opening its own. Pass the store built
+// from REQUEST_LOG_STORE_BACKEND / REQUEST_LOG_STORE_DSN, or nil to leave
+// logging plugins without a persistence target.
+//
+// Safe to call only at startup, before serving traffic and before LoadPlugins,
+// since the writer is injected into plugins as they are built. The store is
+// owned by the caller (closed on shutdown); the gateway only hands it to
+// plugins.
+func (g *Gateway) SetRequestLogWriter(w requestlog.Writer) {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	g.requestLogWriter = w
+}
+
 // Observability returns the current observability.Provider. Always
 // non-nil; defaults to NoOp.
 func (g *Gateway) Observability() observability.Provider {
@@ -282,7 +299,7 @@ func (g *Gateway) ReloadConfig(ctx context.Context, cfg Config) error {
 	if err != nil {
 		return fmt.Errorf("invalid config: %w", err)
 	}
-	plugins, err := buildPluginManager(cfg.Plugins)
+	plugins, err := g.buildPluginManager(cfg.Plugins)
 	if err != nil {
 		return err
 	}
@@ -327,7 +344,7 @@ func (g *Gateway) LoadPlugins() error {
 	g.mu.RLock()
 	configs := append([]PluginConfig(nil), g.config.Plugins...)
 	g.mu.RUnlock()
-	plugins, err := buildPluginManager(configs)
+	plugins, err := g.buildPluginManager(configs)
 	if err != nil {
 		return err
 	}
@@ -342,7 +359,11 @@ func (g *Gateway) LoadPlugins() error {
 	return nil
 }
 
-func buildPluginManager(configs []PluginConfig) (*plugin.Manager, error) {
+func (g *Gateway) buildPluginManager(configs []PluginConfig) (*plugin.Manager, error) {
+	g.mu.RLock()
+	sharedLogWriter := g.requestLogWriter
+	g.mu.RUnlock()
+
 	plugins := plugin.NewManager()
 	for _, pc := range configs {
 		if !pc.Enabled {
@@ -354,6 +375,15 @@ func buildPluginManager(configs []PluginConfig) (*plugin.Manager, error) {
 			return nil, fmt.Errorf("unknown plugin: %s", pc.Name)
 		}
 		p := factory()
+		// Hand the shared request-log store to plugins that record through it,
+		// before Init so Init can direct persistence at it. Plugins never open
+		// their own store from config, so no request-supplied value reaches the
+		// filesystem.
+		if sharedLogWriter != nil {
+			if r, ok := p.(requestlog.WriterReceiver); ok {
+				r.SetRequestLogWriter(sharedLogWriter)
+			}
+		}
 		if err := p.Init(pc.Config); err != nil {
 			_ = plugins.Close()
 			_ = p.Close()
 
@@ -20,6 +20,7 @@ import (
 	"github.com/ferro-labs/ai-gateway/internal/logging"
 	"github.com/ferro-labs/ai-gateway/internal/metrics"
 	cacheplugin "github.com/ferro-labs/ai-gateway/internal/plugins/cache"
+	"github.com/ferro-labs/ai-gateway/internal/requestlog"
 	"github.com/ferro-labs/ai-gateway/mcp"
 	"github.com/ferro-labs/ai-gateway/models"
 	"github.com/ferro-labs/ai-gateway/plugin"
@@ -2889,6 +2890,90 @@ func init() {
 	plugin.RegisterFactory("test-plugin", func() plugin.Plugin {
 		return &testPlugin{name: "test-plugin", typ: plugin.TypeGuardrail}
 	})
+	plugin.RegisterFactory("test-log-receiver", func() plugin.Plugin {
+		p := &logReceiverPlugin{}
+		lastLogReceiver = p
+		return p
+	})
+}
+
+// lastLogReceiver holds the most recently constructed logReceiverPlugin. The
+// receiver tests below run sequentially and build exactly one, so this is a
+// safe way to reach the instance buildPluginManager created and injected.
+var lastLogReceiver *logReceiverPlugin
+
+// logReceiverPlugin records the writer the gateway injects, to prove
+// buildPluginManager hands the shared request-log store to receiver plugins.
+type logReceiverPlugin struct {
+	injected    requestlog.Writer
+	wasInjected bool
+}
+
+func (p *logReceiverPlugin) Name() string                                   { return "test-log-receiver" }
+func (p *logReceiverPlugin) Type() plugin.PluginType                        { return plugin.TypeLogging }
+func (p *logReceiverPlugin) Init(map[string]any) error                      { return nil }
+func (p *logReceiverPlugin) Execute(context.Context, *plugin.Context) error { return nil }
+func (p *logReceiverPlugin) Close() error                                   { return nil }
+func (p *logReceiverPlugin) SetRequestLogWriter(w requestlog.Writer) {
+	p.injected = w
+	p.wasInjected = true
+}
+
+func loadReceiverPlugin(t *testing.T, gw *Gateway) *logReceiverPlugin {
+	t.Helper()
+	lastLogReceiver = nil
+	if err := gw.LoadPlugins(); err != nil {
+		t.Fatalf("LoadPlugins failed: %v", err)
+	}
+	if lastLogReceiver == nil {
+		t.Fatal("receiver plugin was never constructed")
+	}
+	return lastLogReceiver
+}
+
+// The shared request-log store the gateway holds is injected into a receiver
+// plugin as it loads.
+func TestGateway_InjectsRequestLogWriterIntoPlugins(t *testing.T) {
+	rec := &recordingLogWriter{}
+	gw, _ := New(Config{
+		Strategy: StrategyConfig{Mode: ModeSingle},
+		Targets:  []Target{{VirtualKey: mockProviderName}},
+		Plugins: []PluginConfig{
+			{Name: "test-log-receiver", Type: "logging", Stage: "after_request", Enabled: true},
+		},
+	})
+	gw.SetRequestLogWriter(rec)
+
+	p := loadReceiverPlugin(t, gw)
+	if p.injected != requestlog.Writer(rec) {
+		t.Fatalf("plugin injected writer = %v, want the store set via SetRequestLogWriter", p.injected)
+	}
+}
+
+// Without a shared store, buildPluginManager does not call the receiver at all,
+// so the plugin decides its own fallback rather than being handed nil.
+func TestGateway_NoRequestLogWriter_PluginNotInjected(t *testing.T) {
+	gw, _ := New(Config{
+		Strategy: StrategyConfig{Mode: ModeSingle},
+		Targets:  []Target{{VirtualKey: mockProviderName}},
+		Plugins: []PluginConfig{
+			{Name: "test-log-receiver", Type: "logging", Stage: "after_request", Enabled: true},
+		},
+	})
+
+	p := loadReceiverPlugin(t, gw)
+	if p.wasInjected {
+		t.Fatalf("SetRequestLogWriter was called with %v; want no injection when the gateway has no store", p.injected)
+	}
+}
+
+type recordingLogWriter struct {
+	entries []requestlog.Entry
+}
+
+func (w *recordingLogWriter) Write(_ context.Context, e requestlog.Entry) error {
+	w.entries = append(w.entries, e)
+	return nil
 }
 
 func TestGateway_LoadPlugins(t *testing.T) {
 
@@ -78,11 +78,8 @@ func (h *Handlers) getKey(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	masked := *key
-	masked.Key = maskKey(masked.Key)
-
 	w.Header().Set("Content-Type", "application/json")
-	_ = json.NewEncoder(w).Encode(masked)
+	_ = json.NewEncoder(w).Encode(key)
 }
 
 //nolint:gocyclo // Query parsing + filtering/sorting logic is intentionally centralized for the endpoint.
 
@@ -121,100 +121,48 @@ func (h *Handlers) logsStats(w http.ResponseWriter, r *http.Request) {
 		return
 	}
 
-	baseQuery := requestlog.Query{
-		Limit:    200,
-		Offset:   0,
+	query := requestlog.Query{
 		Stage:    r.URL.Query().Get("stage"),
 		Model:    r.URL.Query().Get("model"),
 		Provider: r.URL.Query().Get("provider"),
 		Since:    since,
 	}
 
-	result, err := h.Logs.List(r.Context(), baseQuery)
+	stats, err := h.Logs.Stats(r.Context(), query)
 	if err != nil {
 		writeError(w, http.StatusInternalServerError, "failed to compute request log stats", "server_error", "internal_error")
 		return
 	}
 
-	entries := make([]requestlog.Entry, 0, min(result.Total, logsStatsMaxScannedEntries))
-	entries = append(entries, result.Data...)
-	for len(entries) < result.Total && len(entries) < logsStatsMaxScannedEntries {
-		baseQuery.Offset = len(entries)
-		next, listErr := h.Logs.List(r.Context(), baseQuery)
-		if listErr != nil {
-			writeError(w, http.StatusInternalServerError, "failed to compute request log stats", "server_error", "internal_error")
-			return
-		}
-		if len(next.Data) == 0 {
-			break
-		}
-		remaining := logsStatsMaxScannedEntries - len(entries)
-		if remaining <= 0 {
-			break
-		}
-		if len(next.Data) > remaining {
-			next.Data = next.Data[:remaining]
-		}
-		entries = append(entries, next.Data...)
-	}
-	truncated := len(entries) < result.Total
-
-	byStage := map[string]int{}
-	byProvider := map[string]int{}
-	byModel := map[string]int{}
-	errorCount := 0
-	tokens := 0
-	for _, entry := range entries {
-		stage := entry.Stage
-		if stage == "" {
-			stage = unknownLabel
-		}
-		byStage[stage]++
-
-		provider := entry.Provider
-		if provider == "" {
-			provider = unknownLabel
-		}
-		byProvider[provider]++
-
-		model := entry.Model
-		if model == "" {
-			model = unknownLabel
-		}
-		byModel[model]++
-
-		if entry.ErrorMessage != "" || stage == "on_error" {
-			errorCount++
-		}
-		tokens += entry.TotalTokens
-	}
-
-	byProvider = limitCounts(byProvider, limit)
-	byModel = limitCounts(byModel, limit)
-
 	w.Header().Set("Content-Type", "application/json")
 	_ = json.NewEncoder(w).Encode(map[string]any{
 		"summary": map[string]any{
-			"total_entries":     len(entries),
-			"error_entries":     errorCount,
-			"total_tokens":      tokens,
-			"truncated":         truncated,
-			"available_entries": result.Total,
-			"scan_limit":        logsStatsMaxScannedEntries,
+			"total_entries": stats.TotalEntries,
+			"error_entries": stats.ErrorEntries,
+			"total_tokens":  stats.TotalTokens,
 		},
-		"by_stage":    byStage,
-		"by_provider": byProvider,
-		"by_model":    byModel,
+		"by_stage":    nonNilCounts(stats.ByStage),
+		"by_provider": nonNilCounts(limitCounts(stats.ByProvider, limit)),
+		"by_model":    nonNilCounts(limitCounts(stats.ByModel, limit)),
 		"filters": map[string]any{
 			"limit":    limit,
-			"stage":    baseQuery.Stage,
-			"model":    baseQuery.Model,
-			"provider": baseQuery.Provider,
+			"stage":    query.Stage,
+			"model":    query.Model,
+			"provider": query.Provider,
 			"since":    r.URL.Query().Get("since"),
 		},
 	})
 }
 
+// nonNilCounts keeps an absent dimension encoding as {} rather than null, which
+// clients index into without checking.
+func nonNilCounts(input map[string]int) map[string]int {
+	if input == nil {
+		return map[string]int{}
+	}
+	return input
+}
+
 func limitCounts(input map[string]int, limit int) map[string]int {
 	if limit <= 0 || len(input) <= limit {
 		return input
Original file line number	Diff line number	Diff line change
`@@ -181,9 +181,7 @@`
`181`	`181`	`"enabled": true,`
`182`	`182`	`"config": {`
`183`	`183`	`"level": "info",`
`184`		`- "persist": false,`
`185`		`- "backend": "sqlite",`
`186`		`- "dsn": "ferrogw-requests.db"`
	`184`	`+ "persist": false`
`187`	`185`	`}`
`188`	`186`	`},`
`189`	`187`	`{`
Original file line number	Diff line number	Diff line change
`@@ -78,11 +78,8 @@ func (h Handlers) getKey(w http.ResponseWriter, r http.Request) {`
`78`	`78`	`return`
`79`	`79`	`}`
`80`	`80`
`81`		`- masked := *key`
`82`		`- masked.Key = maskKey(masked.Key)`
`83`		`-`
`84`	`81`	`w.Header().Set("Content-Type", "application/json")`
`85`		`- _ = json.NewEncoder(w).Encode(masked)`
	`82`	`+ _ = json.NewEncoder(w).Encode(key)`
`86`	`83`	`}`
`87`	`84`
`88`	`85`	`//nolint:gocyclo // Query parsing + filtering/sorting logic is intentionally centralized for the endpoint.`