feat: console packaging: image + Helm/k8s/compose + deploy-lint CI (#363) (#489)

* feat: console packaging: image + Helm/k8s/compose artifacts + deploy-lint CI (#363)

Ships the reference PACKAGING for the stateless console, and the CI gate that
validates it (no docker/helm/kubectl was available locally, so the lint job is
the validation oracle -- the same CI-as-oracle pattern as the redis differential
test).

- Dockerfile.console: the SAME static-musl, distroless/nonroot recipe as the
  cache Dockerfile, staging `ironcache-console` (no data VOLUME; sets
  IRONCACHE_CONSOLE_HTTP_ADDR=0.0.0.0:9180 so a kubelet/LB probe can reach the
  otherwise-loopback listener; CMD ["run"]).
- Helm: an OPTIONAL stateless console (console.enabled, default off) rendering a
  Deployment + Service + (2+ replicas) PodDisruptionBudget, with a `console:`
  values block and console.* label/selector helpers (component=console, so the
  Deployment selector never collides with the StatefulSet's). Chart 0.1.0 -> 0.2.0.
- deploy/k8s/ironcache-console.yaml: the same as raw manifests (Secret +
  Deployment + Service + PDB).
- deploy/compose/docker-compose.console.yml + config/console-nginx.conf: two
  stateless replicas behind an nginx LB, as an overlay onto a cache compose file.
- .github/workflows/deploy-lint.yml: a NEW gate that runs on any PR touching
  deploy/** or a Dockerfile -- `helm lint` + `kubeconform` on the rendered chart
  (defaults AND console.enabled=true), `kubeconform` on the raw manifests,
  `hadolint` on both Dockerfiles, and `docker compose config` on the compose
  files. It also guards the EXISTING server manifests from drift.

All console manifests are stateless (no PVC/identity/headless service), match the
server's securityContext/anti-affinity/probe conventions, and wire the console's
real interface (port 9180; /livez + /readyz; IRONCACHE_CONSOLE_SEEDS /
_PROMETHEUS_URL / _NODE_USER / _NODE_PASSWORD_FILE / _READ_TOKEN). DEPLOY.md
documents deploying the console in each form.

NOTE: the release.yml/image.yml entry that actually BUILDS + publishes the console
image is tag-triggered (not PR-CI-validatable), so it is a deliberate follow-up.

🤖 Generated with Claude Code
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: ELares <zeke@butlr.io>

* ci: pass a placeholder cluster secret to the compose lint step (#363)

`docker compose config` fails on docker-compose.cluster.yml because its
${IRONCACHE_CLUSTER_SECRET:?} is a fail-fast required var; deploy-lint checks
file STRUCTURE not real secrets, so set a placeholder in the step env.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ELares <zeke@butlr.io>

* fix: give the console a distinct app name to end the cache selector collision (#363)

Two independent adversarial reviews found the same real defect: the cache-side
Service / PDB / ServiceMonitor / StatefulSet selectors are `{name, instance}`
only, and the console pods carried `{name, instance, ..., component=console}` --
a SUPERSET -- so the cache selectors ALSO matched the console pods. Worst case:
the cache PDB (which selects pods with no port filter) formed a cross-controller
disruption budget spanning the StatefulSet AND the console Deployment, which can
corrupt eviction accounting and block a cache node drain; the cache Services were
saved only incidentally by a named-port mismatch. (Both reviews also confirmed the
opposite, more-worrying axis is fine: the console does ZERO runtime filesystem
writes -- the only fs writes are in #[cfg(test)] -- so readOnlyRootFilesystem:true
with no writable volume is correct and needs no emptyDir /tmp.)

Fix: the console now uses a DISTINCT `app.kubernetes.io/name` (`<name>-console`)
instead of the cache's `<name>`, so its labels are no longer a superset of the
cache selector and the cache selectors cannot match console pods -- in EVERY
direction (PDB, Services, ServiceMonitor, StatefulSet). This is done entirely on
the console side, so it does NOT change (and break) the cache's immutable
StatefulSet selector, keeping `helm upgrade` from 0.1.0 clean. Applied in both the
Helm console helpers and the raw ironcache-console.yaml.

Also hardened deploy-lint: the helm-template job now asserts that
`console.enabled=true` actually RENDERS the console resources (kubeconform is
happy with zero objects, so a future broken `console.enabled` gate would otherwise
pass green).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: ELares <zeke@butlr.io>

---------

Signed-off-by: ELares <zeke@butlr.io>
Co-authored-by: ELares <zeke@butlr.io>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: 3 people

.github/workflows/deploy-lint.yml

@@ -0,0 +1,139 @@
+# SPDX-License-Identifier: MIT OR Apache-2.0
+#
+# Deploy-artifact lint gate (#363). The repo ships production deployment artifacts
+# (the Helm chart, the raw k8s manifests, the docker-compose files, the two
+# Dockerfiles) that no other workflow validates: image.yml only runs on `v*` tags,
+# and rust.yml lints Rust, not YAML. This gate lints them on every PR that TOUCHES
+# them (the `paths:` filter), so a malformed chart / manifest / Dockerfile is caught
+# before it ships rather than when an operator first applies it.
+#
+# It renders the chart (default AND with the console enabled) and validates the
+# output against the Kubernetes schemas, validates the raw manifests the same way,
+# and hadolints both Dockerfiles. Third-party actions are tag-pinned to match the
+# CI-workflow convention (rust.yml / docs.yml use `@v4`-style tags; the release/
+# image workflows use SHA pins).
+name: deploy-lint
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    paths:
+      - "deploy/**"
+      - "Dockerfile"
+      - "Dockerfile.console"
+      - ".github/workflows/deploy-lint.yml"
+
+permissions:
+  contents: read
+
+env:
+  # Validate against a recent stable Kubernetes API surface.
+  KUBE_VERSION: "1.30.0"
+  KUBECONFORM_VERSION: "0.6.7"
+  HELM_VERSION: "v3.15.4"
+
+jobs:
+  helm:
+    name: helm lint + kubeconform (rendered chart)
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - uses: azure/setup-helm@v4
+        with:
+          version: ${{ env.HELM_VERSION }}
+      - name: helm lint (defaults + console enabled)
+        run: |
+          set -euo pipefail
+          helm lint deploy/helm/ironcache
+          helm lint deploy/helm/ironcache \
+            --set console.enabled=true --set console.replicas=2
+      - name: Install kubeconform
+        run: |
+          set -euo pipefail
+          curl -fsSL \
+            "https://github.com/yannh/kubeconform/releases/download/v${KUBECONFORM_VERSION}/kubeconform-linux-amd64.tar.gz" \
+            | tar -xz kubeconform
+          sudo install -m 0755 kubeconform /usr/local/bin/kubeconform
+          kubeconform -v
+      - name: helm template | kubeconform (both value sets)
+        run: |
+          set -euo pipefail
+          # -ignore-missing-schemas skips CRDs (ServiceMonitor) that have no core
+          # schema; -strict rejects unknown fields in the core resources.
+          for extra in "" "--set console.enabled=true --set console.replicas=2"; do
+            echo "::group::helm template ${extra:-(defaults)}"
+            rendered="$(helm template ic deploy/helm/ironcache $extra)"
+            printf '%s\n' "$rendered" \
+              | kubeconform -strict -summary -verbose \
+                  -ignore-missing-schemas \
+                  -kubernetes-version "${KUBE_VERSION}"
+            if [ -n "$extra" ]; then
+              # Guard against a broken `console.enabled` gate silently rendering
+              # nothing (kubeconform is happy with zero console objects, so assert
+              # the console Deployment/Service actually appeared).
+              printf '%s\n' "$rendered" | grep -q "ic-ironcache-console" \
+                || { echo "ERROR: console.enabled=true rendered no console resources"; exit 1; }
+            fi
+            echo "::endgroup::"
+          done
+
+  k8s:
+    name: kubeconform (raw manifests)
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install kubeconform
+        run: |
+          set -euo pipefail
+          curl -fsSL \
+            "https://github.com/yannh/kubeconform/releases/download/v${KUBECONFORM_VERSION}/kubeconform-linux-amd64.tar.gz" \
+            | tar -xz kubeconform
+          sudo install -m 0755 kubeconform /usr/local/bin/kubeconform
+      - name: kubeconform deploy/k8s
+        run: |
+          set -euo pipefail
+          kubeconform -strict -summary -verbose \
+            -ignore-missing-schemas \
+            -kubernetes-version "${KUBE_VERSION}" \
+            deploy/k8s/*.yaml
+
+  docker:
+    name: hadolint (Dockerfiles)
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - name: hadolint Dockerfile
+        uses: hadolint/hadolint-action@v3.1.0
+        with:
+          dockerfile: Dockerfile
+      - name: hadolint Dockerfile.console
+        uses: hadolint/hadolint-action@v3.1.0
+        with:
+          dockerfile: Dockerfile.console
+
+  compose:
+    name: docker compose config (validate)
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - name: Validate every compose file parses + resolves
+        env:
+          # The cluster compose uses a fail-fast required var (${IRONCACHE_CLUSTER_SECRET:?});
+          # deploy-lint only checks the file STRUCTURE, not real secrets, so a
+          # placeholder lets `config` interpolate. Add more here if a compose file
+          # introduces another required var.
+          IRONCACHE_CLUSTER_SECRET: deploy-lint-placeholder
+        run: |
+          set -euo pipefail
+          for f in deploy/compose/docker-compose*.yml; do
+            echo "::group::docker compose -f $f config"
+            # `config` fully parses + interpolates + schema-checks the file; -q
+            # suppresses the rendered output, leaving only errors.
+            docker compose -f "$f" config -q
+            echo "::endgroup::"
+          done

CHANGELOG.md

@@ -169,8 +169,16 @@ release.
   a different `/api/timeseries` window (the fix is a shared `prometheus_url`); the console is otherwise
   already stateless (header-token auth, independently-polled topology, per-replica `/livez` + `/readyz`
   probes) and safe to run as N replicas behind an LB. DEPLOY.md documents the stateless-behind-a-load-
-  balancer HA topology. The reference container image + Helm/k8s manifests are tracked follow-up
-  packaging work under #363.
+  balancer HA topology.
+- Console packaging (issue #363): a container image (`Dockerfile.console`, the same static-musl,
+  distroless/nonroot recipe as the cache image) plus reference deployment artifacts for the stateless
+  console: an optional Helm Deployment + Service + PDB (`console.enabled=true`), a raw
+  `deploy/k8s/ironcache-console.yaml`, and a `deploy/compose/docker-compose.console.yml` overlay (two
+  replicas behind an nginx LB). A new `deploy-lint` CI workflow validates all of it on every PR that
+  touches it (`helm lint` + `kubeconform` on the rendered chart and raw manifests, `hadolint` on both
+  Dockerfiles, `docker compose config` on the compose files), so the manifests cannot silently rot.
+  DEPLOY.md documents deploying the console in each form. (The tag-triggered release/image-publish
+  entry that BUILDS the console image is a tracked follow-up.)
 - Reference least-privilege console aclfile (issue #367): `deploy/aclfile.console.example` ships two
   scoped IronCache ACL users the console authenticates as when it dials nodes: `console_monitor`
   (read-only, exactly the poll loop's PING/INFO/CLIENT LIST, no key access, no mutation) and

DEPLOY.md

@@ -3,11 +3,20 @@
 
 This guide covers the production deployment artifacts that ship in this repo:
 
-- A multi-stage, non-root container image (`Dockerfile`), published to GHCR by
+- Two multi-stage, non-root container images (`Dockerfile` for the cache,
+  `Dockerfile.console` for the console), published to GHCR by
   `.github/workflows/image.yml`.
-- `deploy/compose/` -- single-node and 3-node Raft cluster docker-compose files.
-- `deploy/helm/ironcache/` -- a Helm chart deploying a StatefulSet.
-- `deploy/k8s/` -- the same StatefulSet as raw, Helm-free YAML.
+- `deploy/compose/` -- single-node and 3-node Raft cluster docker-compose files,
+  plus `docker-compose.console.yml` (two stateless console replicas behind an LB).
+- `deploy/helm/ironcache/` -- a Helm chart deploying the cache StatefulSet and an
+  optional stateless console Deployment (`console.enabled=true`).
+- `deploy/k8s/` -- the same StatefulSet as raw, Helm-free YAML, plus
+  `ironcache-console.yaml` (the console Deployment + Service).
+
+The deploy artifacts are lint-gated on every PR that touches them
+(`.github/workflows/deploy-lint.yml`: `helm lint` + `kubeconform` on the rendered
+chart and the raw manifests, `hadolint` on both Dockerfiles, `docker compose
+config` on the compose files).
 
 It maps every knob to the REAL config key, explains the ports, and is honest
 about what was validated offline versus what needs a live cluster to confirm.
@@ -204,6 +213,28 @@ kubectl create namespace ironcache
 kubectl -n ironcache apply -f deploy/k8s/ironcache.yaml
 ```
 
+### The console (optional, stateless)
+
+The monitoring/management console (epic #352) is a SEPARATE, stateless workload,
+off by default. It is a Deployment of N identical replicas behind a Service (no
+PVC, no per-pod identity), so it scales horizontally and any replica serves any
+request; see "Console HA" under section 7 for the statelessness details.
+
+- **Helm:** enable it in the same release with `--set console.enabled=true`
+  (tune `console.replicas`, `console.seeds`, `console.prometheusUrl`, and the
+  `console.nodePasswordSecret` / `console.tokensSecret` references). It renders a
+  Deployment + Service + (at 2+ replicas) a PodDisruptionBudget.
+- **Raw manifests:** `kubectl -n ironcache apply -f deploy/k8s/ironcache-console.yaml`
+  (edit the Secret placeholders + the seeds/Prometheus env first).
+- **docker-compose:** overlay `docker-compose.console.yml` (two replicas + an nginx
+  LB) onto a cache compose file.
+
+In every form: point the console at the cache client Service/nodes
+(`IRONCACHE_CONSOLE_SEEDS`), a SHARED Prometheus for consistent history
+(`IRONCACHE_CONSOLE_PROMETHEUS_URL`), and the least-privilege node user
+(`console_monitor`, section 6); set a read token so the privileged API is not open;
+and keep the console Service behind a VPN-locked LB (#369).
+
 ---
 
 ## 6. Enabling auth and TLS

Dockerfile.console

@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: MIT OR Apache-2.0
+#
+# Production container image for the IronCache CONSOLE (epic #352, issue #363).
+#
+# The console is a SEPARATE, STATELESS monitoring/management server from the
+# `ironcache` data-plane binary; it discovers a deployment, aggregates a
+# cluster-wide view, and serves a dashboard + `/api/*`, staying OUT of the client
+# data path. This image mirrors the server image (Dockerfile): it STAGES the SAME
+# fully-static musl `ironcache-console` binary the release pipeline builds (the
+# `*-unknown-linux-musl` targets, `crt-static`, no libc dependency) onto a minimal,
+# nonroot, distroless runtime. There is no Rust toolchain in either stage, so the
+# final image is tiny and cannot drift from the released, attested artifact. The
+# dashboard assets (HTML/CSS/JS/fonts) are compiled INTO the binary (include_str!/
+# include_bytes!), so nothing else needs staging. Unlike the server image there is
+# NO data VOLUME (the console holds no durable state).
+#
+# Local / manual multi-arch build (after the release tarballs are unpacked under
+# dist/, see .github/workflows/image.yml for the exact layout):
+#   docker buildx build --platform linux/amd64,linux/arm64 \
+#     -f Dockerfile.console -t ghcr.io/elares/ironcache-console:dev --load .
+#
+# Port:
+#   9180  the single HTTP listener: /livez + /readyz + /metrics + the UI + /api/*
+#         (IRONCACHE_CONSOLE_HTTP_ADDR). It defaults to loopback, so this image
+#         sets it to 0.0.0.0:9180 below or a kubelet/LB probe could not reach it.
+
+# --- stage: take the prebuilt static musl binary for the target arch ----------
+# `alpine` is a throwaway staging filesystem to chmod the binary; it never reaches
+# the final image. TARGETARCH (amd64 | arm64) is set by buildx, so one Dockerfile
+# serves both arches.
+FROM alpine:3 AS stage
+ARG TARGETARCH
+WORKDIR /stage
+# dist/amd64/ironcache-console and dist/arm64/ironcache-console are the fully-static
+# musl binaries unpacked from the release tarballs by the image-publish CI.
+COPY dist/${TARGETARCH}/ironcache-console /stage/ironcache-console
+RUN chmod 0755 /stage/ironcache-console
+
+# --- final: distroless static, nonroot, least-privilege -----------------------
+# gcr.io/distroless/static:nonroot ships CA certs + a nonroot user (uid/gid 65532)
+# and NOTHING else (no shell, no package manager, no libc): the minimal, secure
+# runtime for a static binary.
+FROM gcr.io/distroless/static:nonroot
+
+# OCI image metadata. The version label is overwritten at build time by the CI
+# `--label org.opencontainers.image.version=<tag>`.
+LABEL org.opencontainers.image.title="ironcache-console"
+LABEL org.opencontainers.image.description="The IronCache monitoring + management console (stateless)."
+LABEL org.opencontainers.image.source="https://github.com/ELares/IronCache"
+LABEL org.opencontainers.image.licenses="MIT OR Apache-2.0"
+
+COPY --from=stage /stage/ironcache-console /usr/local/bin/ironcache-console
+
+# Run as the distroless nonroot user (uid 65532), never root. The console needs no
+# writable filesystem (stateless), so it pairs with readOnlyRootFilesystem: true.
+USER 65532:65532
+
+# Document the single HTTP port (EXPOSE is metadata only; it does not publish).
+EXPOSE 9180
+
+# Bind all interfaces so a kubelet probe / load balancer can reach the listener
+# (the binary defaults to loopback 127.0.0.1:9180, which is unreachable in a
+# container). Mirrors how the server image sets IRONCACHE_DATA_DIR. Override any
+# other knob via IRONCACHE_CONSOLE_* env vars or a mounted TOML.
+ENV IRONCACHE_CONSOLE_HTTP_ADDR=0.0.0.0:9180
+
+ENTRYPOINT ["/usr/local/bin/ironcache-console"]
+CMD ["run"]

deploy/compose/config/console-nginx.conf

@@ -0,0 +1,29 @@
+# SPDX-License-Identifier: MIT OR Apache-2.0
+#
+# nginx reverse proxy / load balancer in front of the two stateless IronCache
+# console replicas (deploy/compose/docker-compose.console.yml, issue #363). Any
+# request may go to either replica (the console is stateless), and nginx marks a
+# replica down + retries the other on a connection error, so a dead replica is
+# transparently routed around. This is the HTTP health surface for the compose
+# topology (the distroless console image has no curl for a container healthcheck).
+
+upstream ironcache_console {
+    server console-1:9180 max_fails=3 fail_timeout=10s;
+    server console-2:9180 max_fails=3 fail_timeout=10s;
+}
+
+server {
+    listen 9180;
+
+    location / {
+        proxy_pass http://ironcache_console;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        # Retry the other replica on a connection-level failure (a dead pod), so a
+        # single replica loss is invisible to the client.
+        proxy_next_upstream error timeout http_502 http_503 http_504;
+        proxy_connect_timeout 2s;
+    }
+}

deploy/compose/docker-compose.console.yml

@@ -0,0 +1,55 @@
+# SPDX-License-Identifier: MIT OR Apache-2.0
+#
+# The IronCache CONSOLE as TWO stateless replicas behind an nginx load balancer
+# (epic #352, issue #363). This is an OVERLAY: run it alongside a cache (the
+# single-node docker-compose.yml or the 3-node docker-compose.cluster.yml), e.g.
+#
+#   docker compose \
+#     -f docker-compose.cluster.yml \
+#     -f docker-compose.console.yml up -d
+#   curl localhost:9180/readyz          # served by whichever replica the LB picks
+#
+# The two replicas are IDENTICAL and stateless (Bearer-header auth, per-replica
+# derived topology, per-replica readiness), so the LB can route any request to
+# either and transparently drop a dead one. History is kept consistent by pointing
+# every replica at ONE shared Prometheus (IRONCACHE_CONSOLE_PROMETHEUS_URL); leave
+# it unset only for a single replica. Set IRONCACHE_CONSOLE_SEEDS to the cache
+# node(s) and IRONCACHE_CONSOLE_READ_TOKEN to gate the privileged API.
+#
+# Note: the console image is distroless (no shell/curl), so there is no per-container
+# HTTP healthcheck; the nginx LB is the HTTP health surface (it proxies /readyz).
+
+name: ironcache-console
+
+# The common console replica config (both replicas are identical).
+x-console: &console
+  image: ghcr.io/elares/ironcache-console:latest
+  command: [run]
+  environment:
+    IRONCACHE_CONSOLE_HTTP_ADDR: 0.0.0.0:9180
+    # Point at the cache node(s). Defaults to the single-node compose service name.
+    IRONCACHE_CONSOLE_SEEDS: ${IRONCACHE_CONSOLE_SEEDS:-ironcache:6379}
+    # A SHARED Prometheus keeps history consistent across the two replicas.
+    IRONCACHE_CONSOLE_PROMETHEUS_URL: ${IRONCACHE_CONSOLE_PROMETHEUS_URL:-}
+    # Dial nodes as the least-privilege user (deploy/aclfile.console.example).
+    IRONCACHE_CONSOLE_NODE_USER: ${IRONCACHE_CONSOLE_NODE_USER:-console_monitor}
+    IRONCACHE_CONSOLE_NODE_PASSWORD_FILE: ${IRONCACHE_CONSOLE_NODE_PASSWORD_FILE:-}
+    # Set a read token so the privileged API is not open behind the LB.
+    IRONCACHE_CONSOLE_READ_TOKEN: ${IRONCACHE_CONSOLE_READ_TOKEN:-}
+  restart: unless-stopped
+
+services:
+  console-1:
+    <<: *console
+  console-2:
+    <<: *console
+  console-lb:
+    image: nginx:1.27-alpine
+    ports:
+      - "9180:9180"
+    volumes:
+      - ./config/console-nginx.conf:/etc/nginx/conf.d/default.conf:ro
+    depends_on:
+      - console-1
+      - console-2
+    restart: unless-stopped

deploy/helm/ironcache/Chart.yaml

@@ -8,7 +8,8 @@ description: >-
   preserves Raft quorum.
 type: application
 # The chart version. Bump on chart changes (independent of the app version).
-version: 0.1.0
+# 0.2.0: adds the optional stateless console Deployment (console.enabled, #363).
+version: 0.2.0
 # The default IronCache image tag this chart targets. Override via
 # image.tag in values.yaml. Quoted so a calendar version is not parsed as a float.
 appVersion: "latest"