sbob: ship Pixie's own ContainerProfiles, with the network allowlists that keep them quiet - #103
Open
ConstanzeTU wants to merge 184 commits into
Open
sbob: ship Pixie's own ContainerProfiles, with the network allowlists that keep them quiet#103ConstanzeTU wants to merge 184 commits into
ConstanzeTU wants to merge 184 commits into
Conversation
…e fix) Root cause of the flaky dx-steered capture (dc_snoop/http erratically 0 while light tables always land): OrderExportAll fans out ~20 tables concurrently, each OrderQuery issued ONE unbounded PxL query over the full ~600s control window against the single node-local PEM (pem-direct). QueryFor only set start_time, so every query re-scanned [sliceStart, now] and post-filtered — the heavy tables materialize huge result sets on a saturated PEM and lose the fixed 180s deadline race, dropping out; the cheap tables (redis/conn/stack) return instantly and survive. Reconcile fingerprint: the same dc_snoop query returns 2459 rows in isolation but 0 + 1 err under the fan-out. Fix (durable — removes the data-volume↔deadline coupling, not just tunes it): - pxl.QueryFor: bound the PEM source scan on BOTH sides. Emit a relative end_time (floored toward now so nothing real is clipped; the exact upper bound stays enforced by the df.time_ < sliceEnd nanos post-filter) whenever sliceEnd is in the past. Live-edge slices keep scanning to now (no end_time), preserving prior behavior for the most-recent window. - controller.OrderQuery: walk the capture window in OrderChunk-sized sub-windows (default 60s, env ADAPTIVE_ORDER_CHUNK_SEC), each a both-sides bounded query, so no single query re-materializes the whole window. captureSpan adaptively halves any chunk that still fails with a transient (deadline/overload) error down to orderMinChunk (1s); non-transient errors (missing dark table) surface immediately without wasteful splitting. Overlapping/retried spans dedupe in the ReplacingMergeTree evidence tables, so re-pulls are idempotent. One aggregated reconcile row per table (not per chunk). Chunks run sequentially per table, so OrderExportAll's per-table concurrency is unchanged while each table now issues cheap bounded queries instead of one firehose — reliable capture without needing the global inflight throttle set. Tests: queryfor end_time present for past windows / absent at the live edge; OrderQuery chunking, single aggregated reconcile row, adaptive subdivision on transient error, no-split on non-transient error, termination at min-chunk.
… (dc_snoop) The dx-steered OrderExportAll path applied only a partial comm denylist and NO namespace filter to the node-scoped dark-vector tables — unlike the shipped cron preset (script/presets dc_snoop.pxl __DC_SNOOP_EXCLUSION__, built from presets.go defaultExcludeNamespaces + defaultExcludeComms). So every dc_snoop capture drowned in infra dcache churn: on a real k3s node a single window returned ~54k rows dominated by ConfigReloader/iptables/CNI(host-local,bridge,flannel,loopback)/host daemons(systemd-udevd,dbus-daemon,tailscaled)/kubevuln — burying the salient attack specimens (whoami/cat/getent reading /etc/shadow + the SA token). - Extend darkExcludeCommsDefault with the host/CNI/node daemons that were leaking (systemd-udevd, host-local, bridge, flannel, loopback, bandwidth, dbus-daemon, mount, umount, tailscaled, grpc_health_pro, kubevuln, opm, kube-proxy, …). - Add darkExcludeNamespacesDefault + darkNamespaceExclusion(), applied in the IsDarkVector branch AFTER PodEnrichPxL resolves df.namespace, dropping infra namespaces (pl, kube-system, clickhouse, …). Blank-namespace transient rows survive (each `!=` is true for ''), so the attack's short-lived children — which resolve blank — are never dropped. Overridable via DC_SNOOP_EXCLUDE_NAMESPACES. Kept in sync with script/presets.go. Tests: infra namespaces + host/CNI comms dropped; df.namespace never pinned to the alert pod (node-scoped); env override replaces the default list.
… depth cap) Live RCA on aeprod54: the chunk fix is correct in isolation (pem unit suite — dc_snoop 54k, redis/conn/stack written per-chunk) but UNSAFE under the dx steering firehose. dx does generic collect-per-alert, so OrderExportAll (20 tables) fires on every noisy pl system pod continuously; all land on the ONE node-local PEM (pem-direct) → it saturates → 100% DeadlineExceeded. captureSpan then split every timeout into two narrower retries, amplifying a busy PEM into a query storm where nothing completes (observed: "0 ordered pixie rows written" across the whole run; draining dx + restarting AE → pem-direct instantly serves again). Make subdivision safe: - Circuit-breaker: orderTimeoutStreak (atomic) counts CONSECUTIVE transient failures; any success resets it. Above orderBreakerTrip (8) captureSpan stops subdividing — a saturated PEM must not be flooded with retries. It still splits a genuinely-oversized window on a healthy PEM (the reset keeps that path live). - Depth cap: maxOrderSplitDepth (3) bounds one chunk to ≤2^3 leaf queries even if it keeps timing out (was ~64 splitting 60s→1s). Tests: a 10-chunk all-timeout window stays <60 queries (ungated ≈640); a single transient failure still recovers (breaker resets on success, no latch). NOTE (deployment, not code): the firehose root also needs dx steering scoped so it doesn't fire 20-table captures on every noisy pl/system-pod alert — tracked separately for dx-agent.
Live RCA (aeprod55): every dx-steered capture in the e2e returned 0 rows, and the reconcile showed why — all 36 ordered captures had ~512ns-wide windows (width_s=0), so they matched no pixie rows. /export/start already reaches back controlExportLookback, but a control client that keys the /query window on a single finding's event_time sends lo≈hi (a sub-microsecond span). That passes the lo<hi validation yet captures nothing. handleQuery now widens any window narrower than minControlQueryWindow (5s) to controlExportLookback ending at hi — a point-in-time referral still captures the evidence leading up to it. hi is preserved; comfortably-wide windows pass through unchanged. Isolated /query probes (proper windows) already proved the capture path works — dc_snoop 54k→16k filtered, redis/conn/stack per-chunk; this makes the dx-driven path robust to degenerate windows too. Tests: a 512ns window is widened to >=5s (hi preserved); a 120s window is untouched. NOTE (dx-agent): dx should send a real window (or use /export/start) rather than a point window per finding — tracked separately. This is the AE-side safety net.
The bootstrap manifest was a replicas:0 Deployment with minimal env (EXPORT_MODE= auto, no pem-direct, no throttle) — it never ran and could not do node-local pem-direct. Replace it with the working config that the e2e RCA validated: - DaemonSet (one-per-node) so each pod queries its OWN node's vizier-pem at HOST_IP:50305 (pem-direct: node-local, desync-immune). - dx-steered: EXPORT_MODE=never + CONTROL_ADDR=:9100 + the control Service (internalTrafficPolicy:Local so dx reaches its co-located AE). - PEM-protection: ADAPTIVE_MAX_INFLIGHT_QUERIES_GLOBAL=4 and ADAPTIVE_ORDER_CHUNK_SEC =600 (one query per table, no window pre-chunking) so the AE never saturates the single node-local PEM it shares with dx. See RCA_ae_capture_20260803. Secret still seeded per-cluster (unchanged).
…efault; trim comments - queryfor.go: add darkExcludeCommSubstrings (kworker/ksoftirqd/rcu_/… — kernel threads with variable suffixes exact-match misses) applied via px.logicalNot( px.contains); add pause + systemd-logind exact. Workload comms (redis-*) untouched. - controller.go: defaultOrderChunk 60s -> 600s (one query per table; pre-chunking 10x-amplified queries on the single node-local PEM). - Strip verbose comments across queryfor.go/controller.go/server.go + the AE manifest. Test: kernel-thread substrings dropped, workload comms kept, pause dropped.
Deploys the dx-daemon DaemonSet + Service into honey and mirrors the pl->honey secrets (jwt-signing-key, cluster-id, cloud-addr, api-key, clickhouse http-url) via a before-hook, replacing the hand-applied manifest used in the e2e. Deploy with: skaffold deploy -f k8s/vizier/dx/skaffold.yaml CH http-url defaults to the soc clickhouse Service; override with DX_CH_HTTP_URL.
Replaces the imperative seed-secret + patch-cloud-addr + sed-image +
kubectl-apply sequence with a single skaffold module:
skaffold deploy -f k8s/vizier/adaptive_export/skaffold.yaml
- kustomize overlay reuses bootstrap/adaptive_export_{role,deployment}
and pins the image via images: (ghcr aeprod tag) instead of sed.
- before-hook patches PL_CLOUD_ADDR :443 and seeds
pl-adaptive-export-secrets ONLY when PIXIE_API_KEY/PX_API_KEY is set,
never clobbering an existing secret with an empty key.
- LoadRestrictionsNone so the overlay can reuse the bootstrap manifests
in place (no duplication/drift).
Pairs with the dx-daemon skaffold (k8s/vizier/dx). Bump the AE image by
editing newTag in kustomization.yaml.
…aths
The AE/dx skaffold configs lived inside their overlay dirs with kustomize
paths: [.], which skaffold resolves against the shell CWD (repo root), not
the config-file dir -> 'unable to find kustomization.yaml in /.../pixie'.
Match the repo convention instead (skaffold/skaffold_vizier.yaml et al.):
skaffold configs live in skaffold/ and reference overlays by repo-root-
relative kustomize paths. Overlays stay in k8s/vizier/{adaptive_export,dx}.
skaffold deploy -f skaffold/skaffold_adaptive_export.yaml
skaffold deploy -f skaffold/skaffold_dx.yaml # run from repo root
- dx overlay gains a kustomization.yaml (was rawYaml).
- both validated with 'skaffold render' from repo root (image overrides +
RBAC/DaemonSet/Service resolve).
dx image 0.3.0-public3 -> 0.4.0-ssotforest-rc6 (forest scope + evidence_graph + isTableAbsent; broker no longer blinds the verdict). AE aeprod57 -> aeprod63 (upid + OOM firehose-collapse + px.any dark-export fixes). Add DX_FOREST_SCOPE, DX_PRECORRELATE_GRAPH, DX_EVIDENCE_GRAPH_CH so dx populates forensic_db.dx_evidence_graph. Verified live on a pemdq1 rig: dc_snoop + evidence_graph populate; DX_BENCH=pemdirect (already set) keeps dx off the shared broker so AE's export doesn't DeadlineExceed.
…nce_graph Root cause of 'attack fires but dx_evidence_graph stays empty': with DX_PRECORRELATE_GRAPH=1 the workup pulls the per-anomaly full-evidence set into memory and at the 1Gi limit dx is OOM-killed (exit 137) mid-workup, BEFORE writing the graph — then crash-loops, so no edges ever land. Reproduced on a pemdq1 rig: dx received the referral (comm=ls/sh rule=R0001) then died OOMKilled x4. 2Gi clears it (verified: graph 23->34, 32 malignant). Request 256Mi->512Mi. Operational note (not a manifest change): the vector->dx sink can wedge when the dx pod bounces (findings stop arriving, no referral) — bounce the node-01 vector pod after any dx redeploy. Also seen: transient node-01 PEM restart -> pemdirect 'connection refused' -> BLIND verdicts (edges still write via generic-malignant).
…flood Fresh-PG validation of the 2Gi memory fix surfaced a SECOND bug: the bobctl attacks/redis-oss.yaml kill-chain fires ~31 distinct comm/rule anomalies on redis-master-0, and dx with 4 concurrent workers SIGSEGVs (exit 139, no Go panic = hard crash in the concurrent workup/pemdirect path) — crash-loops, graph stays empty. Serializing workups (DX_WORKERS=1) eliminates it: restarts=0, graph 0->32 (30 malignant) on a fresh pemdq1 rig with the full kill-chain. The 2Gi fix (previous commit) handles OOM; this handles the concurrency crash. Root fix for the race (so >1 worker is safe) tracked separately.
…aid to 4 The DX_WORKERS=1 workaround is no longer needed: the crash was a nil qes.Timing deref in pxapi handleStats (fixed in rc8 via pixie@6422c0508782 + dx nil-rs guard + TriagePull recover), not a dx concurrency defect (race-detector floods clean). Restore the default 4 workers and the fixed image. Memory stays 2Gi (real precorrelate need).
Sync the entlein/dx#138 fixes into the skaffold-deployed lab manifest: - image rc8 -> optdbg2 (non-garble; rc8 predates the manifest+pushdown code and the garble release crashes exit 139 under load — fault 2, unresolved) - memory 2Gi -> 3Gi (fault 1: 1.3GB/round peak, OOM below) - DX_FOREST_PUSHDOWN=1 + depth 4 (fault 3: PxL lineage pushdown frees the PEM so AE exports dc_snoop under load; validated restarts=0 over 6+ rounds, dc_snoop 0->1777).
…ult 2 fixed) rc13 = garble -literals (dropped -tiny, the SIGSEGV cause). Obfuscated + survives the kill-chain (restarts=0). Replaces the non-garble optdbg2 debug tag.
… + metrics (#97) The trigger's strict forward-only high-water-mark on the content event_time could silently halt AE forever (F8/AE-9, loadtest E8): one far-future row jumped the cursor past all real data, and out-of-order / clock-skewed / restart-buried rows were dropped with no signal. Fix: - Bounded lookback (ADAPTIVE_TRIGGER_LOOKBACK_SEC, default 300; 0 = legacy strict HWM): each poll scans [watermark-lookback, inf) and a bounded insertion-ordered LRU of row fingerprints (dedup.go) makes re-seen rows exactly-once. Includes in-window paging (catchup floor) so backlogs wider than PollLimit still drain. - Wall-clock poison clamp (ADAPTIVE_TRIGGER_MAX_SKEW_SEC, default 3600): a normalized event_time past now+skew is emitted once but never advances the cursor; an already-poisoned persisted watermark is clamped at load, so E8 recovers with no manual ALTER TABLE + restart. - Metrics on the default prometheus registry (metrics.go), served via the shared services/metrics /metrics handler in cmd/main.go (AE_PPROF_ADDR mux + optional AE_METRICS_ADDR listener): ae_trigger_watermark_ns{table,hostname}, ae_trigger_below_watermark_total, ae_trigger_event_time_rejected_total. normalizeEventTimeNanos stays as the first line of defense; the monotonic happy path with LOOKBACK=0 is byte-identical to before (existing suite runs unchanged). New tests: late-arrival exactly-once, below-lookback bound, E8 poison non-halt + self-recovery, strict-mode regression, dedup LRU unit tests. Fixes #97
Flip the dx→AE control surface (:9100) to secure-by-default so the bearer
JWT + control payloads no longer cross the CNI in cleartext.
- TLS default-ON. Mounted /certs/server.{crt,key} (service-tls-certs) win;
else AE self-generates an ephemeral in-memory ECDSA P-256 self-signed cert
(1y, SAN localhost/127.0.0.1/::1/node) so TLS works with zero extra secrets.
Plaintext ONLY via explicit CONTROL_INSECURE=true (loud WARN).
- Auth default-ON whenever PL_JWT_SIGNING_KEY is present (drops the extra
CONTROL_REQUIRE_AUTH gate). No key + no CONTROL_INSECURE => fail-closed:
the control HTTP surface refuses to start; the rest of AE keeps running.
- CONTROL_TLS / CONTROL_REQUIRE_AUTH become deprecated no-ops (warn if set);
CONTROL_TLS_CERT/KEY kept as overrides; new CONTROL_INSECURE opt-out.
control/tls.go: TLSConfig(cert,key,hosts) + selfSignedCert + certToPEM helper.
control/tls_test.go: self-gen serves TLS /healthz, TLS rejects unauthenticated,
mounted-cert load path, plaintext opt-out path. BUILD.bazel srcs updated.
Manifests: AE deployment already mounts /certs + PL_JWT_SIGNING_KEY (no
CONTROL_TLS to drop) — added a secure-by-default note. dx-daemon
AE_CONTROL_ADDR http:// -> https:// (dx client skip-verifies).
Stacks on #92 (fix/ae-protocol-export-pxexport); does not touch #97 code.
…' error) %(taggerdate:raw) is empty for a lightweight release tag → create_manifest_update emits 'timestamp: ,' → jq syntax error → the vizier release-metadata step fails even though the image built + pushed. Fall back to the tagged commit's committer date.
Standalone GraphWidget bundle (no src/ui changes) rendering the dx evidence graph with drill-down: graph edges -> investigation manifest -> consulted raw forensic rows. Reads forensic_db in ClickHouse via px.DataFrame(clickhouse_dsn). - evidence_graph: severity-weighted pod->pod edges; px.Pod() stamps ST_POD_NAME so nodes deep-link to px/pod via the widget's built-in deepLinkURLFromSemanticType. - investigation_detail: manifest row(s), case_window bounds via px.pluck_int64. - consulted_rows: demo.md 'H3 dc_snoop reconstruction for the alert pod. - vis.json: Graph over evidence_graph (edgeWeightColumn=confidence, edgeColorColumn=max_severity, edgeHoverInfo=investigation_id/condition/criteria/ edge_kind), plus manifest + consulted-rows Table widgets; investigation_id var is the zoom. - README: 3-level zoom, load-into-UI steps, clickhouse_dsn feasibility (YES) + templated-read / hostname-partition / ns-start_time caveats. Static-validated only; needs live-UI validation on a cluster carrying forensic_db.
Enhance the existing widget in place (drop the parallel dx/evidence_graph bundle): - drill-able pod nodes (px.Pod -> ST_POD_NAME -> double-click deep-links to px/pod via the GraphWidget's built-in deepLinkURLFromSemanticType; NO src/ui change) - L2 investigation_detail: the manifest (verdict / case_window / evidence_hash / findings) - L3 consulted_rows: the raw forensic rows dx considered (demo.md §H reconstruction) - investigation_id vis variable = the zoom key; keeps the forensic_analyst read DSN. Needs live-UI validation on a cluster with populated dx_evidence_graph/manifest.
…orensic_db L3 consulted_rows referenced df.pod/namespace/container which exist in NO raw table (dc_snoop has only time_,pid,comm,t,file,hostname,event_time) -> the 'Column pod not found' compile error that killed the whole view. Project the event_time+hostname intersection (present in every raw_table), filter by hostname (host_filter, was pod_filter/PX_POD). raw_table default dc_snoop -> kubescape_logs: the px ClickHouseSourceNode only returns rows for UInt64 event_time; dc_snoop/redis_events/conn_stats are DateTime64 and read back 0 despite millions in CH. kubescape_logs (and the UInt64 dx_evidence_* graph/manifest tables) are the readable ones.
… not event_time/hostname The defensive event_time+hostname+investigation_id projection returned content-free rows (a nanosecond int + node name + blank) -- looked like random fields. Fix: consulted_rows pins to kubescape_logs (the one px-readable table) and px.plucks the real evidence from the RuntimeK8sDetails / RuntimeProcessDetails JSON columns: namespace/pod/container + comm/cmdline + RuleID + the alert message. Drops the raw_table var (dc_snoop/redis_events are DateTime64-unreadable anyway). Validated live: redis/redis-master-0 R0001/R0002/R0006/R0008/R0010/R0011 with real messages.
Re-point L1/L2 to the deterministic uniqueID join (dx rc15 carries the kubescape uniqueID into the manifest seed). Graph edges = subject_pod ->[process]-> target (process now surfaced: cat/ln/getent); consulted findings join on uniqueID, not the lossy RuleID@timestamp. Reads the dx_kubescape_anomalies + dx_anomaly_findings views (deduped).
… per anomaly dx's consulted findings reference benign background rows (proven: closest redis row is PING/CLIENT LIST, never the attack), so they can't surface the payload. The real command lives in kubescape_logs RuntimeProcessDetails.processTree.cmdline, keyed by uniqueID. Re-point consulted_findings at the dx_anomaly_findings view (now built from that process tree): rule / comm / parent / the actual cmdline / alert -- e.g. R0010 -> '/usr/bin/cat /etc/shadow', R1008 -> 'sh -c getent hosts xmr.pool.minergate.com'.
…ld -> rc18 L2 (consulted_records) now joins each finding to its actual record: exact (time_) join for pixie redis_events/conn_stats (rc18 makes finding.time_ == source.time_), row_identity content for dc_snoop/dns/process. Reveals the COMPLETE evidence set per anomaly -- attack (cat /etc/shadow, anomalous.dns.query, mnt_payload/drifted_bob) AND benign (PING, 127.0.0.1, proc/self/stat) -- the pre-correlation completeness guarantee. Bump k8s/vizier/dx/dx-daemon.yaml to dx rc18 (the per-row timestamp fix chain rc15-18).
PodEnrichPxL set namespace + pod on the native (socket_tracer) tables but
NOT hostname — only the stack_trace branch stamped it. So conn_stats /
http_events / dns_events / redis_events (and every protocol table) landed
in ClickHouse with an EMPTY hostname, even though hostname is the LEADING
ORDER BY column on all of them.
Consequences that this fixes:
* px reads of these tables filter WHERE hostname=<node>; empty hostname
matched nothing (the reads only worked via a join that sourced hostname
elsewhere).
* the pixie-io#136 order-UUID pre-correlation views could not expose a real
hostname without an order-JOIN, and that join blocked the (hostname,
event_time) primary-key pushdown (validated on rig 6a841cf7, CH 24.8).
Fix: PodEnrichPxL's native path also emits
df.hostname = px.upid_to_node_name(df.upid)
— the same UDF stack_trace already uses; valid on any upid-bearing table.
Dark-vector tables (raw pid, no upid) are unchanged; their node stamping
is a separate follow-up. Tests: statement-count oracles +1 line.
Two AE changes for the order-UUID pre-correlation dashboard: 1. dx_order_seeds table (schema.sql + KnownTables + OperatorOwnedTables): dx INSERTs one row per referral (evidence-loss fix — dx coalesces same-pod anomalies so most write no manifest). The dx_anomaly_orders view windows every uniqueID from this. ReplacingMergeTree ORDER BY (unique_id, rule_id), 30d TTL. 2. PodEnrichPxL dark-vector branch stamps hostname via px.upid_to_node_name on the same process_stats upid pod/ns already come from, so dc_snoop (and the other dark tables) carry hostname and become px-readable. Transient pids that miss process_stats resolve blank — same accepted limitation as pod/ns.
Cherry-picked rather than merged: the pull request branched from an older tip and a merge would have reverted the process-forest view and the profile work. Permissions are read-only and narrow as proposed: list and get on nodes, pods, services and the Cilium node objects, and get on the two objects in honey the job reads its own parameters from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014GDT6HWiFmRxmUaGFSKjPY
The adaptive-export source no longer lives in this tree. The manifests stay
here beside the rest of the vizier deployment; only the image reference moves.
ghcr.io/k8sstormcenter/vizier-adaptive_export_image:0.14.19-aeprod95
-> docker.io/tanzeee/adaptive-export:0.15.0-rc1
(index sha256:0bec613d86b5d272564f2107354eff5cac266ebfa4cfe4c62e792f84894245b9,
linux/amd64 + linux/arm64)
Same code as aeprod95. The repository is private, so the DaemonSet names a pull
secret and the skaffold module creates it in pl from the operators own registry
config before applying. No credential is read from or written to this repo.
The adaptive-export source no longer lives in this tree. The manifests stay
here beside the rest of the vizier deployment; only the image reference moves.
ghcr.io/k8sstormcenter/vizier-adaptive_export_image:0.14.19-aeprod95
-> docker.io/tanzeee/adaptive-export:0.15.0-rc1
(index sha256:0bec613d86b5d272564f2107354eff5cac266ebfa4cfe4c62e792f84894245b9,
linux/amd64 + linux/arm64)
Same code as aeprod95. The repository is private, so the DaemonSet names a pull
secret and the skaffold module creates it in pl from the operators own registry
config before applying. No credential is read from or written to this repo.
The adaptive-export source no longer lives in this tree. The manifests stay
here beside the rest of the vizier deployment; only the image reference moves.
ghcr.io/k8sstormcenter/vizier-adaptive_export_image:0.14.19-aeprod95
-> docker.io/tanzeee/adaptive-export:0.15.0-rc1
(index sha256:0bec613d86b5d272564f2107354eff5cac266ebfa4cfe4c62e792f84894245b9,
linux/amd64 + linux/arm64)
Same code as aeprod95. The repository is private, so the DaemonSet names a pull
secret and the skaffold module creates it in pl from the operators own registry
config before applying. No credential is read from or written to this repo.
Image 0.15.0-rc1 -> 0.15.0-rc2 (index sha256:701835e53381b42385393108da788ef9597dbfd7a318987114c2df7a10d1c08a). ADAPTIVE_TRIGGER_POLL=false stops the 250ms read of forensic_db.kubescape_logs on every node, roughly 345,000 queries per node per day. Captures still happen on every control-surface request, which is a superset of what the poll found. Nothing outside adaptive_export reads forensic_db.adaptive_attribution, so no view or report loses rows. Remove the env var to restore the poll.
Image 0.15.0-rc1 -> 0.15.0-rc2 (index sha256:701835e53381b42385393108da788ef9597dbfd7a318987114c2df7a10d1c08a). ADAPTIVE_TRIGGER_POLL=false stops the 250ms read of forensic_db.kubescape_logs on every node, roughly 345,000 queries per node per day. Captures still happen on every control-surface request, which is a superset of what the poll found. Nothing outside adaptive_export reads forensic_db.adaptive_attribution, so no view or report loses rows. Remove the env var to restore the poll.
Shared query client with token refresh, and a soft heap limit from the cgroup. Connection reclaim is now a function of elapsed time rather than of how much traffic the cluster is producing. Index sha256:3b1358d7b0a8925098dce6a6862184c2db3a340adeee705ecf0fc5ea745b62d7
The 0.15.0 images were built from the adaptive_export tree on main, which is behind this branch. main has no /dx/rows handler, so a deploy of 0.15.0 would have 404d every row handover from dx. Back to the last image built from this branch until the 0.15.0 line is rebuilt from the right tree.
Rebuilt from this branch, so /dx/rows and the rest of the surface aeprod95 carries are present. Supersedes the 0.15.0 line, which was built from main. Index sha256:8c33b6d2135eb488e0e938dea5ce432432c26fdab89a2e4606c8e05b688acae1
Built from the adaptive_export tree on fix/ae-protocol-export-pxexport, which is the branch every aeprod image is cut from. The 0.15.1 image came from this branch instead, which is a sibling off main and behind the AE line. Index sha256:11fa81836ad9d11cce332caec30110c1aef4114a11d84d9bcc56ca906116da06
… with no endpoints
Found on k3s-1 while testing 0.15.2-rc1, and it predates that image by hours.
The adaptive-export-control Service there was a stale object from the original
px-deploy bootstrap, created 2026-08-31, carrying the generic vizier selector:
{app: pl-monitoring, component: vizier, name: adaptive-export, vizier-bootstrap: true}
No adaptive-export pod carries those labels, so the endpoint list was empty.
This manifest declares selector {name: adaptive-export}, which is correct, but
kubectl apply could not repair the live object: it has no last-applied
annotation, so the selector map is merged into rather than replaced.
The result was silent. AE was listening on 9100 the whole time. Every row dx
handed over was refused at the transport, dropped without retry, and the only
trace was one log line per batch on the dx side: 11,803 failures and not one
success, at about a thousand an hour since the hand-off path shipped.
Two guards:
- before deploy, replace the selector when it differs from the one declared
here. A JSON patch, because a merge patch adds keys instead of replacing the
map and reports no change.
- after deploy, wait briefly for endpoints and exit non-zero if there are none,
printing the live selector and the pod labels. An empty endpoint list means
every hand-off is being discarded, and that should stop a deploy rather than
be discovered hours later.
Reported by the k3s-1 session, who found and repaired it on that cluster.
All eleven bridged evidence tables now collapse a row collected twice, and the image carries the migration that converts the four that were plain MergeTree, preserving their rows. Leader-only, enabled by ADAPTIVE_MIGRATE_ENGINES.
Converts the four bridged evidence tables that shipped as plain MergeTree into ReplacingMergeTree, preserving their rows. Runs on the cluster-setup leader only and is a no-op once converted, so it stays on.
The engine migration now verifies distinct sorting-key tuples before swapping, and refuses to swap if any were lost. The raw row count is expected to fall on a table that had duplicates; distinct keys are not.
amqp_events, mux_events and tls_events are now bridged like every other evidence table: identity column, a sorting key that can identify a row, a dx_ord__ join view, and conversion of the existing tables on the leader. Index sha256 below; fourteen bridged tables.
amqp, mux and tls declare the columns the PEM actually emits and are keyed on them. The migration now rebuilds when the live sorting key differs from the schema, so those three convert a second time — for the key rather than the engine. Dial count published as a metric. This is the pinned adaptive-export for both clusters.
The schema apply defers a view whose base is not ready, which it has never actually done: the predicate tested for CREATE VIEW and every view is written CREATE OR REPLACE VIEW, so 27 views took the fatal path instead. A non-leader applying during the leader migration crash-looped rather than deferring. Index sha256:09ea9b0975c9df0fe4ce7cd93bdc7167fe9180ecc54b85ea325ada5e8a950350
A table awaiting the leader migration is no longer treated as schema drift, so a non-leader pod boots instead of crash-looping and the rollout can reach the leader. 0.16.1 deadlocked k3s-1 for exactly that reason and 0.16.2 only moved the fatal one step later. Index sha256:bce661ebcb417bd1edcab617e89811f7eca26049929ea0e695c610ab6e4f0fc9
dx/shadow_profiles: stop OOM-killing ClickHouse — read the pinned view
dns_resolve compares row_time against the raw window bounds. Seven dx_orders reads move to dx_src__orders. profile_coverage and signatures read dx_profiles__latest, dx_rogueartifacts__latest and dx_trust__latest. evidence_graph and shadow_trace expose seconds since the newest source row, so a quiet panel and a dead one read differently.
pxl_scripts/dx: read typed views; add freshness to two panels
The view carries authored and learned profiles together once the mirror reads the authored marker. Rows written before that carry neither value and are excluded rather than assumed learned.
dx: remove the dx-kpi CronJob manifest
pxl_scripts/dx/shadow_profiles: show only learned profiles
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pixie's own workloads are governed by signed ContainerProfiles that ship with the manifests and bind by pod-template label, so they are in force from the first container start rather than learned on each cluster. This branch is where those profiles live and where they are improved.
One canonical set
The profiles used to exist in three places that had already drifted: the operator overlay's own copy, a set inlined into the operator helm template, and a third folder that no kustomization referenced. There is now one folder,
k8s/vizier/sbob/profiles/, holding all eighteen. The operator overlay references those files directly, and the helm template is generated from them byk8s/vizier/sbob/gen-operator-helm-template.py, which has a--checkmode for CI.Network allowlists
Every profile shipped with zero ingress entries, so any inbound connection raised an unexpected-ingress alert, and the egress gaps were all service traffic. A service address carries no pod labels before translation, so a selector can never match it. Each profile now allows, in both directions: the Pixie pods by their shared label, the kubescape node-agent, both OLM operators, kube-dns, service references for the apiserver, kube-dns and the six Pixie services, and the host entity for kubelet probes. Services are named by reference rather than by address, so the profiles work on any cluster.
Kubelet probes arrive from the node's pod-network gateway, and the host entity resolves to the local node only, so a pod that moves node is probed from an address no profile allows. That was 5356 ingress alerts on one demo cluster, all of them the OLM catalog operator being probed after a reschedule. Every profile with a declared probe or metrics port now allows the k3s gateway range on exactly those ports.
Verified
On a k3s flannel playground with node-agent v0.1.0-rogue8 and the current soc rule set, all profiles bound:
Two operational notes. An authored profile marked completed is frozen in storage, so updating one on a running cluster needs a delete and re-apply; a plain apply reports success and changes nothing. And the OLM catalog registry pod still has no profile, which is the only ungoverned workload left in the infrastructure namespaces.
The branch also carries the dx viewer scripts and the pinned image tags that the cloud releases are cut from.