Skip to content

Commit c832c8a

Browse files
committed
fix "total" logic, fix target selection, add protection label
1 parent a9ecf50 commit c832c8a

3 files changed

Lines changed: 35 additions & 0 deletions

File tree

README.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,24 @@ Protection is enforced in two places:
7575
- **Validating webhook** — rejects `Experiment` objects whose `spec.selector.namespace` is in the protected list at admission time.
7676
- **Controller** — filters out any pods in protected namespaces during target selection, even for cluster-scoped selectors.
7777

78+
## Safety: Pod-level Opt-out
79+
80+
Individual pods can be excluded from all chaos experiments by adding the annotation `chaos.kreicer.dev/ignore: "true"`. This is useful for pods running critical in-flight work (e.g., database migrations, stateful leaders) that must not be interrupted.
81+
82+
```bash
83+
kubectl annotate pod <pod-name> chaos.kreicer.dev/ignore=true
84+
```
85+
86+
Or in the pod template:
87+
88+
```yaml
89+
metadata:
90+
annotations:
91+
chaos.kreicer.dev/ignore: "true"
92+
```
93+
94+
The annotated pod is removed from the eligible list before selection. If all matching pods carry the annotation, the run transitions to `Skipped` automatically.
95+
7896
## Run locally (against Kind or Minikube)
7997

8098
### Prerequisites
@@ -203,6 +221,8 @@ kubectl describe experimentrun <run-name>
203221

204222
Events use `Normal` type for successful transitions (`PreviewGenerated`, `Approved`, `Running`, `Completed`) and `Warning` for failure states (`Failed`, `Expired`).
205223

224+
The `TOTAL` column in `kubectl get expruns` is populated as soon as targets are selected during the `PreviewGenerated` phase, so you can see how many pods will be affected before the run executes.
225+
206226
## Safe Deletion
207227

208228
`Experiment` objects carry a finalizer (`chaos.omen.com/finalizer`). When an `Experiment` is deleted, the controller first deletes all owned `ExperimentRun`s and waits for them to be removed before releasing the finalizer. This prevents orphaned runs from executing chaos actions after the parent is gone.

api/v1alpha1/experiment_types.go

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,12 @@ import (
2020
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
2121
)
2222

23+
const (
24+
// IgnoreAnnotation is the pod annotation that opts a pod out of target selection.
25+
// Set to "true" on any pod to exclude it from all chaos experiments.
26+
IgnoreAnnotation = "chaos.kreicer.dev/ignore"
27+
)
28+
2329
// RunPolicyType defines how often an Experiment executes.
2430
// +kubebuilder:validation:Enum=Once;Repeat
2531
type RunPolicyType string

internal/controller/experiment_controller.go

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,7 @@ func (r *ExperimentReconciler) scheduleRun(ctx context.Context, experiment *chao
274274
run.Status.Phase = chaosv1alpha1.PhasePreviewGenerated
275275
run.Status.PreviewTargets = targets
276276
run.Status.ScheduledAt = &scheduledAt
277+
run.Status.Summary = &chaosv1alpha1.RunSummary{Total: len(targets)}
277278
if err := r.Status().Update(ctx, run); err != nil {
278279
return ctrl.Result{}, err
279280
}
@@ -425,10 +426,18 @@ func (r *ExperimentReconciler) selectTargets(ctx context.Context, experiment *ch
425426
if pod.Status.Phase != corev1.PodRunning {
426427
continue
427428
}
429+
// Skip pods already marked for deletion (terminating).
430+
if !pod.DeletionTimestamp.IsZero() {
431+
continue
432+
}
428433
// Self-exclusion: skip pods belonging to the Omen operator itself.
429434
if pod.Labels[omenAppLabel] == omenAppName {
430435
continue
431436
}
437+
// Skip pods that have opted out of chaos via annotation.
438+
if pod.Annotations[chaosv1alpha1.IgnoreAnnotation] == "true" {
439+
continue
440+
}
432441
eligible = append(eligible, pod.Name)
433442
}
434443

0 commit comments

Comments
 (0)