Skip to content

Race condition when rotating multiple Vault dynamic secrets causes identical restartedAt annotation and skipped rollout in Kubernetes #1228

@duong-se

Description

@duong-se

Describe the bug
We are experiencing a potential race condition when using multiple Vault dynamic secrets (e.g., Database secrets and GCP secrets) in the same Kubernetes workload via Vault Secrets Operator (VSO).

If two dynamic secrets are rotated within an extremely short time window (nanoseconds to sub-second), both secret updates may trigger a rollout restart at effectively the same timestamp (e.g., 2026-02-27T14:26:59Z).

Since VSO updates the annotation:
vso.secrets.hashicorp.com/restartedAt
with second-level precision, it is possible that both secret rotations set the exact same annotation value.

As a result:

  • The first secret rotation (e.g., database) updates the annotation and triggers a rollout restart.
  • The second secret rotation (e.g., GCP) happens within the same second and updates the annotation with the same value.
  • Kubernetes does not detect any change in the pod template because the annotation value is identical.
  • The second rollout restart is effectively ignored.
    This leads to inconsistent pod states where:
  • Some pods load old GCP credentials.
  • Some pods load new GCP credentials.
  • Database credentials may already be updated.
    This creates a temporary but critical inconsistency across replicas.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions