Persistent State Management

Persistent State Management

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/axis/degraded_contract_test.go
• cmd/axis/testdata/context_show_corrupt_state.golden
• cmd/axis/testdata/skills_corrupt_store.golden
• internal/api/degraded_contract_test.go
• internal/api/testdata/knowledge_corrupt_persistence.golden
• internal/api/turboquant_contract_test.go
• internal/failures/expiry.go
• internal/failures/failures_test.go
• internal/failures/record.go
• internal/failures/scope.go
• internal/persist/recovery_test.go
• internal/state/state.go
• internal/state/state_test.go

AXIS maintains a persistent view of the cluster's dynamic state in a local JSON file. Unlike the static configuration found in nodes.yaml, this state tracks transient information such as resource reservations, active executions, heartbeat timestamps, and failure history. This persistence allows AXIS to maintain consistency across CLI invocations and daemon restarts.

State File Location and Structure

The primary state file is located at ~/.axis/state.json internal/state/state.go:82-85. It is managed by the ClusterState struct, which serves as the root container for all persistent telemetry.

Data Model: ClusterState

The ClusterState struct internal/state/state.go:61-70 includes:

• Nodes: A map of NodeState objects, keyed by node name.
• Failures: A failures.Store containing records of past execution issues internal/state/state.go:65.
• Observations: Historical performance data (e.g., Peak RAM) used for future placement heuristics internal/state/state.go:66.
• Decisions: A circular buffer of the last 20 placement decisions to provide context for AI agents internal/state/state.go:67.
• TaskHistory: A log of recent task executions internal/state/state.go:68.

Node Tracking: NodeState

Each node in the cluster has a corresponding NodeState internal/state/state.go:17-29 that tracks:

• ReservedMB: The total RAM currently committed to active tasks on that node.
• ActiveExecs: A list of unique execution IDs currently running.
• ExecHeartbeatAt: A map of timestamps for each execution ID, used to detect "zombie" reservations if a process crashes without releasing resources internal/state/state.go:24.
• ExecOwnerPID: The local Process ID of the AXIS process that initiated the execution internal/state/state.go:25.

Sources: internal/state/state.go:17-70, internal/state/state.go:82-85

---

State Lifecycle and Maintenance

AXIS does not just read and write the state file; it actively maintains it to ensure the cluster view remains accurate even after hardware failures or ungraceful shutdowns.

The Maintain Pipeline

The Maintain() function internal/state/state.go:163-209 is the central logic for state cleanup. It is typically called immediately after Load() and before the state is used for placement decisions internal/state/state_test.go:40-44.

Feature	Logic	Threshold
Zombie Reclaim	Reclaims RAM from executions that haven't sent a heartbeat internal/state/state.go:193-197.	2 minutes (execHeartbeatStaleAfter)
Soft Reclaim	Reduces large reservations to a 1GB "per-exec cap" if they exceed a time threshold internal/state/state.go:77.	45 minutes (staleReservationReclaimAfter)
Hard Expiry	Completely drops node state if no activity is recorded for a long period internal/state/state.go:179-183.	24 hours (staleReservationHardExpiry)
Failure Pruning	Removes expired failure records from the immune system internal/state/state.go:166-169.	Variable (Exponential Backoff)

State Persistence Flow

This diagram illustrates how the state package mediates between the on-disk JSON and the active system memory.

State Transition and Maintenance Flow

graph TD
    subgraph "Disk Space"
        JSON["~/.axis/state.json"]
    end

    subgraph "Code Space (internal/state)"
        LOAD["Load()"]
        MAINTAIN["Maintain()"]
        SAVE["Save()"]
        MIGRATE["runMigrations()"]
    end

    JSON -->|Read| LOAD
    LOAD -->|Check Version| MIGRATE
    MIGRATE -->|Transform| MAINTAIN
    LOAD --> MAINTAIN
    MAINTAIN -->|Prune Failures| PRUNE["failures.Store.Prune()"]
    MAINTAIN -->|Reclaim Zombies| RECLAIM["reclaimStaleReservation()"]
    RECLAIM -->|Write| SAVE
    SAVE -->|Atomic Write| JSON

Loading

Sources: internal/state/state.go:87-122, internal/state/state.go:163-209, internal/state/state.go:75-80

---

Failure Memory and Schema Migrations

AXIS uses a sophisticated "Immune System" to prevent scheduling tasks on nodes that have recently failed for specific workloads.

Schema Migration: Tombstones to FailureRecords

In older versions, AXIS used a simple TombstoneEntry internal/state/state.go:39-45. The system now migrates these automatically to the more robust failures.Store during Load() internal/state/state.go:115-119.

• Logic: Legacy tombstones are parsed, and if the task pattern suggests an LLM workload (e.g., "ollama"), they are classified as FailureBackendMisfit internal/state/state.go:136-137. Otherwise, they are treated as FailureExecCrash internal/state/state.go:139.

Failure Record Lifecycle

Failures are stored in internal/failures/record.go. When a failure is recorded via Record() internal/failures/record.go:33-75, AXIS calculates an expiry time using exponential backoff internal/failures/expiry.go:12-27.

• Base Expiry: 24 hours internal/failures/expiry.go:8.
• Max Expiry: 7 days internal/failures/expiry.go:9.
• Backoff: The duration doubles for every consecutive failure in the same scope internal/failures/expiry.go:20-25.

Sources: internal/state/state.go:127-156, internal/failures/record.go:33-75, internal/failures/expiry.go:7-27

---

Atomic Recovery and Quarantine

To prevent state corruption from breaking the entire CLI, AXIS implements an atomic write pattern and a quarantine recovery system.

Atomic Writes

AXIS never overwrites state.json directly. Instead, it uses WriteFileAtomic internal/persist/recovery_test.go:60-94:

1. Writes to a .tmp file.
2. Calls Sync() on the file handle to ensure bits are on disk internal/persist/recovery_test.go:104-107.
3. Renames the temp file to state.json.
4. Calls Sync() on the parent directory to ensure the directory entry is updated internal/persist/recovery_test.go:108-114.

Quarantine Recovery

If json.Unmarshal fails during Load(), AXIS does not exit with an error. Instead, it triggers QuarantineCorruptFile internal/state/state.go:97-101:

1. The corrupt file is renamed to state.json.corrupt-<TIMESTAMP> internal/persist/recovery_test.go:27-29.
2. A fresh, empty ClusterState is returned so the system can continue operating internal/state/state.go:100.
3. A RecoveryWarning is injected into the snapshot, which appears in axis status and the API internal/api/degraded_contract_test.go:81-86.

Recovery System Mapping

graph TD
    subgraph "FileSystem Space"
        FILE["state.json"]
        TMP["state.json.tmp-xyz"]
        CORRUPT["state.json.corrupt-2023..."]
    end

    subgraph "Logic (internal/persist)"
        WA["WriteFileAtomic()"]
        QCF["QuarantineCorruptFile()"]
    end

    subgraph "API / CLI"
        WARN["RecoveryWarning"]
    end

    WA -->|1. Create| TMP
    TMP -->|2. Rename| FILE
    FILE -->|3. Parse Error| QCF
    QCF -->|4. Rename| CORRUPT
    QCF -->|5. Emit| WARN

Loading

Sources: internal/persist/recovery_test.go:11-43, internal/persist/recovery_test.go:60-129, internal/state/state.go:96-103, internal/api/degraded_contract_test.go:26-57

---