Skip to content

Commit 0fb0c56

Browse files
farhan-syahfarhan-syah
committed
docs: add cross-shard transactions and corruption quarantine
Add cross-shard-transactions.rdx covering the Calvin sequencer architecture: why 2PC was replaced, the sequencer/scheduler/executor pipeline, OLLP optimistic read-set validation, admission control, and session variable reference. Add corruption-quarantine.rdx covering the two-strike CRC detection rule, quarantine file naming, registry persistence across restarts, affected engines, operator CLI runbook, and S3/local storage config. Extend existing docs with supporting content: - multi-raft: document three Raft group kinds (data, meta, sequencer) and sequencer group isolation behaviour - monitoring: add cross-shard txn metrics, memory backpressure metrics, IO priority metrics, CDC drop counter, and quarantine gauges - cluster-operations: add cross_shard_txn session variable examples and debug HTTP endpoints - backup-recovery: document S3-compatible object storage config for snapshots and quarantine (independent of cold_storage) - change-streams: document gap-detection fields in poll response - sql/explain: show Calvin sequencer preamble in EXPLAIN output - sql/ddl: document REINDEX and REINDEX CONCURRENTLY - sql/update-delete: document RETURNING clause for UPDATE and DELETE - sql/fusion-rrf: clarify two supported pairwise fusion modes and add SEARCH...USING FUSION() shorthand syntax Register both new pages in oxidoc.toml sidebar.
1 parent 49b6117 commit 0fb0c56

12 files changed

Lines changed: 427 additions & 4 deletions

‎docs/administration/backup-recovery.rdx‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,3 +43,29 @@ The Event Plane resumes from its LSN watermark — no events are lost.
4343
## WAL Archiving
4444

4545
WAL segments can be archived for point-in-time recovery. Old segments are eligible for cleanup once all records have been flushed to L1 segments.
46+
47+
## Object storage for snapshots and quarantine
48+
49+
By default, snapshots and quarantined segment files are stored on local disk alongside the data directory. Both can be redirected to S3-compatible object storage.
50+
51+
```toml
52+
[snapshot_storage]
53+
endpoint = "https://s3.amazonaws.com"
54+
bucket = "my-nodedb-snapshots"
55+
region = "us-east-1"
56+
prefix = "cluster-prod"
57+
access_key = "AKIA..."
58+
secret_key = "..."
59+
60+
[quarantine_storage]
61+
endpoint = "https://s3.amazonaws.com"
62+
bucket = "my-nodedb-quarantine"
63+
region = "us-east-1"
64+
prefix = "cluster-prod"
65+
access_key = "AKIA..."
66+
secret_key = "..."
67+
```
68+
69+
Omit `endpoint` (or leave it empty) to use the local filesystem — this is the default. Both sections are independent: you can put snapshots on S3 and leave quarantine files local, or vice versa.
70+
71+
The cold data tier (`[cold_storage]`) is a separate config from these two — it covers Parquet archives and timeseries L2 data, not snapshots or quarantine files.

‎docs/administration/cluster-operations.rdx‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,35 @@ Raft leader election handles temporary unavailability.
4242
- **Single node failure** — Raft elects new leaders for affected vShards
4343
- **Minority failure** — Cluster remains available with quorum
4444
- **Majority failure** — Cluster becomes read-only until quorum restores
45+
46+
## Cross-shard transaction mode
47+
48+
```sql
49+
-- Require atomic cross-shard writes (default)
50+
SET cross_shard_txn = 'strict';
51+
52+
-- Opt out of atomicity for bulk loads (each shard commits independently)
53+
SET cross_shard_txn = 'best_effort_non_atomic';
54+
55+
SHOW cross_shard_txn;
56+
```
57+
58+
See [Cross-Shard Transactions](../architecture/cross-shard-transactions) for details on the Calvin sequencer and OLLP.
59+
60+
## Debug endpoints
61+
62+
```bash
63+
# List all Raft groups (data, meta, sequencer)
64+
curl http://localhost:6480/v1/cluster/debug/raft/{group_id}
65+
66+
# QUIC transport diagnostics
67+
curl http://localhost:6480/v1/cluster/debug/transport
68+
69+
# Catalog descriptor dump
70+
curl http://localhost:6480/v1/cluster/debug/catalog/descriptors
71+
72+
# Segments currently in quarantine (corrupt and isolated)
73+
curl http://localhost:6480/v1/cluster/debug/quarantined-segments
74+
```
75+
76+
See [Corruption Quarantine](./corruption-quarantine) for the quarantine runbook.

‎docs/administration/corruption-quarantine.rdx‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
---
2+
title: Corruption Quarantine
3+
description: Automatic detection and isolation of corrupt segments, with operator tooling for inspection and recovery.
4+
---
5+
6+
# Corruption Quarantine
7+
8+
NodeDB automatically detects corrupt segments using CRC32C checksums and isolates them so one bad segment cannot take down an entire collection or shard.
9+
10+
## How it works — two-strike rule
11+
12+
**First CRC failure** on a segment: log a warning and retry the read once.
13+
14+
**Second failure** on the same segment: the segment is quarantined —
15+
16+
1. The file is renamed to `<original-path>.quarantined.<unix_ts_ms>`
17+
2. The segment ID is recorded in the quarantine registry
18+
3. Subsequent reads of that segment return a typed `SegmentQuarantined` error
19+
4. All other segments in the collection continue serving reads normally
20+
21+
On restart, NodeDB scans the data directory for `*.quarantined.*` files and rebuilds the registry automatically — quarantine state survives restarts.
22+
23+
The two-strike rule prevents transient I/O errors (flipped bit on a warm SSD) from quarantining healthy segments, while ensuring persistently corrupt segments are isolated after the first retry.
24+
25+
## Affected engines
26+
27+
Quarantine is wired into reads for:
28+
29+
- **Columnar** — segment scan, retention scan, prior-value read
30+
- **FTS** — redb backend byte retrieval
31+
- **Raft snapshots** — snapshot chunk install
32+
- **Vector** — wrapper present; currently no production read sites (segments held in-memory; quarantine activates if disk-resident vector segments are added in future)
33+
34+
## Inspect quarantined segments
35+
36+
```bash
37+
curl http://localhost:6480/v1/cluster/debug/quarantined-segments
38+
```
39+
40+
Response:
41+
42+
```json
43+
{
44+
"segments": [
45+
{
46+
"segment_id": "col-00042",
47+
"engine": "columnar",
48+
"collection": "events",
49+
"quarantined_at_unix_ms": 1746480000000,
50+
"strikes": 2,
51+
"last_error": "FooterCrcMismatch"
52+
}
53+
]
54+
}
55+
```
56+
57+
An empty `segments` array means no segments are currently quarantined.
58+
59+
## Metrics
60+
61+
| Metric | Type | Description |
62+
|---|---|---|
63+
| `nodedb_segments_quarantined_total{engine,collection}` | counter | Cumulative segments quarantined since startup |
64+
| `nodedb_segments_quarantined_active{engine,collection}` | gauge | Segments currently in quarantine |
65+
66+
Alert on `nodedb_segments_quarantined_total` increasing, or `nodedb_segments_quarantined_active > 0`.
67+
68+
## Recovery
69+
70+
A quarantined segment means data in that segment is unreadable. Options:
71+
72+
**Option 1 — Restore from backup.** If you have a recent backup, restore it. The quarantined file is preserved as-is until you delete it manually.
73+
74+
```sql
75+
RESTORE TENANT acme FROM '/backups/acme-latest.bak';
76+
```
77+
78+
**Option 2 — Rebuild the index.** For vector and FTS indexes, the index can be rebuilt from the source data without data loss.
79+
80+
```sql
81+
REINDEX CONCURRENTLY my_collection;
82+
```
83+
84+
**Option 3 — Drop and repopulate.** If the collection can be repopulated from an upstream source, drop and recreate it.
85+
86+
**After recovery:** the `.quarantined.<ts>` files can be deleted manually once you've confirmed data is restored. NodeDB does not auto-delete them.
87+
88+
## Storage location
89+
90+
By default quarantined files stay alongside their originals on local disk. To archive quarantined files to object storage instead, configure `quarantine_storage` — see [Backup & Recovery](./backup-recovery#quarantine-storage).

‎docs/administration/monitoring.rdx‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,49 @@ curl http://localhost:6480/health/ready # WAL recovered, ready for queries
5252
| Connection | Active connections, auth failures |
5353
| Replication | Raft log lag, replication latency |
5454
| Storage | WAL fsync latency, segment count, compaction debt |
55+
56+
## Cross-shard transaction metrics
57+
58+
| Metric | Type | Description |
59+
| ------ | ---- | ----------- |
60+
| `nodedb_sequencer_epochs_total` | counter | Epochs proposed by the sequencer |
61+
| `nodedb_sequencer_epoch_duration_ms` | histogram | Time to drain and propose each epoch |
62+
| `nodedb_sequencer_admitted_txns_total{outcome}` | counter | Admission outcomes: `admitted`, `rejected_conflict`, `rejected_inbox_full`, `rejected_txn_too_large`, `rejected_fanout_too_wide`, `rejected_tenant_quota`, `rejected_not_leader` |
63+
| `nodedb_sequencer_inbox_depth` | gauge | Pending transactions in the sequencer inbox |
64+
| `nodedb_calvin_scheduler_lock_wait_ms_total{vshard}` | counter | Cumulative lock-wait time per shard |
65+
| `nodedb_calvin_executor_txn_duration_ms{vshard}` | histogram | Per-shard execution time for cross-shard txns |
66+
| `nodedb_calvin_ollp_retries_total{predicate_class,outcome}` | counter | OLLP retry outcomes (`succeeded`, `retried`, `exhausted`, `circuit_open`, `tenant_budget_exceeded`) |
67+
| `nodedb_calvin_ollp_circuit_state{predicate_class}` | gauge | 0 = closed, 1 = half-open, 2 = open |
68+
| `nodedb_calvin_ollp_backoff_ms{predicate_class}` | gauge | Current OLLP retry backoff delay |
69+
| `nodedb_calvin_infra_abort_total{reason}` | counter | Infrastructure aborts (disk error, OOM, corruption) |
70+
71+
## Memory backpressure metrics
72+
73+
| Metric | Type | Description |
74+
| ------ | ---- | ----------- |
75+
| `nodedb_backpressure_critical_total{engine}` | counter | Write handlers that entered the Critical-pressure flush path |
76+
| `nodedb_backpressure_emergency_total{engine}` | counter | Write handlers rejected by Emergency-pressure |
77+
78+
## IO priority metrics
79+
80+
| Metric | Type | Description |
81+
| ------ | ---- | ----------- |
82+
| `nodedb_io_queue_depth{priority}` | gauge | Pending tasks per IO priority tier (`background`, `normal`, `high`, `critical`) |
83+
| `nodedb_io_wait_ns{priority}` | histogram | Submission-to-completion latency per tier |
84+
85+
## CDC metrics
86+
87+
| Metric | Type | Description |
88+
| ------ | ---- | ----------- |
89+
| `nodedb_cdc_events_dropped_total{tenant,stream}` | counter | Events dropped from a named stream's buffer due to overflow — per stream, not global |
90+
91+
Alert on this increasing for a stream whose consumer is active — it means the consumer is falling behind.
92+
93+
## Corruption quarantine metrics
94+
95+
See [Corruption Quarantine](./corruption-quarantine) for the full quarantine runbook.
96+
97+
| Metric | Type | Description |
98+
| ------ | ---- | ----------- |
99+
| `nodedb_segments_quarantined_total{engine,collection}` | counter | Cumulative segments quarantined since startup |
100+
| `nodedb_segments_quarantined_active{engine,collection}` | gauge | Segments currently in quarantine |

‎docs/architecture/cross-shard-transactions.rdx‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
---
2+
title: Cross-Shard Transactions
3+
description: Calvin sequencer architecture for deterministic, atomically-committed cross-shard writes.
4+
---
5+
6+
# Cross-Shard Transactions
7+
8+
When a write touches rows that hash to more than one vShard, NodeDB routes it through the **Calvin sequencer** — a dedicated Raft-backed coordination layer that guarantees all participating shards commit the transaction in the same order, with no mid-flight aborts.
9+
10+
Single-shard writes bypass this entirely and take the normal per-vShard Raft fast path.
11+
12+
## Why Calvin, not 2PC
13+
14+
Two-phase commit with compensating actions was the earlier design. It was replaced because compensation creates observable intermediate states (a row exists, then disappears), compensation can itself fail, and every read path that crosses an in-flight transaction has to know about pending compensations. The concern spreads outward indefinitely.
15+
16+
Calvin (Thomson et al., SIGMOD 2012) eliminates the problem by validating the full read/write set *before* any shard touches the transaction. Every shard executes against a globally-ordered input log. There is no concept of "shard A committed, shard B failed" — either all shards execute the transaction or none do.
17+
18+
## Architecture
19+
20+
```
21+
Client / Control Plane
22+
│ declares read/write set
23+
▼
24+
SEQUENCER (dedicated Raft group, Control Plane)
25+
│ batches transactions into epochs (default 20 ms)
26+
│ replicates each epoch via Raft — globally ordered
27+
▼
28+
SCHEDULER (per vShard, Control Plane)
29+
│ acquires locks in deterministic global order
30+
│ single-threaded per shard
31+
▼
32+
EXECUTOR (Data Plane, existing engine handlers)
33+
│ executes deterministically using the sequenced batch
34+
│ no application-level aborts — all constraint checks happen upstream
35+
```
36+
37+
### The sequencer Raft group
38+
39+
The sequencer runs as its own independent Raft group (`SEQUENCER_GROUP_ID`), separate from the per-vShard data groups and the metadata group. This means:
40+
41+
- **Failure isolation** — sequencer leader election doesn't disrupt metadata reads, schema operations, or data writes
42+
- **Independent tuning** — epoch duration (default 20 ms) is tuned separately from data-group commit latency
43+
- **Geographic placement** — the sequencer triad can sit on the lowest-latency nodes without constraining where data groups live
44+
45+
### Epochs
46+
47+
The sequencer leader batches incoming transactions into epoch windows (default 20 ms). At the end of each window it:
48+
49+
1. Runs a pre-validation pass — detects intra-batch write-set conflicts; admits the first txn for a conflicting key, rejects others with `SequencerConflict` so the client retries
50+
2. Proposes the validated batch to the sequencer Raft group
51+
3. Once committed, fans the epoch out to each participating vShard's scheduler
52+
53+
### Determinism
54+
55+
All replicas must produce byte-identical WAL output for the same epoch. The executor is forbidden from using wall-clock time, non-seeded randomness, or non-deterministic map iteration on the cross-shard write path. System-time columns (bitemporal `sys_from`, KV TTL expiry, graph HLC ordinals) are seeded from the epoch timestamp so all replicas stamp the same value.
56+
57+
## Session variable
58+
59+
```sql
60+
-- Default: route multi-vShard writes through the sequencer (atomic)
61+
SET cross_shard_txn = 'strict';
62+
63+
-- Explicit opt-out: each shard commits independently (NOT atomic)
64+
SET cross_shard_txn = 'best_effort_non_atomic';
65+
66+
SHOW cross_shard_txn;
67+
```
68+
69+
`best_effort_non_atomic` is intended for bulk loads where atomicity is not required. It is deliberately named to discourage accidental use.
70+
71+
## EXPLAIN output
72+
73+
`EXPLAIN` reports whether a query routes through the sequencer:
74+
75+
```sql
76+
EXPLAIN INSERT INTO orders VALUES (...);
77+
-- cross-shard: sequenced | vshards: [3, 7] | epoch: <assigned at admission> | position: <assigned at admission>
78+
79+
EXPLAIN INSERT INTO local_cache VALUES (...);
80+
-- single-shard: vshard 3
81+
```
82+
83+
## OLLP — value-dependent predicates
84+
85+
For `UPDATE ... WHERE balance > 10000` where the write set depends on a scan result, NodeDB uses Optimistic Lock Location Prediction:
86+
87+
1. The planner runs the query optimistically once to capture the predicted write set
88+
2. Submits to the sequencer with the predicted set
89+
3. At execution time the executor re-scans and compares — if the set changed due to a concurrent commit, it returns `OllpRetryRequired` and retries transparently
90+
91+
To prevent retry storms, each predicate class (the parsed predicate AST, ignoring bound parameters) has:
92+
93+
- **Adaptive backoff** — starts at 10 ms, doubles up to 5 s
94+
- **Circuit breaker** — opens after >50% retry rate over a 60 s window; half-opens after 30 s; closes after 4 consecutive successes
95+
- **Per-tenant budget** — 1000 retries/min per tenant; excess returns `OllpTenantBudgetExceeded`
96+
97+
Static predicates (`WHERE id IN (...)`, `WHERE id = ?`) compute a deterministic write set at parse time and never enter the OLLP path.
98+
99+
## Admission limits
100+
101+
The sequencer enforces caps to prevent a single transaction from monopolising the cluster:
102+
103+
| Cap | Default | Error on exceed |
104+
|---|---|---|
105+
| `max_plans_bytes_per_txn` | 1 MiB | `TxnTooLarge` |
106+
| `max_participating_vshards_per_txn` | 64 | `FanoutTooWide` |
107+
| `max_txns_per_epoch` | 1024 | queued to next epoch |
108+
| `max_bytes_per_epoch` | 16 MiB | queued to next epoch |
109+
| Per-tenant inbox quota | inbox / 8 | `TenantQuotaExceeded` |
110+
111+
## Failure modes
112+
113+
| Failure | Behaviour |
114+
|---|---|
115+
| Sequencer leader dies | Raft re-elects; in-flight inbox submissions dropped; clients see `Unavailable` and retry |
116+
| Sequencer follower dies | No client-visible impact; quorum remains |
117+
| Scheduler shard crashes | Rebuilt from sequencer log on restart; shard unavailable until rebuild completes |
118+
| Executor panic mid-apply | Locks held; shard restarts and replays the txn from WAL (determinism ensures identical result) |
119+
| Network partition | Partitioned shard stops applying new epochs; serves reads with snapshot semantics; catches up on heal |
120+
121+
## Metrics
122+
123+
| Metric | Type | Description |
124+
|---|---|---|
125+
| `nodedb_sequencer_epochs_total` | counter | Epochs proposed |
126+
| `nodedb_sequencer_epoch_duration_ms` | histogram | Time to drain + propose each epoch |
127+
| `nodedb_sequencer_admitted_txns_total{outcome}` | counter | Per-outcome admission counts (`admitted`, `rejected_conflict`, `rejected_inbox_full`, `rejected_txn_too_large`, `rejected_fanout_too_wide`, `rejected_tenant_quota`, `rejected_not_leader`) |
128+
| `nodedb_sequencer_inbox_depth` | gauge | Pending txns in the inbox |
129+
| `nodedb_calvin_scheduler_lock_wait_ms_total{vshard}` | counter | Cumulative lock-wait time per shard |
130+
| `nodedb_calvin_executor_txn_duration_ms{vshard}` | histogram | Per-shard execution time |
131+
| `nodedb_calvin_ollp_retries_total{predicate_class,outcome}` | counter | OLLP retry outcomes |
132+
| `nodedb_calvin_ollp_circuit_state{predicate_class}` | gauge | 0=closed, 1=half-open, 2=open |
133+
| `nodedb_calvin_ollp_backoff_ms{predicate_class}` | gauge | Current OLLP backoff delay |
134+
| `nodedb_calvin_infra_abort_total{reason}` | counter | Infrastructure-level aborts (disk error, OOM, etc.) |

‎docs/architecture/multi-raft.rdx‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,27 @@ Each Raft group handles:
2525

2626
Writes are linearizable within each Raft group.
2727

28+
## Raft group kinds
29+
30+
NodeDB runs three kinds of Raft groups simultaneously:
31+
32+
| Kind | Purpose | Count |
33+
|---|---|---|
34+
| **Data** | One per vShard — replicates WAL entries for that shard's data | One per vShard |
35+
| **Meta** | Cluster membership, catalog, schema | One per cluster |
36+
| **Sequencer** | Cross-shard transaction ordering (Calvin epoch log) | One per cluster |
37+
38+
Each kind has independent leader election. A sequencer leader failure does not affect data-group leaders, and vice versa.
39+
40+
### Sequencer Raft group
41+
42+
The sequencer group exists solely to produce a globally-ordered log of cross-shard transaction batches (epochs). It has its own dedicated group ID outside the data-group range so it can never accidentally alias a vShard. See [Cross-Shard Transactions](./cross-shard-transactions) for how the sequencer group interacts with the scheduler and executor.
43+
44+
Single-shard writes never touch the sequencer group — they go directly through the relevant data-group's Raft.
45+
2846
## Advantages of Multi-Raft
2947

3048
- **Independent leaders** — different vShards can have leaders on different nodes, distributing write load
3149
- **Parallel commits** — vShards commit independently, no global ordering bottleneck
3250
- **Granular failover** — a node failure only triggers leader election for the vShards it led, not the entire cluster
51+
- **Failure isolation** — sequencer leader election is independent of data and meta group elections

‎docs/real-time/change-streams.rdx‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,21 @@ WITH (DELIVERY = 'kafka', BROKERS = 'localhost:9092', TOPIC = 'orders');
4545

4646
**HTTP long-poll** — `GET /v1/streams/{stream}/poll?group={group}&limit=100`.
4747

48+
The poll response includes gap-detection fields alongside the events:
49+
50+
```json
51+
{
52+
"events": [...],
53+
"evicted_since_last_poll": 0,
54+
"oldest_available_lsn": 19240
55+
}
56+
```
57+
58+
- `evicted_since_last_poll` — number of events dropped from the buffer since the previous poll call. A non-zero value means the consumer fell behind and the stream's ring buffer wrapped; events in the gap are permanently lost for this consumer.
59+
- `oldest_available_lsn` — the lowest LSN still present in the stream buffer. Consumers can compare this against their last-seen LSN to detect gaps without waiting for the next event to arrive.
60+
61+
The `nodedb_cdc_events_dropped_total{tenant,stream}` counter tracks drops per named stream (not globally). Alert on this increasing for a stream whose consumer is active.
62+
4863
## Streaming Materialized Views
4964

5065
```sql

0 commit comments

Comments
 (0)