docs: align documentation with current implementation

farhan-syah · farhan-syah · commit 81aeb6345c7b · 2026-04-24T07:01:41.000+08:00
Update docs across all major sections to match implemented behavior:

- Graph: require IN '&lt;collection&gt;' clause on GRAPH INSERT/DELETE EDGE;
  document that omitting IN is a parse error
- SQL: document INSERT ... ON CONFLICT (DO NOTHING / DO UPDATE SET)
  with EXCLUDED semantics; clarify that UPSERT is equivalent to ON
  CONFLICT DO UPDATE over all columns; note trigger op-tag derivation
  from storage prior-bytes, not surface SQL verb
- DDL: document two-phase DROP COLLECTION lifecycle — tombstone,
  UNDROP within retention window, PURGE for immediate hard-delete;
  add retention knobs and _system.dropped_collections / l2_cleanup_queue
  inspection queries; note cascade to materialized views and
  change-stream consumer groups
- CSR index: reflect tenant partitioning via ShardedCsrIndex, structural
  tenant isolation in B-Tree keys, LocalNodeId partition-tag boundary
  enforcement, and label width change (u16 → u32)
- Columnar: add PK point-get and ORDER BY examples; document that plain
  INSERT raises unique_violation on PK conflict
- KV: add UPSERT, ON CONFLICT DO NOTHING, and ON CONFLICT DO UPDATE
  examples alongside existing INSERT
- WAL: document tombstone GC during checkpoint and crash-safe replay
- Backup/recovery: note catalog reconstruction and tombstone-set
  embedding in backup envelopes
- Change streams: clarify DROP stream teardown and name-reuse semantics
- Error codes: expand CONSTRAINT_VIOLATION description with SQLSTATE
  and ON CONFLICT alternatives
- fusion-rrf, graph-queries: add required VECTOR_FIELD clause to
  GraphRAG VECTOR_SEARCH examples
diff --git a/docs/administration/backup-recovery.rdx b/docs/administration/backup-recovery.rdx
@@ -13,6 +13,8 @@ BACKUP TENANT acme TO '/backups/acme-2026-04.bak';
 
 Backups cover all 7 engines: documents, indexes, vectors, graph edges, KV tables, timeseries, and CRDT state. Encrypted with AES-256-GCM using the tenant WAL key.
 
+The backup envelope embeds catalog rows and the source tombstone set alongside segment data, so a restored snapshot reconstructs the catalog deterministically and refuses to resurrect collections tombstoned before the backup was taken. Each `StoredCollection` row carries a `size_bytes_estimate` field surfaced through `_system.dropped_collections` for sizing the L2 cleanup queue before `PURGE`.
+
 ## Validate
 
 ```sql
diff --git a/docs/architecture/wal.rdx b/docs/architecture/wal.rdx
@@ -28,6 +28,8 @@ The WAL ensures durability. Every write is persisted to the WAL before being ack
 
 **Segmented** — The WAL rolls over to a new segment file automatically. Old segments are eligible for cleanup once all records have been flushed to L1 segments.
 
+**Tombstone GC** — Each checkpoint garbage-collects WAL rows for collections that have been hard-deleted (tombstoned), so tombstone records do not accumulate across restarts. On replay, the startup path merges persisted WAL tombstones with tombstones extracted from the WAL itself — a crash mid-purge cannot resurrect a dropped collection.
+
 **Encryption** — Optional AES-256-GCM encryption at the page level. Key management is external.
 
 ## Crash Recovery
diff --git a/docs/data-modeling/btree-index.rdx b/docs/data-modeling/btree-index.rdx
@@ -29,4 +29,4 @@ CREATE UNIQUE INDEX ON users FIELDS email;
 
 ## Graph Edge Storage
 
-Graph edges are persisted in redb B-Trees with forward and reverse indexes. The key format (`src_id\x00label\x00dst_id`) enables efficient prefix scans for outbound traversal.
+Graph edges are persisted in redb B-Trees with forward and reverse indexes, keyed by `(tenant_id: u32, "src\x00label\x00dst")` tuples. Tenant isolation is a first-class key component, not a lexical prefix; the composite portion enables prefix scans for outbound traversal within a tenant.
diff --git a/docs/data-modeling/collections.rdx b/docs/data-modeling/collections.rdx
@@ -67,7 +67,7 @@ CREATE SEARCH INDEX ON products FIELDS title, description ANALYZER 'english';
 CREATE SPATIAL INDEX ON products FIELDS location;
 
 -- Add graph edges for relationships
-GRAPH INSERT EDGE FROM 'products:p1' TO 'products:p2' TYPE 'similar';
+GRAPH INSERT EDGE IN 'products' FROM 'products:p1' TO 'products:p2' TYPE 'similar';
 ```
 
 ## Converting Between Engines
diff --git a/docs/data-modeling/columnar.rdx b/docs/data-modeling/columnar.rdx
@@ -60,16 +60,27 @@ CREATE COLLECTION locations TYPE COLUMNAR (
 ## Queries
 
 ```sql
+-- Point-get by primary key hits the segment PK index (not a full scan).
+SELECT * FROM logs WHERE ts = '2026-04-24T10:00:00Z';
+
 SELECT level, COUNT(*) FROM logs
 WHERE ts > now() - INTERVAL '1 hour'
 GROUP BY level ORDER BY COUNT(*) DESC;
 
+-- ORDER BY is supported on columnar scans
+SELECT ts, host, message FROM logs
+WHERE level = 'error'
+ORDER BY ts DESC
+LIMIT 100;
+
 -- Window functions
 SELECT host, message,
        ROW_NUMBER() OVER (PARTITION BY host ORDER BY ts DESC) AS rank
 FROM logs;
 ```
 
+Plain `INSERT` on a columnar collection raises `unique_violation` on primary-key conflict; use `UPSERT` or `INSERT ... ON CONFLICT (pk) DO UPDATE SET col = EXCLUDED.col` for overwrite semantics.
+
 ## HTAP Bridge
 
 Combine strict (OLTP) with columnar (OLAP):
diff --git a/docs/data-modeling/csr-index.rdx b/docs/data-modeling/csr-index.rdx
@@ -10,21 +10,26 @@ CSR (Compressed Sparse Row) is the graph engine's core index format. It stores a
 ## Layout
 
 ```
-out_offsets: Vec<u32>        [num_nodes + 1]  — offset into target array per node
-out_targets: DenseArray<u32> [num_edges]      — destination node IDs (contiguous, mmap-capable)
-out_labels:  DenseArray<u16> [num_edges]      — edge labels (parallel array)
-out_weights: Option<DenseArray<f64>>           — optional, allocated only when weighted
+CsrIndex (per tenant):
+  out_offsets:  Vec<u32>                 [num_nodes + 1]  — offset into target array per node
+  out_targets:  DenseArray<u32>          [num_edges]      — destination node IDs (contiguous, mmap-capable)
+  out_labels:   DenseArray<u32>          [num_edges]      — edge labels (parallel array)
+  out_weights:  Option<DenseArray<f64>>                   — optional, allocated only when weighted
 
-in_offsets / in_targets / in_labels / in_weights — symmetric for inbound
+  in_offsets / in_targets / in_labels / in_weights — symmetric for inbound
 ```
 
+## Tenant Partitioning
+
+The in-memory index is `ShardedCsrIndex` — one `CsrIndex` per tenant. Algorithms and traversals receive a single tenant's partition; there is no lexical tenant prefix on node names. Each `CsrIndex` is assigned a unique partition tag at construction, and public APIs that return dense node indices hand out `LocalNodeId { id, partition_tag }`. Using a node id from one partition with another partition's API panics at the boundary.
+
 ## Memory Efficiency
 
-At 1 billion edges, CSR uses ~10 GB vs ~60 GB for naive adjacency lists (6x improvement). Node IDs are interned as `u32`, labels as `u16`.
+At 1 billion edges, CSR uses ~10 GB vs ~60 GB for naive adjacency lists (6x improvement). Node IDs are interned as `u32`, labels as `u32`.
 
 ## Storage
 
-Edges are persisted in a redb B-Tree with forward and reverse indexes. The CSR index is built at query time for bulk operations. Writes go to a mutable buffer and become visible immediately. Compaction merges the buffer into dense CSR arrays when the buffer exceeds 10% of the dense size.
+Edges are persisted in a redb B-Tree with forward and reverse indexes, both keyed by `(tenant_id: u32, "src\x00label\x00dst")` tuples. Tenant isolation is structural (first-class key component), not lexical. The CSR index is built at query time for bulk operations. Writes go to a mutable buffer and become visible immediately. Compaction merges the buffer into dense CSR arrays when the buffer exceeds 10% of the dense size.
 
 ## Graph Operations
 
diff --git a/docs/data-modeling/graph.rdx b/docs/data-modeling/graph.rdx
@@ -24,7 +24,8 @@ CREATE COLLECTION people;
 INSERT INTO people (id, name) VALUES ('alice', 'Alice');
 INSERT INTO people (id, name) VALUES ('bob', 'Bob');
 
-GRAPH INSERT EDGE FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020, weight: 0.9 };
+GRAPH INSERT EDGE IN 'people' FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020, weight: 0.9 };
+-- The IN '<collection>' clause is required — edges live on a named collection.
 ```
 
 ## Traversal
@@ -103,6 +104,7 @@ Combines vector similarity with graph traversal in one query:
 ```sql
 GRAPH RAG FUSION ON entities
   QUERY $embedding
+  VECTOR_FIELD 'embedding'
   VECTOR_TOP_K 50
   EXPANSION_DEPTH 2
   EDGE_LABEL 'related_to'
diff --git a/docs/data-modeling/kv.rdx b/docs/data-modeling/kv.rdx
@@ -20,9 +20,22 @@ Purpose-built hash-indexed store with O(1) point lookups, native TTL, and second
 ```sql
 CREATE COLLECTION sessions TYPE KEY_VALUE (key TEXT PRIMARY KEY);
 
--- Insert with TTL
+-- Insert with TTL; plain INSERT raises unique_violation (23505) on duplicate key.
 INSERT INTO sessions { key: 'sess_abc', user_id: 'alice', role: 'admin', ttl: 3600 };
 
+-- Set-or-overwrite (Redis SET semantics)
+UPSERT INTO sessions { key: 'sess_abc', user_id: 'alice', role: 'admin', ttl: 3600 };
+
+-- Set-if-absent (Redis SETNX semantics)
+INSERT INTO sessions { key: 'sess_abc', user_id: 'alice', role: 'admin', ttl: 3600 }
+ON CONFLICT DO NOTHING;
+
+-- Conditional merge: bump counter on conflict, EXCLUDED references the incoming row.
+INSERT INTO sessions (key, user_id, role, hits) VALUES ('sess_abc', 'alice', 'admin', 1)
+ON CONFLICT (key) DO UPDATE SET
+  role = EXCLUDED.role,
+  hits = sessions.hits + 1;
+
 -- Get by key
 SELECT * FROM sessions WHERE key = 'sess_abc';
 
diff --git a/docs/introduction/quickstart.rdx b/docs/introduction/quickstart.rdx
@@ -84,7 +84,7 @@ CREATE COLLECTION social;
 INSERT INTO social (id, name) VALUES ('alice', 'Alice');
 INSERT INTO social (id, name) VALUES ('bob', 'Bob');
 
-GRAPH INSERT EDGE FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020 };
+GRAPH INSERT EDGE IN 'social' FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020 };
 
 GRAPH TRAVERSE FROM 'alice' DEPTH 2;
 
diff --git a/docs/real-time/change-streams.rdx b/docs/real-time/change-streams.rdx
@@ -18,6 +18,9 @@ WITH (URL = 'https://hooks.example.com/orders');
 CREATE CHANGE STREAM user_state ON users WITH (COMPACTION = 'key', KEY = 'id');
 
 DROP CHANGE STREAM order_events;
+-- Drop atomically tears down the stream's consumer groups and persisted offset
+-- rows. Recreating a stream with the same name starts from the head; it does
+-- not resume at stale offsets.
 SHOW CHANGE STREAMS;
 ```
 
diff --git a/docs/reference/error-codes.rdx b/docs/reference/error-codes.rdx
@@ -29,7 +29,7 @@ NodeDB uses `NodeDbError` — a struct with numeric `ErrorCode`, human-readable
 | ---------------------- | ---- | --------------------------- |
 | `COLLECTION_NOT_FOUND` | 1100 | Check collection name       |
 | `TYPE_MISMATCH`        | 1020 | Check data types in query   |
-| `CONSTRAINT_VIOLATION` | 1000 | Duplicate key — use UPSERT  |
+| `CONSTRAINT_VIOLATION` | 1000 | Duplicate key on plain INSERT maps to SQLSTATE `23505` (`unique_violation`). Use `UPSERT` or `INSERT ... ON CONFLICT DO NOTHING / DO UPDATE` for non-error semantics. |
 | `DEADLINE_EXCEEDED`    | 1002 | Query took too long         |
 | `INSUFFICIENT_BALANCE` | 1022 | Not enough for TRANSFER     |
 | `OVERFLOW`             | 1021 | i64 overflow on INCR        |
diff --git a/docs/sql/ddl.rdx b/docs/sql/ddl.rdx
@@ -28,11 +28,27 @@ CREATE COLLECTION locations TYPE COLUMNAR (geom GEOMETRY SPATIAL_INDEX, name VAR
 -- Convenience alias
 CREATE TIMESERIES metrics;
 
-DROP COLLECTION users;
+DROP COLLECTION users;          -- soft-delete: tombstoned, still restorable
+UNDROP COLLECTION users;        -- restore while within the retention window
+DROP COLLECTION users PURGE;    -- admin-only: immediate hard-delete, not restorable
 SHOW COLLECTIONS;
 DESCRIBE users;
 ```
 
+`DROP COLLECTION` is two-phase. A plain drop marks the collection as tombstoned and enqueues background L2 cleanup after a configurable retention window; within that window, `UNDROP COLLECTION` restores the collection and its data. Admins can bypass the window with `DROP ... PURGE`.
+
+```sql
+-- Retention window knobs
+ALTER SYSTEM SET collection_gc_retention = '7d';
+ALTER TENANT acme SET collection_gc_retention = '24h';
+
+-- Inspect the lifecycle
+SELECT name, dropped_at, size_bytes_estimate FROM _system.dropped_collections;
+SELECT * FROM _system.l2_cleanup_queue;
+```
+
+Dropping a collection also cascades downstream catalog state: dependent materialized views are unregistered, change-stream consumer groups are torn down, and the L2 cleanup worker reclaims segment files.
+
 ## Schema Evolution
 
 ```sql
@@ -76,6 +92,8 @@ DROP TRIGGER notify ON orders;
 SHOW TRIGGERS;
 ```
 
+The op tag (INSERT vs UPDATE) is derived from storage prior-bytes, not the surface SQL verb. An `UPSERT` or `INSERT ... ON CONFLICT (pk) DO UPDATE` that overwrites an existing row fires `AFTER UPDATE`; the same statement against a non-existent key fires `AFTER INSERT`. `ON CONFLICT DO NOTHING` emits no event on conflict.
+
 ## Functions & Procedures
 
 ```sql
diff --git a/docs/sql/fusion-rrf.rdx b/docs/sql/fusion-rrf.rdx
@@ -26,7 +26,7 @@ LIMIT 10;
 
 ```sql
 GRAPH RAG FUSION ON entities
-  QUERY $embedding VECTOR_TOP_K 50 EXPANSION_DEPTH 2
+  QUERY $embedding VECTOR_FIELD 'embedding' VECTOR_TOP_K 50 EXPANSION_DEPTH 2
   EDGE_LABEL 'related_to' FINAL_TOP_K 10 RRF_K (60.0, 35.0);
 ```
 
diff --git a/docs/sql/graph-queries.rdx b/docs/sql/graph-queries.rdx
@@ -8,10 +8,12 @@ description: GRAPH commands, MATCH pattern matching, and algorithm execution in
 ## Edge Operations
 
 ```sql
-GRAPH INSERT EDGE FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020 };
-GRAPH DELETE EDGE FROM 'alice' TO 'bob' TYPE 'knows';
+GRAPH INSERT EDGE IN 'edges' FROM 'alice' TO 'bob' TYPE 'knows' PROPERTIES { since: 2020 };
+GRAPH DELETE EDGE IN 'edges' FROM 'alice' TO 'bob' TYPE 'knows';
 ```
 
+The `IN '<collection>'` clause is **required** — edges are overlays on a named document collection. Statements without `IN` fail to parse.
+
 ## Traversal
 
 ```sql
@@ -59,6 +61,6 @@ Available: `pagerank`, `wcc`, `label_propagation`, `lcc`, `sssp`, `betweenness`,
 
 ```sql
 GRAPH RAG FUSION ON entities
-  QUERY $embedding VECTOR_TOP_K 50 EXPANSION_DEPTH 2
+  QUERY $embedding VECTOR_FIELD 'embedding' VECTOR_TOP_K 50 EXPANSION_DEPTH 2
   EDGE_LABEL 'related_to' FINAL_TOP_K 10 RRF_K (60.0, 35.0);
 ```
diff --git a/docs/sql/insert-upsert.rdx b/docs/sql/insert-upsert.rdx
@@ -21,9 +21,28 @@ INSERT INTO users { name: 'Alice', email: 'alice@example.com', age: 30 };
 INSERT INTO archive SELECT * FROM orders WHERE created_at < '2025-01-01';
 ```
 
+Plain `INSERT` is strict: a duplicate primary key raises `unique_violation` (SQLSTATE `23505`). Use `ON CONFLICT` or `UPSERT` for "insert if absent" or "insert or overwrite" semantics.
+
+## INSERT ... ON CONFLICT
+
+```sql
+-- Skip rows that would collide with an existing PK (no error)
+INSERT INTO users (id, name) VALUES ('u1', 'Alice')
+ON CONFLICT DO NOTHING;
+
+-- Overwrite selected fields on conflict. EXCLUDED refers to the incoming row;
+-- bare column names refer to the existing row.
+INSERT INTO users (id, name, login_count) VALUES ('u1', 'Alice', 1)
+ON CONFLICT (id) DO UPDATE SET
+  name        = EXCLUDED.name,
+  login_count = users.login_count + EXCLUDED.login_count;
+```
+
+`ON CONFLICT DO UPDATE` fires `AFTER UPDATE` triggers (not `AFTER INSERT`) when a row is overwritten — the trigger op tag is derived from storage prior-bytes, not the surface SQL verb.
+
 ## UPSERT
 
-Insert or merge if the document ID already exists:
+Insert or overwrite if the primary key already exists:
 
 ```sql
 UPSERT INTO users (id, name, role) VALUES ('u1', 'Alice', 'admin');
@@ -32,6 +51,8 @@ UPSERT INTO users (id, name, role) VALUES ('u1', 'Alice', 'admin');
 UPSERT INTO users { id: 'u1', name: 'Alice', role: 'admin' };
 ```
 
+`UPSERT` is equivalent to `INSERT ... ON CONFLICT (<pk>) DO UPDATE SET <all-columns> = EXCLUDED.<col>`. It fires `AFTER UPDATE` on overwrite and `AFTER INSERT` on first write.
+
 ## Bulk Import
 
 ```sql

Original file line number	Diff line number	Diff line change
`@@ -29,4 +29,4 @@ CREATE UNIQUE INDEX ON users FIELDS email;`
`29`	`29`
`30`	`30`	`## Graph Edge Storage`
`31`	`31`
`32`		-Graph edges are persisted in redb B-Trees with forward and reverse indexes. The key format (`src_id\x00label\x00dst_id`) enables efficient prefix scans for outbound traversal.
	`32`	+Graph edges are persisted in redb B-Trees with forward and reverse indexes, keyed by `(tenant_id: u32, "src\x00label\x00dst")` tuples. Tenant isolation is a first-class key component, not a lexical prefix; the composite portion enables prefix scans for outbound traversal within a tenant.