Skip to content

Commit 0ab4460

Browse files
saubyksaubyk
committed
docs: added a user guide for db migration
1 parent 1a30ac2 commit 0ab4460

File tree

1 file changed

+224
-0
lines changed

1 file changed

+224
-0
lines changed

‎docs/db_migration_guide.md‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
# `LND Database Upgrade Guide`: A Two-Stage Migration for Node Operators
2+
3+
*Table of Contents*
4+
- [Overview](#overview)
5+
- [Operation Modes](#operation-modes)
6+
- [Stage 1: Migration from bbolt to SQLite/Postgres (kvdb)](#stage-1-migration-from-bbolt-to-sqlitepostgres-kvdb)
7+
- [Choosing Your Target Backend](#choosing-your-target-backend)
8+
- [Postgres kvdb Migration](#postgres-kvdb-migration)
9+
- [SQLite kvdb Migration](#sqlite-kvdb-migration)
10+
- [Stage 2: Migration from kvdb to Relational Database](#stage-2-migration-from-kvdb-to-relational-database)
11+
- [Subsystem Readiness](#subsystem-readiness)
12+
- [Known Limitations and Edge Cases](#known-limitations-and-edge-cases)
13+
- [Best Practices for Node Operators](#best-practices-for-node-operators)
14+
- [Choosing the Right Path](#choosing-the-right-path)
15+
- [Timing Your Migration](#timing-your-migration)
16+
- [Backup and Validation](#backup-and-validation)
17+
- [Implementation Examples](#implementation-examples)
18+
- [Migrating to SQLite kvdb](#migrating-to-sqlite-kvdb)
19+
- [Migrating Invoices to Relational Backend](#migrating-invoices-to-relational-backend)
20+
- [Future Improvements](#future-improvements)
21+
- [Conclusion](#conclusion)
22+
23+
---
24+
25+
## Overview
26+
27+
The `LND Database Upgrade` process enables node operators to migrate from the legacy **bbolt** key-value store to modern **SQLite** or **PostgreSQL** backends—first in **kvdb mode**, then ultimately to a **relational database** format. This two-stage migration addresses long-standing performance bottlenecks, and lays the foundation for scalable, maintainable node operations.
28+
29+
This guide explains the migration path, backend trade-offs, subsystem readiness, and best practices for safely upgrading your LND database.
30+
31+
---
32+
33+
## Operation Modes
34+
35+
The LND database upgrade operates in two sequential stages, each with distinct tooling and implications:
36+
37+
```mermaid
38+
flowchart TD
39+
subgraph "Step 1: Migration from bbolt to kvdb (via lndinit)"
40+
direction TB
41+
Bbolt["Bbolt (kvdb)"] --> SQLite["SQLite (kvdb)"]
42+
Bbolt --> Postgres["Postgres (kvdb)"]
43+
end
44+
45+
subgraph "Step 2: Migration from kvdb to Native SQL (via lnd)"
46+
direction TB
47+
M1_Sqlite["Migration #1 (lnd v0.19)\nInvoices"]
48+
M1_Sqlite --> M2_Sqlite["Migration #2 (lnd v0.20)\nGraph"]
49+
M2_Sqlite --> M3_Sqlite["Migration #3 (lnd v0.21)\nPayments"]
50+
51+
M1_Postgres["Migration #1 (lnd v0.19)\nInvoices"]
52+
M1_Postgres --> M2_Postgres["Migration #2 (lnd v0.20)\nGraph"]
53+
M2_Postgres --> M3_Postgres["Migration #3 (lnd v0.21)\nPayments"]
54+
end
55+
56+
%% Define links between subgraphs externally
57+
SQLite --> M1_Sqlite
58+
Postgres --> M1_Postgres
59+
60+
%% Styles
61+
style Bbolt fill:#e2e3e5,stroke:#383d41
62+
style SQLite fill:#d4edda,stroke:#155724
63+
style M1_Sqlite fill:#d4edda,stroke:#155724
64+
style M2_Sqlite fill:#d4edda,stroke:#155724
65+
style M3_Sqlite fill:#d4edda,stroke:#155724
66+
style Postgres fill:#cce5ff,stroke:#004085
67+
style M1_Postgres fill:#cce5ff,stroke:#004085
68+
style M2_Postgres fill:#cce5ff,stroke:#004085
69+
style M3_Postgres fill:#cce5ff,stroke:#004085
70+
71+
```
72+
73+
- **Stage 1**: Migrate from bbolt to a SQL-based **kvdb** backend using the [lndinit](https://github.com/lightninglabs/lndinit/blob/main/docs/data-migration.md) tool.
74+
- **Stage 2**: Incrementally migrate subsystem data (invoices, graph, payments, etc.) from **kvdb** to SQL native **relational tables** as support becomes available.
75+
76+
---
77+
78+
## Stage 1: Migration from bbolt to SQLite/Postgres (kvdb)
79+
80+
LND cannot run with mixed backends, so all users must first leave bbolt behind. This stage uses the `lndinit` utility to perform an offline migration.
81+
82+
### Choosing Your Target Backend
83+
84+
| Backend | Performance (in kvdb mode) | Default in LND? | Long-Term Viability |
85+
|-----------|--------------------------|------------------|----------------------|
86+
| Postgres | Mediocre | No | ✅ |
87+
| SQLite | Good | **Yes (future)** | ✅ |
88+
89+
> 💡 **Recommendation**: Unless you require Postgres for infrastructure reasons, **migrate to SQLite kvdb** as your interim backend.
90+
91+
### Postgres kvdb Migration
92+
93+
A migration script is available via `lndinit`:
94+
95+
- [Postgres migration script](https://github.com/lightninglabs/lndinit/blob/main/docs/data-migration.md#using-postgres-as-the-destination-remote-database)
96+
- **Caveat**: Users report degraded performance in kvdb mode. Only proceed if you plan to **immediately follow with Stage 2** to migrate the available data stores to relational DB.
97+
This will mitigate the poor Postgres performance on kvdb.
98+
99+
### SQLite kvdb Migration
100+
101+
- [SQLite migration script](https://github.com/lightninglabs/lndinit/blob/main/docs/data-migration.md#using-sqlite-as-the-destination-remote-database)
102+
- **Advantage**: Maintains good performance while waiting for full relational migration.
103+
104+
---
105+
106+
## Stage 2: Migration from kvdb to Relational Database
107+
108+
This stage unlocks true SQL performance by restructuring data into relational tables. Migration is **per-subsystem** and **incremental**.
109+
110+
### Subsystem Readiness
111+
112+
| Subsystem | Relational Backend | Migration Script | Status |
113+
|---------------------|--------------------|------------------|--------|
114+
| Invoices | ✅ Available | ✅ | Available with **v0.19** |
115+
| Graph | ✅ Available | ✅ | Available with **v0.20** |
116+
| Payments | 🚧 In Progress | Planned | Targeted with **v0.21**|
117+
| Btcwallet | 🚧 In Progress | Planned | Targeted with **v0.21**|
118+
| Forwarding History | ❌ TBD | ❌ TBD | Future work |
119+
120+
---
121+
122+
## Known Limitations and Edge Cases
123+
124+
- **Single database engine required**: LND requires a single consistent backend.
125+
You cannot run invoices in relational mode while payments remain in kvdb *unless*
126+
both are on the same SQL engine (e.g., SQLite).
127+
- **Data loss risk**: Always **back up your `data/` directory** before migration.
128+
- **Downtime required**: Stage 1 requires LND to be offline. Stage 2 is done at startup, requiring a LND restart.
129+
- **Postgres kvdb performance**: Postgres performance on kvdb is sub-optimal. It is
130+
recommended to make the stage 2 migration immediately to avoid performance bottlenecks. Certain RPCs like `listpayments` may not perform well on Postgres if the node has a lot of payments data. If your node operation is heavy
131+
on payments and `listpayments` performance is critical for you, we'd recommend
132+
not doing any migration and wait till version 0.21.0 is released.
133+
- **No migration path between SQL backend**: Once migrated to either Postgres or
134+
SQLite, it is not possible to switch to the other, so choose your target backend carefully.
135+
---
136+
137+
## Best Practices for Node Operators
138+
139+
### Choosing the Right Path
140+
141+
- **For most users**: Choose SQLite, then migrate. Later, adopt relational backends subsystem-by-subsystem.
142+
- **Enterprise/Postgres users**: Recommend waiting to start the migration
143+
until **payments relational backend** is ready, then perform **Stage 1 + Stage 2 in quick succession**.
144+
145+
### Timing Your Migration
146+
147+
- Perform migrations during **low-activity periods**.
148+
- Monitor LND release notes for relational db support of different subsystems.
149+
150+
### Backup and Validation
151+
152+
1. Stop LND.
153+
2. Backup entire `~/.lnd/` directory.
154+
3. Run migration with `lndinit`.
155+
4. Start LND with new backend flags, to execute stage 2 migrations.
156+
5. Validate node health: channels, balance, invoice/payment history.
157+
158+
---
159+
160+
## Implementation Examples
161+
162+
### Migrating to SQLite kvdb
163+
164+
```bash
165+
# Stop LND
166+
lnd --shutdown
167+
168+
# Backup
169+
cp -r ~/.lnd ~/lnd-backup-$(date +%Y%m%d)
170+
171+
# Run migration (e.g. sqlite)
172+
lndinit --debuglevel info migrate-db \
173+
--source.bolt.data-dir /home/myuser/.lnd/data \
174+
--dest.backend sqlite \
175+
--dest.sqlite.data-dir /home/myuser/.lnd/data --network mainnet
176+
177+
# Start LND with SQLite backend
178+
lnd --db.backend=sqlite
179+
```
180+
181+
> 📝 Add `db.backend=sqlite` to your `lnd.conf` to make it persistent.
182+
183+
### Migrating Invoices to Relational Backend
184+
185+
Once on LND v0.19+ with SQLite/Postgres:
186+
187+
```bash
188+
# Ensure backend is set
189+
echo "db.backend=sqlite" >> ~/.lnd/lnd.conf
190+
191+
# Start LND — invoice migration runs automatically
192+
lnd
193+
```
194+
195+
Check logs for:
196+
```
197+
Migrating invoices from kvdb to relational format...
198+
Invoice migration completed successfully.
199+
```
200+
201+
---
202+
203+
## Future Improvements
204+
205+
The LND team is actively working on:
206+
207+
- **Payments relational backend** and migration tooling (Stage 2)
208+
- **Btcwallet relational backend** and migration tooling (Stage 2)
209+
- **Forwarding history** relational schema (long-term)
210+
- **Automatic detection** of migration readiness in `lnd`
211+
212+
Node operators should monitor:
213+
- [LND GitHub Releases](https://github.com/lightningnetwork/lnd/releases)
214+
- [lndinit repository](https://github.com/lightninglabs/lndinit)
215+
216+
---
217+
218+
## Conclusion
219+
220+
The LND database upgrade is a strategic two-stage process designed to eliminate bbolt performance limitations while ensuring data integrity and operational continuity.
221+
222+
By **first migrating to SQLite/Postgres kvdb** and **then adopting relational backends incrementally**, node operators can achieve significant performance gains—especially for payment-heavy workloads—without rushing into unstable configurations.
223+
224+
Choose your path wisely, back up rigorously, and stay informed. The future of LND is relational, and this guide ensures you get there safely.

0 commit comments

Comments
 (0)