tapdb: add implementation of supplycommit.CommitmentTracker and supplycommit.StateMachineStore #1508

Roasbeef · 2025-05-03T04:02:50Z

In this PR, we add a concrete implementation of the two fundamental interfaces that will drive the state machine implemented in #1464.

We start with a state machine on disk for a given group key that also points to the latest supply commitment. When we want to add new elements to the supply tree, we create a new state transition, and then queue pending updates associated with this state transition. Once we've signed and broadcasted the commitment transaction, we then apply the state transition: insert all the updates into the tree, update the state machine to point to that new tree, and finalize the state transition.

See the readme for more details.

ffranr · 2025-05-29T23:03:17Z

tapdb/sqlc/migrations/000036_asset_commit.up.sql

+-- Table storing individual update events associated with a pending transition.
+CREATE TABLE supply_update_events (
+    event_id INTEGER PRIMARY KEY,
+
+    -- Reference to the state transition this event is part of.
+    transition_id BIGINT NOT NULL REFERENCES supply_commit_transitions(transition_id) ON DELETE CASCADE,


an update event can't be persisted unless it references an existing transition, is that correct?
If so, then if we're waiting for tx conf then new update events can't be persisted?

If true, then I don't think it matters a great deal, but from a UX perspective, we will need to return an error to the user along the lines of: unconfirmed supply tx, can't accept new asset ignore right now.

That's a great insight. As of right now, if everything is spun up together and a user attempts to add a new update while we're creating a new commitment on-disk, then this'll fail.

I think we can fix this just by removing the NOT NULL here. We'll need some additional follow ups in the state machine though:

Any time an update comes through in a state, attempt to insert it to disk w/ a nil transition_id.

Once we circle back to the default state, then read all the "dangling" update from disk, and assign them to a state transition (in a single db txn).

Alternatively, we can create some other table, then re-insert them in the default state.

I can either do this in another PR (as it affects this one, and the next), or do the incremental change here. Which is preferred?

guggero

Great stuff! Did a first pass to load some context, will continue with the other PRs in the saga.

tapdb/sqlc/migrations/000036_asset_commit.up.sql

guggero · 2025-06-03T12:19:32Z

tapdb/sqlc/queries/supply_commit.sql

+) VALUES (
+    @group_key,
+    -- Use the target state ID if found, otherwise use the default state ID.
+    coalesce((SELECT id FROM target_state), (SELECT id FROM default_state)),


Hmm, I'm a bit surprised this works. So the actual value returned from a sub query is NULL if there was no row found?

Hehe, yeah I went through a few versions of this. I had one which used a CTE as above as part of the update (so a CTE that inserts, then returns a value), but sqlite doesn't allow CTE statements in insert/update like say postgres does.

To answer the question, we basically want either the ID of the target state, or the default if a target isn't set. So on the caller side, we just don't specify the state_name field if we want the default state to be inserted. If no state is set, then target_state has no rows, so trying to select on it returns nothing. In that case the default_state query will always return a value (as we set up the table as such in the migration).

tapdb/supply_commit.go

guggero · 2025-06-03T12:25:56Z

tapdb/supply_commit.go

+	assetSpec asset.Specifier) lfn.Result[supplycommit.PreCommits] {
+
+	groupKey := assetSpec.UnwrapGroupKeyToPtr()
+	if groupKey == nil {


nit: UnwrapGroupKeyOrErr()? Not that it shortens the code by any meaningful amount, just a bit more explicit.

Or, I guess since we're only ever using the group key part of the specifier, why don't we change the param to be a public key directly? Just to have some semantic meaning around what the key is?

Switched everything over to use UnwrapGroupKeyOrErr.

I used asset.Specifier as most other similar interfaces across the codebase are starting to take that value, to force an author/writer to make sure there's logic to handle the various combos.

tapdb/supply_commit.go

guggero · 2025-06-03T19:17:51Z

tapdb/supply_commit.go

+	// NULL (e.g., initial commit before ApplyStateTransition ran), use the
+	// empty root.
+	var rootNode *mssmt.BranchNode
+	if commit.SupplyRootHash == nil || !commit.SupplyRootSum.Valid {


Check length instead of nil?

guggero · 2025-06-03T19:18:59Z

tapdb/supply_commit.go

+// its associated chain confirmation details.
+func fetchCommitment(ctx context.Context, db SupplyCommitStore,
+	commitID sql.NullInt64, groupKeyBytes []byte,
+) (lfn.Option[supplycommit.RootCommitment], lfn.Option[commitmentChainInfo], error) { //nolint:lll


Does the root commitment here really need to be an option? Doesn't look like it's ever None in a non-error case.

I have it as an option as in supplycommit.SupplyStateTransition, it's an option. It's an option here as the very first time we go to create a state transition, OldCommitment won't yet exist.

We can return a value of None in the very fist error case: if we query and it doesn't exist (ErrNoRows), then noneRootCommit with a nil error is returned.

guggero · 2025-06-03T19:20:53Z

tapdb/supply_commit_test.go

+// assertPendingTransitionExists asserts that a pending (non-finalized)
+// transition exists.
+//
+//nolint:lll


We might as well add this to the top of the file so it counts for its entirety 😅

Heh, I prefer to just sprinkle it as needed. The linter comments don't really both me when reading/reviewing code.

In this instance, I added it as there's no good place to split the line w/o it looking awkward. I'd rather just throw this on and move along instead of spending additional time on the more marginal aspects of our code style.

guggero · 2025-06-03T19:26:04Z

tapdb/supply_commit.md

+    }
+```
+
+## State Machine Persistence Logic


Very nice doc!

… tree state machine In this commit, we add a new migration for the persistence layer of the supply tree state machine. We track a state machine that points to the latest supply commitment. We update the supply commit via a new state transition, which refrerences a series of updates. Once we're ready to apply, we'll apply the updates, mark the transition as finalized, and finally update the relevant pointers. We also make a change to modify mint_anchor_uni_commitments to point to a supply commitment. This lets us keep track of the set of pre commitments that aren't yet spent.

… new spent_by field

…and supplycommit.StateMachineStore In this commit, we add concrete implementation of both CommitmentTracker and StateMachineStore. This implements the persistence layer for the supply commit state machine. The main method to understand is ApplyStateTransition, as it implements the atomic update of all the various components (tree, state transition, etc) on disk.

lightninglabs-deploy · 2025-06-24T02:40:02Z

@Roasbeef, remember to re-request review from reviewers when ready

Roasbeef added the supply tree label May 3, 2025

levmi assigned Roasbeef May 5, 2025

levmi added this to Taproot-Assets Project Board May 5, 2025

github-project-automation bot moved this to 🆕 New in Taproot-Assets Project Board May 5, 2025

levmi moved this from 🆕 New to 🏗 In progress in Taproot-Assets Project Board May 5, 2025

Roasbeef force-pushed the supply-commit-machine branch from 4778c90 to d718094 Compare May 5, 2025 22:46

Roasbeef force-pushed the universe-supply-trees branch 2 times, most recently from 522c3ee to d786104 Compare May 5, 2025 23:04

Roasbeef force-pushed the supply-commit-machine branch from d718094 to 1db2104 Compare May 5, 2025 23:08

Roasbeef force-pushed the universe-supply-trees branch from d786104 to af5f8f5 Compare May 5, 2025 23:13

Roasbeef force-pushed the supply-commit-machine branch 3 times, most recently from e67b58f to 6b99f70 Compare May 5, 2025 23:54

Roasbeef force-pushed the universe-supply-trees branch from af5f8f5 to 64516c9 Compare May 5, 2025 23:54

Roasbeef force-pushed the supply-commit-machine branch from 6b99f70 to 5056daa Compare May 6, 2025 01:19

Roasbeef force-pushed the universe-supply-trees branch from 64516c9 to 6e0b6a4 Compare May 6, 2025 01:19

Roasbeef force-pushed the supply-commit-machine branch from 5056daa to 5c20025 Compare May 6, 2025 01:27

Roasbeef force-pushed the universe-supply-trees branch from 6e0b6a4 to aa782c8 Compare May 6, 2025 01:27

levmi added this to the v0.7 milestone May 8, 2025

Roasbeef mentioned this pull request May 22, 2025

RPC: Add Ignore Tree Endpoints #1554

Draft

guggero self-requested a review May 28, 2025 06:56

ffranr self-requested a review May 29, 2025 22:25

ffranr reviewed May 29, 2025

View reviewed changes

guggero reviewed Jun 3, 2025

View reviewed changes

Roasbeef force-pushed the universe-supply-trees branch from aa782c8 to 36e3753 Compare June 9, 2025 23:40

Roasbeef force-pushed the supply-commit-machine branch 2 times, most recently from bf61350 to 16ae70c Compare June 10, 2025 00:31

Roasbeef force-pushed the universe-supply-trees branch from 36e3753 to 1c17470 Compare June 10, 2025 00:31

Roasbeef force-pushed the supply-commit-machine branch from 16ae70c to 67728da Compare June 10, 2025 01:12

Roasbeef force-pushed the universe-supply-trees branch from 1c17470 to b5f3d5e Compare June 10, 2025 01:12

Roasbeef force-pushed the supply-commit-machine branch from c2c394f to b1173b9 Compare June 18, 2025 01:51

Roasbeef force-pushed the universe-supply-trees branch from dd1d318 to 87fefbc Compare June 18, 2025 01:51

Roasbeef force-pushed the supply-commit-machine branch from b1173b9 to c6f9962 Compare June 18, 2025 02:06

Roasbeef force-pushed the universe-supply-trees branch from 87fefbc to 570e477 Compare June 18, 2025 02:06

Roasbeef force-pushed the supply-commit-machine branch from c6f9962 to da0df38 Compare June 18, 2025 23:02

Roasbeef force-pushed the universe-supply-trees branch from 570e477 to 6584d4f Compare June 18, 2025 23:02

Roasbeef force-pushed the supply-commit-machine branch from da0df38 to ee045a1 Compare June 18, 2025 23:25

Roasbeef force-pushed the universe-supply-trees branch from 6584d4f to d1ae6c4 Compare June 18, 2025 23:25

Roasbeef force-pushed the supply-commit-machine branch from ee045a1 to 598067d Compare June 18, 2025 23:51

Roasbeef force-pushed the universe-supply-trees branch 2 times, most recently from bca0e22 to faae3fc Compare June 19, 2025 00:03

Roasbeef force-pushed the supply-commit-machine branch from 598067d to 9f03d81 Compare June 20, 2025 23:49

Roasbeef force-pushed the universe-supply-trees branch from faae3fc to 3b3420a Compare June 20, 2025 23:49

Roasbeef force-pushed the supply-commit-machine branch from 9f03d81 to e77d0cf Compare June 21, 2025 00:08

Roasbeef force-pushed the universe-supply-trees branch from 3b3420a to 8fff268 Compare June 21, 2025 00:08

Roasbeef force-pushed the supply-commit-machine branch from e77d0cf to 3044a87 Compare June 21, 2025 00:12

Roasbeef force-pushed the universe-supply-trees branch from 8fff268 to c668391 Compare June 21, 2025 00:12

Roasbeef force-pushed the supply-commit-machine branch from 3044a87 to 9854971 Compare June 21, 2025 00:20

Roasbeef force-pushed the universe-supply-trees branch from c668391 to 8b06b69 Compare June 21, 2025 00:20

Roasbeef added 8 commits June 20, 2025 18:22

universe: add Encode+Decode methods for BurnLeaf

533f311

tapdb/sqlc: update mint_anchor_uni_commitments queries to account for…

cd5022d

… new spent_by field

tapdb/sqlc: add new supply_commit.sql queries

cdff615

tapdb/sqlc: generate sqlc structs

b692815

tapdb: add tests for SupplyCommitMachine

65b27b2

tapdb: add supply_commit.md documentation

37bdcfa

Roasbeef force-pushed the supply-commit-machine branch from 9854971 to 37bdcfa Compare June 21, 2025 01:22

Roasbeef force-pushed the universe-supply-trees branch from 8b06b69 to b8988d3 Compare June 21, 2025 01:22

tapdb: add implementation of supplycommit.CommitmentTracker and supplycommit.StateMachineStore #1508

Are you sure you want to change the base?

tapdb: add implementation of supplycommit.CommitmentTracker and supplycommit.StateMachineStore #1508

Uh oh!

Conversation

Roasbeef commented May 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guggero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lightninglabs-deploy commented Jun 24, 2025

Uh oh!

Uh oh!