[feature]: Store the channel force close error message in the closed channels database

[saubyk]

Currently the data stored in the closed channel database points to the type of channel closure e.g. local force close or remote force close, but not specifically the reason which may have triggered it.
Some examples can be:

• User initiated
• Low on reserve

Typically these reasons are printed on the console logs, which is hard to track for a user. If logged in the database, it can provide some additional insight to the user on the reason for force close, especially for the cases where the closure has been triggered automatically.

The ClosedChannels rpc should also be updated to include the reason in the ChannelCloseSummary response

[shaurya947]

Hey @saubyk this seems like a very useful addition! I recently had a channel force close by my own node while I was away and it was quite unclear as to why that happened.

I would like to give this a shot. I'll do some digging today to come up with a high-level plan of the required code changes, and share it when it's ready 🙂

[saubyk]

Hi @shaurya947 thanks for taking the initiative
Please post your plan for the change as a comment here. Looking forward :)

[shaurya947]

I spent a few hours looking through the code. There are many cases under which a channel may be marked as force closed in the DB:

• local force close
  • user initiated
  • link error
    • commitment point from remote doesn't match last unrevoked commit point
    • error when attempting to settle upstream HTLC
    • unable to accept new commitment (due to sig error)
• remote force close
• abandoning a channel
  • direct RPC call
  • as part of BatchOpenChannel
• funding errors
  • error before funding TX confirmed
  • timeout while waiting for funding TX conf

There may be more and/or I may have misinterpreted some cases. IMO the ones of interest here are:

• local force close
  • link error
    • commitment point from remote doesn't match last unrevoked commit point
    • error when attempting to settle upstream HTLC
    • unable to accept new commitment (due to sig error)
• remote force close

I would need some assistance from someone more experienced with the codebase regarding the following:

1. It's not obvious to me how to ascertain specific reasons for remote force close. I see that a lot of link failure messages are sent to the peer but I don't see where they may be processed and acted upon?
2. What would be the best way to pipe channel force close reason through the channel state machine? The force closing seems to happen inside chain_arbitrator.go#ForceCloseContract() where the force close request is queued. But the actual marking of the channel as closed in the DB seems to happen within channel_arbitrator.go as various ChainEvents are observed. What would be a good design/architecture to bridge the two instances?

[saubyk]

Great analysis so far @shaurya947
cc: @yyforyongyu

[yyforyongyu]

I'd recommend take a look at this transition flow to understand it more. Notice that we've already have CloseType and ChanStatusFlags to give us some insights about the FC. What's nice to have is to somehow record the state steps that have happened from calling this method.

[shaurya947]

Thanks for the links @yyforyongyu ! I'll try to absorb all of this over the next few days and come back with a plan and/or more questions 🙂

[shaurya947]

Hey @yyforyongyu I looked through the state transitions. I've got a few more questions for you. Sorry that this is a lengthy comment. I've underlined the 4 exact spots that are the actual questions:

Could you please confirm that there is no feasible way to extract a specific reason for a remote force closure? I say this because the path for registering a remote force closure is as follows:

1. lnd/contractcourt/chain_watcher.go

   Line 623 in ac3e7be

   ok, err = c.handleKnownRemoteState(

2. lnd/contractcourt/chain_watcher.go

   Line 758 in ac3e7be

   err := c.dispatchRemoteForceClose(

3. lnd/contractcourt/chain_watcher.go

   Line 1127 in ac3e7be

   case sub.RemoteUnilateralClosure <- &RemoteUnilateralCloseInfo{

4. lnd/contractcourt/channel_arbitrator.go

   Line 566 in ac3e7be

   go c.channelAttendant(bestHeight)

5. lnd/contractcourt/channel_arbitrator.go

   Line 2684 in ac3e7be

   case uniClosure := <-c.cfg.ChainEvents.RemoteUnilateralClosure:

6. lnd/contractcourt/channel_arbitrator.go

   Line 2742 in ac3e7be

   _, _, err = c.advanceState(

7. lnd/contractcourt/channel_arbitrator.go

   Line 1403 in ac3e7be

   nextState, closeTx, err := c.stateStep(

8. lnd/contractcourt/channel_arbitrator.go

   Line 892 in ac3e7be

   case remoteCloseTrigger:

9. lnd/contractcourt/channel_arbitrator.go

   Line 893 in ac3e7be

   nextState = StateContractClosed

So basically, our observer that's watching for on-chain events recognizes a broadcast of the commitment TX by the remote party, and in response to that notification, our state machine transitions to StateContractClosed and we don't have any actionable info as to why the remote party chose to broadcast their TX. Is that accurate? If so, I can stop worrying about the remote force closure and focus my attention only on local force closure.

---

As for local force closures, there seem to be 2 paths invoking the userTrigger:

• CloseChannel RPC request:

  1. lnd/rpcserver.go

     Line 2438 in ac3e7be

     func (r *rpcServer) CloseChannel(in *lnrpc.CloseChannelRequest,

  2. lnd/rpcserver.go

     Line 2529 in ac3e7be

     closingTx, err := chainArbitrator.ForceCloseContract(

  3. lnd/contractcourt/chain_arbitrator.go

     Line 1067 in ac3e7be

     func (c *ChainArbitrator) ForceCloseContract(chanPoint wire.OutPoint) (*wire.MsgTx, error) {

  4. lnd/contractcourt/chain_arbitrator.go

     Line 1091 in ac3e7be

     case arbitrator.forceCloseReqs <- &forceCloseReq{

  5. lnd/contractcourt/channel_arbitrator.go

     Line 2838 in ac3e7be

     case closeReq := <-c.forceCloseReqs:

  6. lnd/contractcourt/channel_arbitrator.go

     Line 2853 in ac3e7be

     nextState, closeTx, err := c.advanceState(

• Link failure in brontide.go:

  1. lnd/peer/brontide.go

     Line 3080 in ac3e7be

     func (p *Brontide) handleLinkFailure(failure linkFailureReport) {

  2. lnd/peer/brontide.go

     Line 3103 in ac3e7be

     closeTx, err := p.cfg.ChainArb.ForceCloseContract(

  3. ...Rest of steps same as above

Does it seem correct that a link-failure force close uses the exact same trigger as a user-requested force close? Even if we were to record the states within the arbitrator's log, there doesn't seem to be a way to distinguish between these two paths unless we use different triggers/states.. Is that correct?

---

There also seem to be 2 paths invoking chainTrigger:

• new block notification being received
  • there are no on-chain actions required -> state remains as StateDefault so we can ignore this case

    lnd/contractcourt/channel_arbitrator.go

    Line 849 in ac3e7be

    if len(chainActions) == 0 && trigger == chainTrigger {

  • there are on-chain actions required (HTLC timeout or settle) -> state changes to StateBroadcastCommit so we need to consider this case

    lnd/contractcourt/channel_arbitrator.go

    Line 872 in ac3e7be

    case chainTrigger:

• resolution signal being received

  lnd/contractcourt/channel_arbitrator.go

  Line 2815 in ac3e7be

  case <-c.resolutionSignal:

  
  As far as I can tell a resolution signal can only be received AFTER StateContractClosed which means we can ignore this case because our channel has already been marked closed. Is that accurate?

---

All that being said, my plan is as follows. As a final question, I would love your feedback on this plan:

• Decide on one of the following two options (so we have knowledge of the triggers in addition to knowledge of the states):
  • divide the StateBroadcastCommit state into further substates as necessary such as StateBroadcastCommitUserTrigger, StateBroadcastCommitLinkFailureTrigger (if we wish to distinguish between link failure vs user initiated), and StateBroadcastCommitChainTrigger (could be broken down further depending on exact reason such as HTLC timeout etc.)
  • OR when storing the states within the Arbitrator, also store the trigger responsible for the state change (could be a struct or a 2-byte flag with first byte being state and second byte being trigger)
• Extend the ArbitratorLog interface to add a States function that returns a slice of all the states up to CurrentState

  lnd/contractcourt/briefcase.go

  Line 60 in ac3e7be

  type ArbitratorLog interface {

• When committing a new state using the CommitState function, instead of overriding a single state, we instead append to a slice of states and store that new slice. We do the append iff the new state is not the same as the last state so that the slice doesn't grow unbounded with a number of redundant states.

  lnd/contractcourt/briefcase.go

  Line 543 in ac3e7be

  func (b *boltArbitratorLog) CommitState(s ArbitratorState) error {

• We also update the CurrentState function accordingly to return the last element of the slice

  lnd/contractcourt/briefcase.go

  Line 502 in ac3e7be

  func (b *boltArbitratorLog) CurrentState(tx kvdb.RTx) (ArbitratorState, error) {

• When marking a channel as closed with the initiator being local, we examine the log slice here

  lnd/contractcourt/channel_arbitrator.go

  Line 2661 in ac3e7be

  err = c.cfg.MarkChannelClosed(

  and based on the triggers within the slice, we can mark the channel as one of following:
  • user initiated
  • link error
  • HTLC timeout or on-chain settlement required

[shaurya947]

Hey @yyforyongyu just wanted to make sure this didn't disappear from your radar 🙂