lnwallet+contractcourt: gracefully handle auto force close post data …

[Roasbeef]

…loss

In this commit, update the start up logic to gracefully handle a seemingly rare case. In this case, a peer detects local data loss with a set of active HTLCs. These HTLCs then eventually expire (they may or may not actually "exist"), causing a force close decision. Before this PR, this attempt would fail with a fatal error that can impede start up.

To better handle such a scenario, we'll now catch the error when we fail to force close due to entering the DLP and instead terminate the state machine at the broadcast state. When a commitment transaction eventually confirms, we'll play it as normal.

Fixes #7984

[yyforyongyu]

I think we should catch the error here to make sure we can finish startup,

lnd/contractcourt/channel_arbitrator.go

Line 534 in 90effda

case errNoResolutions:

And no advancing the state in channel arbitrator. Once data loss is detected, we'll wait for the remote to force close it. Then the chain arbitrator will detect it and send the event to RemoteUnilateralClosure, which will be picked up and handled by the channel arbitrator to clean up the resolutions.

[Roasbeef]

I think we should catch the error here to make sure we can finish startup,

True, we want to see have the channel arb active so it can get the on chain close signal.

[Roasbeef]

And no advancing the state in channel arbitrator.

Ah so you mean don't state step at all and instead wait for the on chain confirmation? Each time a block is mined though, we may end up triggering the go-to-chain decision, so I think we still need to catch this error and just have the state machine terminate as is. I updated the commit to just go back to StateBroadcastCommit, so it terminates there. Otherwise, in StateDefault, we'll do checkLocalChainActions, eventually deciding to force close again once the next block comes around.

[Roasbeef]

Pushed up a new commit:

• test case added to ensure we start up
• modified to not advance to next state, just stay in StateBroadcastCommit

[yyforyongyu]

pending linter fix, otherwise LGTM🙏

I think going back to either StateDefault or StateBroadcastCommit works. Once in DLP mode, there isn't much we can do here but listening to remote force close and possible breach, and these two triggers can both be handled as long as we don't go beyong StateBroadcastCommit.

A future PR may wanna investigate and modify the ChainArbitrator.Start to never stop when one of the channel arbitrators fails to start,

lnd/contractcourt/chain_arbitrator.go

Lines 707 to 709 in 90effda

	if err := arbitrator.Start(startState); err != nil {
	stopAndLog()
	return err

I think unless it's related to db failure, the system should start normally so it can operate on other channels.