[feature]: reduce rescan bandwidth for mobile

[JssDWt]

Is your feature request related to a problem? Please describe.
LND does historical rescans for spent utxos on startup. This happens on every startup, regardless of the state of the previous rescan. For mobile devices this is problematic, because it costs a lot of bandwidth. With Breez, we're seeing users apps consuming gigabytes of data in a matter minutes due to these historical rescans.

The culprit, I believe, is in the funding manager. Let me walk you through the process.
We're discussing zero conf channels here, as context.

• The funding manager calls advanceFundingState for every channel.
• Then it calls the stateStep based on the current channel state.
• If the current state is addedToGraph, and the channel is a zero conf channel, it will call waitForZeroConfChannel.
• call waitForFundingWithTimeout
• call waitForFundingConfirmation
• call RegisterConfirmationsNtfn

In this case we're using neutrino.

• call RegisterConf on the tx notifier
• That will initiate a HistoricalConfDispatch here
• And push a rescan update to the notificationRegistry
• historicalConfDetails are requested
• It will start rescanning the chain from tip to broadcast height of the channel here

Note that this will happen on EVERY start. Regardless of whether LND has seen the funding transaction being confirmed before. If the user has a channel created 100k blocks ago, this will rescan the chain 100k blocks, costing gigabytes of data in a short period of time. The state machine switched on here has addedToGraph as its final state. There is no "we have seen the funding transaction before, don't worry about it" state, which I think should be added if the confirmation of the funding transaction is already known.

Describe the solution you'd like
Please ensure that restarts with zero conf channels and neutrino don't rescan the chain on every restart.

[JssDWt]

Noting here that on Breez large chunks of neutrino filters are removed on disk to save storage space. So where otherwise neutrino could load previously loaded filters from disk, in the mobile case they will have to be fetched from a remote peer instead.

[Roasbeef]

LND does historical rescans for spent utxos on startup. This happens on every startup, regardless of the state of the previous rescan.

Are you disabling the height hint cache? It's purpose is to checkpoint where the last scan ended, to prevent starting from scratch each time.

[Roasbeef]

There is no "we have seen the funding transaction before, don't worry about it" state, which I think should be added if the confirmation of the funding transaction is already known.

We currently implement that by deleting the channel state once things are fully added to the graph:

lnd/funding/manager.go

Lines 1246 to 1255 in 94f7ed4

	// We delete the channel opening state from our internal
	// database as the opening process has succeeded. We can do
	// this because we assume the AuthenticatedGossiper queues the
	// announcement messages, and persists them in case of a daemon
	// shutdown.
	err = f.deleteChannelOpeningState(&channel.FundingOutpoint)
	if err != nil {
	return fmt.Errorf("error deleting channel state: %w",
	err)
	}

So then it should fail here on restart:

lnd/funding/manager.go

Lines 1115 to 1124 in 94f7ed4

	if err == channeldb.ErrChannelNotFound {
	// Channel not in fundingManager's opening database,
	// meaning it was successfully announced to the
	// network.
	// TODO(halseth): could do graph consistency check
	// here, and re-add the edge if missing.
	log.Debugf("ChannelPoint(%v) with chan_id=%x not "+
	"found in opening database, assuming already "+
	"announced to the network",
	channel.FundingOutpoint, pendingChanID)

[JssDWt]

Are you disabling the height hint cache? It's purpose is to checkpoint where the last scan ended, to prevent starting from scratch each time.

No. it's not disabled.

We currently implement that by deleting the channel state once things are fully added to the graph:

I see. I think the issue is then that that point is simply never reached. Scanning 100k blocks backwards on a mobile device is pretty much impossible because users tend to close the app before that's complete. I think the main issue here is that it's scanning the chain backwards. I cannot see how the height hint cache helps here, because there would never be any progress on the start height?