peer: eliminate circular waiting by calling maybeSendNodeAnn async

[Roasbeef]

In this commit, we attempt to fix circular waiting scenario introduced inadvertently when fixing a race condition
scenario. With that PR, we added a new channel that would block Disconnect, and WaitForDisconnect to ensure that only until the Start method has finished would those calls be allowed to succeed.

The issue is that if the server is trying to disconnect a peer due to a concurrent connection, but Start is blocked on maybeSendNodeAnn, which then wants to grab the main server mutex, then Start can never exit, which causes startReady to never be closed, which then causes the server to be blocked.

This PR attempts to fix the issue by calling maybeSendNodeAnn in a goroutine, so it can grab the server mutex and not block the Start method.

Fixes #7924

Fixes #7928

Fixes #7866

Fixes #7917

[Roasbeef]

IMO this isn't an ideal solution, but pretty sure this is the interaction at hand.

Here're some supporting goroutine stack traces:

Server blocked on grabbing the mutex to execute the InboundPeerConnected call back:

285 @ 0x43df0e 0x44f3b8 0x44f38f 0x46cf65 0x48d81d 0x48ecf1 0x48ecd2 0x18da3df 0x4710e1
#    0x46cf64    sync.runtime_SemacquireMutex+0x24                    runtime/sema.go:77
#    0x48d81c    sync.(*Mutex).lockSlow+0x15c                        sync/mutex.go:171
#    0x48ecf0    sync.(*Mutex).Lock+0x30                            sync/mutex.go:90
#    0x48ecd1    sync.(*RWMutex).Lock+0x11                        sync/rwmutex.go:147
#    0x18da3de    github.com/lightningnetwork/lnd.(*server).InboundPeerConnected+0xfe    github.com/lightningnetwork/lnd/server.go:3520

Peer goroutines unable to be torn down (concurrent connection case), as waiting on startReady to be closed:

1 @ 0x43df0e 0x44e385 0x18196fd 0x181c27a 0x4710e1
#    0x18196fc    github.com/lightningnetwork/lnd/peer.(*Brontide).Disconnect+0x9c    github.com/lightningnetwork/lnd/peer/brontide.go:1115
#    0x181c279    github.com/lightningnetwork/lnd/peer.(*Brontide).readHandler+0x10b9    github.com/lightningnetwork/lnd/peer/brontide.go:1697

1 @ 0x43df0e 0x44e385 0x18196fd 0x18df825 0x18da9ba 0x4710e1
#    0x18196fc    github.com/lightningnetwork/lnd/peer.(*Brontide).Disconnect+0x9c    github.com/lightningnetwork/lnd/peer/brontide.go:1115
#    0x18df824    github.com/lightningnetwork/lnd.(*server).removePeer+0x104        github.com/lightningnetwork/lnd/server.go:4309
#    0x18da9b9    github.com/lightningnetwork/lnd.(*server).InboundPeerConnected+0x6d9    github.com/lightningnetwork/lnd/server.go:3586

Peer goroutine can't finish starting up as it needs to call maybeSendAnn, which wants the server mutex:

2 @ 0x43df0e 0x44f3b8 0x44f38f 0x46cf65 0x48d81d 0x48ecf1 0x48ecd2 0x18d6a8f 0x18dcc65 0x18192d1 0x1816ac5 0x18dd0b8 0x4710e1
#    0x46cf64    sync.runtime_SemacquireMutex+0x24                    runtime/sema.go:77
#    0x48d81c    sync.(*Mutex).lockSlow+0x15c                        sync/mutex.go:171
#    0x48ecf0    sync.(*Mutex).Lock+0x30                            sync/mutex.go:90
#    0x48ecd1    sync.(*RWMutex).Lock+0x11                        sync/rwmutex.go:147
#    0x18d6a8e    github.com/lightningnetwork/lnd.(*server).genNodeAnnouncement+0x8e    github.com/lightningnetwork/lnd/server.go:2972
#    0x18dcc64    github.com/lightningnetwork/lnd.(*server).peerConnected.func1+0x84    github.com/lightningnetwork/lnd/server.go:3857
#    0x18192d0    github.com/lightningnetwork/lnd/peer.(*Brontide).maybeSendNodeAnn+0xb0    github.com/lightningnetwork/lnd/peer/brontide.go:1069
#    0x1816ac4    github.com/lightningnetwork/lnd/peer.(*Brontide).Start+0xae4        github.com/lightningnetwork/lnd/peer/brontide.go:675
#    0x18dd0b7    github.com/lightningnetwork/lnd.(*server).peerInitializer+0x137        github.com/lightningnetwork/lnd/server.go:3977

So now Start can't finish, which means that Disconnect can't be called.

[yyforyongyu]

I think this works. There's also an old PR #7525 which removes maybeSendNodeAnn as it seems it's not needed?

[guggero]

LGTM 🎉

Either this approach or the one in #7525 seem fine to me.

[Crypt-iQ]

nice catch 🎣