Production grade lnd setup for High Availability and Fault Tolerance as a payment gateway.

[sanjay-shah]

Hello,

Can someone please guide me or point me in the right direction with "Best Practices" to setup lnd for High Availability(HA) and Fault Tolerance(FT). This is for accepting high volume tiny incoming and outgoing payments for an online Sportsbook and casino.

To be clear, I'm not asking best practices for routing node.

Here are key points I'm concerned about:

1. What if the lnd node crashes for some reason, how do we switch to different standby node instantly?
2. What is the recommended gRPC reverse proxy for lnd gRPC calls?
3. Will bolt.db be right choice for our usecase?
4. Is there a risk of ddos or dos if we open the gRPC port on a public IP? If yes, how can we prevent it?
5. How to discourage people from opening channels to us and using our node as routing node for their re-balance activity? Is this a real risk or problem?
6. Hardware/VM requirements in terms of Memory/CPU etc for both lnd and bitcoind
7. Is bitcoind full node or pruned node recommended. We will also enable on-chain invoice/payments at a future date.

Any other things I should be concerned about?

Thanks in advance for your help!

[guggero]

I'm not sure there are any "best practices" for that written down anywhere, since that always kind of depends on the use case. But giving a shot at answering your questions for your specific setup.

1. The same node (=same seed/identity public key) can only run once with the current lnd architecture. But if you use a remote database backend (for example Postgres), you can have a "hot-standby" setup where you have one "hot" node running and one or more "standby" nodes waiting to take over if the hot one fails. That setup would require an etcd cluster just for the leader election functionality though, but in case of a kubernetes setup that would already be available.
2. From what I've heard nginx works pretty well with gRPC. Depending on the use case you could also use Aperture but that has fewer features for plain reverse proxying than you might need. What kind of requests are you looking to proxy? User requests or just admin/management requests?
3. No, I would not recommend that. If you use a remote storage backend as mentioned in 1), the files on disk for each node would be ephemeral. The alternative is to use a mounted network file system which would be passed on from a failing node to a standby node. But then the standby node could only begin to start up once the previous one has fully shut down and released the persisted volume, which introduces a delay. That delay is increased even further by bbolt as opening a DB always performs a number of persistence checks which can take a while. So if you decide to go with a shared file system approach then I would recommend using SQLite instead.
4. Still not sure why you'd want to open up the gRPC port to the internet. But you could filter any requests that don't have a Macaroon: or Grpc-Metadata-Macaroon: HTTP header, that should filter out all automated stuff that's hitting any public HTTP(S) port.
5. You can increase your minimum channel size to a very high number. Since that only applies to incoming channels, that would disallow anyone from opening one. The alternative is to use the channel acceptor API where you get full control over what incoming channel open requests you accept or reject.
6. The CPU/memory requirement scales with the number of channels and number of peer connections (and also somewhat with the number of RPC requests made to the node). In practice a node with quite a number of channels runs well in a kubernetes environment with 8GB of RAM and 3 vCPUs. When using a file based database backend (e.g. SQLite) then the number of IOPS of the mounted storage can have a big impact on performance as well. bitcoind requires less CPU and the RAM depends on settings like the mempool size or the DB cache size. Something between 2 and 6 GB should work well.
7. Definitely don't run a pruned node if you're looking for maximum performance. If lnd requests a block that bitcoind doesn't have, then it'll go and fetch it from the network which can introduce a big delay. This can increase times for chain rescan operations quite significantly. Also in practice it has proven to be necessary to have a bitcoind node for every lnd node, since bitcoind only has a single internal thread to handle RPC requests. So if multiple lnd nodes use the same bitcoind that can introduce slowdowns as well.

Any other things I should be concerned about?

Yes, how to handle backups. If you back up your storage backend (whether that's file based SQLite or a Postgres cluster), you can't simply restore a backup if things go wrong, as you can never be certain you have the absolute latest state in that backup. Therefore making sure the DB cluster cannot easily corrupt or get out of sync is way more important than the actual backup of it by itself. Please read this document to get more info around the topic of backup and safety around channel management: https://github.com/lightningnetwork/lnd/blob/master/docs/safety.md

[sanjay-shah]

Thanks @guggero for very detailed explanation.

1. We'll go with k8s and Postgres DB.
2. We'll start with nginx, also exploring Envoy proxy. Any gotchas or pointers for Envoy? "User requests or just admin/management requests?" -- At this point: just Invoices and Payments. These are internal gRPC calls made from server apps to generate invoice, listen to invoice status. And to send payment to users via either sendpayment or payinvoice.
3. We'll go with Postgres.
4. As you mentioned, I think it's really a bad idea to open gRPC on public IP. I was being lazy and wanted to re-use the same lnd for receiving payments from geyser.fund - They only support invoices from gRPC :(. We'll spin a separate lnd instance for it.
5. Got it! We'll start with minchansize=100000000, eventually move to ChannelAcceptor.
6. Thanks for the hardware spec. We'll start with 8GB-3vCPUs k8s worker nodes.
7. Full node it is!

Didn't realize the nuance of backups is so important. Will reach-out as we make further progress.

Once again thanks for being so awesome and taking time from your busy schedule to help community adoption of lightning.

[hieblmi]

Re 1., here are some thoughts on lightning node fail-over strategies:

• https://suredbits.com/failover-strategies-for-lightning-nodes
• https://suredbits.com/lightning-201-failover-node-demonstration

They might be dated but could still be useful.

[sanjay-shah]

Thanks @hieblmi
Your article definitely helped me. As a novice, initially I was considering spinning multiple standalone lnd node to achieve scalability which is totally a bad idea. @guggero has rightly pointed us to the best HA/FT solution. Once I find some free time, after all the implementations, I will create a detailed blog post incorporating all of Oliver's recommendation with detailed commands.