Skip to content

Commit

Permalink
Tweaks around libp2p pubsub seen messages cache (#3773)
Browse files Browse the repository at this point in the history
Refs: #3770
Depends on: #3771

Recent libp2p versions (we started to use them in
#3771) introduced a way to
set the seen messages cache TTL and strategy. Here we leverage those
settings to reduce the excessive message flooding effect that sometimes
occurs on mainnet. This pull request consists of two steps

### Use longer TTL for pubsub seen messages cache

Once a message is received and validated, pubsub re-broadcasts it to
other peers and puts it into the seen messages cache. This way,
subsequent arrivals of the same message are not re-broadcasted
unnecessarily. This mechanism is important for the network to avoid
excessive message flooding. The default value used by libp2p is 2
minutes. However, Keep client messaging sessions are quite
time-consuming so, we use a longer TTL of 5 minutes to reduce flooding
risk even further. Worth noting that this time cannot be too long as the
cache may grow excessively and impact memory consumption.

### Use `LastSeen` as seen messages cache strategy

By default, the libp2p seen messages cache uses the `FirstSeen` strategy
which expires an entry once TTL elapses from when it was added. This
means that if a single message is being received frequently and
consistently, pubsub will re-broadcast it every TTL, rather than never
re-broadcasting it.

In the context of the Keep client which additionally uses app-level
retransmissions, that often leads to a strong message amplification in
the broadcast channel which causes a significant increase in the network
load.

As the problem is quite common (see
libp2p/go-libp2p-pubsub#502), the libp2p team
added a new `LastSeen` strategy which behaves differently. This strategy
expires an entry once TTL elapses from the last time the message was
touched by a cache write (`Add`) or read (`Has`) operation. That gives
the desired behavior of never re-broadcasting a message that was already
seen within the last TTL period. This reduces the risk of unintended
over-amplification.
  • Loading branch information
tomaszslabon authored Feb 8, 2024
2 parents bc29dbc + a712112 commit 499a3d5
Showing 1 changed file with 14 additions and 0 deletions.
14 changes: 14 additions & 0 deletions pkg/net/libp2p/channel_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,25 @@ import (
"github.com/keep-network/keep-core/pkg/net"
"github.com/keep-network/keep-core/pkg/net/retransmission"
pubsub "github.com/libp2p/go-libp2p-pubsub"
pubsubtc "github.com/libp2p/go-libp2p-pubsub/timecache"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peerstore"
)

const (
libp2pPeerOutboundQueueSize = 256
libp2pValidationQueueSize = 4096
// libp2pSeenMessagesTTL is the time-to-live used for pubsub seen messages
// cache. Once a message is received and validated, pubsub re-broadcasts it
// to other peers and puts it into the seen messages cache. This way,
// subsequent arrivals of the same message are not re-broadcasted
// unnecessarily. This mechanism is important for the network to avoid
// excessive message flooding. The default value used by libp2p is 2 minutes.
// However, Keep client messaging sessions are quite time-consuming so,
// we use a longer TTL to reduce flooding risk even further. Worth noting
// that this time cannot be too long as the cache may grow excessively and
// impact memory consumption.
libp2pSeenMessagesTTL = 5 * time.Minute
)

type channelManager struct {
Expand Down Expand Up @@ -51,6 +63,8 @@ func newChannelManager(
pubsub.WithMessageSignaturePolicy(pubsub.StrictSign),
pubsub.WithPeerOutboundQueueSize(libp2pPeerOutboundQueueSize),
pubsub.WithValidateQueueSize(libp2pValidationQueueSize),
pubsub.WithSeenMessagesStrategy(pubsubtc.Strategy_LastSeen),
pubsub.WithSeenMessagesTTL(libp2pSeenMessagesTTL),
)
if err != nil {
return nil, err
Expand Down

0 comments on commit 499a3d5

Please sign in to comment.