Skip to content

Commit

Permalink
Merge branch 'doc/system-design-spec' into 'dev'
Browse files Browse the repository at this point in the history 
doc: update system design spec

See merge request ergo/rosen-bridge/rosenet!51
  • Loading branch information
@vorujack
vorujack committed Oct 26, 2024
2 parents e5adb29 + b9e1b83 commit 21e4e7b
Show file tree
Hide file tree
Showing 2 changed files with 52 additions and 114 deletions.
2 changes: 2 additions & 0 deletions .changeset/nice-humans-reflect.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
---
---
164 changes: 50 additions & 114 deletions specs/system-design.md
View file
Original file line number Diff line number Diff line change
@@ -1,140 +1,76 @@
# RoseNet System Design Spec
# RoseNet System Design Specification

## Introduction
This document outlines the core principles, assumptions, and features underlying the development of RoseNet.

RoseNet, a decentralized p2p protocol empowering Rosen Bridge guards, aims to
provide a secure and reliable communication network. This document outlines the
design principles, assumptions, and features of RoseNet.

| Status | creation date | last update date | version |
| Status | Creation Date | Last Update Date | Version |
| ------ | ------------- | ---------------- | ------- |
| Draft | 25 Dec 2023 | 30 Apr 2024 | Draft-3 |
| Draft | 25 Dec 2023 | 19 Oct 2024 | Draft-4 |

Author: @mkermani144
Contributors: N/A

## Overview

RoseNet serves as a critical component in enabling secure communication between
Rosen Bridge guards. This section provides a high-level overview of RoseNet's
purpose and its role within the broader system.

## RoseNet in action
## Introduction to RoseNet

The current RoseNet implementation utilizes libp2p internally. Before delving
into the system design specifics, let's establish a comprehensive understanding
of RoseNet's functionality.
RoseNet is a decentralized peer-to-peer (p2p) protocol designed to empower Rosen Bridge guards by providing a secure and reliable communication network.

In RoseNet, there are two distinct node types: Normal nodes (referred to as
"nodes" in this document) and relays. The conceptual framework mirrors that of
libp2p, wherein, to facilitate communication between two nodes, a well-known,
public intermediary called a relay is essential. This relay efficiently relays
traffic between the communicating nodes.
## RoseNet in Action

For a clearer grasp of RoseNet's operation, consider the following example
involving NodeA connecting to RoseNet through RelayR:
RoseNet's current implementation utilizes libp2p internally. Before delving into specific system design details, let's establish a comprehensive understanding of RoseNet's functionality.

1. Following minimal configurations, particularly defining the RelayR
multiaddress, NodeA attempts to establish a connection with RelayR.
2. If NodeA lacks whitelisting in RelayR, the connection remains unestablished,
and NodeA cannot join RoseNet.
3. Upon NodeA's successful whitelisting in RelayR and subsequent connection to
RelayR, it initiates the dissemination of its multiaddress across RoseNet.
This dissemination employs a gossip-based protocol within libp2p, although
specific details are not a focal point in this document. The key takeaway is
that, over time, every node becomes aware of NodeA's presence in RoseNet,
enabling the exchange of messages—NodeA can both send and receive messages
seamlessly.
4. _TODO: messaging_

## Features Summary
### Node Types

Implementation status:
🏁 Not started
🚧 Under development
✅ Finished

| Status | Feature | Current State |
| ------ | ----------------------------- | ------------------------------------------------------------------ |
| 🚧 | Topology (ZeroLayer) | Starts as a star, but the final topology is to be determined |
| 🚧 | Topology (RoseLayer) | [To be determined] |
| 🏁 | Fast Message Delivery | Guaranteed within a 5-minute latency window |
| 🏁 | Message Delivery Verification | Message delivery is verified through an ack mechanism |
| ✅ | Whitelisting | Configurable for both relays and nodes, being mandatory for relays |
| ✅ | Communication Security | All messages are end-to-end encrypted |
| ✅ | Sybil Attack Mitigation | Relays only accept connections from whitelisted nodes |
| 🏁 | Network Data Availability | Support planned in case of introducing state to the network |
| 🚧 | Scalability | Up to 40 non-relay nodes |
| - | Decentralization | Mostly decentralized, but some relay nodes are required |
| ✅ | Discoverability (Nodes) | Achieved after connecting to the relay |
| 🚧 | NAT Traversal | Supports both public and private nodes |
| 🏁 | Monitoring & Analytics | Tracks connected nodes, relays, messaging rates, etc. |
| 🏁 | Message Routing | [To be determined] |
| 🚧 | Burst Messaging | [To be determined] |
| ✅ | Sending Large Messages | Delivery of messages up to 100kB guaranteed |
| 🏁 | Idle Network State | [To be determined] |

### Unsupported Features:

| Feature | Current state |
| -------------------------------- | ----------------------------------------------------------------------------------- |
| Openness | New nodes cannot join RoseNet until whitelisted in at least one relay |
| High throughput | Only sending messages over different channels in parallel are meant to be supported |
| Consistent messaging order | Not guaranteed and not planned |
| Relays count constraints support | Not planned, new relays must be added on demand |
| Persistence | Not planned |
| Discoverability (relays) | Relays are not meant to be well-known ip addresses and don't need to be discovered |
| Backward-compatibility | All RoseNet nodes and relays must use the same version of RoseNet protocol |

## RoseNet design principles
RoseNet consists of two distinct node types:

### Plug and Play

RoseNet is designed for easy use with minimal configuration. In the RoseNet APIs, many libp2p configurations are intentionally inaccessible, forming a fundamental aspect of its design philosophy. The intention is for RoseNet to be employed without extensive customization, relying on configurations only when necessary for optimal node performance. If your customization requirements surpass the scope of RoseNet APIs, consider transitioning to direct utilization of libp2p for more granular control over network behavior.

### Limited Access Network
1. Normal nodes (referred to as "nodes" in this document)
2. Relays

Rosen Bridge guards, comprising a select group of trusted entities, form the backbone of RoseNet. Designed primarily for Rosen Bridge guards, RoseNet operates as an exclusive, invite-only network. To connect to RoseNet, a node must be whitelisted in at least one relay. This intentional limitation comes with its own set of advantages and disadvantages, outlined below.
This conceptual framework mirrors that of libp2p, where communication between two private nodes requires a well-known, public intermediary called a relay to efficiently route traffic.

#### Pros
### Connection Process

- **Enhanced Security:** The "invitation-only" requirement acts as a robust barrier against potential malicious nodes, significantly bolstering the network's security.
- **Trust Simplicity:** The high-security nature of RoseNet obviates the need for complex reputation systems, streamlining trust dynamics within the network.
- **Limited Scale, Limited Issues:** The closed nature of RoseNet results in a smaller node count, mitigating scaling challenges commonly faced by open peer-to-peer networks. This ensures a more manageable and efficient network infrastructure tailored to the intended scale of RoseNet.
Consider the following example of NodeA connecting to RoseNet through RelayR:

#### Cons
1. NodeA attempts to establish a connection with RelayR using minimal configurations, primarily the RelayR multiaddress.
2. If NodeA is not whitelisted in RelayR, the connection fails, and NodeA cannot join RoseNet.
3. Upon successful whitelisting and connection to RelayR, NodeA initiates the dissemination of its multiaddress across RoseNet using a gossip-based protocol within libp2p.
4. Over time, every node becomes aware of NodeA's presence in RoseNet, enabling seamless message exchange.
5. If NodeA can identify its own public IP, even behind a NAT, it may perform NAT traversal and connect directly to other nodes via hole punching.
6. NodeA can then send or receive direct or pubsub messages, based on application layer usage.

- **Limited Applicability:** While RoseNet excels in security and selectivity, its closed design may limit its suitability for applications that thrive on openness. Many decentralized applications, by nature, aim to cater to a broad user base, making RoseNet less ideal for such use cases.
- **Distributing New Whitelisted Nodes:** The current implementation relies on a manual mechanism for distributing new whitelisted nodes. This means that relays must perform consensus manually outside of the network, posing challenges in terms of scalability and automation.
- **Whitelist Conflicts:** There may be instances of whitelist conflicts, potentially causing issues within the network. Careful coordination and resolution mechanisms are essential to address these conflicts effectively.
- **Centralized Trust Establishment:** The reliance on whitelisting through relays introduces a centralized trust establishment point, which could be a potential vulnerability if not managed carefully.
- **User Onboarding Complexity:** The invite-only model might introduce complexities in onboarding new users and nodes, potentially hindering network growth.
## RoseNet Design Principles

This nuanced perspective aims to provide a comprehensive understanding of RoseNet's intentional limitations, facilitating informed decision-making based on its unique strengths and drawbacks.

## Features

### Whitelisting

In RoseNet, the strategic implementation of whitelisting plays a pivotal role, with its application spanning both nodes and relays. This necessity stems from the fundamental design principles governing RoseNet. To gain a deeper understanding of the advantages and disadvantages associated with whitelisting, one must delve into the limited access network design principle.

When engaging in the whitelisting process, careful consideration of the following factors is crucial:

- **Essential Role in Relays:** Whitelists are indispensable for the proper functioning of relays, serving as the cornerstone of the network's design.
### Plug and Play

- **Decentralization via Relays:** The number of relays directly impacts the decentralization of the network. To maintain a decentralized characteristic, it is advisable to authorize a substantial number of relays, each sanctioned by different entities.
RoseNet is designed for easy use with minimal configuration. Many libp2p configurations in the RoseNet APIs are preset with reasonable defaults, allowing for immediate use without extensive customization.

- **Distribution Mechanism Complexity:** Establishing a reliable whitelisting distribution mechanism is imperative. Questions such as when RelayB should replicate the whitelisting of NodeX after RelayA has done so, and how to ensure trustworthiness against malicious nodes, require thoughtful consideration.
### Limited Access Network

- **Guarding Against Malicious Relays:** As RoseNet expands, the potential for the emergence of malicious relays increases. Nodes incorporate whitelisting mechanisms as a defensive measure. In the event of malicious relays, nodes must configure their whitelists to exclusively accept connections from trusted parties.
RoseNet operates as an exclusive, invite-only network primarily designed for Rosen Bridge guards. To connect to RoseNet, a node must be whitelisted in at least one relay.

- **Resilience to Whitelist Conflicts:** The network's resilience hinges on its ability to navigate whitelist conflicts gracefully. In a network where all nodes and relays can whitelist each other, conflicts regarding trusted entities can easily arise, necessitating a robust conflict resolution mechanism.
### Throughput Limitation

This enhanced approach to whitelisting not only ensures the smooth operation of RoseNet but also fortifies its resilience against potential challenges, thereby contributing to a more secure and reliable decentralized network.
By default, RoseNet nodes have limited throughput for both pubsub and direct messages. This preventive measure aims to avoid potential resource (CPU/memory) outages, which could lead to non-deterministic states for nodes and, over time, the entire network. These throughput limits are configurable based on specific use cases.

## TODO
## Features Summary

- [ ] Describe the features in detail
- [ ] Stateful vs Stateless
- [ ] RoseNet use case in Rosen Bridge guards
- [ ] ZeroLayer vs RoseLayer
| Feature | Description |
| ----------------------------- | ------------------------------------------------------------------------- |
| Fast Message Delivery | Low overhead for sending messages with successful hole punching |
| Message Delivery Verification | Implemented through an acknowledgment mechanism |
| Whitelisting | Configurable for both relays and nodes; mandatory for relays |
| Communication Security | End-to-end encryption for all messages |
| Sybil Attack Mitigation | Relays only accept connections from whitelisted nodes |
| Scalability | Guaranteed scale of up to 40 running non-relay nodes |
| Decentralization | Mostly decentralized, with some relay nodes required |
| Discoverability (Nodes) | Achieved after connecting to the relay |
| NAT Traversal | Implemented through the DCUtR protocol |
| Burst Messaging | Supports concurrent sending of numerous messages within throughput limits |

### Unsupported Features

| Feature | Current State |
| -------------------------- | --------------------------------------------------------------- |
| Openness | New nodes require whitelisting in at least one relay to join |
| Consistent Messaging Order | Not guaranteed and not planned |
| Backward-compatibility | All nodes and relays must use the same RoseNet protocol version |
| Message Routing | Not planned |

0 comments on commit 21e4e7b

Please sign in to comment.