From c59fb3c47a14440c2ebfba8d20bb93e24c2ba306 Mon Sep 17 00:00:00 2001 From: swarnabhasinha Date: Wed, 11 Dec 2024 14:54:15 +0530 Subject: [PATCH 1/8] feat(docs): guide to run node --- docs/run-node/6-flare-systems.mdx | 112 ++++++++++++++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 docs/run-node/6-flare-systems.mdx diff --git a/docs/run-node/6-flare-systems.mdx b/docs/run-node/6-flare-systems.mdx new file mode 100644 index 00000000..f093a624 --- /dev/null +++ b/docs/run-node/6-flare-systems.mdx @@ -0,0 +1,112 @@ +--- +sidebar_position: 6 +slug: flare-systems +title: Flare Systems +description: Flare Systems +--- + +This guide contains sample deployment configuration for infrastructure providers to enable participation in Flare Systems Protocol ([FSP](https://dev.flare.network/network/fsp/)). + +Currently supported sub-protocols are [FTSOv2](https://flare.network/wp-content/uploads/FTSOv2-White-Paper.pdf) and [FDC](https://flare.network/wp-content/uploads/FDC_WP_171024_02.pdf). + +## Overview + +A data provider system for FSP consists of the following components: + +1. **Flare System Client** + Responsible for all interactions with FSO smart contracts, including voting round data collection and submission for all protocols, voter registration, and additional system tasks. +2. **C-chain Indexer** + Captures FSP-related blockchain transactions and events. Used by clients for data calculation and action triggers. +3. **FTSO Client** + Supports FTSO anchor feeds. Provides protocol votes and median data to System Client for submission. +4. **Fast Updates Client** + Supports FTSO block-latency feeds. Collects and submits data to Fast Updates contracts. +5. **Feed Value Provider** + Gathers data from exchanges and provides current values (prices) for a given set of feeds. +6. **FDC Client** + Interacts with FDC suite and provides FDC protocol voting round data to SystemClient for submission. + +:::info + +Reference implementations are provided for all components except **Feed Value Provider**, providers are encouraged to use own implementation conforming to the [REST API spec](https://github.com/flare-foundation/ftso-v2-example-value-provider?tab=readme-ov-file#example-usage). + +::: + +## Deployment + +### Prerequisites + +:::warning + +Complete all key registration steps in [REGISTRATION.md](https://github.com/flare-foundation/flare-systems-deployment/blob/main/docs/REGISTRATION.md), or else deployed services will not work. + +::: + +1. Running FDC Client + This requires access to underlying chain infrastructure. You can perform the necessary deployment using information in [this repository](https://github.com/flare-foundation/fdc-suite-deployment). + +2. System Dependencies + The following system dependencies are required to run this deployment: + - [jq](https://jqlang.github.io/jq/) + - [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html) + - [Docker](https://www.docker.com/) + - Bash (macOs only for updated version), install by running `brew install bash` + +### Generate configuration + +Copy the code in this [sample configuration file](https://github.com/flare-foundation/flare-systems-deployment/blob/main/.env.example) into a `.env` file at the root of your project and fill out the values as appropriate. + +Then copy the code in this [shell script](https://github.com/flare-foundation/flare-systems-deployment/blob/main/populate_config.sh) (`.sh` file) into a `populate_config.sh` file at your project's root directory. + +Run `./populate_config.sh` from your root to populate configs for provider stack. + +:::note + +You'll need to rerun this command if you change your `.env` file. + +::: + +### Start deployment + +Copy this [docker configuration code](https://github.com/flare-foundation/flare-systems-deployment/blob/main/docker-compose.yaml) into a `docker-compose.yaml` file at the root of your project. + +You can then everything up with `docker compose up -d` and stop everything with `docker compose down`. + +:::info + +Database is persisted in a named docker volume. If you need to wipe database you need to remove the volume manually. + +When the upstream codebase is changed new images need to be pulled with `docker compose pull`. + +::: + +### Feed Value Provider + +The Feed Value Provider instance for FTSO protocol is not part of `docker-compose.yaml` and needs to be created separately. + +Start your own implementation or alternatively use the example implementation for testing: + +```bash +docker run --rm -it \ + --publish "0.0.0.0:3101:3101" \ + --network "ftso-v2-deployment_default" \ + ghcr.io/flare-foundation/ftso-v2-example-value-provider +``` + +### FDC Suite + +Underlying infrastructure required for FDC protocol is not part of this project and needs to be setup seperately. This can be done via [fdc-suite-deployment](https://github.com/flare-foundation/fdc-suite-deployment). + +
+ Potential Issues + + 1. `system-client` is not doing anything. + It is likely that your entity is not registered as a voter for the current reward epoch. There is a time window for voter registration on every reward epoch, and if you leave things running you should eventually see `RegisterVoter success` in the logs. It should then start submitting data successfully in the **following** reward epoch. + + 2. `system-client` fails to fetch sub-protocol data. + The `c-chain-indexer` may still be indexing data, once it's finished you should see `Indexer at block x` in the logs. If it's still processing blocks you will see various errors across all services. + + 3. `fast-updates` client is not doing anything. + If you have low weight it may take some time until you are selected to submit an update. + +
From 55b99b501f729d4bb538afb9795dc763686fb0c4 Mon Sep 17 00:00:00 2001 From: swarnabhasinha Date: Wed, 11 Dec 2024 14:57:00 +0530 Subject: [PATCH 2/8] fix: typo --- docs/run-node/6-flare-systems.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/run-node/6-flare-systems.mdx b/docs/run-node/6-flare-systems.mdx index f093a624..7d5aed48 100644 --- a/docs/run-node/6-flare-systems.mdx +++ b/docs/run-node/6-flare-systems.mdx @@ -95,7 +95,7 @@ docker run --rm -it \ ### FDC Suite -Underlying infrastructure required for FDC protocol is not part of this project and needs to be setup seperately. This can be done via [fdc-suite-deployment](https://github.com/flare-foundation/fdc-suite-deployment). +Underlying infrastructure required for FDC protocol is not part of this project and needs to be setup separately. This can be done via [fdc-suite-deployment](https://github.com/flare-foundation/fdc-suite-deployment).
Potential Issues From 336f4f7977ec78c2509526daf333645a9f176a1f Mon Sep 17 00:00:00 2001 From: Dinesh Pinto Date: Thu, 12 Dec 2024 14:40:13 +0400 Subject: [PATCH 3/8] feat(docs): improve scaling getting started --- docs/ftso/scaling/2-getting-started.mdx | 74 +++++++++++++++++++------ 1 file changed, 57 insertions(+), 17 deletions(-) diff --git a/docs/ftso/scaling/2-getting-started.mdx b/docs/ftso/scaling/2-getting-started.mdx index d7cca594..5b0825a3 100644 --- a/docs/ftso/scaling/2-getting-started.mdx +++ b/docs/ftso/scaling/2-getting-started.mdx @@ -22,23 +22,23 @@ import FetchAnchorFeedsPy from "!!raw-loader!/examples/developer-hub-python/fetc import FetchAndVerifyAnchorOnchainJs from "!!raw-loader!/examples/developer-hub-javascript/fetch_and_verify_anchor_onchain.js"; import FtsoV2AnchorFeedConsumer from "!!raw-loader!/examples/developer-hub-solidity/FtsoV2AnchorFeedConsumer.sol"; -Scaling provides offchain access to anchor feeds, using Flare's network of 100 independent data providers and a full commit-reveal process every 90 seconds. +Scaling enables offchain access to anchor feeds by leveraging Flare's network of 100 independent data providers and a robust commit-reveal process every 90 seconds. -This section demonstrates how to consume anchor feeds on Flare. The key steps to read an anchor feed are: +To read anchor feeds on Flare, follow these key steps: 1. **Fetch anchor feed data offchain:** - Query the Data Availability (DA) Layer to retrieve anchor feeds along with cryptographic proofs. + Use the Data Availability (DA) Layer API to retrieve anchor feeds and their associated cryptographic proofs. 2. **Verify the proof onchain:** - Validate the proof provided by the DA Layer onchain, ensuring it matches the finalized version recorded by [Scaling](/ftso/scaling/overview). + Validate the provided proof onchain to ensure the data matches the finalized version committed by [Scaling](/ftso/scaling/overview). 3. **Use the feed data onchain:** After verification, integrate the feed data into your onchain application logic. -Scaling only stores **commitments to feed data** onchain. The complete feed data resides offchain but can be verified onchain against a Merkle root. This ensures data integrity and guards against tampering. +Scaling only stores **commitments to feed data** onchain. Complete feed data resides offchain but can be verified against an onchain Merkle root, ensuring data integrity and tamper resistance. ## DA Layer API URLs @@ -48,20 +48,20 @@ The public DA Layer endpoints are rate-limited. To request an API key for higher ::: -The DA Layer provides a set of API endpoints for querying offchain data from Flare protocols. +The DA Layer provides API endpoints for querying offchain data from Flare protocols. | Network | Base URL | | ----------------------- | ---------------------------------------------- | | Flare Mainnet | `https://flr-data-availability.flare.network/` | | Songbird Canary-Network | `https://sgb-data-availability.flare.network/` | -All networks have the same API, for a full list of endpoints see [Data Availability API Reference](/fdc/reference/data-availability-api). +All networks have the same API structure. For a full list of endpoints see [Data Availability API Reference](/fdc/reference/data-availability-api). ## Fetching anchor feed data -The DA Layer API allows querying the value and proof for multiple feeds from the same voting epoch in a single request. Use the `anchor-feeds-with-proof` POST endpoint to retrieve pricing data. +The DA Layer API allows querying values and proofs for multiple feeds from the same voting epoch in a single request. Use the `anchor-feeds-with-proof` POST endpoint to retrieve pricing data. -For example, to get the feed value of `FLR/USD`, `BTC/USD` and `ETH/USD` at the latest voting round: +To fetch the feed values for FLR/USD, BTC/USD, and ETH/USD at the latest voting round, use the following command: @@ -97,18 +97,18 @@ For example, to get the feed value of `FLR/USD`, `BTC/USD` and `ETH/USD` at the -### API Response +#### API response structure The response contains JSON objects for each feed, with the following fields: -- `votingRoundId`: ID of the current voting round, each lasting 90s (see the [Flare Systems Explorer](https://flare-systems-explorer.flare.network/voting-epoch)). -- `id`: ID of the feed (see [list of anchor feeds](/ftso/scaling/anchor-feeds)). -- `value`: Integer value of the feed. -- `turnoutBIPS`: Percentage (in basis points) of voting weight which contributed towards the finalized value. -- `decimals`: Number of decimal places for the feed. -- `proof`: Merkle proof array for data verification. +- `votingRoundId`: The voting round ID (each round lasts 90 seconds; see the [Flare Systems Explorer](https://flare-systems-explorer.flare.network/voting-epoch)). +- `id`: The feed ID (refer to the [list of anchor feeds](/ftso/scaling/anchor-feeds)). +- `value`: The integer value of the feed. +- `turnoutBIPS`: The percentage of voting weight (in basis points) that contributed to the finalized value. +- `decimals`: The number of decimal places for the feed. +- `proof`: The Merkle proof array for data verification. -An example response for BTC/USD: +#### Example Response (for BTC/USD) ```json [ @@ -130,6 +130,46 @@ An example response for BTC/USD: The floating point value of a feed can be calculated by dividing the `value` by 10^`decimals`. For example, if the feed value of BTC/USD is `6900420` and the decimal is `2`, the floating point feed value is `69004.20`. +### Fetching Timestamps + +The `ftso/anchor-feeds-with-proof` endpoint returns a `votingRoundId`. Each voting round lasts for a fixed duration of **90 seconds**. To determine the **starting timestamp**, use the `fsp/status` GET endpoint: + +```bash +curl -X 'GET' \ + 'https://flr-data-availability.flare.network/api/v0/fsp/status' \ + -H 'accept: application/json' +``` + +:::warning + +- The timestamps returned correspond to the **start** of the voting round, which lasts for **90 seconds**. +- Prices for a given voting round are finalized at the **end** of the round, calculated as `start_timestamp + 90s`. + ::: + +#### Example Response + +```json +{ + "active": { + "voting_round_id": 839641, + "start_timestamp": 1733997690 + }, + "latest_fdc": { + "voting_round_id": -1, + "start_timestamp": -1 + }, + "latest_ftso": { + "voting_round_id": 839640, + "start_timestamp": 1733997600 + } +} +``` + +The response indicates that: + +- The **currently active voting round** has `voting_round_id` [839641](https://flare-systems-explorer.flare.network/voting-epoch/839641), which **started at** `1733997690`. +- The **most recently finalized FTSO voting round** has `voting_round_id` [839640](https://flare-systems-explorer.flare.network/voting-epoch/839640), which **started at** `1733997600`. + ## Verifying proof onchain To verify feed data onchain, use the [`FtsoV2Interface`](/ftso/solidity-reference/FtsoV2Interface). This interface offers the [`verifyFeedData`](/ftso/solidity-reference/FtsoV2Interface#verifyfeeddata) method to validate feed data and proof against the onchain Merkle root. From 7d317e43d6c6f5b2ca22d62c0b32b0f0ee579aab Mon Sep 17 00:00:00 2001 From: swarnabhasinha Date: Thu, 12 Dec 2024 21:11:15 +0530 Subject: [PATCH 4/8] feat: add diagram and fix the guide --- docs/run-node/6-flare-systems.mdx | 75 ++++++++++++------- static/img/run-node-fsp/fsp_run_node_dark.svg | 1 + .../img/run-node-fsp/fsp_run_node_light.svg | 1 + 3 files changed, 52 insertions(+), 25 deletions(-) create mode 100644 static/img/run-node-fsp/fsp_run_node_dark.svg create mode 100644 static/img/run-node-fsp/fsp_run_node_light.svg diff --git a/docs/run-node/6-flare-systems.mdx b/docs/run-node/6-flare-systems.mdx index 7d5aed48..af421190 100644 --- a/docs/run-node/6-flare-systems.mdx +++ b/docs/run-node/6-flare-systems.mdx @@ -5,20 +5,33 @@ title: Flare Systems description: Flare Systems --- +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import ThemedImage from "@theme/ThemedImage"; +import useBaseUrl from "@docusaurus/useBaseUrl"; + This guide contains sample deployment configuration for infrastructure providers to enable participation in Flare Systems Protocol ([FSP](https://dev.flare.network/network/fsp/)). -Currently supported sub-protocols are [FTSOv2](https://flare.network/wp-content/uploads/FTSOv2-White-Paper.pdf) and [FDC](https://flare.network/wp-content/uploads/FDC_WP_171024_02.pdf). +Currently supported sub-protocols are [FTSOv2](https://dev.flare.network/ftso/overview) and [FDC](https://dev.flare.network/fdc/overview). ## Overview + + A data provider system for FSP consists of the following components: 1. **Flare System Client** - Responsible for all interactions with FSO smart contracts, including voting round data collection and submission for all protocols, voter registration, and additional system tasks. + Responsible for all interactions with FTSO smart contracts, including voting round data collection and submission for all protocols, voter registration, and additional system tasks. 2. **C-chain Indexer** Captures FSP-related blockchain transactions and events. Used by clients for data calculation and action triggers. 3. **FTSO Client** - Supports FTSO anchor feeds. Provides protocol votes and median data to System Client for submission. + Supports FTSO anchor feeds. Provides protocol votes and median data to the System Client for submission. 4. **Fast Updates Client** Supports FTSO block-latency feeds. Collects and submits data to Fast Updates contracts. 5. **Feed Value Provider** @@ -28,7 +41,7 @@ A data provider system for FSP consists of the following components: :::info -Reference implementations are provided for all components except **Feed Value Provider**, providers are encouraged to use own implementation conforming to the [REST API spec](https://github.com/flare-foundation/ftso-v2-example-value-provider?tab=readme-ov-file#example-usage). +Reference implementations are provided for all components except **Feed Value Provider**. Providers are encouraged to use their own implementation conforming to the [REST API spec](https://github.com/flare-foundation/ftso-v2-example-value-provider?tab=readme-ov-file#example-usage). ::: @@ -36,21 +49,33 @@ Reference implementations are provided for all components except **Feed Value Pr ### Prerequisites -:::warning +1. Registration -Complete all key registration steps in [REGISTRATION.md](https://github.com/flare-foundation/flare-systems-deployment/blob/main/docs/REGISTRATION.md), or else deployed services will not work. + Each Entity in the FSP system must have the following five addresses registered: -::: + - `Identity address`: This is the entity's main address. It should be stored in cold storage, as it will only ever be needed during the initial setup and any future rotation of other keys. + - `Submit address`: This is used for `submit1`, `submit2`, and `submit3` transactions. It can be stored in hot storage, as its private key will be added to your environment variables (`.env`) file during deployment. + - `Submit signatures`: This is used for `submitSignatures` transactions. It can be stored in hot storage, as its private key will be added to the `.env` file during deployment, + - `Signing policy`: This is used for reward epoch registration and relay transactions as well as payload signing in some parts of the system. It can be stored in hot storage, as its private key will be added to the `.env` file during deployment. + - `Delegation address`: This is used for wrapping your tokens and getting delegation. It should be stored in cold storage, as it will not be needed in day to day protocol participation. + + :::info + + You can register the above keys via the `EntityManager` contract by first calling `propose...Address` with your identity address and then calling `confirm...Address` with the proposed address. -1. Running FDC Client - This requires access to underlying chain infrastructure. You can perform the necessary deployment using information in [this repository](https://github.com/flare-foundation/fdc-suite-deployment). + All 5 addresses will need to be funded for gas fees. -2. System Dependencies + ::: + +2. Running FDC Client + This requires access to the underlying chain infrastructure. You can perform the necessary deployment using the information in [this repository](https://github.com/flare-foundation/fdc-suite-deployment). + +3. System Dependencies The following system dependencies are required to run this deployment: - [jq](https://jqlang.github.io/jq/) - [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html) - [Docker](https://www.docker.com/) - - Bash (macOs only for updated version), install by running `brew install bash` + - Bash (macOS only for an updated version), install by running `brew install bash` ### Generate configuration @@ -58,7 +83,7 @@ Copy the code in this [sample configuration file](https://github.com/flare-found Then copy the code in this [shell script](https://github.com/flare-foundation/flare-systems-deployment/blob/main/populate_config.sh) (`.sh` file) into a `populate_config.sh` file at your project's root directory. -Run `./populate_config.sh` from your root to populate configs for provider stack. +Run `./populate_config.sh` from your root to populate configs for the provider stack. :::note @@ -70,11 +95,11 @@ You'll need to rerun this command if you change your `.env` file. Copy this [docker configuration code](https://github.com/flare-foundation/flare-systems-deployment/blob/main/docker-compose.yaml) into a `docker-compose.yaml` file at the root of your project. -You can then everything up with `docker compose up -d` and stop everything with `docker compose down`. +You can then start everything up with `docker compose up -d` and stop everything with `docker compose down`. :::info -Database is persisted in a named docker volume. If you need to wipe database you need to remove the volume manually. +The database is persisted in a named docker volume. If you need to wipe the database you need to remove the volume manually. When the upstream codebase is changed new images need to be pulled with `docker compose pull`. @@ -84,7 +109,7 @@ When the upstream codebase is changed new images need to be pulled with `docker The Feed Value Provider instance for FTSO protocol is not part of `docker-compose.yaml` and needs to be created separately. -Start your own implementation or alternatively use the example implementation for testing: +Start your own implementation or use the example implementation for testing: ```bash docker run --rm -it \ @@ -95,18 +120,18 @@ docker run --rm -it \ ### FDC Suite -Underlying infrastructure required for FDC protocol is not part of this project and needs to be setup separately. This can be done via [fdc-suite-deployment](https://github.com/flare-foundation/fdc-suite-deployment). +The underlying infrastructure required for FDC protocol is not part of this project and needs to be set up separately. This can be done via [fdc-suite-deployment](https://github.com/flare-foundation/fdc-suite-deployment). + +## Potential Issues + +- **`system-client` is not doing anything** -
- Potential Issues + Llikely, your entity is not registered as a voter for the current reward epoch. There is a time window for voter registration on every reward epoch, and if you leave things running you should eventually see `RegisterVoter success` in the logs. It should then start submitting data successfully in the **following** reward epoch. - 1. `system-client` is not doing anything. - It is likely that your entity is not registered as a voter for the current reward epoch. There is a time window for voter registration on every reward epoch, and if you leave things running you should eventually see `RegisterVoter success` in the logs. It should then start submitting data successfully in the **following** reward epoch. +- **`system-client` fails to fetch sub-protocol data** - 2. `system-client` fails to fetch sub-protocol data. - The `c-chain-indexer` may still be indexing data, once it's finished you should see `Indexer at block x` in the logs. If it's still processing blocks you will see various errors across all services. + The `c-chain-indexer` may still be indexing data, once it's finished you should see `Indexer at block x` in the logs. If it's still processing blocks you will see various errors across all services. - 3. `fast-updates` client is not doing anything. - If you have low weight it may take some time until you are selected to submit an update. +- **`fast-updates` client is not doing anything** -
+ If you have low weight it may take some time until you are selected to submit an update. diff --git a/static/img/run-node-fsp/fsp_run_node_dark.svg b/static/img/run-node-fsp/fsp_run_node_dark.svg new file mode 100644 index 00000000..551421f8 --- /dev/null +++ b/static/img/run-node-fsp/fsp_run_node_dark.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/static/img/run-node-fsp/fsp_run_node_light.svg b/static/img/run-node-fsp/fsp_run_node_light.svg new file mode 100644 index 00000000..596e69ce --- /dev/null +++ b/static/img/run-node-fsp/fsp_run_node_light.svg @@ -0,0 +1 @@ + \ No newline at end of file From fd3ee5301406f15fedbe815081a3f7d5e02b0e0a Mon Sep 17 00:00:00 2001 From: Dinesh Pinto Date: Thu, 12 Dec 2024 20:22:03 +0400 Subject: [PATCH 5/8] fix(docs): clarify diagram style guide --- CONTRIBUTING.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a4c013da..adb878a2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -97,14 +97,15 @@ Follow the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/ When contributing diagrams, follow the established visual style: -| **Element** | **Light Mode** | **Dark Mode** | -| ------------------------- | -------------- | ------------- | -| **Border Width** | 1px | 1px | -| **Border Color** | `#E7125E` | `#EF4A82` | -| **Arrow Width** | 1px | 1px | -| **Arrow Color** | `#595959` | `#FFFFFF` | -| **Onchain Border Style** | Solid | Solid | -| **Offchain Border Style** | Dashed | Dashed | +| **Element** | **Light Mode** | **Dark Mode** | +| ---------------------------- | -------------- | ------------- | +| **Arrow Width** | 1px | 1px | +| **Arrow Color** | `#595959` | `#FFFFFF` | +| **Border Width** | 1px | 1px | +| **Border Color (Highlight)** | `#E7125E` | `#EF4A82` | +| **Border Color (Normal)** | `#595959` | `#FFFFFF` | +| **Onchain Border Style** | Solid | Solid | +| **Offchain Border Style** | Dashed | Dashed | **Tip:** Use the dashed style **only if** both onchain and offchain elements are displayed. From 5103442b41d4ca6289af5c23639c1bc46ed88f00 Mon Sep 17 00:00:00 2001 From: swarnabhasinha Date: Fri, 13 Dec 2024 01:37:26 +0530 Subject: [PATCH 6/8] fix: diagrams --- static/img/run-node-fsp/fsp_run_node_dark.svg | 2 +- static/img/run-node-fsp/fsp_run_node_light.svg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/static/img/run-node-fsp/fsp_run_node_dark.svg b/static/img/run-node-fsp/fsp_run_node_dark.svg index 551421f8..0cf566ff 100644 --- a/static/img/run-node-fsp/fsp_run_node_dark.svg +++ b/static/img/run-node-fsp/fsp_run_node_dark.svg @@ -1 +1 @@ - \ No newline at end of file + \ No newline at end of file diff --git a/static/img/run-node-fsp/fsp_run_node_light.svg b/static/img/run-node-fsp/fsp_run_node_light.svg index 596e69ce..420513c3 100644 --- a/static/img/run-node-fsp/fsp_run_node_light.svg +++ b/static/img/run-node-fsp/fsp_run_node_light.svg @@ -1 +1 @@ - \ No newline at end of file + \ No newline at end of file From a073ff8443fa80c6e2c95680d20bc3b36e660d95 Mon Sep 17 00:00:00 2001 From: Dinesh Pinto Date: Fri, 13 Dec 2024 02:18:37 +0400 Subject: [PATCH 7/8] feat: update fsp deployment guide --- docs/run-node/6-flare-systems-provider.mdx | 499 +++++++++++++++++++++ docs/run-node/6-flare-systems.mdx | 137 ------ 2 files changed, 499 insertions(+), 137 deletions(-) create mode 100644 docs/run-node/6-flare-systems-provider.mdx delete mode 100644 docs/run-node/6-flare-systems.mdx diff --git a/docs/run-node/6-flare-systems-provider.mdx b/docs/run-node/6-flare-systems-provider.mdx new file mode 100644 index 00000000..bafa8dda --- /dev/null +++ b/docs/run-node/6-flare-systems-provider.mdx @@ -0,0 +1,499 @@ +--- +sidebar_position: 6 +title: Flare Systems Provider +description: Provide data for Flare's enshrined FTSO and FDC protocols. +--- + +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; +import ThemedImage from "@theme/ThemedImage"; +import useBaseUrl from "@docusaurus/useBaseUrl"; + +This guide provides a sample deployment configuration for infrastructure providers participating in the [Flare Systems Protocol (FSP)](https://dev.flare.network/network/fsp). +The FSP consists of the [Flare Time Series Oracle](https://dev.flare.network/ftso/overview) and [Flare Data Connector](https://dev.flare.network/fdc/overview). + +## Overview + + + +An FSP provider system consists of the following components: + +1. **Flare System Client**: Manages interactions with FTSO smart contracts, including data collection, submission, voter registration, and system tasks. +2. **C-chain Indexer**: Tracks FSP-related blockchain transactions and events, enabling data calculations and action triggers. +3. **FTSO Client**: Provides anchor feed submissions and median data to the System Client. +4. **Fast Updates Client**: Submits block-latency feeds to Fast Updates contracts. +5. **Feed Value Provider**: Retrieves data from exchanges and supplies current feed values (prices). +6. **FDC Client**: Provides FDC protocol voting round data to the System Client. + +## Hardware requirements + +To deploy all of Flare's enshrined data protocols to a single HW instance: + + + + + | | **Requirement** | + |:--------------------------|:----------------| + | **CPU** | 16/32 cores | + | **RAM** | 64 GB | + | **Disk space** | 4 TB SSD | + + + + + | | **Requirement** | + |:--------------------------|:----------------| + | **CPU** | 8 cores | + | **RAM** | 16 GB | + | **Disk space** | 100 GB SSD | + + + + + | | **Requirement** | + |:--------------------------|:-----------------| + | **CPU** | 16/32 cores | + | **RAM** | 64 GB | + | **Disk space** | 4 TB SSD | + + + + + | | **Requirement** | + |:--------------------------|:----------------| + | **CPU** | 8 cores | + | **RAM** | 16 GB | + | **Disk space** | 100 GB SSD | + + + + +- **Disk speed:** 1200 MB/s read and 600 MB/s write, or higher +- **Network speed:** 40 Mbps, or higher + +## Prerequisites + +Ensure you have the following tools installed: + +- [Docker Engine](https://docs.docker.com/engine/install/) +- [yarn](https://yarnpkg.com/) +- [jq](https://jqlang.github.io/jq/) +- [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html) + +## Registration + +:::info + +Registration only needs to be performed once. + +::: + +### Register addresses + +Each entity in the FSP system must register the following five addresses: + +- `Identity`. Main identity account of the voter. On mainnet this should be held in cold storage. This account is required for initial data provider setup, but is not used during each voting round. + +- `Submit`. Used for sending commit and reveal transactions. + +- `SubmitSignatures`. Used for sending voting round result signature transactions. (To avoid nonce conflicts, **Flare System Client** uses multiple accounts for submitting transactions). + +- `SigningPolicy`. Used for signature generation during the voting round, and reward epoch signing policy signing (it's a system protocol ran once during reward epoch to establish reward epoch settings, including valid voters and their weights). + +- `Delegation`. Account to which community should delegate funds (using `WNat` contract) to increase the vote power of the voter (identity/entity) - and also to later get the rewards. + +To register you can directly call the `EntityManager` contract. For the contract address, see the [Solidity Reference](/network/solidity-reference). The relevant functions on the `EntityManager` contract are: + +- [`proposeSubmitAddress`](/network/fsp/solidity-reference/IEntityManager#proposesubmitaddress) and [`confirmSubmitAddressRegistration`](/network/fsp/solidity-reference/IEntityManager#confirmsubmitaddressregistration) +- [`proposeSubmitSignaturesAddress`](/network/fsp/solidity-reference/IEntityManager#proposesubmitsignaturesaddress) and [`confirmSubmitSignaturesAddressRegistration`](/network/fsp/solidity-reference/IEntityManager#confirmsubmitsignaturesaddressregistration) +- [`proposeSigningPolicyAddress`](/network/fsp/solidity-reference/IEntityManager#proposesigningpolicyaddress) and [`confirmSigningPolicyAddressRegistration`](/network/fsp/solidity-reference/IEntityManager#confirmsigningpolicyaddressregistration) +- [`proposeDelegationAddress`](/network/fsp/solidity-reference/IEntityManager#proposedelegationaddress) and [`confirmDelegationAddressRegistration`](/network/fsp/solidity-reference/IEntityManager#confirmdelegationaddressregistration) + +First call the `propose...` with your `Identity` address and then the `confirm...` with the proposed address. + +All addresses need to be funded for gas fees. The `Delegation` account is used for establishing voter power, which can be achieved by wrapping funds directly or by delegation from other accounts. Wrapping can be done via: + +- [Development Portal](https://governance.dev.aflabs.org/) for Flare Testnet Coston2 and Songbird Testnet Coston +- [Flare Portal](https://portal.flare.network) for Flare Mainnet and Songbird Canary-Network. + +:::warning + +The protocol operation uses normalized weights, and the delegation account should have at least 150 Wrapped Native Tokens to obtain a non-zero vote power. + +| **Network** | **Native Token** | **Wrapped Native Token** | +| ----------------------- | ---------------- | ------------------------ | +| Flare Mainnet | FLR | WFLR | +| Flare Testnet Coston2 | C2FLR | WC2FLR | +| Songbird Canary-Network | SGB | WSGB | +| Songbird Testnet Coston | CFLR | WCFLR | + +The contract addresses of the Wrapped Native Tokens (WNat) are in the [Solidity Reference](/network/solidity-reference). + +::: + +### Register sortition key + +Generate a sortition key to use in Fast Updates. You can use the [fast-updates/go-client](https://github.com/flare-foundation/fast-updates/tree/main/go-client) docker image to generate a key: + +```bash +docker run --rm ghcr.io/flare-foundation/fast-updates/go-client:latest keygen +``` + +Accounts for Fast Updates submission need to be generated and funded with gas fees. +These can be any accounts **not used** for the five accounts in [register addresses](#register-addresses). +You are recommended to use **three** separate accounts to avoid nonce conflicts. + +Register the generated sortition key via the `EntityManager` contract by calling [`registerPublicKey`](/network/fsp/solidity-reference/IEntityManager#registerpublickey) with your identity address. + +
+Automated registration on testnets. + +:::danger + +Automated registration can expose private keys. Only use on testnets. + +::: + +On testnets, you can register your entity addresses with the [`register-entities.ts`](https://github.com/flare-foundation/flare-smart-contracts-v2/blob/main/deployment/tasks/register-entities.ts) script and your sortition public key with the [`register-public-keys.ts`](https://github.com/flare-foundation/flare-smart-contracts-v2/blob/main/deployment/tasks/register-public-keys.ts) scripts. + +Start by cloning and building [flare-smart-contracts-v2](https://github.com/flare-foundation/flare-smart-contracts-v2/): + +```bash +git clone https://github.com/flare-foundation/flare-smart-contracts-v2/ +cd flare-smart-contracts-v2 +yarn +yarn c +``` + +Create an `entities.json` file with the following account addresses and private keys: + +```json +[ + { + "identity": { + "address": "
", + "privateKey": "" + }, + "submit": { + "address": "
", + "privateKey": "" + }, + "submitSignatures": { + "address": "
", + "privateKey": "" + }, + "signingPolicy": { + "address": "
", + "privateKey": "" + }, + "delegation": { + "address": "
", + "privateKey": "" + }, + "sortitionPrivateKey": "" + } +] +``` + +Setup the following variables in `.env`: + + + + + ```text title=".env" + ENTITIES_FILE_PATH="" + COSTON2_RPC= + CHAIN_CONFIG="coston2" + ``` + + + + + ```text title=".env" + ENTITIES_FILE_PATH="" + SONGBIRD_RPC= + CHAIN_CONFIG="songbird" + ``` + + + + + ```text title=".env" + ENTITIES_FILE_PATH="" + COSTON_RPC= + CHAIN_CONFIG="coston" + ``` + + + + +Run the registration tasks: + + + + + ```bash + yarn hardhat --network coston2 register-entities + yarn hardhat --network coston2 register-public-keys + ``` + + + + + ```bash + yarn hardhat --network songbird register-entities + yarn hardhat --network songbird register-public-keys + ``` + + + + + ```bash + yarn hardhat --network coston register-entities + yarn hardhat --network coston register-public-keys + ``` + + + +
+ +## Setup FDC + +The FDC suite integrates blockchain data sources through a network of blockchain nodes, indexers, and API servers. This setup allows secure and efficient data verification. The blockchain data flow is of two types: + +- **UTXO-Based Chains** (Bitcoin, Dogecoin, Ripple): Use an indexer to create a local database from the blockchain, exposing data through a Verifier API server. + +- **EVM Chains** (Ethereum, Flare, Songbird): Directly query the RPC node using the Verifier API server. + +### Required components + +You can use the following components to run a full FDC suite. + +- Blockchain nodes (Docker images): + + - Bitcoin - [flarefoundation/bitcoin](https://hub.docker.com/r/flarefoundation/bitcoin) + - Dogecoin - [flarefoundation/dogecoin](https://hub.docker.com/r/flarefoundation/dogecoin) + - Ripple - [flarefoundation/rippled](https://hub.docker.com/r/flarefoundation/rippled) + - Ethereum - [ethereum/client-go](https://hub.docker.com/r/ethereum/client-go) and [prysm](https://docs.prylabs.network/docs/install/install-with-docker) + +- Indexers and verification servers for: + + - BTC - [flare-foundation/verifier-utxo-indexer](https://github.com/flare-foundation/verifier-utxo-indexer) and [flare-foundation/verifier-indexer-api](https://github.com/flare-foundation/verifier-indexer-api) + - DOGE - [flare-foundation/verifier-utxo-indexer](https://github.com/flare-foundation/verifier-utxo-indexer) and [flare-foundation/verifier-indexer-api](https://github.com/flare-foundation/verifier-indexer-api) + - XRP - [flare-foundation/verifier-xrp-indexer](https://github.com/flare-foundation/verifier-xrp-indexer) and [flare-foundation/verifier-indexer-api](https://github.com/flare-foundation/verifier-indexer-api) + +- EVM verifier - [flare-foundation/evm-verifier](https://github.com/flare-foundation/evm-verifier) + +:::info + +- FLR and SGB nodes are required for EVM verification but are not listed here. +- You are not required to use exactly these components, eg. if you already have a compatible Bitcoin RPC node, you can configure your deployment to run everything else except for Bitcoin node. +- You can split the deployment across multiple servers. + +::: + +Start by cloning the [fdc-suite-deployment](https://github.com/flare-foundation/fdc-suite-deployment.git) repo: + +```bash +git clone https://github.com/flare-foundation/fdc-suite-deployment.git +cd fdc-suite-deployment +``` + +### Configuring blockchain nodes + +#### Bitcoin (BTC) + +Generate the authentication password: + +```bash +cd nodes-mainnet/btc +./generate-password.sh +``` + +Sample output: + +```plaintext +password: c021cae645db6d3371b26ced94c8d17a5d9f3accbf3591d8b4c0be19623e5662 +String to be appended to bitcoin.conf: +rpcauth=admin:a0956d81a2344f1602d9ed7b82ef3118$2caf19c9cf27937f728f600fc14e8db97f80218d727e331a57c3cfc55b3e17fe +Your password: +c021cae645db6d3371b26ced94c8d17a5d9f3accbf3591d8b4c0be19623e5662 +``` + +Alternatively you can configure the username and password manually: + +```bash +./rpcauth.py +``` + +#### Dogecoin (DOGE) + +Same configuration process as Bitcoin. + +```bash +cd nodes-mainnet/doge +./generate-password.sh +``` + +#### Ripple (XRP) + +No additional configuration required. + +#### Ethereum (ETH) + +Generate JWT for Authentication: + +```bash +openssl rand -hex 32 > nodes-mainnet/eth/jwt.hex +``` + +:::warning + +Blockchain nodes expose all ports by default. + +::: + +### Configuring indexers and verifiers + +Copy `.env.example` into `.env`: + +```bash +cp .env.example .env +``` + +#### Configure required keys + +- **RPC Nodes Authentication:** Use credentials generated earlier. If you run blockchain nodes and verifiers on the same server, you can use the ip `172.17.0.1` to reach the nodes. + +- **Start Block Number:** Set `*_START_BLOCK_NUMBER` to a block finalized 14 days ago. This needs to be set the first time when you start the indexers to avoid indexing too much data. FDC requires at least 14 days of history. On later restarts indexers will start indexing from the latest block in the database. + +- **Testnet Mode:** Set `TESTNET=true` if using testnets. +- **API Keys:** Configure `VERIFIER_API_KEYS` with comma-separated API keys. One or more comma separated keys can be configured. You will likely need at least one key for FDC client that will call verifier api servers. +- **Database Passwords:** Set `*_DB_PASSWORD` variables to random strings. These are used internally for the indexer database. + +#### Generate configuration files + +```bash +./generate-config.sh +``` + +This script populates config files from `*.example` templates in: + +- `verifiers/btc/` +- `verifiers/doge/` +- `verifiers/xrp/` +- `evm-verifier/` + +### Start services + +#### Start blockchain nodes + +1. Navigate to the node directory, e.g., `nodes-mainnet/btc` + +2. Start the node: + + ```bash + docker compose up -d + ``` + +Repeat this process for all blockchain nodes you plan to run. + +#### Start indexers and verifiers + +1. Navigate to the appropriate verifier directory, e.g., `verifiers/btc` + +2. Start the verifier: + ```bash + docker compose up -d + ``` + +Repeat this process for all verifiers you plan to run. + +## Setup FTSO + +### Feed value provider + +:::warning + +The reference implementation of Feed Value Provider is **only provided for testing purposes and should not be relied on**. +Data providers are expected to reimplement this with their own data sources, following the [REST API specification](https://github.com/flare-foundation/ftso-v2-example-value-provider?tab=readme-ov-file#example-usage). + +::: + +Start your own feed value provider, or **(for testing only)** use the reference feed value provider: + +```bash +docker run --rm -it --publish "0.0.0.0:3101:3101" --network "ftso-v2-deployment_default" ghcr.io/flare-foundation/ftso-v2-example-value-provider +``` + +Once the container is running, you can find the API spec at: http://localhost:3101/api-doc + +#### Troubleshooting + +For initial testing a fixed value provider can be used that simply returns a constant instead of reading data from external data sources. It can be started by setting the variable `VALUE_PROVIDER_IMPL=fixed`: + +```bash +docker run --rm -it --env VALUE_PROVIDER_IMPL=fixed --publish "0.0.0.0:3101:3101" --network "ftso-v2-deployment_default" ghcr.io/flare-foundation/ftso-v2-example-value-provider +``` + +You should see the following line in the logs: + +```plaintext +WARN [FixedFeed] Initializing FixedFeed, will return 0.01 for all feeds. +``` + +## Generate configuration + +1. Copy the sample configuration file: + + ```bash + cp .env.example .env + ``` + +2. Fill out all the values using the information from [Registration](#registration), [Setup FDC](#setup-fdc) and [Setup FTSO](#setup-ftso). + +3. Generate the required configurations, you will need to rerun this command if you change your `.env` file: + + ```bash + ./populate_config.sh + ``` + +## Deploy + +Since `docker-compose.yaml` is already provided, you can start everything with: + +```bash +docker compose up -d +``` + +To stop: + +```bash +docker compose down +``` + +To pull the latest images: + +```bash +docker compose pull +``` + +## Troubleshooting + +- **`system-client` is not doing anything** + + Llikely, your entity is not registered as a voter for the current reward epoch. There is a time window for voter registration on every reward epoch, and if you leave things running you should eventually see `RegisterVoter success` in the logs. It should then start submitting data successfully in the **following** reward epoch. + +- **`system-client` fails to fetch sub-protocol data** + + The `c-chain-indexer` may still be indexing data, once it's finished you should see `Indexer at block x` in the logs. If it's still processing blocks you will see various errors across all services. + +- **`fast-updates` client is not doing anything** + + If you have low weight it may take some time until you are selected to submit an update. diff --git a/docs/run-node/6-flare-systems.mdx b/docs/run-node/6-flare-systems.mdx deleted file mode 100644 index af421190..00000000 --- a/docs/run-node/6-flare-systems.mdx +++ /dev/null @@ -1,137 +0,0 @@ ---- -sidebar_position: 6 -slug: flare-systems -title: Flare Systems -description: Flare Systems ---- - -import Tabs from "@theme/Tabs"; -import TabItem from "@theme/TabItem"; -import ThemedImage from "@theme/ThemedImage"; -import useBaseUrl from "@docusaurus/useBaseUrl"; - -This guide contains sample deployment configuration for infrastructure providers to enable participation in Flare Systems Protocol ([FSP](https://dev.flare.network/network/fsp/)). - -Currently supported sub-protocols are [FTSOv2](https://dev.flare.network/ftso/overview) and [FDC](https://dev.flare.network/fdc/overview). - -## Overview - - - -A data provider system for FSP consists of the following components: - -1. **Flare System Client** - Responsible for all interactions with FTSO smart contracts, including voting round data collection and submission for all protocols, voter registration, and additional system tasks. -2. **C-chain Indexer** - Captures FSP-related blockchain transactions and events. Used by clients for data calculation and action triggers. -3. **FTSO Client** - Supports FTSO anchor feeds. Provides protocol votes and median data to the System Client for submission. -4. **Fast Updates Client** - Supports FTSO block-latency feeds. Collects and submits data to Fast Updates contracts. -5. **Feed Value Provider** - Gathers data from exchanges and provides current values (prices) for a given set of feeds. -6. **FDC Client** - Interacts with FDC suite and provides FDC protocol voting round data to SystemClient for submission. - -:::info - -Reference implementations are provided for all components except **Feed Value Provider**. Providers are encouraged to use their own implementation conforming to the [REST API spec](https://github.com/flare-foundation/ftso-v2-example-value-provider?tab=readme-ov-file#example-usage). - -::: - -## Deployment - -### Prerequisites - -1. Registration - - Each Entity in the FSP system must have the following five addresses registered: - - - `Identity address`: This is the entity's main address. It should be stored in cold storage, as it will only ever be needed during the initial setup and any future rotation of other keys. - - `Submit address`: This is used for `submit1`, `submit2`, and `submit3` transactions. It can be stored in hot storage, as its private key will be added to your environment variables (`.env`) file during deployment. - - `Submit signatures`: This is used for `submitSignatures` transactions. It can be stored in hot storage, as its private key will be added to the `.env` file during deployment, - - `Signing policy`: This is used for reward epoch registration and relay transactions as well as payload signing in some parts of the system. It can be stored in hot storage, as its private key will be added to the `.env` file during deployment. - - `Delegation address`: This is used for wrapping your tokens and getting delegation. It should be stored in cold storage, as it will not be needed in day to day protocol participation. - - :::info - - You can register the above keys via the `EntityManager` contract by first calling `propose...Address` with your identity address and then calling `confirm...Address` with the proposed address. - - All 5 addresses will need to be funded for gas fees. - - ::: - -2. Running FDC Client - This requires access to the underlying chain infrastructure. You can perform the necessary deployment using the information in [this repository](https://github.com/flare-foundation/fdc-suite-deployment). - -3. System Dependencies - The following system dependencies are required to run this deployment: - - [jq](https://jqlang.github.io/jq/) - - [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html) - - [Docker](https://www.docker.com/) - - Bash (macOS only for an updated version), install by running `brew install bash` - -### Generate configuration - -Copy the code in this [sample configuration file](https://github.com/flare-foundation/flare-systems-deployment/blob/main/.env.example) into a `.env` file at the root of your project and fill out the values as appropriate. - -Then copy the code in this [shell script](https://github.com/flare-foundation/flare-systems-deployment/blob/main/populate_config.sh) (`.sh` file) into a `populate_config.sh` file at your project's root directory. - -Run `./populate_config.sh` from your root to populate configs for the provider stack. - -:::note - -You'll need to rerun this command if you change your `.env` file. - -::: - -### Start deployment - -Copy this [docker configuration code](https://github.com/flare-foundation/flare-systems-deployment/blob/main/docker-compose.yaml) into a `docker-compose.yaml` file at the root of your project. - -You can then start everything up with `docker compose up -d` and stop everything with `docker compose down`. - -:::info - -The database is persisted in a named docker volume. If you need to wipe the database you need to remove the volume manually. - -When the upstream codebase is changed new images need to be pulled with `docker compose pull`. - -::: - -### Feed Value Provider - -The Feed Value Provider instance for FTSO protocol is not part of `docker-compose.yaml` and needs to be created separately. - -Start your own implementation or use the example implementation for testing: - -```bash -docker run --rm -it \ - --publish "0.0.0.0:3101:3101" \ - --network "ftso-v2-deployment_default" \ - ghcr.io/flare-foundation/ftso-v2-example-value-provider -``` - -### FDC Suite - -The underlying infrastructure required for FDC protocol is not part of this project and needs to be set up separately. This can be done via [fdc-suite-deployment](https://github.com/flare-foundation/fdc-suite-deployment). - -## Potential Issues - -- **`system-client` is not doing anything** - - Llikely, your entity is not registered as a voter for the current reward epoch. There is a time window for voter registration on every reward epoch, and if you leave things running you should eventually see `RegisterVoter success` in the logs. It should then start submitting data successfully in the **following** reward epoch. - -- **`system-client` fails to fetch sub-protocol data** - - The `c-chain-indexer` may still be indexing data, once it's finished you should see `Indexer at block x` in the logs. If it's still processing blocks you will see various errors across all services. - -- **`fast-updates` client is not doing anything** - - If you have low weight it may take some time until you are selected to submit an update. From 0de37c934d98386b3b27fd68566f057657d2c154 Mon Sep 17 00:00:00 2001 From: Dinesh Pinto Date: Fri, 13 Dec 2024 02:19:17 +0400 Subject: [PATCH 8/8] fix: mark guide as unlisted --- docs/run-node/6-flare-systems-provider.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/run-node/6-flare-systems-provider.mdx b/docs/run-node/6-flare-systems-provider.mdx index bafa8dda..ee6fdfc3 100644 --- a/docs/run-node/6-flare-systems-provider.mdx +++ b/docs/run-node/6-flare-systems-provider.mdx @@ -2,6 +2,7 @@ sidebar_position: 6 title: Flare Systems Provider description: Provide data for Flare's enshrined FTSO and FDC protocols. +unlisted: true --- import Tabs from "@theme/Tabs";