docs: change readme files to include information about the data-fetcher

README.md

@@ -3,8 +3,9 @@
 <p align="center">Online blockchain browser for viewing and analyzing <a href="https://zksync.io">zkSync Era</a> blockchain.</p>
 
 ## 📌 Overview
-This repository is a monorepo consisting of 3 packages:
-- [Worker](./packages/worker) - an indexer service for [zkSync Era](https://zksync.io) blockchain data. The purpose of the service is to read the data from the blockchain in real time, transform it and fill in it's database with the data in a way that makes it easy to be queried by the [API](./packages/api) service.
+This repository is a monorepo consisting of 4 packages:
+- [Worker](./packages/worker) - an indexer service for [zkSync Era](https://zksync.io) blockchain data. The purpose of the service is to read blockchain data in real time, transform it and fill in it's database with the data in a way that makes it easy to be queried by the [API](./packages/api) service.
+- [Data Fetcher](./packages/data-fetcher) - a service that exposes and implements an HTTP endpoint to retrieve aggregated data for a certain block / range of blocks from the blockchain. This endpoint is called by the [Worker](./packages/worker) service.
 - [API](./packages/api) - a service providing Web API for retrieving structured [zkSync Era](https://zksync.io) blockchain data collected by [Worker](./packages/worker). It connects to the Worker's database to be able to query the collected data.
 - [App](./packages/app) - a front-end app providing an easy-to-use interface for users to view and inspect transactions, blocks, contracts and more. It makes requests to the [API](./packages/api) to get the data and presents it in a way that's easy to read and understand.
 
@@ -20,10 +21,14 @@ flowchart
   subgraph explorer[Block explorer]
     Database[("Block explorer DB<br/>(PostgreSQL)")]
     Worker(Worker service)
+    Data-Fetcher(Data Fetcher service)
     API(API service)
     App(App)
-
+    
+    Worker-."Request aggregated data (HTTP)".->Data-Fetcher
+    Data-Fetcher-."Request data (HTTP)".->Blockchain
     Worker-.Save processed data.->Database
+
     API-.Query data.->Database
     App-."Request data (HTTP)".->API
     App-."Request data (HTTP)".->Blockchain
@@ -32,7 +37,7 @@ flowchart
   Worker-."Request data (HTTP)".->Blockchain
 ```
 
-[Worker](./packages/worker) service is responsible for getting data from blockchain using [zkSync Era JSON-RPC API](https://era.zksync.io/docs/api/api.html), processing it and saving into the database. [API](./packages/api) service is connected to the same database where it gets the data from to handle API requests. It performs only read requests to the database. The front-end [App](./packages/app) makes HTTP calls to the Block Explorer [API](./packages/api) to get blockchain data and to the [zkSync Era JSON-RPC API](https://era.zksync.io/docs/api/api.html) for reading contracts, performing transactions etc.
+[Worker](./packages/worker) service retrieves aggregated data from the [Data Fetcher](./packages/data-fetcher) via HTTP and also directly from the blockchain using [zkSync Era JSON-RPC API](https://era.zksync.io/docs/api/api.html), processes it and saves into the database. [API](./packages/api) service is connected to the same database where it gets the data from to handle API requests. It performs only read requests to the database. The front-end [App](./packages/app) makes HTTP calls to the Block Explorer [API](./packages/api) to get blockchain data and to the [zkSync Era JSON-RPC API](https://era.zksync.io/docs/api/api.html) for reading contracts, performing transactions etc.
 
 ## 🚀 Features
 
@@ -56,12 +61,12 @@ npm install
 ## ⚙️ Setting up env variables
 
 ### Manually set up env variables
-Make sure you have set up all the necessary env variables. Follow [Setting up env variables for Worker](./packages/worker#setting-up-env-variables) and [Setting up env variables for API](./packages/api#setting-up-env-variables) for instructions. For the [App](./packages/app) package you might want to edit environment config, see [Environment configs](./packages/app#environment-configs).
+Make sure you have set up all the necessary env variables. Follow setting up env variables instructions for [Worker](./packages/worker#setting-up-env-variables), [Data Fetcher](./packages/data-fetcher#setting-up-env-variables) and [API](./packages/api#setting-up-env-variables). For the [App](./packages/app) package you might want to edit environment config, see [Environment configs](./packages/app#environment-configs).
 
 ### Build env variables based on your [zksync-era](https://github.com/matter-labs/zksync-era) local repo setup
 Make sure you have [zksync-era](https://github.com/matter-labs/zksync-era) repo set up locally. You must have your environment variables files present in the [zksync-era](https://github.com/matter-labs/zksync-era) repo at `/etc/env/*.env` for the build envs script to work.
 
-The following script sets `.env` files for [Worker](./packages/worker) and [API](./packages/api) packages as well as environment configuration file for [App](./packages/app) package based on your local [zksync-era](https://github.com/matter-labs/zksync-era) repo setup.
+The following script sets `.env` files for [Worker](./packages/worker), [Data Fetcher](./packages/data-fetcher) and [API](./packages/api) packages as well as environment configuration file for [App](./packages/app) package based on your local [zksync-era](https://github.com/matter-labs/zksync-era) repo setup.
 ```bash
 npm run hyperchain:configure
 ```
@@ -75,7 +80,7 @@ To create a database run the following command:
 npm run db:create
 ```
 
-To run all the packages (`Worker`, `API` and front-end `App`) in `development` mode run the following command from the root directory.
+To run all the packages (`Worker`, `Data Fetcher`, `API` and front-end `App`) in `development` mode run the following command from the root directory.
 ```bash
 npm run dev
 ```
@@ -100,7 +105,7 @@ To get block-explorer connected to your ZK Stack Hyperchain you need to set up a
 
 ## 🔍 Verify Block Explorer is up and running
 
-To verify front-end `App` is running open http://localhost:3010 in your browser. `API` should be available at http://localhost:3020. `Worker` - http://localhost:3001.
+To verify front-end `App` is running open http://localhost:3010 in your browser. `API` should be available at http://localhost:3020, `Worker` at http://localhost:3001 and `Data Fetcher` at http://localhost:3040.
 
 ## 🕵️‍♂️ Testing
 Run unit tests for all packages:

packages/data-fetcher/README.md

@@ -1,7 +1,7 @@
 # zkSync Era Block Explorer Data Fetcher
 ## Overview
 
-`zkSync Era Block Explorer Data Fetcher` service exposes and implements an HTTP endpoint to retrieve all data for a certain block / range of blocks from the blockchain. This endpoint is called by the [Block Explorer Worker](/packages/worker) service.
+`zkSync Era Block Explorer Data Fetcher` service exposes and implements an HTTP endpoint to retrieve aggregated data for a certain block / range of blocks from the blockchain. This endpoint is called by the [Block Explorer Worker](/packages/worker) service.
 
 ## Installation
 

packages/worker/README.md

@@ -1,7 +1,7 @@
 # zkSync Era Block Explorer Worker
 ## Overview
 
-`zkSync Era Block Explorer Worker` is an indexer service for zkSync Era blockchain data. The purpose of the service is to read the data from the blockchain in real time, transform it and fill in it's database with the data in a way that makes it easy to read by the [Block explorer API](/packages/api).
+`zkSync Era Block Explorer Worker` is an indexer service for zkSync Era blockchain data. It retrieves aggregated data from the [Data Fetcher](/packages/data-fetcher) via HTTP and also directly from the blockchain using [zkSync Era JSON-RPC API](https://era.zksync.io/docs/api/api.html), processes it and saves into the database in a way that makes it easy to read by the [Block Explorer API](/packages/api).
 
 ## Installation
 
@@ -16,6 +16,7 @@ $ npm install
 cp .env.example .env
 ```
 - In order to tell the service where to get the blockchain data from set the value of the `BLOCKCHAIN_RPC_URL` env var to your blockchain RPC API URL. For zkSync Era testnet it can be set to `https://zksync2-testnet.zksync.dev`. For zkSync Era mainnet - `https://zksync2-mainnet.zksync.io`.
+- To retrieve aggregated blockchain data for a certain block, the Worker service calls the [Data Fetcher](/packages/data-fetcher) service via HTTP. To specify Data Fetcher URL use `DATA_FETCHER_URL` env variable. By default, it is set to `http://localhost:3040` which is a default value for the local environment.
 - Set up env variables for Postgres database connection. By default it points to `localhost:5432` and database name is `block-explorer`.
 You need to have a running Postgres server, set the following env variables to point the service to your database:
   - `DATABASE_HOST`

scripts/setup-hyperchain-config.ts

@@ -21,6 +21,12 @@ const buildAppConfig = (zkSyncEnvs: { [key: string]: string }) => ({
   }]
 });
 
+const buildDataFetcherConfig = (zkSyncEnvs: { [key: string]: string }) => {
+  return {
+    BLOCKCHAIN_RPC_URL: zkSyncEnvs.API_WEB3_JSON_RPC_HTTP_URL || "",
+  }
+};
+
 const buildWorkerConfig = (zkSyncEnvs: { [key: string]: string }) => {
   const dbConfig = parseConnectionString(zkSyncEnvs.DATABASE_URL);
   return {
@@ -29,6 +35,7 @@ const buildWorkerConfig = (zkSyncEnvs: { [key: string]: string }) => {
     DATABASE_USER: dbConfig.user || "",
     DATABASE_PASSWORD: dbConfig.password || "",
     DATABASE_NAME: "block-explorer",
+    DATA_FETCHER_URL: "http://localhost:3040",
   }
 };
 
@@ -77,12 +84,18 @@ const buildEnvFileContent = (json: { [key: string]: string | number }) => Object
   const envs = dotenv.parse(readFileSync(selectedEnvFilePath));
   const appConfig = buildAppConfig(envs);
   const workerConfig = buildWorkerConfig(envs);
+  const dataFetcherConfig = buildDataFetcherConfig(envs);
   const apiConfig = buildApiConfig(envs);
 
   writeFileSync(path.join(__dirname, "../packages/app/src/configs/hyperchain.config.json"), JSON.stringify(appConfig, null, 2));
   console.log("Updated app config at app/src/configs/hyperchain.config.json");
+
+  writeFileSync(path.join(__dirname, "../packages/data-fetcher/.env"), buildEnvFileContent(dataFetcherConfig));
+  console.log("Updated data-fetcher env file at data-fetcher/.env");
+
   writeFileSync(path.join(__dirname, "../packages/worker/.env"), buildEnvFileContent(workerConfig));
   console.log("Updated worker env file at worker/.env");
+
   writeFileSync(path.join(__dirname, "../packages/api/.env"), buildEnvFileContent(apiConfig));
   console.log("Updated api env file at api/.env");
 })();