forked from Concordium/concordium-node
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
It has historical use, but it is no longer relevant.
- Loading branch information
Showing
9 changed files
with
203 additions
and
164 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,10 @@ | ||
| # P2P Layer implementation in Rust | ||
| `master` [](https://gitlab.com/Concordium/p2p-client/commits/master) `develop` [](https://gitlab.com/Concordium/p2p-client/commits/develop) | ||
| # Concordium node implementation | ||
|
|
||
| `master` [](https://gitlab.com/Concordium/concordium-node/commits/master). | ||
|
|
||
| ## General usage information | ||
| This repository relies on git submodules for internal component dependencies, so do remember to clone recursively or use `git submodule update --init --recursive` after having cloned it. | ||
| This repository relies on git submodules for some internal and external component dependencies. | ||
| Do remember to clone recursively or use `git submodule update --init --recursive` after having cloned it. | ||
|
|
||
| ## Dependencies to build the project | ||
| * Rust (stable 1.45.2 for using static libraries) | ||
|
|
@@ -17,7 +19,7 @@ This repository relies on git submodules for internal component dependencies, so | |
| * PostGreSQL >= 10 | ||
|
|
||
| ### Optional dependencies | ||
| * Stack (and GHC-8.8.4, if not building using static libraries) | ||
| * [Haskell stack](https://docs.haskellstack.org/en/stable/README/) if **not** building using static libraries | ||
| * capnp (for running `s11n_capnp` enabled benches only) | ||
|
|
||
| ## Supported features | ||
|
|
@@ -38,147 +40,89 @@ This repository relies on git submodules for internal component dependencies, so | |
|
|
||
| ## Building the node | ||
|
|
||
| See instructions in | ||
| [consensus](./deps/internal/consensus/consensus-rust/README.md) for how to link | ||
| the node. | ||
|
|
||
| ## Setting up basic local build environment | ||
| Install the needed dependencies from the list above, and run the script (requires that the user executing is has sudo privileges) `scripts/local-setup-unix-deps.sh` and pay special attention to setting the right version of GHC (see [build scripts](/scripts/local-setup-unix-deps.sh#L28) for details). | ||
|
|
||
| Alternatively use `--features=static` to build statically against the haskell dependencies (only available on Linux, and requries that you download them using [scripts/download-static-libs.sh](/scripts/download-static-libs.sh) before first compilation, and whenever the pointer to any internal dependencies are updated). | ||
|
|
||
| ## Installing genesis data | ||
| Unpack the relevant set of genesis data and private baker data from [genesis-data/](/genesis-data) to the correct OS folder (e.g. on Linux this would be `$HOME/.local/share/concordium`). This determines how many bakers you need to run for the network to be able to work properly. | ||
|
|
||
| ## Running the library as a binary (usable via gRPC) | ||
| ```bash | ||
| $> cargo run -- --debug | ||
| ``` | ||
|
|
||
| ## Running all tests | ||
| ```bash | ||
| $> cargo test --all | ||
| ``` | ||
| The [build.rs](./build.rs) script links with a number of Haskell libraries and | ||
| supports a number of configuration options that can be configured in two | ||
| different ways, either via environment variables or Cargo features. | ||
|
|
||
| ## Obtaining documentation | ||
| The output is placed in `target/doc` by default. | ||
| ```bash | ||
| $> cargo doc | ||
| ``` | ||
| ### Features | ||
|
|
||
| ## Nix | ||
| Currently this project only sports support for Nix on Linux platforms. | ||
| ### Development | ||
| All `zsh` wrapper functions wrap around `nix-shell`, and if dropping into a `nix-shell` directly remember to use the cargo flag `--features=static` to build against the static libraries (`nix-shell` will automatically pull these down from S3). | ||
| ### Install binaries as a package | ||
| ```bash | ||
| $> scripts/download-static-libs.sh | ||
| $> nix-env -f . -i | ||
| ``` | ||
| The package supports the following features related to linking with the Haskell component. | ||
|
|
||
| ## Docker-Compose | ||
| ### Building docker images | ||
| To build the stable image built in a Jenkins pipeline (it gets tagged `latest`, if not changed in the line shown below, so it matches the image hosted on docker-hub - and as the layers will have a newer version, it won't download from docker-hub unless the locally built image is removed via e.g. `docker image rmi ..`). It passes the local `ssh-agent` into the docker build environment for the needed stages to download internal crates with git directly. This image builds on `192549843005.dkr.ecr.eu-west-1.amazonaws.com/concordium/base` so make sure to have either built this locally (check [devops:base-images/build-base.sh](https://gitlab.com/Concordium/devops/blob/master/base-images/build-base-docker.sh) for the syntax and current version), or have access to AWS ECR to pull it. | ||
| ```bash | ||
| $> git clone -b master --single-branch [email protected]:Concordium/tools/baker_id_gen.git baker_id_gen # Only needed once, as it's a vital component to scaling the bakers inside docker-compose | ||
| $> scripts/download-genesis-data.sh | ||
| $> scripts/download-genesis-complementary-bundle.sh | ||
| $> echo $(cd deps/internal/consensus && git rev-parse HEAD) > CONSENSUS_VERSION | ||
| $> DOCKER_BUILDKIT=1 docker build -f scripts/dev-client.Dockerfile -t concordium/dev-client:latest --ssh default . --no-cache | ||
| ``` | ||
| ### Latest stable from master branch | ||
| For a local docker compose setup, a docker-compose.yml file has been provided in the root of this repository. It uses a image hosted in Docker hub built automatically upon push to the master branch. | ||
| - `static`: This feature enables linking with static Haskell libraries. | ||
| They are expected to be available in `../deps/static-libs/linux/vanilla`. | ||
| The [../scripts/download-static-libs.sh](../scripts/download-static-libs.sh)` | ||
| script will download them into the correct location. | ||
|
|
||
| For the most simple and common setup, simply run the below command in the root of the checked out repository | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose up --scale baker=5 --force-recreate | ||
| ``` | ||
| - `profiling`: This will link with Haskell libraries built with profiling support. This option implies `static`, with the difference | ||
| that the libraries must be available in `../deps/static-libs/linux/profiling`. | ||
|
|
||
| By default none of these features are enabled. | ||
|
|
||
| For more complicated setups the EXTRA_ARGS environment variable can be set. | ||
| ### Environment variables | ||
|
|
||
| ### Latest unstable from develop branch | ||
| For a local docker compose setup, a docker-compose.develop.yml file has been provided in the root of this repository. It uses a image hosted in Docker hub built automatically upon push to the develop branch. | ||
| Environment variables only apply to the default build. This links with shared Haskell libraries. | ||
|
|
||
| For the most simple and common setup, simply run the below command in the root of the checked out repository | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.develop.yml up --scale baker=5 --force-recreate | ||
| ``` | ||
| - `CONCORDIUM_HASKELL_ROOT` should, if present, be a directory containing | ||
| - libHSconcordium-consensus-0.1.0.0.so | ||
| - libHSconcordium-base-0.1.0.0.so | ||
| - libHSlmdb-0.2.5.so | ||
|
|
||
| ### Latest debug from custom branch | ||
| For a local docker compose setup, a docker-compose.debug.yml file has been provided in the root of this repository. It uses a image hosted in Docker hub built manually from a specific branch. These builds must be considered extremely volatile! | ||
| This only applies to non-windows platforms. It is not used on other platforms. | ||
| On Windows the Concordium haskell package is built with a `standalone` option which embeds all dependent libraries into one single DLL. | ||
| This option does not work on linux, hence the more elaborate linking process. | ||
|
|
||
| For the most simple and common setup, simply run the below command in the root of the checked out repository | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.debug.yml up --scale baker=5 --force-recreate | ||
| ``` | ||
| If this flag is not present the build script will try to automatically discover all the Haskell libraries by using `stack path`. | ||
|
|
||
| ### Latest debug from custom branch (with smart contract support) | ||
| For a local docker compose setup, a docker-compose.debug-sc.yml file has been provided in the root of this repository. It uses a image hosted in Docker hub built manually from a specific branch. These builds must be considered extremely volatile! | ||
| - `HASKELL_RTS_VARIANT` applies to all platforms and is the prefix of the GHC runtime library that should be linked. | ||
| This defaults to `libHSrts_thr-` for the threaded version. | ||
|
|
||
| For the most simple and common setup, simply run the below command in the root of the checked out repository | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.debug-sc.yml up --scale baker=5 --force-recreate | ||
| ``` | ||
| - `HASKELL_GHC_LIBDIR` applies to all platforms and should be the location of | ||
| the GHC libraries directory (by GHC libraries we mean the base runtime system | ||
| libraries, base haskell libraries, and dependenencies). | ||
| If not given the script will try to discover the value by running `stack ghc -- --print-libdir`. | ||
|
|
||
| ### CAVEATS | ||
|
|
||
| - Note that [build.rs](./build.rs) **will not** automatically build the Haskell | ||
| dependencies, it will only try to discover them. Building should be done before | ||
| by running `stack build` inside | ||
| [../concordium-consensus/](../concordium-consensus/) or running | ||
|
|
||
| ```console | ||
| stack --stack-yaml ../concordium-consensus/stack.yaml build | ||
| ``` | ||
|
|
||
| Once the node is built it can be run as | ||
| ```console | ||
| cargo run -- | ||
| ``` | ||
| or | ||
| ```console | ||
| cargo run --release -- | ||
| ``` | ||
| to be run in release mode for improved performance. Note that the | ||
| [concordium-consensus](../concordium-consensus/) dependency is the same regardless of how the | ||
| node itself is built, the `--release` only applies to the optimization of the Rust node xcomponents. | ||
|
|
||
| For more complicated setups the EXTRA_ARGS environment variable can be set. | ||
| - The node built with Haskell library auto-discovery is not suitable for distribution to other | ||
| machines. It is a dynamically linked binary with a large number of shared library dependencies | ||
| that have | ||
|
|
||
| ## Middleware local development mode | ||
| The PostGreSQL instance is exposed on port 5432/tcp and the username is `concordium`, password: `concordium`, and database name is `concordium`. | ||
|
|
||
| ### Running the local development version from the stable master branch | ||
| Use docker-compose if you only need a middle-ware enabled set of nodes to test on | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.middleware.yml up --scale baker=5 --force-recreate | ||
| ``` | ||
|
|
||
| Remember to clean out PostGreSQL data between runs using | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.middleware.yml down | ||
| ``` | ||
|
|
||
| ### Running the local development version from the unstable develop branch (middleware) | ||
| Use docker-compose if you only need a middle-ware enabled set of nodes to test on | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.develop.middleware.yml up --scale baker=5 --force-recreate | ||
| ``` | ||
|
|
||
| Remember to clean out PostGreSQL data between runs using | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.develop.middleware.yml down | ||
| ``` | ||
|
|
||
| ### Delay baker startup if PostGreSQL starts too slowly | ||
| If PostGreSQL starts too slowly the baker enabled for logging to it can be delayed by using the variable `DB_SLEEP` | ||
|
|
||
|
|
||
| ## Wallet local development mode | ||
| The PostGreSQL instance is exposed on port 5432/tcp and the username is `concordium`, password: `concordium`, and database name is `concordium`. | ||
| The wallet-proxy is mapped on port 14000/tcp. | ||
|
|
||
| ### Running the local development version from the stable master branch | ||
| Use docker-compose if you only need a middle-ware enabled set of nodes to test on | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.wallet-dev.yml up --scale baker=5 --force-recreate | ||
| ``` | ||
| ## Installing genesis data | ||
| Unpack the relevant set of genesis data and private baker data from [genesis-data/](/genesis-data) to the correct OS folder (e.g. on Linux this would be `$HOME/.local/share/concordium`). This determines how many bakers you need to run for the network to be able to work properly. | ||
|
|
||
| Remember to clean out PostGreSQL data between runs using | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.wallet-dev.yml down | ||
| # Running all tests | ||
| ```console | ||
| $> cargo test --all | ||
| ``` | ||
|
|
||
| ### Running the local development version from the unstable develop branch | ||
| Use docker-compose if you only need a middle-ware enabled set of nodes to test on | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.develop.wallet-dev.yml up --scale baker=5 --force-recreate | ||
| # Obtaining documentation | ||
| The output is placed in `target/doc` by default. | ||
| ```console | ||
| $> cargo doc | ||
| ``` | ||
|
|
||
| Remember to clean out PostGreSQL data between runs using | ||
| ```bash | ||
| $> NUM_BAKERS=5 DESIRED_PEERS=4 docker-compose -f docker-compose.develop.wallet-dev.yml down | ||
| To automatically open the browser with the documentation once it is generated to | ||
| ```console | ||
| $> cargo doc --open | ||
| ``` | ||
|
|
||
| ### Delay baker startup if PostGreSQL starts too slowly | ||
| If PostGreSQL starts too slowly the baker enabled for logging to it can be delayed by using the variable `DB_SLEEP` (the wallet-proxy has a default value of 30 set to delay start until PostGreSQL is up). | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.