From 4807141ef1984d4f9406c87738b9cef94bd561cc Mon Sep 17 00:00:00 2001 From: Eyal Rundstein Date: Tue, 14 Mar 2023 14:26:44 +0200 Subject: [PATCH 01/12] README changes --- CONTRIBUTING.md | 101 ++++++++++++++---------------------------------- 1 file changed, 30 insertions(+), 71 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3e053b8eb7..8fae19a41e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,91 +1,50 @@ -# Contributing to concord-bft + -The concord-bft project team welcomes contributions from the community. If you wish to contribute code and you have not -signed our contributor license agreement (CLA), our bot will update the issue when you open a Pull Request. For any -questions about the CLA process, please refer to our [FAQ](https://cla.vmware.com/faq). +# Concord-BFT: A Distributed Trust Infrastructure -## Contribution Flow +[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) +[![clang-tidy](https://github.com/vmware/concord-bft/workflows/clang-tidy/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3Aclang-tidy) +[![Build Status](https://github.com/vmware/concord-bft/workflows/Release%20build%20(gcc)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Release+build+%28gcc%29") +[![Build Status](https://github.com/vmware/concord-bft/workflows/Debug%20build%20(gcc)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Debug+build+%28gcc%29") +[![Build Status](https://github.com/vmware/concord-bft/workflows/Release%20build%20(clang)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Release+build+%28clang%29") +[![Build Status](https://github.com/vmware/concord-bft/workflows/Debug%20build%20(clang)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Debug+build+%28clang%29") +[![codecoverage](https://github.com/vmware/concord-bft/actions/workflows/codecoverage.yml/badge.svg)](https://github.com/vmware/concord-bft/actions/workflows/codecoverage.yml) -This is a rough outline of what a contributor's workflow looks like: + -- Fork the repository -- Create a topic branch on *your fork*\* from where you want to base your work -- Make commits of logical units -- Make sure your commit messages are in the proper format (see below) -- Push your changes to a topic branch in your fork of the repository -- Submit a pull request + -\* Make sure that you create a branch on your fork and not the main repository. -If you push your changes together with your PR from a branch you created on the main repository, it would -trigger the CI twice (on push and on PR), resulting in unecessary load on Github Actions. +## Overview -Example: +**Concord-bft** is a generic state machine replication library that can handle malicious (byzantine) replicas. -``` shell -git remote add upstream https://github.com/vmware/concord-bft.git -git checkout -b my-new-feature master -git commit -a -git push origin my-new-feature -``` +BFT-based systems require substantial communication between nodes and, thus, don’t scale well. Project Concord-bft solves this problem by simplifying and streamlining communication between nodes, enabling greater scalability while increasing overall network throughput. -### Staying In Sync With Upstream +Project Concord’s BFT engine obtains significant scaling improvements via three major advances: -When your branch gets out of sync with the vmware/master branch, use the following to update: +1. It uses a linear communication consensus protocol while many other BFT consensus protocols (including PBFT) require quadratic communication +2. It exploits optimism to provide a common case fast-path execution +3. It uses modern cryptographic algorithms (BLS threshold signatures) -``` shell -git checkout my-new-feature -git fetch -a -git pull --rebase upstream master -git push --force-with-lease origin my-new-feature -``` +Its implementation is based on the algorithm described in the paper [SBFT: a Scalable Decentralized Trust Infrastructure for +Blockchains](https://arxiv.org/pdf/1804.01626.pdf). -### Updating pull requests +It is designed to be used as a core building block for replicated distributed data stores, and is especially suited to serve as the basis of permissioned Blockchain systems. -If your PR fails to pass CI or needs changes based on code review, you'll most likely want to squash these changes into -existing commits. +For a real-life integration example, please take a look at [Project Concord](https://github.com/vmware/concord), a highly scalable and energy-efficient distributed trust infrastructure for consensus and smart contract execution. -If your pull request contains a single commit or your changes are related to the most recent commit, you can simply -amend the commit. +Start with example usage here: https://github.com/concord-bft/examples/ -``` shell -git add . -git commit --amend -git push --force-with-lease origin my-new-feature -``` +## Documentation -If you need to squash changes into an earlier commit, you can use: +See the github [wiki](https://github.com/concord-bft/wiki/) for detailed explanation. -``` shell -git add . -git commit --fixup -git rebase -i --autosquash master -git push --force-with-lease origin my-new-feature -``` +## Community -Be sure to add a comment to the PR indicating your new changes are ready to review, as GitHub does not generate a -notification when you git push. +[Concord-BFT Slack](https://concordbft.slack.com/). -### Formatting Commit Messages +Request a Slack invitation via . -We follow the conventions on [How to Write a Git Commit Message](http://chris.beams.io/posts/git-commit/). +## License -Be sure to include any related GitHub issue references in the commit message. See -[GFM syntax](https://guides.github.com/features/mastering-markdown/#GitHub-flavored-markdown) for referencing issues -and commits. - -### Formatting Code - -We *mostly* follow the guidelines outlined by the -[Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). -If you use emacs, consider installing -[google-c-style.el](https://raw.githubusercontent.com/google/styleguide/gh-pages/google-c-style.el). - -Our code style has the following exceptions that differ from the google C++ style guidelines. - - * We use `camelCase` function and method names, rather than `PascalCase`. - * We allow, and encourage the use of exceptions. - * We allow and prefer `#pragma once` over include guards - -## Reporting Bugs and Creating Issues - -When opening a new issue, try to roughly follow the commit message format conventions above. +Concord-bft is available under the [Apache 2 license](LICENSE). From 52b975554c378439b26fecd602c80182c1fd9f26 Mon Sep 17 00:00:00 2001 From: Eyal Rundstein Date: Tue, 14 Mar 2023 16:55:57 +0200 Subject: [PATCH 02/12] update readme, revert contribution doc --- CONTRIBUTING.md | 101 ++++++++++++++------- README.md | 229 +++--------------------------------------------- 2 files changed, 84 insertions(+), 246 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8fae19a41e..8d1ccb66b3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,50 +1,91 @@ - +# Contributing to concord-bft -# Concord-BFT: A Distributed Trust Infrastructure +The concord-bft project team welcomes contributions from the community. If you wish to contribute code and you have not +signed our contributor license agreement (CLA), our bot will update the issue when you open a Pull Request. For any +questions about the CLA process, please refer to our [FAQ](https://cla.vmware.com/faq). -[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) -[![clang-tidy](https://github.com/vmware/concord-bft/workflows/clang-tidy/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3Aclang-tidy) -[![Build Status](https://github.com/vmware/concord-bft/workflows/Release%20build%20(gcc)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Release+build+%28gcc%29") -[![Build Status](https://github.com/vmware/concord-bft/workflows/Debug%20build%20(gcc)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Debug+build+%28gcc%29") -[![Build Status](https://github.com/vmware/concord-bft/workflows/Release%20build%20(clang)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Release+build+%28clang%29") -[![Build Status](https://github.com/vmware/concord-bft/workflows/Debug%20build%20(clang)/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3A"Debug+build+%28clang%29") -[![codecoverage](https://github.com/vmware/concord-bft/actions/workflows/codecoverage.yml/badge.svg)](https://github.com/vmware/concord-bft/actions/workflows/codecoverage.yml) +## Contribution Flow - +This is a rough outline of what a contributor's workflow looks like: - +- Fork the repository +- Create a topic branch on *your fork*\* from where you want to base your work +- Make commits of logical units +- Make sure your commit messages are in the proper format (see below) +- Push your changes to a topic branch in your fork of the repository +- Submit a pull request -## Overview +\* Make sure that you create a branch on your fork and not the main repository. +If you push your changes together with your PR from a branch you created on the main repository, it would +trigger the CI twice (on push and on PR), resulting in unecessary load on Github Actions. -**Concord-bft** is a generic state machine replication library that can handle malicious (byzantine) replicas. +Example: -BFT-based systems require substantial communication between nodes and, thus, don’t scale well. Project Concord-bft solves this problem by simplifying and streamlining communication between nodes, enabling greater scalability while increasing overall network throughput. +```shell +git remote add upstream https://github.com/vmware/concord-bft.git +git checkout -b my-new-feature master +git commit -a +git push origin my-new-feature +``` -Project Concord’s BFT engine obtains significant scaling improvements via three major advances: +### Staying In Sync With Upstream -1. It uses a linear communication consensus protocol while many other BFT consensus protocols (including PBFT) require quadratic communication -2. It exploits optimism to provide a common case fast-path execution -3. It uses modern cryptographic algorithms (BLS threshold signatures) +When your branch gets out of sync with the vmware/master branch, use the following to update: -Its implementation is based on the algorithm described in the paper [SBFT: a Scalable Decentralized Trust Infrastructure for -Blockchains](https://arxiv.org/pdf/1804.01626.pdf). +```shell +git checkout my-new-feature +git fetch -a +git pull --rebase upstream master +git push --force-with-lease origin my-new-feature +``` -It is designed to be used as a core building block for replicated distributed data stores, and is especially suited to serve as the basis of permissioned Blockchain systems. +### Updating pull requests -For a real-life integration example, please take a look at [Project Concord](https://github.com/vmware/concord), a highly scalable and energy-efficient distributed trust infrastructure for consensus and smart contract execution. +If your PR fails to pass CI or needs changes based on code review, you'll most likely want to squash these changes into +existing commits. -Start with example usage here: https://github.com/concord-bft/examples/ +If your pull request contains a single commit or your changes are related to the most recent commit, you can simply +amend the commit. -## Documentation +```shell +git add . +git commit --amend +git push --force-with-lease origin my-new-feature +``` -See the github [wiki](https://github.com/concord-bft/wiki/) for detailed explanation. +If you need to squash changes into an earlier commit, you can use: -## Community +```shell +git add . +git commit --fixup +git rebase -i --autosquash master +git push --force-with-lease origin my-new-feature +``` -[Concord-BFT Slack](https://concordbft.slack.com/). +Be sure to add a comment to the PR indicating your new changes are ready to review, as GitHub does not generate a +notification when you git push. -Request a Slack invitation via . +### Formatting Commit Messages -## License +We follow the conventions on [How to Write a Git Commit Message](http://chris.beams.io/posts/git-commit/). -Concord-bft is available under the [Apache 2 license](LICENSE). +Be sure to include any related GitHub issue references in the commit message. See +[GFM syntax](https://guides.github.com/features/mastering-markdown/#GitHub-flavored-markdown) for referencing issues +and commits. + +### Formatting Code + +We *mostly* follow the guidelines outlined by the +[Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html). +If you use emacs, consider installing +[google-c-style.el](https://raw.githubusercontent.com/google/styleguide/gh-pages/google-c-style.el). + +Our code style has the following exceptions that differ from the google C++ style guidelines. + +* We use `camelCase` function and method names, rather than `PascalCase`. +* We allow, and encourage the use of exceptions. +* We allow and prefer `#pragma once` over include guards + +## Reporting Bugs and Creating Issues + +When opening a new issue, try to roughly follow the commit message format conventions above. diff --git a/README.md b/README.md index e3a905adfc..8fae19a41e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ - -# Concord-BFT: a Distributed Trust Infrastructure +# Concord-BFT: A Distributed Trust Infrastructure [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![clang-tidy](https://github.com/vmware/concord-bft/workflows/clang-tidy/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3Aclang-tidy) @@ -15,11 +14,18 @@ - ## Overview **Concord-bft** is a generic state machine replication library that can handle malicious (byzantine) replicas. +BFT-based systems require substantial communication between nodes and, thus, don’t scale well. Project Concord-bft solves this problem by simplifying and streamlining communication between nodes, enabling greater scalability while increasing overall network throughput. + +Project Concord’s BFT engine obtains significant scaling improvements via three major advances: + +1. It uses a linear communication consensus protocol while many other BFT consensus protocols (including PBFT) require quadratic communication +2. It exploits optimism to provide a common case fast-path execution +3. It uses modern cryptographic algorithms (BLS threshold signatures) + Its implementation is based on the algorithm described in the paper [SBFT: a Scalable Decentralized Trust Infrastructure for Blockchains](https://arxiv.org/pdf/1804.01626.pdf). @@ -27,227 +33,18 @@ It is designed to be used as a core building block for replicated distributed da For a real-life integration example, please take a look at [Project Concord](https://github.com/vmware/concord), a highly scalable and energy-efficient distributed trust infrastructure for consensus and smart contract execution. +Start with example usage here: https://github.com/concord-bft/examples/ -## Releases - -We abide by [semantic versioning](https://semver.org/). Public APIs can change -at any time until we reach version 1.0. We will, however, bump the minor version -when a backwards incompatible change is made. - - [v0.5](https://github.com/vmware/concord-bft/releases/tag/v0.5) - -## Install and Build (Ubuntu Linux 18.04) - -Concord-BFT supports two kinds of builds: native and docker. - -The docker build is **strongly recommended**. - -### Docker - -* Install the latest docker. -* Optional: [configure docker as non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user). -* Build: -```sh -cd concord-bft -make -make test -``` -Run `make help` to see more commands. - -Note: -* The output binaries are stored in the host's `concord-bft/build`. -* `Makefile` is configurable. -For example, if you want to use another compiler you may pass it to the `make`: -``` -make CONCORD_BFT_CONTAINER_CXX=g++ \ - CONCORD_BFT_CONTAINER_CC=gcc \ - build -``` - -Other build options, including passthrough options for CMake, are defined in the Makefile and prefixed with `CONCORD_BFT_`. Variables that are capable of being overridden on the commandline are set with the Make conditional operator `?=` and are at the beginning of [Makefile](Makefile). Please check that file for options. - -#### Select comm module -One option that is worth calling out explicitly is the communication (transport) library. Transport defaults to TLS and can be configured explicitly by setting the `CONCORD_BFT_CMAKE_TRANSPORT` flag. The flag defaults to **TLS**, but also supports **UDP** and **TCP**. These can be useful because the use of pinned certificates for TLS requires an out of band setup. - -See [create_tls_certs.sh](scripts/linux/create_tls_certs.sh) for an example. This script is used in apollo tests. For production usage, an out of band deployment for each replica must be used to avoid revealing private keys to each replica. - -### Native - -```sh -git clone https://github.com/vmware/concord-bft -cd concord-bft -sudo ./install_deps_release.sh # Installs all dependencies and 3rd parties -mkdir build -cd build -cmake .. -make -sudo make test -``` - -By default the build chooses the active compiler on the native build platform. In order to force compilation by clang you can use the following command. -```sh -CC=clang CXX=clang++ cmake .. -``` - -In order to turn on or off various options, you need to change your cmake configuration. This is -done by passing arguments to cmake with a `-D` prefix: e.g. `cmake -DBUILD_TESTING=OFF`. Note that -make must be run afterwards to build according to the configuration. Please see [CMakeLists.txt](CMakeLists.txt) for configurable options. - -#### Select comm module -One option that is worth calling out explicitly is the communication (transport) library. - -We support both UDP and TCP communication. UDP is the default. In order to -enable TCP communication, build with `-DBUILD_COMM_TCP_PLAIN=TRUE` in the cmake -instructions shown above. If set, the test client will run using TCP. If you -wish to use TCP in your application, you need to build the TCP module as -mentioned above and then create the communication object using CommFactory and -passing PlainTcpConfig object to it. - -We also support TCP over TLS communication. To enable it, change the -`BUILD_COMM_TCP_TLS` flag to `TRUE` in the main CMakeLists.txt file. When -running simpleTest using the testReplicasAndClient.sh - there is no need to create TLS certificates manually. The script will use the `create_tls_certs.sh` (located under the scripts/linux folder) to create certificates. The latter can be used to create TLS files for any number of replicas, e.g. when extending existing tests. - -As we used pinned certificates for TLS, the user will have to manually provide these. THey can use the [create_tls_certs.sh](scripts/linux/create_tls_certs.sh) script as an example. - - -### C++ Linter - -The C++ code is statically checked by `clang-tidy` as part of the [CI](https://github.com/vmware/concord-bft/actions?query=workflow%3Aclang-tidy). -
To check code before submitting PR, please run `make tidy-check`. - -[Detailed information about clang-tidy checks](https://clang.llvm.org/extra/clang-tidy/checks/list.html). - - -#### (Optional) Python client - -The python client is required for running tests. If you do not want to install python, you can -configure the build of concord-bft by running `cmake -DBUILD_TESTING=OFF ..` from the `build` -directory for native builds, and `CONCORD_BFT_CMAKE_BUILD_TESTING=TRUE make` for docker builds. - -The python client requires python3(>= 3.5) and trio, which is installed via pip. +## Documentation - python3 -m pip install --upgrade trio - - -#### Adding a new dependency or tool - -The CI builds and runs tests in a docker container. To add a new dependency or tool, follow the steps below: - -* Rebase against master -* In order to add/remove dependencies update the file - [install_deps_release.sh](https://github.com/vmware/concord-bft/blob/master/install_deps_release.sh) -* Build new release/debug images: `make build-docker-images` -* Check images current version in - [Makefile](https://github.com/vmware/concord-bft/blob/master/Makefile#L5) - and - [Makefile](https://github.com/vmware/concord-bft/blob/master/Makefile#L10) -* Tag the new images (at least one): - * For release: `docker tag concord-bft:latest concordbft/concord-bft:` - * For debug: `docker tag concord-bft-debug:latest concordbft/concord-bft-debug:` -
where version is `current version + 1`. -* Update the version in the Makefile -* Make sure that Concord-BFT is built and tests pass with the new images: `RUN_WITH_DEBUG_IMAGE=TRUE make - clean build test` -* Ask one of the maintainers for a temporary write permission to Docker Hub - repository(you need to have a [Docker ID](https://docs.docker.com/docker-id/)) -* Push the images (at least one): - * For release: `docker push concordbft/concord-bft:` - * For debug: `docker push concordbft/concord-bft-debug:` -* Create a PR for the update: - * The PR must contain only changes related to the updates in the image - * PR's summary has to be similar to `Docker update to version release= debug=` - * PR's message has to list the changes made in the image content and - preferably the reason - * Submit the PR - -Important notes: -1. Adding dependencies or tools directly to the `Dockerfile` is strongly not recommended because it breaks the native build support. -2. If any tools are installed during the build but not needed for the actual compilation/debugging/test execution(for example, `git`), please remove them(`Dockerfile` is an example). The reason is that the image is supposed to be as tiny as possible. - - -## Apollo testing framework - - -The Apollo framework provides utilities and advanced testing scenarios for validating -Concord BFT's correctness properties, regardless of the running application/execution engine. -For the purposes of system testing, we have implemented a "Simple Key-Value Blockchain" (SKVBC) -test application which runs on top of the Concord BFT consensus engine. -
- -Apollo enables running all test suites (without modification) against any supported BFT network -configuration (in terms of n, f, c and other parameters). -
- -Various crash or byzantine failure scenarios are also covered -(including faulty replicas and/or network partitioning). -
- -Apollo test suites run regularly as part of Concord BFT's continuous integration pipeline. - -Please find more details about the Apollo framework [here](tests/apollo/README.md) - -## Run examples - - -### Simple test application (4 replicas and 1 client on a single machine) - -Tests are compiled into in the build directory and can be run from anywhere as -long as they aren't moved. - -Run the following from the top level concord-bft directory: - - ./build/tests/simpleTest/scripts/testReplicasAndClient.sh - -### Using simple test application via Python script - -You can use the simpleTest.py script to run various configurations via a simple -command line interface. -Please find more information [here](./tests/simpleTest/README.md) - -### Example application -This example demo application shows some capabilities of the Concord-BFT consensus-based byzantine fault-tolerant state machine replication library. -For Concord-BFT users who are interested in learning more about Concord-BFT and its uses, this application offers a demonstration and instruction. -Overall, any blockchain application based on concord-bft consensus may be created using this example application. -
- -Use the [test_example.sh](example/scripts/test_example.sh) script to run the example application. This script is also used to perform this demo via the command line interface with different configurations. -
- -Please see [here](example/README.md) for more information about the example/demo application. - -## Directory Structure - - -- [bftengine](./bftengine): concord-bft codebase - - [include](./bftengine/include): external interfaces of concord-bft (to be used by client applications) - - [src](./bftengine/src): internal implementation of concord-bft - - [tests](./bftengine/tests): tests and usage examples -- [threshsign](./threshsign): crypto library that supports digital threshold signatures - - [include](./threshsign/include): external interfaces of threshsign (to be used by client applications) - - [src](./threshsign/src): internal implementation of threshsign - - [tests](./threshsign/tests): tests and usage examples -- [scripts](./scripts): build scripts -- [tests](./tests): BFT engine system tests - -## Contributing - - -The concord-bft project team welcomes contributions from the community. If you wish to contribute code and you have not -signed our contributor license agreement (CLA), our bot will update the issue when you open a Pull Request. For any -questions about the CLA process, please refer to our [FAQ](https://cla.vmware.com/faq). For more detailed information, -refer to [CONTRIBUTING.md](CONTRIBUTING.md). - -## Notes -The library calls `std::terminate()` when it cannot continue in a safe manner. -In that way, users can install a handler that does something different than just calling `std::abort()`. +See the github [wiki](https://github.com/concord-bft/wiki/) for detailed explanation. ## Community - [Concord-BFT Slack](https://concordbft.slack.com/). Request a Slack invitation via . ## License -concord-bft is available under the [Apache 2 license](LICENSE). +Concord-bft is available under the [Apache 2 license](LICENSE). From 3341634c0165577c3f278d02c9a8d0840681b0cd Mon Sep 17 00:00:00 2001 From: eyalrund <92918350+eyalrund@users.noreply.github.com> Date: Wed, 15 Mar 2023 17:25:16 +0200 Subject: [PATCH 03/12] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8fae19a41e..947ca76f55 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ It is designed to be used as a core building block for replicated distributed da For a real-life integration example, please take a look at [Project Concord](https://github.com/vmware/concord), a highly scalable and energy-efficient distributed trust infrastructure for consensus and smart contract execution. -Start with example usage here: https://github.com/concord-bft/examples/ +Start with example usage here: https://github.com/vmware/concord-bft/tree/master/examples ## Documentation From 4882d0a84324cc596ce7b10de99474c57ac514c7 Mon Sep 17 00:00:00 2001 From: eyalrund <92918350+eyalrund@users.noreply.github.com> Date: Wed, 15 Mar 2023 17:26:07 +0200 Subject: [PATCH 04/12] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 947ca76f55..e3ccf6a836 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ Start with example usage here: https://github.com/vmware/concord-bft/tree/master ## Documentation -See the github [wiki](https://github.com/concord-bft/wiki/) for detailed explanation. +See the github [wiki](https://github.com/eyalrund/concord-bft/wiki) for detailed explanation. ## Community From b103a17917244b7ccac21beb761db991d22bdf61 Mon Sep 17 00:00:00 2001 From: eyalrund <92918350+eyalrund@users.noreply.github.com> Date: Wed, 15 Mar 2023 17:51:10 +0200 Subject: [PATCH 05/12] Update design.md --- docs/design.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/design.md b/docs/design.md index 05e0ac0970..0a1783e843 100644 --- a/docs/design.md +++ b/docs/design.md @@ -22,6 +22,7 @@ explanatory. Important structures will be mentioned explicitly below. All user interfaces live in the `bftEngine` namespace. # Clients + A default client, [SimpleClient](../bftengine/include/bftengine/SimpleClient.hpp), is included with concord-bft. It is just one manner of implementing a client, and is @@ -32,6 +33,7 @@ Python](../util/pyclient/bft_client.py) that we use for testing that does just that. ## Client Messages + Client request and reply messages are prefixed with a `ClientRequestMsgHeader` or `ClientReplyMsgHeader` respectively. Each header, like all other messages in concord-bft is implemented as a packed C struct, meaning that there are no padding @@ -111,7 +113,7 @@ committed. This is the hook that allows arbitrary SMR implemented in concord-bft to result in execution of application specific operations at all replicas in a total order. -# ReplicaConfig +## ReplicaConfig ReplicaConfig contains most configurable attributes of concord-bft and should be created by the application and passed into `Replica::createReplica(...)`. Note that other configurable attributes are set as part of the communication From 83bc622c174789e8a0e52cf796fee058c921f1e3 Mon Sep 17 00:00:00 2001 From: eyalrund <92918350+eyalrund@users.noreply.github.com> Date: Thu, 16 Mar 2023 09:06:55 +0200 Subject: [PATCH 06/12] Update README.md --- examples/README.md | 56 +++++----------------------------------------- 1 file changed, 6 insertions(+), 50 deletions(-) diff --git a/examples/README.md b/examples/README.md index a11f2e4c4b..988f129f75 100644 --- a/examples/README.md +++ b/examples/README.md @@ -8,67 +8,23 @@ This example/demo will provide end to end functionality of Concord byzantine fau **Note:** Initially, we plan to do this using an open source key-value blockchain replica. However, once the setup is finished, we'll also use alternative open source execution engines, such Ethereum, WASM, etc. -## Install and Build (Ubuntu Linux 18.04 or above) -Concord-BFT supports two kinds of builds are, -* Docker -* Native - -The docker build is **strongly recommended** - - -### Docker - -* [Install the latest docker] (https://docs.docker.com/engine/install/ubuntu/). -* Optional: [configure docker as non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user). -* Build concord-bft -```sh -git clone https://github.com/vmware/concord-bft -cd concord-bft -make -``` -Run `make help` to see more commands. - -Note: -* If you face any format related issues than run `make format` before running `make` command to format the changes. -* The output binaries are stored in the host's `concord-bft/build`. -* Default compiler is clang and clang++. -* `Makefile` is configurable. If you want to use another compiler you may pass it to the `make`. -For example, -``` -make CONCORD_BFT_CONTAINER_CXX=g++ \ - CONCORD_BFT_CONTAINER_CC=gcc \ - build -``` -Other build options, including passthrough options for CMake, are defined in the [Makefile](../Makefile) and prefixed with `CONCORD_BFT_`. - - -### Native - -For native development environment, need to install some dependencies mentioned in [install_deps_release.sh](../install_deps_release.sh) script. -```sh -git clone https://github.com/vmware/concord-bft -cd concord-bft -sudo ./install_deps_release.sh # Install all dependencies and 3rd parties -mkdir build -cd build -cmake .. -make -``` +## Install and Build +See [Getting Started](https://github.com/eyalrund/concord-bft/wiki/Getting-Started) for instructions. ## Run Example/Demo -This simple example can be run from the script [test_osexample.sh](scripts/test_osexample.sh) under `example/script` directory. +This simple example can be run from the script [test_osexample.sh](scripts/test_osexample.sh) under `examples/script` directory. Or, it can be directly run from python script [test_osexample.py](scripts/test_osexample.py.in). ### Simple scenario (4 replicas and 1 client) This demo can be run from inside the container once the build is finished. -* Run the [test_osexample.sh](scripts/test_osexample.sh) script from the container's `build/example/scripts` directory. +* Run the [test_osexample.sh](scripts/test_osexample.sh) script from the container's `build/examples/scripts` directory. ```sh ./test_osexample.sh ``` -* Run the [test_osexample.py](scripts/test_osexample.py.in) script from the container's `build/example/scripts` directory with default commandline arguments `-bft n=4,cl=1`, where n is number of replicas and cl is number of client. +* Run the [test_osexample.py](scripts/test_osexample.py.in) script from the container's `build/examples/scripts` directory with default commandline arguments `-bft n=4,cl=1`, where n is number of replicas and cl is number of client. It is a configurable script which can be configured according to the need. ```sh ./test_osexample.py -bft n=4,cl=1 @@ -110,4 +66,4 @@ In future we are planning to include open source execution engines for this demo ## License -This example/demo part of concord-bft is available under the [Apache 2 license](../LICENSE). \ No newline at end of file +This example/demo part of concord-bft is available under the [Apache 2 license](../LICENSE). From 55badd64a2d5d87fcd949fa71889c1d8d33f8a25 Mon Sep 17 00:00:00 2001 From: eyalrund <92918350+eyalrund@users.noreply.github.com> Date: Thu, 16 Mar 2023 09:09:50 +0200 Subject: [PATCH 07/12] Update README.md --- examples/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/examples/README.md b/examples/README.md index 988f129f75..f904bf71e3 100644 --- a/examples/README.md +++ b/examples/README.md @@ -32,12 +32,12 @@ It is a configurable script which can be configured according to the need. #### Explanation of the [test_osexample.sh](scripts/test_osexample.sh) -On our example, we will use the script [test_osexample.py](scripts/test_osexample.py.in) and this script is used for following things, +In our example, we will use the script [test_osexample.py](scripts/test_osexample.py.in) and this script is used for the following things, * It is used to generate the keys for replica's. For more [refer](../tools/README.md) -* It is used to generate TLS certificates used in for TLSTCP communication which we are using as a default mode of communication. For more [refer](../scripts/linux/create_tls_certs.sh) -* Running multiple replica's. Here we are running 4 replica's. +* It is used to generate TLS certificates used for TLS/TCP communication which we are using as a default mode of communication. For more info [refer](../scripts/linux/create_tls_certs.sh) +* Running multiple replica's. Here we are running 4 replica's * Running client -* For resources cleanup. +* For resources cleanup ## Directory Structure. @@ -47,10 +47,10 @@ On our example, we will use the script [test_osexample.py](scripts/test_osexampl - [msg-configs](./msg-configs): Message configurations for read/write messages. - [scripts](./scripts): Build scripts, replica config file, etc. -## How to use msg configs +## How to use message configs Initially, we are planning to do this using an open source key-value blockchain replica. Message configs are used to set some parameters which will further used to create ClientRequestMsg. -All the message configurations can be found [msg-configs](./msg-configs) folder. +All messages configurations can be found [msg-configs](./msg-configs) folder. Following are the parameters from msg config file, * **type**: This is a message type parameters i.e. it shows the type of message Write, Read and Both. According to this parameter client request msg will be created. @@ -62,7 +62,7 @@ Following are the parameters from msg config file, ## Future Plans -In future we are planning to include open source execution engines for this demo, such as Ethereum, WASM, etc. +In future we plan to include open source execution engines for this demo, such as Ethereum, WASM, etc. ## License From 55b5787967f8a46f1d97bf7e3a08a0e01cd3d281 Mon Sep 17 00:00:00 2001 From: eyalrund <92918350+eyalrund@users.noreply.github.com> Date: Thu, 16 Mar 2023 09:24:57 +0200 Subject: [PATCH 08/12] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e3ccf6a836..01cd80f5ce 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ It is designed to be used as a core building block for replicated distributed da For a real-life integration example, please take a look at [Project Concord](https://github.com/vmware/concord), a highly scalable and energy-efficient distributed trust infrastructure for consensus and smart contract execution. -Start with example usage here: https://github.com/vmware/concord-bft/tree/master/examples +Start with example usage [here](https://github.com/vmware/concord-bft/tree/master/examples). ## Documentation From 879ca369807c5ac1296af0bb4846467a525c2c2c Mon Sep 17 00:00:00 2001 From: erundstein Date: Wed, 22 Mar 2023 07:09:03 +0000 Subject: [PATCH 09/12] fix wiki link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 01cd80f5ce..0b734d16d5 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,7 @@ Start with example usage [here](https://github.com/vmware/concord-bft/tree/maste ## Documentation -See the github [wiki](https://github.com/eyalrund/concord-bft/wiki) for detailed explanation. +See the github [wiki](https://github.com/vmware/concord-bft/wiki) for detailed explanation. ## Community From 92a4dbbbc999249d7107c492706c89122d5ad5bf Mon Sep 17 00:00:00 2001 From: erundstein Date: Wed, 22 Mar 2023 09:34:34 +0000 Subject: [PATCH 10/12] remove ref to archived concord project --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index 0b734d16d5..a315195236 100644 --- a/README.md +++ b/README.md @@ -31,8 +31,6 @@ Blockchains](https://arxiv.org/pdf/1804.01626.pdf). It is designed to be used as a core building block for replicated distributed data stores, and is especially suited to serve as the basis of permissioned Blockchain systems. -For a real-life integration example, please take a look at [Project Concord](https://github.com/vmware/concord), a highly scalable and energy-efficient distributed trust infrastructure for consensus and smart contract execution. - Start with example usage [here](https://github.com/vmware/concord-bft/tree/master/examples). ## Documentation From 05b669bed371405a1b4abdcbb2cd2e552b7e4fdb Mon Sep 17 00:00:00 2001 From: erundstein Date: Wed, 22 Mar 2023 10:00:12 +0000 Subject: [PATCH 11/12] minor changes --- README.md | 2 +- examples/README.md | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index a315195236..8d6c94d1b5 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# Concord-BFT: A Distributed Trust Infrastructure +# Concord-BFT: A Distributed Trust Framework [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![clang-tidy](https://github.com/vmware/concord-bft/workflows/clang-tidy/badge.svg)](https://github.com/vmware/concord-bft/actions?query=workflow%3Aclang-tidy) diff --git a/examples/README.md b/examples/README.md index f904bf71e3..4e08aba58b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,15 +1,15 @@ # Overview -We are providing an open-source illustration and demonstration of example/demo. This will help us to understand the usage of the concord-bft library. Since concord-bft is an open-source consensus, this application will also assist us in comprehending the internals of concord-bft. +We provide an open-source illustration and demonstration of example/demo. This will help demonstrating the usage of the concord-bft library. Since concord-bft is an open-source consensus, this application also assists in comprehending the internals of concord-bft. -It is responsible for assisting and guiding large number of concord-bft users around the globe, including those who want to understand more about concord-bft, its usage, and how it works. +It is designed to assist and guide large number of concord-bft users around the globe, including those who want to understand more about concord-bft, its usage, and how it works. -This example/demo will provide end to end functionality of Concord byzantine fault tolerant state machine replication library about its usage. This can be used as a library to build a distributed trust infrastructure that is extremely scalable and low-power for smart contract execution. +This example/demo demonstrates end to end functionality of Concord byzantine fault tolerant state machine replication library. It can be used as a library to build a distributed trust infrastructure that is extremely scalable and low-power for smart contract execution. -**Note:** Initially, we plan to do this using an open source key-value blockchain replica. However, once the setup is finished, we'll also use alternative open source execution engines, such Ethereum, WASM, etc. +**Note:** Initially, we plan to do this using an open source key-value blockchain replica. However, once the setup is finished, we'll also use alternative open source execution engines, such as EVM, WASM, etc. ## Install and Build -See [Getting Started](https://github.com/eyalrund/concord-bft/wiki/Getting-Started) for instructions. +See [Getting Started](https://github.com/vmware/concord-bft/wiki/Getting-Started) for instructions. ## Run Example/Demo @@ -32,7 +32,7 @@ It is a configurable script which can be configured according to the need. #### Explanation of the [test_osexample.sh](scripts/test_osexample.sh) -In our example, we will use the script [test_osexample.py](scripts/test_osexample.py.in) and this script is used for the following things, +In our example, we use the script [test_osexample.py](scripts/test_osexample.py.in) and this script is used to achieve the following: * It is used to generate the keys for replica's. For more [refer](../tools/README.md) * It is used to generate TLS certificates used for TLS/TCP communication which we are using as a default mode of communication. For more info [refer](../scripts/linux/create_tls_certs.sh) * Running multiple replica's. Here we are running 4 replica's @@ -48,7 +48,7 @@ In our example, we will use the script [test_osexample.py](scripts/test_osexampl - [scripts](./scripts): Build scripts, replica config file, etc. ## How to use message configs -Initially, we are planning to do this using an open source key-value blockchain replica. +Initially, we plan to do this using an open source key-value blockchain replica. Message configs are used to set some parameters which will further used to create ClientRequestMsg. All messages configurations can be found [msg-configs](./msg-configs) folder. @@ -62,7 +62,7 @@ Following are the parameters from msg config file, ## Future Plans -In future we plan to include open source execution engines for this demo, such as Ethereum, WASM, etc. +In future we plan to include open source execution engines for this demo, such as EVM, WASM, etc. ## License From ebaa9de49baf489f64dcfb00d56b1bcf9ff6fbe1 Mon Sep 17 00:00:00 2001 From: erundstein Date: Sun, 2 Apr 2023 15:55:21 +0000 Subject: [PATCH 12/12] test --- bftengine/src/bftengine/ReplicaImp.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/bftengine/src/bftengine/ReplicaImp.cpp b/bftengine/src/bftengine/ReplicaImp.cpp index 0d14f44213..ee9ac2febb 100644 --- a/bftengine/src/bftengine/ReplicaImp.cpp +++ b/bftengine/src/bftengine/ReplicaImp.cpp @@ -198,12 +198,14 @@ void ReplicaImp::messageHandler(std::unique_ptr msg) { return; } } + if constexpr (std::is_same_v) { if (validateMessage(trueTypeObj.get())) { onMessage(std::move(trueTypeObj)); } return; } + if (!isCollectingState()) { // The message validation of few messages require time-consuming processes like // digest calculation, signature verification etc. Such messages are identified