From c7edf61c8a8f8b1b7441c1dd157b634c99e2318d Mon Sep 17 00:00:00 2001 From: "mergify[bot]" <37929162+mergify[bot]@users.noreply.github.com> Date: Tue, 14 Mar 2023 11:20:52 -0300 Subject: [PATCH] Reorganize existing guides (backport #492) (#528) * Reorganize existing guides (#492) * Moves all guides to a guids folder * Moves the actual intro to the README.md of the intro folder * Add guides to the guide README.md * Fixing links in docs/README.md (cherry picked from commit fa98b5ece71f65fc25dfb804de907e13bb7d55c2) # Conflicts: # docs/README.md # docs/guides/java.md # docs/guides/kotlin.md # docs/introduction/README.md # docs/introduction/what-is-cometbft.md * solving conflicts * Add guides to the guide README.md --------- Co-authored-by: Lasaro --- docs/README.md | 7 +- docs/guides/README.md | 15 + docs/{tutorials => guides}/go-built-in.md | 0 docs/{tutorials => guides}/go.md | 0 docs/{introduction => guides}/install.md | 0 docs/{tutorials => guides}/java.md | 6 +- docs/{tutorials => guides}/kotlin.md | 6 +- docs/{introduction => guides}/quick-start.md | 4 +- .../upgrading-from-tm.md | 0 docs/introduction/README.md | 327 +++++++++++++++++- docs/tutorials/README.md | 10 - 11 files changed, 342 insertions(+), 33 deletions(-) create mode 100644 docs/guides/README.md rename docs/{tutorials => guides}/go-built-in.md (100%) rename docs/{tutorials => guides}/go.md (100%) rename docs/{introduction => guides}/install.md (100%) rename docs/{tutorials => guides}/java.md (99%) rename docs/{tutorials => guides}/kotlin.md (99%) rename docs/{introduction => guides}/quick-start.md (97%) rename docs/{introduction => guides}/upgrading-from-tm.md (100%) delete mode 100644 docs/tutorials/README.md diff --git a/docs/README.md b/docs/README.md index 66d7d8bb4a..cc36e176f2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,13 +16,12 @@ CometBFT serves blockchain applications. More formally, CometBFT performs Byzantine Fault Tolerant (BFT) State Machine Replication (SMR) for arbitrary deterministic, finite state machines. -For more background, see [What is CometBFT?](introduction/what-is-cometbft.md). +For more background, see [What is CometBFT?](introduction/README.md#what-is-cometbft). -To get started quickly with an example application, see the -[quick start guide](introduction/quick-start.md). +To get started quickly with an example application, see the [quick start guide](guides/quick-start.md). To upgrade from Tendermint Core v0.34.x to CometBFT v0.34.x, please see our -[upgrading instructions](./introduction/upgrading-from-tm.md). +[upgrading instructions](./guides/upgrading-from-tm.md). To learn about application development on CometBFT, see the [Application Blockchain Interface](https://github.com/cometbft/cometbft/tree/v0.34.x/spec/abci). diff --git a/docs/guides/README.md b/docs/guides/README.md new file mode 100644 index 0000000000..7c8b0a9e5b --- /dev/null +++ b/docs/guides/README.md @@ -0,0 +1,15 @@ +--- +order: false +parent: + order: 2 +--- + +# Guides + +- [Installing CometBFT](./install.md) +- [Quick-start using CometBFT](./quick-start.md) +- [Upgrading from Tendermint to CometBFT](./upgrading-from-tm.md) +- [Creating a built-in application in Go](./go-built-in.md) +- [Creating an external application in Go](./go.md) +- [Creating an external application in Java](./java.md) +- [Creating an external application in Kotlin](./kotlin.md) diff --git a/docs/tutorials/go-built-in.md b/docs/guides/go-built-in.md similarity index 100% rename from docs/tutorials/go-built-in.md rename to docs/guides/go-built-in.md diff --git a/docs/tutorials/go.md b/docs/guides/go.md similarity index 100% rename from docs/tutorials/go.md rename to docs/guides/go.md diff --git a/docs/introduction/install.md b/docs/guides/install.md similarity index 100% rename from docs/introduction/install.md rename to docs/guides/install.md diff --git a/docs/tutorials/java.md b/docs/guides/java.md similarity index 99% rename from docs/tutorials/java.md rename to docs/guides/java.md index 526d3a2c68..ed5dfa30a4 100644 --- a/docs/tutorials/java.md +++ b/docs/guides/java.md @@ -1,6 +1,6 @@ - +--- +order: 6 +--- # Creating an application in Java diff --git a/docs/tutorials/kotlin.md b/docs/guides/kotlin.md similarity index 99% rename from docs/tutorials/kotlin.md rename to docs/guides/kotlin.md index c311e9d71e..25f3b6bfca 100644 --- a/docs/tutorials/kotlin.md +++ b/docs/guides/kotlin.md @@ -1,6 +1,6 @@ - +--- +order: 7 +--- # Creating an application in Kotlin diff --git a/docs/introduction/quick-start.md b/docs/guides/quick-start.md similarity index 97% rename from docs/introduction/quick-start.md rename to docs/guides/quick-start.md index 06d2df0aee..21dcdc91d6 100644 --- a/docs/introduction/quick-start.md +++ b/docs/guides/quick-start.md @@ -11,7 +11,7 @@ works and want to get started right away, continue. ## Install -See the [install instructions](./install.md). +See the [install guide](./install.md). ## Initialization @@ -95,7 +95,7 @@ First create four Ubuntu cloud machines. The following was tested on Digital Ocean Ubuntu 16.04 x64 (3GB/1CPU, 20GB SSD). We'll refer to their respective IP addresses below as IP1, IP2, IP3, IP4. -Then, `ssh` into each machine and install CometBFT following the [instructions](./install.md). +Then, `ssh` into each machine and install CometBFT following the [guide](./install.md). Next, use the `cometbft testnet` command to create four directories of config files (found in `./mytestnet`) and copy each directory to the relevant machine in the cloud, so that each machine has `$HOME/mytestnet/node[0-3]` directory. diff --git a/docs/introduction/upgrading-from-tm.md b/docs/guides/upgrading-from-tm.md similarity index 100% rename from docs/introduction/upgrading-from-tm.md rename to docs/guides/upgrading-from-tm.md diff --git a/docs/introduction/README.md b/docs/introduction/README.md index 504c17b700..acafe992f5 100644 --- a/docs/introduction/README.md +++ b/docs/introduction/README.md @@ -5,22 +5,327 @@ parent: order: 1 --- -# Overview +# What is CometBFT -## Quick Start +CometBFT is software for securely and consistently replicating an +application on many machines. By securely, we mean that CometBFT works +as long as less than 1/3 of machines fail in arbitrary ways. By consistently, +we mean that every non-faulty machine sees the same transaction log and +computes the same state. Secure and consistent replication is a +fundamental problem in distributed systems; it plays a critical role in +the fault tolerance of a broad range of applications, from currencies, +to elections, to infrastructure orchestration, and beyond. -Get CometBFT up and running quickly with the -[quick-start guide](./quick-start.md)! +The ability to tolerate machines failing in arbitrary ways, including +becoming malicious, is known as Byzantine fault tolerance (BFT). The +theory of BFT is decades old, but software implementations have only +became popular recently, due largely to the success of "blockchain +technology" like Bitcoin and Ethereum. Blockchain technology is just a +reformalization of BFT in a more modern setting, with emphasis on +peer-to-peer networking and cryptographic authentication. The name +derives from the way transactions are batched in blocks, where each +block contains a cryptographic hash of the previous one, forming a +chain. -## Install +CometBFT consists of two chief technical components: a blockchain +consensus engine and a generic application interface. +The consensus engine, +which is based on [Tendermint consensus algorithm][tendermint-paper], +ensures that the same transactions are +recorded on every machine in the same order. The application interface, +called the Application BlockChain Interface (ABCI), delivers the transactions +to applications for processing. Unlike other +blockchain and consensus solutions, which come pre-packaged with built +in state machines (like a fancy key-value store, or a quirky scripting +language), developers can use CometBFT for BFT state machine +replication of applications written in whatever programming language and +development environment is right for them. -Detailed [installation instructions](./install.md). +CometBFT is designed to be easy-to-use, simple-to-understand, highly +performant, and useful for a wide variety of distributed applications. -## Upgrading from Tendermint Core +## CometBFT vs. X -See our [upgrading guidelines](./upgrading-from-tm.md) if you are interested in -switching to CometBFT from Tendermint Core. +CometBFT is broadly similar to two classes of software. The first +class consists of distributed key-value stores, like Zookeeper, etcd, +and consul, which use non-BFT consensus. The second class is known as +"blockchain technology", and consists of both cryptocurrencies like +Bitcoin and Ethereum, and alternative distributed ledger designs like +Hyperledger's Burrow. -## What is CometBFT +### Zookeeper, etcd, consul -Dive into [what CometBFT is and why](./what-is-cometbft.md)! +Zookeeper, etcd, and consul are all implementations of key-value stores +atop a classical, non-BFT consensus algorithm. Zookeeper uses an +algorithm called Zookeeper Atomic Broadcast, while etcd and consul use +the Raft log replication algorithm. A +typical cluster contains 3-5 machines, and can tolerate crash failures +in less than 1/2 of the machines (e.g., 1 out of 3 or 2 out of 5), +but even a single Byzantine fault can jeopardize the whole system. + +Each offering provides a slightly different implementation of a +featureful key-value store, but all are generally focused around +providing basic services to distributed systems, such as dynamic +configuration, service discovery, locking, leader-election, and so on. + +CometBFT is in essence similar software, but with two key differences: + +- It is Byzantine Fault Tolerant, meaning it can only tolerate less than 1/3 + of machines failing, but those failures can include arbitrary behavior - + including hacking and malicious attacks. +- It does not specify a + particular application, like a fancy key-value store. Instead, it + focuses on arbitrary state machine replication, so developers can build + the application logic that's right for them, from key-value store to + cryptocurrency to e-voting platform and beyond. + +### Bitcoin, Ethereum, etc + +[Tendermint consensus algorithm][tendermint-paper], adopted by CometBFT, +emerged in the tradition of cryptocurrencies like Bitcoin, +Ethereum, etc. with the goal of providing a more efficient and secure +consensus algorithm than Bitcoin's Proof of Work. In the early days, +Tendermint consensus-based blockchains had a simple currency built in, and to participate in +consensus, users had to "bond" units of the currency into a security +deposit which could be revoked if they misbehaved -this is what made +Tendermint consensus a Proof-of-Stake algorithm. + +Since then, CometBFT has evolved to be a general purpose blockchain +consensus engine that can host arbitrary application states. That means +it can be used as a plug-and-play replacement for the consensus engines +of other blockchain software. So one can take the current Ethereum code +base, whether in Rust, or Go, or Haskell, and run it as an ABCI +application using CometBFT. Indeed, [we did that with +Ethereum](https://github.com/cosmos/ethermint). And we plan to do +the same for Bitcoin, ZCash, and various other deterministic +applications as well. + +Another example of a cryptocurrency application built on CometBFT is +[the Cosmos network](http://cosmos.network). + +### Other Blockchain Projects + +[Fabric](https://github.com/hyperledger/fabric) takes a similar approach +to CometBFT, but is more opinionated about how the state is managed, +and requires that all application behavior runs in potentially many +docker containers, modules it calls "chaincode". It uses an +implementation of [PBFT](http://pmg.csail.mit.edu/papers/osdi99.pdf). +from a team at IBM that is [augmented to handle potentially +non-deterministic +chaincode](https://drops.dagstuhl.de/opus/volltexte/2017/7093/pdf/LIPIcs-OPODIS-2016-24.pdf). +It is possible to implement this docker-based behavior as an ABCI app in +CometBFT, though extending CometBFT to handle non-determinism +remains for future work. + +[Burrow](https://github.com/hyperledger/burrow) is an implementation of +the Ethereum Virtual Machine and Ethereum transaction mechanics, with +additional features for a name-registry, permissions, and native +contracts, and an alternative blockchain API. It uses CometBFT as its +consensus engine, and provides a particular application state. + +## ABCI Overview + +The [Application BlockChain Interface +(ABCI)](https://github.com/cometbft/cometbft/tree/main/abci) +allows for Byzantine Fault Tolerant replication of applications +written in any programming language. + +### Motivation + +Thus far, all blockchains "stacks" (such as +[Bitcoin](https://github.com/bitcoin/bitcoin)) have had a monolithic +design. That is, each blockchain stack is a single program that handles +all the concerns of a decentralized ledger; this includes P2P +connectivity, the "mempool" broadcasting of transactions, consensus on +the most recent block, account balances, Turing-complete contracts, +user-level permissions, etc. + +Using a monolithic architecture is typically bad practice in computer +science. It makes it difficult to reuse components of the code, and +attempts to do so result in complex maintenance procedures for forks of +the codebase. This is especially true when the codebase is not modular +in design and suffers from "spaghetti code". + +Another problem with monolithic design is that it limits you to the +language of the blockchain stack (or vice versa). In the case of +Ethereum which supports a Turing-complete bytecode virtual-machine, it +limits you to languages that compile down to that bytecode; while the +[list](https://github.com/pirapira/awesome-ethereum-virtual-machine#programming-languages-that-compile-into-evm) +is growing, it is still very limited. + +In contrast, our approach is to decouple the consensus engine and P2P +layers from the details of the state of the particular +blockchain application. We do this by abstracting away the details of +the application to an interface, which is implemented as a socket +protocol. + +### Intro to ABCI + +[CometBFT](https://github.com/cometbft/cometbft), the +"consensus engine", communicates with the application via a socket +protocol that satisfies the ABCI, the CometBFT Socket Protocol. + +To draw an analogy, let's talk about a well-known cryptocurrency, +Bitcoin. Bitcoin is a cryptocurrency blockchain where each node +maintains a fully audited Unspent Transaction Output (UTXO) database. If +one wanted to create a Bitcoin-like system on top of ABCI, CometBFT +would be responsible for + +- Sharing blocks and transactions between nodes +- Establishing a canonical/immutable order of transactions + (the blockchain) + +The application will be responsible for + +- Maintaining the UTXO database +- Validating cryptographic signatures of transactions +- Preventing transactions from spending non-existent transactions +- Allowing clients to query the UTXO database. + +CometBFT is able to decompose the blockchain design by offering a very +simple API (i.e. the ABCI) between the application process and consensus +process. + +The ABCI consists of 3 primary message types that get delivered from the +core to the application. The application replies with corresponding +response messages. + +The messages are specified here: [ABCI Message +Types](https://github.com/cometbft/cometbft/blob/main/proto/tendermint/abci/types.proto). + +The **DeliverTx** message is the work horse of the application. Each +transaction in the blockchain is delivered with this message. The +application needs to validate each transaction received with the +**DeliverTx** message against the current state, application protocol, +and the cryptographic credentials of the transaction. A validated +transaction then needs to update the application state — by binding a +value into a key values store, or by updating the UTXO database, for +instance. + +The **CheckTx** message is similar to **DeliverTx**, but it's only for +validating transactions. CometBFT's mempool first checks the +validity of a transaction with **CheckTx**, and only relays valid +transactions to its peers. For instance, an application may check an +incrementing sequence number in the transaction and return an error upon +**CheckTx** if the sequence number is old. Alternatively, they might use +a capabilities based system that requires capabilities to be renewed +with every transaction. + +The **Commit** message is used to compute a cryptographic commitment to +the current application state, to be placed into the next block header. +This has some handy properties. Inconsistencies in updating that state +will now appear as blockchain forks which catches a whole class of +programming errors. This also simplifies the development of secure +lightweight clients, as Merkle-hash proofs can be verified by checking +against the block hash, and that the block hash is signed by a quorum. + +There can be multiple ABCI socket connections to an application. +CometBFT creates three ABCI connections to the application; one +for the validation of transactions when broadcasting in the mempool, one +for the consensus engine to run block proposals, and one more for +querying the application state. + +It's probably evident that applications designers need to very carefully +design their message handlers to create a blockchain that does anything +useful but this architecture provides a place to start. The diagram +below illustrates the flow of messages via ABCI. + +![abci](../imgs/abci.png) + +## A Note on Determinism + +The logic for blockchain transaction processing must be deterministic. +If the application logic weren't deterministic, consensus would not be +reached among the CometBFT replica nodes. + +Solidity on Ethereum is a great language of choice for blockchain +applications because, among other reasons, it is a completely +deterministic programming language. However, it's also possible to +create deterministic applications using existing popular languages like +Java, C++, Python, or Go, by avoiding +sources of non-determinism such as: + +- random number generators (without deterministic seeding) +- race conditions on threads (or avoiding threads altogether) +- the system clock +- uninitialized memory (in unsafe programming languages like C + or C++) +- [floating point + arithmetic](http://gafferongames.com/networking-for-game-programmers/floating-point-determinism/) +- language features that are random (e.g. map iteration in Go) + +While programmers can avoid non-determinism by being careful, it is also +possible to create a special linter or static analyzer for each language +to check for determinism. In the future we may work with partners to +create such tools. + +## Consensus Overview + +CometBFT adopts [Tendermint consensus][tendermint-paper], +an easy-to-understand, mostly asynchronous, BFT consensus algorithm. +The algorithm follows a simple state machine that looks like this: + +![consensus-logic](../imgs/consensus_logic.png) + +Participants in the algorithm are called **validators**; they take turns +proposing blocks of transactions and voting on them. Blocks are +committed in a chain, with one block at each **height**. A block may +fail to be committed, in which case the algorithm moves to the next +**round**, and a new validator gets to propose a block for that height. +Two stages of voting are required to successfully commit a block; we +call them **pre-vote** and **pre-commit**. + +There is a picture of a couple doing the polka because validators are +doing something like a polka dance. When more than two-thirds of the +validators pre-vote for the same block, we call that a **polka**. Every +pre-commit must be justified by a polka in the same round. +A block is committed when +more than 2/3 of validators pre-commit for the same block in the same +round. + +Validators may fail to commit a block for a number of reasons; the +current proposer may be offline, or the network may be slow. Tendermint consensus +allows them to establish that a validator should be skipped. Validators +wait a small amount of time to receive a complete proposal block from +the proposer before voting to move to the next round. This reliance on a +timeout is what makes Tendermint consensus a weakly synchronous algorithm, rather +than an asynchronous one. However, the rest of the algorithm is +asynchronous, and validators only make progress after hearing from more +than two-thirds of the validator set. A simplifying element of +Tendermint consensus is that it uses the same mechanism to commit a block as it +does to skip to the next round. + +Assuming less than one-third of the validators are Byzantine, Tendermint consensus algorithm +guarantees that safety will never be violated - that is, validators will +never commit conflicting blocks at the same height. To do this it +introduces a few **locking** rules which modulate which paths can be +followed in the flow diagram. Once a validator precommits a block, it is +locked on that block. Then, + +1. it must prevote for the block it is locked on +2. it can only unlock, and precommit for a new block, if there is a + polka for that block in a later round + +## Stake + +In many systems, not all validators will have the same "weight" in the +consensus protocol. Thus, we are not so much interested in one-third or +two-thirds of the validators, but in those proportions of the total +voting power, which may not be uniformly distributed across individual +validators. + +Since CometBFT can replicate arbitrary applications, it is possible to +define a currency, and denominate the voting power in that currency. +When voting power is denominated in a native currency, the system is +often referred to as Proof-of-Stake. Validators can be forced, by logic +in the application, to "bond" their currency holdings in a security +deposit that can be destroyed if they're found to misbehave in the +consensus protocol. This adds an economic element to the security of the +protocol, allowing one to quantify the cost of violating the assumption +that less than one-third of voting power is Byzantine. + +The [Cosmos Network](https://cosmos.network) is designed to use this +Proof-of-Stake mechanism across an array of cryptocurrencies implemented +as ABCI applications. + +[tendermint-paper]: https://arxiv.org/abs/1807.04938 diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md deleted file mode 100644 index 8a7fda4ca1..0000000000 --- a/docs/tutorials/README.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -order: false -parent: - order: 2 ---- - -# Guides - -- [Creating a built-in application in Go](./go-built-in.md) -- [Creating an external application in Go](./go.md)