This document provides a high level tour of the venus implementation of the Venus protocols in Go.
This document assumes a reasonable level of knowledge about the Venus system and protocols, which are not re-explained here. It is complemented by specs (link forthcoming) that describe the key concepts implemented here.
Table of contents
- Background
- Architecture overview
- A tour of the code
- Networking
- Filesystem storage
- Testing
- Dependencies
- Patterns
The venus implementations is the result of combined research and development effort. The protocol spec and architecture evolved from a prototype, and is the result of iterating towards our goals. We are still working on clarifying the architecture and propagating good patterns throughout the code. Please bear with us, and we’d love your help.
Venus borrows a lot from the IPFS project, including some patterns, tooling, and packages. Some benefits of this include:
- the projects encode data in the same way (IPLD, CIDs), easing interoperability;
- the venus project can build on solid libraries like the IPFS commands.
Other patterns, we've evolving for our needs:
- go-ipfs relies heavily on shell-based integration testing; we aim to rely heavily on unit testing and Go-based integration tests.
- The go-ipfs package structure involves a deep hierarchy of dependent implementations; we're moving towards a more Go-idiomatic approach with narrow interfaces defined in consuming packages (see Patterns.
- The term "block" is heavily overloaded: a blockchain block (
types/block.go), but also content-id-addressed blocks in the block service. Blockchain blocks are stored in block service blocks, but are not the same thing.
| |\/
|_|/\
╔══════════════════════════════════════════╗ ╔══════════════════════════════╗
║ ║ ║ ║
║ ║ ║ ║
║ NETWORK (gossipsub, bitswap, etc.) ║ ║ COMMANDS / REST API ║
Network ║ ║ ║ ║
║ ║ ║ ║
╚═════════════════════╦════════════════════╝ ╚══════════════════════════════╝
│ │
┌───────────────┼───────────────┐ ┌──────────────────────┤
│ │ │ │ │
▼ ▼ ▼ ▼ │
┌──────────────┬─────────────────┬─────────────┐ │
┌───────▶│ Storage API │ Retrieval API │ Block API │ │
│ ├──────────────┼─────────────────┼─────────────┤ │
│ │ │ │ │ │
│ │ │ │ │──┐ │
│ │ Storage │ Retrieval │ Block │ │ │
Internal │ │ Protocol │ Protocol │ Protocol │ │ │
API │ └──────────────┴────────┬────────┴─────────────┘ │ │
│ │ │ │ │ │
│ ▼ ▼ ▼ │ ▼
│ ┌───────────────────────────────────────────┐ │ ┌─────────────┬──────────────┐
│ │ │ │ │
│ │ Core API │ │ │ Client API
└─────────▶│ │ └─▶
│ │ │
└───────────────────────────────────────────┘ └────────────────────────────┘
│ │
┌─────────────────┬────┴──────────────┬────────────────┬───────────┴─────┐
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
Core ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ │ │ │ │ │ │ │ │ │
│ Message │ │ Chain Store │ │ Processor │ │ Block │ │ Wallet │
│ Pool │ │ │ │ │ │ Service │ │ │
│ │ │ │ │ │ │ │ │ │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
The Node (node/) object is the "server".
It contains much of the core protocol implementation and plumbing.
The api package contains the API of all the
core building blocks upon which the protocols are implemented.
The implementation of this API is the Node.
At the bottom of the architecture diagram are core services. These are focused implementations of some functionality that don’t achieve much on their own, but are the means to the end. Core services are the bottom level building blocks out of which application functionality can be built.
They are the source of truth in all data.
Core services are mostly found in top-level packages. Most are reasonably well factored and testable in isolation.
Services include (not exhaustive):
- Message pool: hold messages that haven’t been mined into a block yet.
- Chain store: stores & indexes blockchain blocks.
- Chain syncer: syncs chain blocks from the rest of the network.
- Processor: Defines how transaction messages drive state transitions.
- Block service: content-addressed key value store that stores IPLD data, including blockchain blocks as well as the state tree (it’s poorly named).
- Wallet: manages keys.
The venus binary can run in two different modes, either as a long-running daemon exposing a JSON/HTTP RPC API,
or as a command-line interface which interprets and routes requests as RPCs to a daemon.
In typical usage, you start the daemon with venus daemon then use the same binary to issue commands like venus wallet ls,
which are transmitted to the daemon over the HTTP API.
The commands package uses the go-ipfs command library and defines commands as both CLI and JSON entry points.
Commands implement user- and tool-facing functionality. Command implementations should be very, very small. With no logic of their own, they should call just into a single plumbing or porcelain method (never into core APIs directly). The go-ipfs command library introduces some boilerplate which we can reduce with some effort in the future. Right now, some of the command implementations call into the node; this should change.
Tests for commands are generally end-to-end "daemon tests" that exercise CLI. They start some nodes and interact with them through shell commands.
More detail on the individual protocols is coming soon.
Actors are Filecoin’s notion of smart contracts. They are not true smart contracts—with bytecode running on a VM—but instead implemented in Go. It is expected that other implementations will match the behaviour of the Go actors exactly. An ABI describes how inputs and outputs to the VM are encoded. Future work will replace this implementation with a "real" VM.
The Actor struct is the base implementation of actors, with fields common to all of them.
Codeis a CID identifying the actor code, but since these actors are implemented in Go, is actually some fixed bytes acting as an identifier. This identifier selects the kind of actor implementation when a message is sent to its address.Headis the CID of this actor instance’s state.Nonceis a counter of #messages received from that actor. It is only set for account actors (the actors from which external parties send messages); messages between actors in the VM don’t use the nonce.Balanceis FIL balance the actor controls.
Some actors are singletons (e.g. the storage market) but others have multiple instances (e.g. storage miners). A storage miner actor exists for each miner in the Filesystem network. Their structs share the same code CID so they have the same behavior, but have distinct head state CIDs and balance. Each actor instance exists at an address in the state tree. An address is the hash of the actor’s public key.
Actors declare a list of exported methods with ABI types. Method implementations typically load the state tree, perform some query or mutation, then return a value or an error.
Blockchain state is represented in the state tree, which contains the state of all actors. The state tree is a map of address to (encoded) actor structs. The state tree interface exposes getting and setting actors at addresses, and iterating actors. The underlying data structure is a Hash array-mapped trie. A HAMT is also often used to store actor state, eg when the actor wants to store a large map.
The canonical binary encoding used by Venus is CBOR. In Go, structs are CBOR-encoded by reflection. The ABI uses a separate inner encoding, which is manual.
Venus state transitions are driven by messages sent to actors; these are our "transactions". A message is a method invocation on an actor. A message has sender and recipient addresses, and optional parameters such as an amount of filecoin to transfer, a method name, and parameters.
Messages from the same actor go on chain in nonce order. Note that the nonce is only really used by account actors (representing external entities such as humans). The nonce guards against replay of messages entering the VM for execution, but is not used for messages between actors during an external message’s execution.
Driving a state transition means invoking an actor method. One invokes a method on an actor by sending it a message. To send a message the message is created, signed, added to your local node’s message pool broadcast on the network to other nodes, which will add it to their message pool too. Some node will then mine a block and possibly include your message. In Venus, it is essential to remember that sending the message does not mean it has gone on chain or that its outcome has been reflected in the state tree. Sending means the message is available to be mined into a block. You must wait for the message to be included in a block to see its effect.
Read-only methods, or query messages, are the mechanism by which actor state can be inspected.
These messages are executed locally against a read only version of the state tree of the head of the chain.
They never leave the node, they are not broadcast.
The plumbing API exposes MessageSend and MessageQuery for these two cases.
The processor is the entry point for making and validating state transitions represented by the messages. It is modelled Ethereum’s message processing system. The processor manages the application of messages to the state tree from the prior block/s. It loads the actor from which a message came, check signatures, then loads the actor and state to which a message is addressed and passes the message to the VM for execution.
The vm package has the low level detail of calling actor methods. A VM context defines the world visible from an actor implementation while executing.
Venus uses a consensus algorithm called expected consensus. Unlike proof-of-work schemes, expected-consensus is a proof-of-stake model, where probability of mining a block in each round (30 seconds) is proportional to amount of storage a miner has committed to the network. Each round, miners are elected through a probabilistic but private mechanism akin to rolling independent, private, but verifiable dice. The expected number of winners in each round is one, but it could be zero or more than one miner. If a miner is elected, they have the right to mine a block in that round.
Given the probabilistic nature of mining new blocks, more than one block may be mined in any given round. Hence, a new block might have more than one parent block. The parents form a set, which we call a tipset. All the blocks in a tipset are at the same height and share the same parents. Tipsets contain one or more blocks. A null block count indicates the absence of any blocks mined in a previous round. Subsequent blocks are built upon all of the tipset; there is a canonical ordering of the messages in a tipset defining a new consensus state, not directly referenced from any of the tipset’s blocks.
There’s no centrally dispatched event loop. The node starts up all the components, connects them as needed, and waits. Protocols (goroutines) communicate through custom channels. This architecture needs more thought, but we are considering moving more inter-module communication to use iterators (c.f. those in Java). An event bus might also be a good pattern for some cases, though.
The Rust code responsible for sectors and proofs is in the rust-fil-proofs repo.
This repo is included in venus as a Git submodule.
The submodule refers to a specific repository SHA hash.
The install-rust-proofs.sh script, invoked by the deps build step of venus, builds the Rust proofs code and copies binary assets to locations hardcoded in Go interface code.
As an alternative to compiling Rust code locally, the continuous integration server publishes an archive of precompiled binary objects with every successful build of the rust-fil-proofs/master branch.
These releases are identified by the Git submodule SHA. This archive is pushed to the GitHub releases service.
When the FILECOIN_USE_PRECOMPILED_RUST_PROOFS environment variable is set, install-rust-proofs.sh attempts to fetch these assets from GitHub releases
and installs them in the same hardcoded locations required by the Go build.
The build or tarball contains:
libsector_builder_ffi.a(a static library)sector_builder_ffi.h(corresponding C-header file)sector_builder_ffi.pc(a pkg-config manifest, used to specify linker dependencies)paramcache(populates Groth parameter-cacheparamfetch(fetches Groth parameters from IPFS gateway into cache)parameters.json(contains the most recently-published Groth parameters)
The proving algorithms rely on a large binary parameter file known as the Groth parameters.
This file is stored in a cache directory, typically /var/tmp/filecoin-proof-parameters.
When proofs code changes, the params may need to change.
The paramcache program populates the Groth parameter directory by generating the parameters, a slow process (10s of minutes).
If the cache directory is already populated, generation is skipped.
The parampublish program (not part of the precompiled proofs archive) publishes params from the cache directory to a local IPFS node.
The CIDs of the parameter files thus published must be pinned (made continuously available) e.g. by a Protocol Labs pinning service.
The paramfetch program fetches params to local cache directory from IPFS gateway.
The install-rust-proofs.sh script fetches or generates these Groth parameters as necessary when building deps.
Groth parameters in /var/tmp/filecoin-proof-parameters are accessed at venus runtime.
The parameters are identified by the parameters.json file from fil-rust-proofs, which includes a checksum.
For ease of development, venus can be configured to use a test proofs mode, which will cause storage miners to use sectors into which only 1016 bytes of user data can be written. This lowers the computational burden of sealing and generating PoSts.
The genesis.car in fixtures/test/ is configured to use test proofs mode.
Venus relies on libp2p for its networking needs.
libp2p is a modular networking stack for the peer-to-peer era. It offers building blocks to tackle requirements such as peer discovery, transport switching, multiplexing, content routing, NAT traversal, pubsub, circuit relay, etc., most of which Venus uses. Developers can compose these blocks easily to build the networking layer behind their P2P system.
A detailed overview of how Venus uses libp2p can be found in the Networking doc.
The repo, aka fsrepo, is a directory stored on disk containing all necessary information to run a venus daemon, typically at $HOME/.venus.
The repo does not include client data stored by storage miners, which is held instead in the sector base.
The repo does include a JSON config file with preferences on how the daemon should operate,
several key value datastores holding data important to the internal services,
and the keystore which holds private key data for encryption.
The JSON config file is stored at $HOME/.venus/config.json, and can be easily edited using the venus config command.
Users can also edit the file directly at their own peril.
The key-value datastores in the repo include persisted data from a variety of systems within venus.
Most of them hold CBOR encoded data keyed on CID, however this varies.
The key value stores include the badger, chain, deals, and wallet directories under $HOME/.venus.
The purpose of these directories is:
- Badger is a general purpose datastore currently only holding the genesis key, but in the future, almost all our datastores should be merged into this one.
- Chain is where the local copy of the blockchain is stored.
- Wallet is where the user’s Venus wallet information is stored.
The keystore contains the binary encoded peer key for interacting securely over the network.
This data lives in a file at $HOME/.venus/keystore/self.
The venus codebase has a few different testing mechanisms:
unit tests, in-process integration tests, "daemon" integration tests, and a couple of high level functional tests.
Many parts of code have good unit tests. We’d like all parts to have unit tests, but in some places it hasn’t been possible where prototype code omitted testability features.
Historically there has been a prevalence of integration-type testing. Relying only on integration tests can make it hard to verify small changes to internals. We’re driving towards both wide unit test coverage, with integration tests to verifying end-to-end behaviour.
Code generally uses simple manual dependency injection. A component that takes a large number of deps at construction can have them factored into a struct.
Some node integration tests start one or more full nodes in-process. This is useful for fine-grained control over the node being tested. Setup for these tests is a bit difficult and we aim to make it much easier to instantiate and test full nodes in-process.
Daemon tests are end-to-end integration tests that exercise the command interface of the venus binary.
These execute separate venus processes and drive them via the command line.
These tests are mostly under the commands package,
The functional-tests directory contains some Go and Bash scripts which perform complicated multi-node tests on our continuous build.
These are not daemon tests, but run separately.
Some packages have a testing.go file with helpers for setting up tests involving that package’s types.
There is also a top-level testhelpers package with higher level helpers, often used by daemon tests.
We’re in process of creating the venus Automation and Systems Toolkit (FAST) library.
The goal of this is to unify duplicated code paths which bootstrap and drive venus daemons for daemon tests, functional tests,
and network deployment verification, providing a common library for filecoin automation in Go.
Tests are typically run with go run ./build/*.go test.
It passes flags to go test under the hood, so you can provide -run <regex> to run a subset of tests.
By default it will run unit and integration tests, but skip more expensive functional and sectorbuilder tests.
For a complete description of testing flags see Test Categorization.
Vanilla go test also works, after build scripts have built and installed appropriate dependencies.
By default unit tests are enabled when issuing the go test command.
To disable pass -unit=false.
A unit test exercises one or a few code modules exhaustively. Unit tests do not involve complex integrations or non-trivial communication, disk access, or artificial delays. A unit test should complete in well under a second, frequently <10 milliseconds. Unit tests should have no side effects and be executable in parallel
By default integration tests are enabled when issuing the go test command.
To disable pass -integration=false.
An integration test exercises integrated functionality and/or multiple nodes and may include multiple processes, inter-node communication, delays for consensus, nontrivial disk access and other expensive operations. Integration tests may involve multiple processes (daemon tests) or in-process integrations. An individual integration test should complete in seconds.
By default functional tests are disabled when issuing the go test command.
To enable pass -functional.
A functional test is an extensive multi-node orchestration or resource-intensive test that may take minutes to run.
Dependencies in venus are managed as go modules, go's new dependency system.
If you've cloned venus into your GOPATH, you may need to set the GO111MODULES environment variable to on. The build system automatically sets this but your favorite editor or IDE may not work without it.
The project makes heavy use of or is moving towards a few key design patterns, explained here.
venus uses Opencensus-go for stats collection and distributed tracing instrumentation. Stats are exported for consumption via Prometheus.
venus can be configured to collect and export metrics to Prometheus via the MetricsConfig.
The details of this can be found inside the config/ package.
To view metrics from your filecoin node using the default configuration options set the prometheusEnabled value to true, start the filecoin daemon, then visit localhost:9400/metrics in your web-browser.
venus can be configured to collect and export traces to Jaeger via the TraceConfig.
The details of this can be found inside the config/ package.
To collect traces from your venus node using the default configuration options set the jaegerTracingEnabled value to true, start the venus daemon, then follow the Jaeger Getting started guide.