-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
docs: update short guide of dev setup #3285
Changes from all commits
fb1b0ad
56d519d
2bb53aa
008164e
87a9f2f
c3da4f2
eec0d1a
249477e
603f31f
04f5d9a
457c4f2
8ea5d1c
51efffd
a440e86
2435467
e7042e6
1f36e79
c34cb22
e4ce183
26609c6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,68 +1,21 @@ | ||
# Installing dependencies | ||
# Installing development setup for ZKsync | ||
|
||
## TL;DR | ||
|
||
This is a shorter version of setup guide to make it easier subsequent initializations. If it's the first time you're | ||
This is a shorter version of setup guide to make it easier for initializations. If it's the first time you're | ||
initializing the workspace, it's recommended that you read the whole guide below, as it provides more context and tips. | ||
|
||
If you run on 'clean' Ubuntu on GCP: | ||
Just run the following command on Debian-based Linux: | ||
|
||
```bash | ||
# For VMs only! They don't have SSH keys, so we override SSH with HTTPS | ||
git config --global url."https://github.com/".insteadOf [email protected]: | ||
git config --global url."https://".insteadOf git:// | ||
|
||
Comment on lines
-11
to
-13
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These lines are important for VMs, because submodule fetching may break if running on a fresh machine without valid ssh keys; however they're missing in the script. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The standard now really is to declare submodules with https, like we do. I did follow the tree of submodules down and it seems all of them use https. |
||
# Rust | ||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh | ||
|
||
# NVM | ||
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash | ||
|
||
# All necessary stuff | ||
sudo apt-get update | ||
sudo apt-get install -y build-essential pkg-config cmake clang lldb lld libssl-dev libpq-dev apt-transport-https ca-certificates curl software-properties-common | ||
|
||
# Install docker | ||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - | ||
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" | ||
sudo apt install docker-ce | ||
sudo usermod -aG docker ${USER} | ||
|
||
# Start docker. | ||
sudo systemctl start docker | ||
|
||
## You might need to re-connect (due to usermod change). | ||
|
||
# Node & yarn | ||
nvm install 20 | ||
# Important: there will be a note in the output to load | ||
# new paths in your local session, either run it or reload the terminal. | ||
npm install -g yarn | ||
yarn set version 1.22.19 | ||
|
||
# For running unit tests | ||
cargo install cargo-nextest | ||
# SQL tools | ||
cargo install sqlx-cli --version 0.8.1 | ||
|
||
# Foundry ZKsync | ||
curl -L https://raw.githubusercontent.com/matter-labs/foundry-zksync/main/install-foundry-zksync | bash | ||
foundryup-zksync | ||
|
||
# Non CUDA (GPU) setup, can be skipped if the machine has a CUDA installed for provers | ||
# Don't do that if you intend to run provers on your machine. Check the prover docs for a setup instead. | ||
echo "export ZKSYNC_USE_CUDA_STUBS=true" >> ~/.bashrc | ||
# You will need to reload your `*rc` file here | ||
|
||
# Clone the repo to the desired location | ||
git clone [email protected]:matter-labs/zksync-era.git | ||
cd zksync-era | ||
git submodule update --init --recursive | ||
curl -L https://raw.githubusercontent.com/matter-labs/zksync-era/main/docs/src/guides/setup-dev.sh | bash | ||
``` | ||
|
||
Don't forget to look at [tips](#tips). | ||
|
||
## Supported operating systems | ||
## Installation guide | ||
|
||
### Supported operating systems | ||
|
||
ZKsync currently can be launched on any \*nix operating system (e.g. any linux distribution or macOS). | ||
|
||
|
@@ -77,84 +30,23 @@ Rosetta may cause problems that are hard to spot and debug, so make sure to chec | |
|
||
If you are a NixOS user or would like to have a reproducible environment, skip to the section about `nix`. | ||
|
||
## Docker | ||
|
||
Install `docker`. It is recommended to follow the instructions from the | ||
[official site](https://docs.docker.com/install/). | ||
|
||
Note: currently official site proposes using Docker Desktop for Linux, which is a GUI tool with plenty of quirks. If you | ||
want to only have CLI tool, you need the `docker-ce` package and you can follow | ||
[this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04) for Ubuntu. | ||
|
||
Installing `docker` via `snap` or from the default repository can cause troubles. | ||
|
||
You need to install both `docker` and `docker compose`. | ||
|
||
**Note:** `docker compose` is installed automatically with `Docker Desktop`. | ||
|
||
**Note:** On linux you may encounter the following error when you’ll try to work with `zksync`: | ||
|
||
``` | ||
ERROR: Couldn't connect to Docker daemon - you might need to run `docker-machine start default`. | ||
``` | ||
|
||
If so, you **do not need** to install `docker-machine`. Most probably, it means that your user is not added to | ||
the`docker` group. You can check it as follows: | ||
|
||
```bash | ||
docker-compose up # Should raise the same error. | ||
sudo docker-compose up # Should start doing things. | ||
``` | ||
|
||
If the first command fails, but the second succeeds, then you need to add your user to the `docker` group: | ||
### Basic packages | ||
|
||
```bash | ||
sudo usermod -a -G docker your_user_name | ||
``` | ||
|
||
After that, you should logout and login again (user groups are refreshed after the login). The problem should be solved | ||
at this step. | ||
|
||
If logging out does not resolve the issue, restarting the computer should. | ||
|
||
## Node.js & Yarn | ||
|
||
1. Install `Node` (requires version `v20`). The recommended way is via [nvm](https://github.com/nvm-sh/nvm). | ||
2. Install `yarn`. Can be done via `npm install -g yarn`. Make sure to get version 1.22.19 - you can change the version | ||
by running `yarn set version 1.22.19`. | ||
|
||
## clang | ||
|
||
In order to compile RocksDB, you must have LLVM available. On debian-based linux it can be installed as follows: | ||
Install the basic packages: | ||
|
||
On debian-based linux: | ||
|
||
```bash | ||
sudo apt-get install build-essential pkg-config cmake clang lldb lld | ||
sudo apt install git build-essential pkg-config cmake clang lldb lld libssl-dev libpq-dev apt-transport-https ca-certificates software-properties-common | ||
``` | ||
|
||
On macOS: | ||
|
||
You need to have an up-to-date `Xcode`. You can install it directly from `App Store`. With Xcode command line tools, you | ||
get the Clang compiler installed by default. Thus, having XCode you don't need to install `clang`. | ||
|
||
## OpenSSL | ||
|
||
Install OpenSSL: | ||
|
||
On mac: | ||
|
||
```bash | ||
brew install openssl | ||
``` | ||
|
||
On debian-based linux: | ||
|
||
```bash | ||
sudo apt-get install libssl-dev | ||
``` | ||
|
||
## Rust | ||
### Rust | ||
|
||
Install `Rust`'s toolchain version reported in `/rust-toolchain.toml` (also a later stable version should work). | ||
|
||
|
@@ -190,62 +82,85 @@ If you see `x86_64` mentioned in the output, probably you're running (or used to | |
that's the case, you should probably change the way you run terminal, and/or reinstall your IDE, and then reinstall the | ||
Rust toolchain as well. | ||
|
||
## PostgreSQL Client Library | ||
#### Cargo nextest | ||
|
||
For development purposes, you typically only need the PostgreSQL client library, not the full server installation. | ||
Here's how to install it: | ||
|
||
On macOS: | ||
[cargo-nextest](https://nexte.st/) is the next-generation test runner for Rust projects. `zkstack dev test rust` uses | ||
`cargo nextest` by default. | ||
|
||
```bash | ||
brew install libpq | ||
cargo install cargo-nextest --locked | ||
``` | ||
|
||
On Debian-based Linux: | ||
#### SQLx CLI | ||
|
||
SQLx is a Rust library we use to interact with Postgres, and its CLI is used to manage DB migrations and support several | ||
features of the library. | ||
|
||
```bash | ||
sudo apt-get install libpq-dev | ||
cargo install sqlx-cli --version 0.8.1 | ||
``` | ||
|
||
### Cargo nextest | ||
### Docker | ||
|
||
[cargo-nextest](https://nexte.st/) is the next-generation test runner for Rust projects. `zkstack dev test rust` uses | ||
`cargo nextest` by default. | ||
Install `docker`. It is recommended to follow the instructions from the | ||
[official site](https://docs.docker.com/install/). | ||
|
||
```bash | ||
cargo install cargo-nextest | ||
``` | ||
Note: currently official site proposes using Docker Desktop for Linux, which is a GUI tool with plenty of quirks. If you | ||
want to only have CLI tool, you need the `docker-ce` package and you can follow | ||
[this guide](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04) for Ubuntu. | ||
|
||
### SQLx CLI | ||
Installing `docker` via `snap` or from the default repository can cause troubles. | ||
|
||
SQLx is a Rust library we use to interact with Postgres, and its CLI is used to manage DB migrations and support several | ||
features of the library. | ||
You need to install both `docker` and `docker compose`. | ||
|
||
**Note:** `docker compose` is installed automatically with `Docker Desktop`. | ||
|
||
**Note:** On linux you may encounter the following error when you’ll try to work with `zksync`: | ||
|
||
``` | ||
ERROR: Couldn't connect to Docker daemon - you might need to run `docker-machine start default`. | ||
``` | ||
|
||
If so, you **do not need** to install `docker-machine`. Most probably, it means that your user is not added to | ||
the`docker` group. You can check it as follows: | ||
|
||
```bash | ||
cargo install --locked sqlx-cli --version 0.8.1 | ||
docker-compose up # Should raise the same error. | ||
sudo docker-compose up # Should start doing things. | ||
``` | ||
|
||
## Easier method using `nix` | ||
If the first command fails, but the second succeeds, then you need to add your user to the `docker` group: | ||
|
||
Nix is a tool that can fetch _exactly_ the right dependencies specified via hashes. The current config is Linux-only but | ||
it is likely that it can be adapted to Mac. | ||
```bash | ||
sudo usermod -a -G docker your_user_name | ||
``` | ||
|
||
Install `nix`. Enable the nix command and flakes. | ||
After that, you should logout and login again (user groups are refreshed after the login). The problem should be solved | ||
at this step. | ||
|
||
Install docker, rustup and use rust to install SQLx CLI like described above. If you are on NixOS, you also need to | ||
enable nix-ld. | ||
If logging out does not resolve the issue, restarting the computer should. | ||
|
||
Go to the zksync folder and run `nix develop`. After it finishes, you are in a shell that has all the dependencies. | ||
### Node.js & Yarn | ||
|
||
1. Install `Node` (requires version `v20`). The recommended way is via [nvm](https://github.com/nvm-sh/nvm). | ||
2. Install `yarn`. Can be done via `npm install -g yarn`. Make sure to get version 1.22.19 - you can change the version | ||
by running `yarn set version 1.22.19`. | ||
|
||
## Foundry ZKsync | ||
### Foundry ZKsync | ||
|
||
ZKSync depends on Foundry ZKsync (which is is a specialized fork of Foundry, tailored for ZKsync). Please follow this | ||
ZKsync depends on Foundry ZKsync (which is is a specialized fork of Foundry, tailored for ZKsync). Please follow this | ||
[installation guide](https://foundry-book.zksync.io/getting-started/installation) to get started with Foundry ZKsync. | ||
|
||
Foundry ZKsync can also be used for deploying smart contracts. For commands related to deployment, you can pass flags | ||
for Foundry integration. | ||
|
||
## Non-GPU setup | ||
### ZK Stack CLI | ||
|
||
Toolkit for creating and managing ZK Stack chains. `ZK Stack CLI` facilitates the creation and management of ZK Stacks. | ||
Commands are interactive but can also accept arguments via the command line. Follow the installation instructions | ||
[here](https://github.com/matter-labs/zksync-era/tree/main/zkstack_cli#installation). | ||
|
||
### Non-GPU setup | ||
|
||
Circuit Prover requires a CUDA bindings to run. If you still want to be able to build everything locally on non-CUDA | ||
setup, you'll need use CUDA stubs. | ||
|
@@ -270,6 +185,18 @@ RUSTFLAGS as env var, or pass it in `config.toml` (either project level or globa | |
rustflags = ["--cfg=no_cuda"] | ||
``` | ||
|
||
## Easier method using `nix` | ||
|
||
Nix is a tool that can fetch _exactly_ the right dependencies specified via hashes. The current config is Linux-only but | ||
it is likely that it can be adapted to Mac. | ||
|
||
Install `nix`. Enable the nix command and flakes. | ||
|
||
Install docker, rustup and use rust to install SQLx CLI like described above. If you are on NixOS, you also need to | ||
enable nix-ld. | ||
|
||
Go to the zksync folder and run `nix develop`. After it finishes, you are in a shell that has all the dependencies. | ||
|
||
## Tips | ||
|
||
### Tip: `mold` | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
#!/usr/bin/env bash | ||
|
||
main() { | ||
say "Setting up the development environment for ZKsync" | ||
|
||
# All necessary stuff | ||
say "Installing apt dependencies..." | ||
sudo apt update | ||
sudo apt install --yes git build-essential pkg-config cmake clang lldb lld libssl-dev libpq-dev apt-transport-https ca-certificates software-properties-common | ||
|
||
# Rust | ||
say "Installing Rust..." | ||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y | ||
source "$HOME/.cargo/env" | ||
# For running unit tests | ||
say "Installing Nextest..." | ||
cargo install cargo-nextest --locked | ||
# SQL tools | ||
say "Installing SQLx CLI..." | ||
cargo install sqlx-cli --version 0.8.1 | ||
|
||
# Install Docker | ||
say "Installing Docker..." | ||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - | ||
sudo add-apt-repository --yes "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" | ||
sudo apt install --yes docker-ce | ||
|
||
# Start Docker | ||
say "Starting Docker..." | ||
sudo systemctl start docker | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't we need to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. AFAIU the usermod change is just to be able to run docker commands without sudo. In fact, in my test machine, docker is running after this command. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Probably user on your test machine has root access, and ability to run docker commands without sudo is super important, so it must be a part of initialization. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is it important? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Because it's a major security prerequisite. Giving docker root access is a bad idea (and in general, you should never give sudo access to anything unless there is a strong request for it). And, well, it's just not convenient to have always run commands with sudo. Plus it will break e.g. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Regarding security, I'd argue exactly the reverse. With sudo you have to input a password for any docker command, thus requiring human approval. If you add the user to the docker group you effectively allow any app with non-root privileges access to the docker commands. But yeah, if |
||
|
||
# NVM | ||
say "Installing NVM..." | ||
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash | ||
export NVM_DIR="$HOME/.nvm" | ||
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm | ||
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion | ||
nvm install 20 | ||
|
||
# Yarn | ||
say "Installing Yarn..." | ||
npm install -g yarn | ||
yarn set version 1.22.19 | ||
|
||
# Foundry ZKsync | ||
say "Installing Foundry ZKsync..." | ||
curl -L https://raw.githubusercontent.com/matter-labs/foundry-zksync/main/install-foundry-zksync | bash | ||
|
||
# ZK Stack CLI | ||
say "Installing ZK Stack CLI..." | ||
curl -L https://raw.githubusercontent.com/matter-labs/zksync-era/main/zkstack_cli/zkstackup/install | bash | ||
"$HOME/.local/bin/zkstackup" | ||
|
||
# Non CUDA (GPU) setup, can be skipped if the machine has a CUDA installed for provers | ||
# Don't do that if you intend to run provers on your machine. Check the prover docs for a setup instead. | ||
say "Setting up the non-CUDA setup..." | ||
echo "export ZKSYNC_USE_CUDA_STUBS=true" >> "$HOME/.bashrc" | ||
|
||
# Clone the repo | ||
say "Cloning the ZKsync repository..." | ||
git clone --recurse-submodules https://github.com/matter-labs/zksync-era.git | ||
|
||
say "Installation of the development environment for ZKsync complete!" | ||
say "Please reload your shell configuration by running 'source ~/.bashrc'" | ||
brunoffranca marked this conversation as resolved.
Show resolved
Hide resolved
|
||
} | ||
|
||
say() { | ||
echo -e "\033[1;32m$1\033[0m" | ||
} | ||
|
||
main "$@" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am also hesitatnt to suggest blindly running it. If the script will be run on a machine where, e.g., docker is already installed (especially if it was installed in some other way), it can mess up the configuration.
Do we really need to have a script here? If so, I'd probably want to make it fail-proof, e.g. that it won't do stuff that is not needed, but it's a much bigger effort.
The benefit of a code block is that you see the code and can choose what lines you need and what lines you don't need.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we are to keep this, I'd add some big red warning that you should first read the script code and make sure that it does what you need (which though would render it much less helpful).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fair enough, but the original setup guide specifically says that the code is to be run on a fresh GCP instance. So it seems reasonable that we have a fast setup option for those cases at least.
I can add a code block below with what's in the script, so that people can inspect it though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Original -- yes, the new one removed mention of GCP.
Regarding code block -- it would mean duplication, and script will likely get out of sync with the code block.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't really think it's possible to create a fail-proof version of this script (at least without a disproportionate amount of work). And if we don't want duplication then the only possible solution is for me to close this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, I agree, and that's exactly the reason why it wasn't done before.
Was there any reason why we need a script except for making usage simpler?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, just to make usage simpler. I created one for myself, since I have to run fresh dev nodes with some frequency, and thought it would be useful for more people.