From 18a647138d299ef49100bec669eadb888350bb2a Mon Sep 17 00:00:00 2001 From: Stanislav Bezkorovainyi Date: Mon, 26 Apr 2021 15:25:34 +0300 Subject: [PATCH 1/2] Bring the setup docs back --- README.md | 7 +- docs/development.md | 215 +++++++++++++++++++++++++++++++++++++++++++ docs/launch.md | 117 ++++++++++++++++++++++++ docs/setup-dev.md | 216 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 554 insertions(+), 1 deletion(-) create mode 100644 docs/development.md create mode 100644 docs/launch.md create mode 100644 docs/setup-dev.md diff --git a/README.md b/README.md index 87885b687a..57e3195f97 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,12 @@ To learn how to use zkSync, please refer to the [zkSync SDK documentation](https ## Development Documentation -The repository architecture overview is available: [docs/architecture.md](docs/architecture.md). +The following guides for developers are available: + +- Installing development dependencies: [docs/setup-dev.md](docs/setup-dev.md). +- Launching zkSync locally: [docs/launch.md](docs/launch.md). +- Development guide: [docs/development.md](docs/development.md). +- Repository architecture overview: [docs/architecture.md](docs/architecture.md). ## Projects diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000000..68fe14b813 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,215 @@ +# Development guide + +This document covers development-related actions in zkSync. + +## **Initializing the project** + +To setup the main toolkit, `zk`, simply run: + +``` +zk +``` + +You may also configure autocompletion for your shell via: + +``` +zk completion install +``` + +Once all the dependencies were installed, project can be initialized: + +``` +zk init +``` + +This command will do the following: + +- Generate `$ZKSYNC_HOME/etc/env/dev.env` file with settings for the applications. +- Initialize docker containers with `geth` Ethereum node and `postgres` database for local development. +- Download and unpack files for cryptographical backend (`circuit`). +- Generate required smart contracts. +- Compile all the smart contracts. +- Deploy smart contracts to the local Ethereum network. +- Initialize database and apply migrations. +- Insert required data into created database. +- Create “genesis block” for server. + +Initializing may take pretty long, but many steps (such as downloading & unpacking keys and initializing containers) +arerequired to be done only once. + +Usually, it is a good idea to do `zk init` once after each merge to the `dev` branch (as application setup may change). + +**Note:** If after getting new functionality from the `dev` branch your code stopped working and `zk init` doesn’t +help,you may try removing `$ZKSYNC_HOME/etc/env/dev.env` and running `zk init` once again. This may help if the +applicationconfiguration has changed. + +If you don’t need all of the `zk init` functionality, but just need to start/stop containers, use the followingcommands: + +``` +zk up # Set up `geth` and `postgres` containers +zk down # Shut down `geth` and `postgres` containers +``` + +## **Committing changes** + +`zksync` uses pre-commit and pre-push git hooks for basic code integrity checks. Hooks are set up automatically +withinthe workspace initialization process. These hooks will not allow to commit the code which does not pass several +checks. + +Currently the following criteria are checked: + +- Rust code should always be formatted via `cargo fmt`. +- Other code should always be formatted via `zk fmt`. +- Dummy Prover should not be staged for commit (see below for the explanation). + +## **Using Dummy Prover** + +Using the real prover for the development can be not really handy, since it’s pretty slow and resource consuming. + +Instead, one may want to use the Dummy Prover: lightweight version of the prover, which does not actually proveanything, +but acts like it does. + +To enable the dummy prover, run: + +``` +zk dummy-prover enable +``` + +And after that you will be able to use the dummy prover instead of actual prover: + +``` +zk dummy-prover run # Instead of `zk prover` +``` + +**Warning:** `dummy-prover enable` subcommand changes the `Verifier.sol` contract, which is a part +of `git` repository.Be sure not to commit these changes when using the dummy prover! + +If one will need to switch back to the real prover, a following command is required: + +``` +zk dummy-prover disable +``` + +This command will revert changes in the contract and redeploy it, so the actual prover will be usable again. + +Also you can always check the current status of the dummy verifier: + +``` +$ zk dummy-prover status +Dummy Prover status: disabled +``` + +## **Database migrations** + +zkSync uses PostgreSQL as a database backend, and `diesel-cli` for database migrations management. + +Existing migrations are located in `core/lib/storage/migrations`. + +Adding a new migration requires the following actions: + +1. Go to the `storage` folder: + + ``` + cd core/lib/storage + ``` + +2. Generate a blanket migration: + + ``` + diesel migration generate name-of-your-migration + ``` + +3. Implement migration: `up.sql` must contain new changes for the DB, and `down.sql` must revert the migration andreturn + the database into previous state. +4. Run `zk db migrate` to apply migration. +5. Implement corresponding changes in the `storage` crate. +6. Implement tests for new functionality. +7. Run database tests: + +``` +zk test db +``` + +## **Testing** + +- Running the `rust` unit-tests (heavy tests such as ones for `circuit` and database will not be run): + + ``` + zk f cargo test + ``` + +- Running the database tests: + + ``` + zk test db + ``` + +- Running the integration test: + + ``` + zk server # Has to be run in the 1st terminal + zk dummy-prover run # Has to be run in the 2nd terminal + zk test i server # Has to be run in the 3rd terminal + ``` + +- Running the circuit tests: + + ``` + zk test circuit + ``` + +- Running the prover tests: + + ``` + zk test prover + ``` + +- Running the benchmarks: + + ``` + zk f cargo bench + ``` + +- Running the loadtest: + + ``` + zk server # Has to be run in the 1st terminal + zk prover # Has to be run in the 2nd terminal + zk run loadtest # Has to be run in the 3rd terminal + ``` + + **Note**. If you have compilation issues with `sqlx`, then make sure to run `zk up` before running the tests. Also, + ifyou see some tests fail, might need to call `zk db reset` and restart the tests. + +## **Developing circuit** + +- To generate proofs one must have the universal setup files (which are downloaded during the first initialization). +- To verify generated proofs one must have verification keys. Verification keys are generated for specific circuit + &Verifier.sol contract; without these keys it is impossible to verify proofs on the Ethereum network. + +Steps to do after updating circuit: + +1. Update circuit version by updating `KEY_DIR` in your env file (don’t forget to place it to `dev.env.example`) + (lastparts of this variable usually means last commit where you updated circuit). +2. Regenerate verification keys and Verifier contract using `zk run verify-keys gen` command. +3. Pack generated verification keys using `zk run verify-keys pack` command and commit resulting file to repo. + +## **Build and push Docker images to dockerhub** + +``` +zk docker push +``` + +## **Contracts** + +### **Re-build contracts** + +``` +zk contract build +``` + +### **Publish source code on etherscan** + +``` +zk contract publish +``` diff --git a/docs/launch.md b/docs/launch.md new file mode 100644 index 0000000000..b940c18580 --- /dev/null +++ b/docs/launch.md @@ -0,0 +1,117 @@ +# Running the application + +This document covers common scenarios of launching zkSync applications set locally. + +## **Prerequisites** + +Prepare dev environment prerequisites: see + +[Installing dependencies](./setup-dev.md) + +## **Setup local dev environment** + +Setup: + +``` +zk # installs and builds zk itself +zk init +``` + +During the first initialization you have to download around 8 GB of setup files, this should be done once. If you have a +problem on this step of the initialization, see help for the `zk run plonk-setup` command. + +If you face any other problems with the `zk init` command, go to the [Troubleshooting](##Troubleshooting) section at the +end of this file. There are solutions for some common error cases. + +To completely reset the dev environment: + +- Stop services: + + ``` + zk down + ``` + +- Repeat the setup procedure above + +If `zk init` has already been executed, and now you only need to start docker containers (e.g. after reboot), +simplylaunch: + +``` +zk up +``` + +## **(Re)deploy db and contraсts** + +``` +zk contract redeploy +``` + +## **Environment configurations** + +Env config files are held in `etc/env/` + +List configurations: + +``` +zk env +``` + +Switch between configurations: + +``` +zk env +``` + +Default confiruration is `dev.env`, which is generated automatically +from `dev.env.example` during `zk init` commandexecution. + +## **Build and run server + prover locally for development** + +Run server: + +``` +zk server +``` + +Server is configured using env files in `./etc/env` directory. After the first initialization, +file `./etc/env/dev.env`will be created. By default, this file is copied from the `./etc/env/dev.env.example` template. + +Server can produce block of different sizes, the list of available sizes is determined by +the`SUPPORTED_BLOCK_CHUNKS_SIZES` environment variable. Block sizes which will actually be produced by the server can be +configured using the `BLOCK_CHUNK_SIZES` environment variable. + +Note: for proof generation for large blocks requires a lot of resources and an average user machine is only capable +ofcreating proofs for the smallest block sizes. As an alternative, a dummy-prover may be used for development +(see`[development.md](https://hackmd.io/S7hTv1EwSpWu8VCReDmsBg)` for details). + +Run prover: + +``` +zk prover +``` + +Make sure you have environment variables set right, you can check it by running: `zk env`. You should +see `* dev` inoutput. + +## Troubleshooting + +### **SSL error: certificate verify failed** + +**Problem**. `zk init` fails with the following error: + +``` +Initializing download: https://universal-setup.ams3.digitaloceanspaces.com/setup_2%5E20.key +SSL error: certificate verify failed +``` + +**Solution**. Make sure that the version of `axel` on your computer is `2.17.10`. + +### **rmSync is not a function** + +**Problem**. `zk init` fails with the following error: + +``` +fs_1.default.rmSync is not a function +``` + +**Solution**. Make sure that the version of `node.js` installed on your computer is `14.14.0` or higher. diff --git a/docs/setup-dev.md b/docs/setup-dev.md new file mode 100644 index 0000000000..a14a3b921a --- /dev/null +++ b/docs/setup-dev.md @@ -0,0 +1,216 @@ +# Installing dependencies + +## **`Docker`** + +Install `docker`. It is recommended to follow the instructions from the +[official site](https://docs.docker.com/install/). + +Installing `docker` via `snap` or from the default repository can cause troubles. + +You need to install both `docker` and `docker-compose`. + +**Note:** On linux you may encounter the following error when you’ll try to work with `zksync`: + +``` +ERROR: Couldn't connect to Docker daemon - you might need to run `docker-machine start default`. +``` + +If so, you **do not need** to install `docker-machine`. Most probably, it means that your user is not added to +the`docker` group. You can check it as follows: + +```bash +docker-compose up # Should raise the same error. +sudo docker-compose up # Should start doing things. +``` + +If the first command fails, but the second succeeds, then you need to add your user to the `docker` group: + +```bash +sudo usermod -a -G docker your_user_name +``` + +After that, you should logout and login again (user groups are refreshed after the login). The problem should be +solvedat this step. + +If logging out does not help, restarting the computer should. + +## **`Node` & `Yarn`** + +1. Install `Node` (requires version 14.14.0 or higher). Since our team attempts to always use the latest LTS version + of`Node.js`, we suggest you to install [nvm](https://github.com/nvm-sh/nvm). It will allow you to + update `Node.js`version easily in the future. +2. Install `yarn`. Instructions can be found on the [official site](https://classic.yarnpkg.com/en/docs/install/). Check + if `yarn` is installed by running `yarn -v`.If you face any problems when installing `yarn`, it might be the case + that your package manager installed the wrong package.Make sure to thoroughly follow the instructions above on the + official website. It contains a lot of troubleshootingguides in it. +3. Run `yarn global add @vue/cli-service` + +## **`Axel`** + +Install `axel` for downloading keys: + +On mac: + +```bash +brew install axel +``` + +On debian-based linux: + +```bash +sudo apt-get install axel +``` + +Check the version of `axel` with the following command: + +``` +axel --version +``` + +Make sure the version is `2.17.10`. + +## **`Rust`** + +Install the latest `rust` version. + +Instructions can be found on the [official site](https://www.rust-lang.org/tools/install). + +Verify the `rust` installation: + +```bash +rustc --version +rustc 1.48.0 (7eac88abb 2020-11-16) +``` + +### **`lld`** + +Optionally, you may want to optimize the build time with the LLVM linker, `lld`. Make sure you have it installed and +append `"-C", "link-arg=-fuse-ld=lld"` to the `rustflags` in your `.cargo/config` file, so it looks like this: + +``` +[target.x86_64-unknown-linux-gnu] +rustflags = [ + "-C", "link-arg=-fuse-ld=lld", +] +``` + +**Warning:** This is only viable for linux since `lld` doesn’t work on mac. + +## **PSQL** + +Install `psql` CLI tool to interact with postgres. + +On debian-based linux: + +```bash +sudo apt-get install postgresql-client +``` + +## **`Diesel` CLI** + +Install `[diesel](https://diesel.rs/)` CLI (it is used for migrations management only): + +```bash +cargo install diesel_cli --no-default-features --features postgres +``` + +If at the install step you get the linkage errors, install the development version of `libpq`. + +On debian-based linux: + +```bash +sudo apt-get install libpq-dev +``` + +If you still see the errors, install the `build-essential` package. On debian-based linux: + +```bash +sudo apt install build-essential +``` + +## **`sqlx` CLI** + +Also, we need `[sqlx](https://github.com/launchbadge/sqlx)` CLI (it is used to generate database wrappers): + +```bash +cargo install --version=0.2.0 sqlx-cli +``` + +If you face an error `Could not find directory of OpenSSL installation`, then you should do the following. +Theinstruction is targeted on debian-based Linux, but generally, the steps are similar for all OS. + +- Install `libssl-dev`: + +``` +sudo apt install libssl-dev +``` + +- Install OpenSSL. Here is [the instruction for Ubuntu](https://www.spinup.com/installing-openssl-on-ubuntu/), but + thesteps should be similar for the debian-based Linux distros. +- Add `OPENSSL_DIR` variable to your environment. This would typically be `/usr/local/ssl`. You can do this by addingthe + following line to your shell profile file (e.g. `~/.bash_profile`): + +```bash +export OPENSSL_DIR=/usr/local/ssl +``` + +- Install `package-config`: + +```bash +sudo apt-get install -y pkg-config +``` + +## **`solc`** + +You have to install `solc` v0.5.16. Instructions can be found at +[readthedocs](https://solidity.readthedocs.io/en/v0.6.2/installing-solidity.html). + +The simplest option for linux is to use `snap`. + +For mac you can install it as follows: + +```bash +brew update +brew upgrade +brew tap ethereum/ethereum +brew install solidity@5 +``` + +If you're Arch user, download the archived version from [here](https://archive.archlinux.org/packages/s/solidity/) and +install it + +```bash +pacman -U solidity-0.5.14-1-x86_64.pkg.tar.xz +``` + +Finally, to prevent pacman from upgrading it, add this line to your /etc/pacman.conf + +``` +IgnorePkg = solidity +``` + +## **drone cli** + +drone cli used to create promotion jobs [described here](https://docs.drone.io/cli/install/). + +## `cmake` + +Required by `binaryen` to build C++ sources. In order to speed it up, you might want to install `clang` and `lld` too. + +```bash +sudo apt-get install cmake clang lld +``` + +## **Environment** + +Edit the lines below and add them to your shell profile file (e.g. `~/.bash_profile`): + +```bash +# Add path here: +export ZKSYNC_HOME=/path/to/zksync + +export PATH=$ZKSYNC_HOME/bin:$PATH + +# If you're like me, uncomment: +# cd $ZKSYNC_HOME +``` From 38a051df22d4bb5f58a5c25982d813dd00d149d9 Mon Sep 17 00:00:00 2001 From: Stanislav Bezkorovainyi Date: Mon, 26 Apr 2021 16:56:23 +0300 Subject: [PATCH 2/2] Remove bold from headers --- docs/development.md | 20 ++++++++++---------- docs/launch.md | 14 +++++++------- docs/setup-dev.md | 22 +++++++++++----------- 3 files changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/development.md b/docs/development.md index 68fe14b813..2c3c1ef785 100644 --- a/docs/development.md +++ b/docs/development.md @@ -2,7 +2,7 @@ This document covers development-related actions in zkSync. -## **Initializing the project** +## Initializing the project To setup the main toolkit, `zk`, simply run: @@ -50,7 +50,7 @@ zk up # Set up `geth` and `postgres` containers zk down # Shut down `geth` and `postgres` containers ``` -## **Committing changes** +## Committing changes `zksync` uses pre-commit and pre-push git hooks for basic code integrity checks. Hooks are set up automatically withinthe workspace initialization process. These hooks will not allow to commit the code which does not pass several @@ -62,7 +62,7 @@ Currently the following criteria are checked: - Other code should always be formatted via `zk fmt`. - Dummy Prover should not be staged for commit (see below for the explanation). -## **Using Dummy Prover** +## Using Dummy Prover Using the real prover for the development can be not really handy, since it’s pretty slow and resource consuming. @@ -99,7 +99,7 @@ $ zk dummy-prover status Dummy Prover status: disabled ``` -## **Database migrations** +## Database migrations zkSync uses PostgreSQL as a database backend, and `diesel-cli` for database migrations management. @@ -130,7 +130,7 @@ Adding a new migration requires the following actions: zk test db ``` -## **Testing** +## Testing - Running the `rust` unit-tests (heavy tests such as ones for `circuit` and database will not be run): @@ -181,7 +181,7 @@ zk test db **Note**. If you have compilation issues with `sqlx`, then make sure to run `zk up` before running the tests. Also, ifyou see some tests fail, might need to call `zk db reset` and restart the tests. -## **Developing circuit** +## Developing circuit - To generate proofs one must have the universal setup files (which are downloaded during the first initialization). - To verify generated proofs one must have verification keys. Verification keys are generated for specific circuit @@ -194,21 +194,21 @@ Steps to do after updating circuit: 2. Regenerate verification keys and Verifier contract using `zk run verify-keys gen` command. 3. Pack generated verification keys using `zk run verify-keys pack` command and commit resulting file to repo. -## **Build and push Docker images to dockerhub** +## Build and push Docker images to dockerhub ``` zk docker push ``` -## **Contracts** +## Contracts -### **Re-build contracts** +### Re-build contracts ``` zk contract build ``` -### **Publish source code on etherscan** +### Publish source code on etherscan ``` zk contract publish diff --git a/docs/launch.md b/docs/launch.md index b940c18580..1ffc5344d0 100644 --- a/docs/launch.md +++ b/docs/launch.md @@ -2,13 +2,13 @@ This document covers common scenarios of launching zkSync applications set locally. -## **Prerequisites** +## Prerequisites Prepare dev environment prerequisites: see [Installing dependencies](./setup-dev.md) -## **Setup local dev environment** +## Setup local dev environment Setup: @@ -40,13 +40,13 @@ simplylaunch: zk up ``` -## **(Re)deploy db and contraсts** +## (Re)deploy db and contraсts ``` zk contract redeploy ``` -## **Environment configurations** +## Environment configurations Env config files are held in `etc/env/` @@ -65,7 +65,7 @@ zk env Default confiruration is `dev.env`, which is generated automatically from `dev.env.example` during `zk init` commandexecution. -## **Build and run server + prover locally for development** +## Build and run server + prover locally for development Run server: @@ -95,7 +95,7 @@ see `* dev` inoutput. ## Troubleshooting -### **SSL error: certificate verify failed** +### SSL error: certificate verify failed **Problem**. `zk init` fails with the following error: @@ -106,7 +106,7 @@ SSL error: certificate verify failed **Solution**. Make sure that the version of `axel` on your computer is `2.17.10`. -### **rmSync is not a function** +### rmSync is not a function **Problem**. `zk init` fails with the following error: diff --git a/docs/setup-dev.md b/docs/setup-dev.md index a14a3b921a..b753688198 100644 --- a/docs/setup-dev.md +++ b/docs/setup-dev.md @@ -1,6 +1,6 @@ # Installing dependencies -## **`Docker`** +## `Docker` Install `docker`. It is recommended to follow the instructions from the [official site](https://docs.docker.com/install/). @@ -34,7 +34,7 @@ solvedat this step. If logging out does not help, restarting the computer should. -## **`Node` & `Yarn`** +## `Node` & `Yarn` 1. Install `Node` (requires version 14.14.0 or higher). Since our team attempts to always use the latest LTS version of`Node.js`, we suggest you to install [nvm](https://github.com/nvm-sh/nvm). It will allow you to @@ -45,7 +45,7 @@ If logging out does not help, restarting the computer should. official website. It contains a lot of troubleshootingguides in it. 3. Run `yarn global add @vue/cli-service` -## **`Axel`** +## `Axel` Install `axel` for downloading keys: @@ -69,7 +69,7 @@ axel --version Make sure the version is `2.17.10`. -## **`Rust`** +## `Rust` Install the latest `rust` version. @@ -82,7 +82,7 @@ rustc --version rustc 1.48.0 (7eac88abb 2020-11-16) ``` -### **`lld`** +### `lld` Optionally, you may want to optimize the build time with the LLVM linker, `lld`. Make sure you have it installed and append `"-C", "link-arg=-fuse-ld=lld"` to the `rustflags` in your `.cargo/config` file, so it looks like this: @@ -96,7 +96,7 @@ rustflags = [ **Warning:** This is only viable for linux since `lld` doesn’t work on mac. -## **PSQL** +## PSQL Install `psql` CLI tool to interact with postgres. @@ -106,7 +106,7 @@ On debian-based linux: sudo apt-get install postgresql-client ``` -## **`Diesel` CLI** +## `Diesel` CLI Install `[diesel](https://diesel.rs/)` CLI (it is used for migrations management only): @@ -128,7 +128,7 @@ If you still see the errors, install the `build-essential` package. On debian- sudo apt install build-essential ``` -## **`sqlx` CLI** +## `sqlx` CLI Also, we need `[sqlx](https://github.com/launchbadge/sqlx)` CLI (it is used to generate database wrappers): @@ -160,7 +160,7 @@ export OPENSSL_DIR=/usr/local/ssl sudo apt-get install -y pkg-config ``` -## **`solc`** +## `solc` You have to install `solc` v0.5.16. Instructions can be found at [readthedocs](https://solidity.readthedocs.io/en/v0.6.2/installing-solidity.html). @@ -189,7 +189,7 @@ Finally, to prevent pacman from upgrading it, add this line to your /etc/pacman. IgnorePkg = solidity ``` -## **drone cli** +## drone cli drone cli used to create promotion jobs [described here](https://docs.drone.io/cli/install/). @@ -201,7 +201,7 @@ Required by `binaryen` to build C++ sources. In order to speed it up, you might sudo apt-get install cmake clang lld ``` -## **Environment** +## Environment Edit the lines below and add them to your shell profile file (e.g. `~/.bash_profile`):