diff --git a/README.md b/README.md index 87885b687a..57e3195f97 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,12 @@ To learn how to use zkSync, please refer to the [zkSync SDK documentation](https ## Development Documentation -The repository architecture overview is available: [docs/architecture.md](docs/architecture.md). +The following guides for developers are available: + +- Installing development dependencies: [docs/setup-dev.md](docs/setup-dev.md). +- Launching zkSync locally: [docs/launch.md](docs/launch.md). +- Development guide: [docs/development.md](docs/development.md). +- Repository architecture overview: [docs/architecture.md](docs/architecture.md). ## Projects diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000000..2c3c1ef785 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,215 @@ +# Development guide + +This document covers development-related actions in zkSync. + +## Initializing the project + +To setup the main toolkit, `zk`, simply run: + +``` +zk +``` + +You may also configure autocompletion for your shell via: + +``` +zk completion install +``` + +Once all the dependencies were installed, project can be initialized: + +``` +zk init +``` + +This command will do the following: + +- Generate `$ZKSYNC_HOME/etc/env/dev.env` file with settings for the applications. +- Initialize docker containers with `geth` Ethereum node and `postgres` database for local development. +- Download and unpack files for cryptographical backend (`circuit`). +- Generate required smart contracts. +- Compile all the smart contracts. +- Deploy smart contracts to the local Ethereum network. +- Initialize database and apply migrations. +- Insert required data into created database. +- Create “genesis block” for server. + +Initializing may take pretty long, but many steps (such as downloading & unpacking keys and initializing containers) +arerequired to be done only once. + +Usually, it is a good idea to do `zk init` once after each merge to the `dev` branch (as application setup may change). + +**Note:** If after getting new functionality from the `dev` branch your code stopped working and `zk init` doesn’t +help,you may try removing `$ZKSYNC_HOME/etc/env/dev.env` and running `zk init` once again. This may help if the +applicationconfiguration has changed. + +If you don’t need all of the `zk init` functionality, but just need to start/stop containers, use the followingcommands: + +``` +zk up # Set up `geth` and `postgres` containers +zk down # Shut down `geth` and `postgres` containers +``` + +## Committing changes + +`zksync` uses pre-commit and pre-push git hooks for basic code integrity checks. Hooks are set up automatically +withinthe workspace initialization process. These hooks will not allow to commit the code which does not pass several +checks. + +Currently the following criteria are checked: + +- Rust code should always be formatted via `cargo fmt`. +- Other code should always be formatted via `zk fmt`. +- Dummy Prover should not be staged for commit (see below for the explanation). + +## Using Dummy Prover + +Using the real prover for the development can be not really handy, since it’s pretty slow and resource consuming. + +Instead, one may want to use the Dummy Prover: lightweight version of the prover, which does not actually proveanything, +but acts like it does. + +To enable the dummy prover, run: + +``` +zk dummy-prover enable +``` + +And after that you will be able to use the dummy prover instead of actual prover: + +``` +zk dummy-prover run # Instead of `zk prover` +``` + +**Warning:** `dummy-prover enable` subcommand changes the `Verifier.sol` contract, which is a part +of `git` repository.Be sure not to commit these changes when using the dummy prover! + +If one will need to switch back to the real prover, a following command is required: + +``` +zk dummy-prover disable +``` + +This command will revert changes in the contract and redeploy it, so the actual prover will be usable again. + +Also you can always check the current status of the dummy verifier: + +``` +$ zk dummy-prover status +Dummy Prover status: disabled +``` + +## Database migrations + +zkSync uses PostgreSQL as a database backend, and `diesel-cli` for database migrations management. + +Existing migrations are located in `core/lib/storage/migrations`. + +Adding a new migration requires the following actions: + +1. Go to the `storage` folder: + + ``` + cd core/lib/storage + ``` + +2. Generate a blanket migration: + + ``` + diesel migration generate name-of-your-migration + ``` + +3. Implement migration: `up.sql` must contain new changes for the DB, and `down.sql` must revert the migration andreturn + the database into previous state. +4. Run `zk db migrate` to apply migration. +5. Implement corresponding changes in the `storage` crate. +6. Implement tests for new functionality. +7. Run database tests: + +``` +zk test db +``` + +## Testing + +- Running the `rust` unit-tests (heavy tests such as ones for `circuit` and database will not be run): + + ``` + zk f cargo test + ``` + +- Running the database tests: + + ``` + zk test db + ``` + +- Running the integration test: + + ``` + zk server # Has to be run in the 1st terminal + zk dummy-prover run # Has to be run in the 2nd terminal + zk test i server # Has to be run in the 3rd terminal + ``` + +- Running the circuit tests: + + ``` + zk test circuit + ``` + +- Running the prover tests: + + ``` + zk test prover + ``` + +- Running the benchmarks: + + ``` + zk f cargo bench + ``` + +- Running the loadtest: + + ``` + zk server # Has to be run in the 1st terminal + zk prover # Has to be run in the 2nd terminal + zk run loadtest # Has to be run in the 3rd terminal + ``` + + **Note**. If you have compilation issues with `sqlx`, then make sure to run `zk up` before running the tests. Also, + ifyou see some tests fail, might need to call `zk db reset` and restart the tests. + +## Developing circuit + +- To generate proofs one must have the universal setup files (which are downloaded during the first initialization). +- To verify generated proofs one must have verification keys. Verification keys are generated for specific circuit + &Verifier.sol contract; without these keys it is impossible to verify proofs on the Ethereum network. + +Steps to do after updating circuit: + +1. Update circuit version by updating `KEY_DIR` in your env file (don’t forget to place it to `dev.env.example`) + (lastparts of this variable usually means last commit where you updated circuit). +2. Regenerate verification keys and Verifier contract using `zk run verify-keys gen` command. +3. Pack generated verification keys using `zk run verify-keys pack` command and commit resulting file to repo. + +## Build and push Docker images to dockerhub + +``` +zk docker push +``` + +## Contracts + +### Re-build contracts + +``` +zk contract build +``` + +### Publish source code on etherscan + +``` +zk contract publish +``` diff --git a/docs/launch.md b/docs/launch.md new file mode 100644 index 0000000000..1ffc5344d0 --- /dev/null +++ b/docs/launch.md @@ -0,0 +1,117 @@ +# Running the application + +This document covers common scenarios of launching zkSync applications set locally. + +## Prerequisites + +Prepare dev environment prerequisites: see + +[Installing dependencies](./setup-dev.md) + +## Setup local dev environment + +Setup: + +``` +zk # installs and builds zk itself +zk init +``` + +During the first initialization you have to download around 8 GB of setup files, this should be done once. If you have a +problem on this step of the initialization, see help for the `zk run plonk-setup` command. + +If you face any other problems with the `zk init` command, go to the [Troubleshooting](##Troubleshooting) section at the +end of this file. There are solutions for some common error cases. + +To completely reset the dev environment: + +- Stop services: + + ``` + zk down + ``` + +- Repeat the setup procedure above + +If `zk init` has already been executed, and now you only need to start docker containers (e.g. after reboot), +simplylaunch: + +``` +zk up +``` + +## (Re)deploy db and contraсts + +``` +zk contract redeploy +``` + +## Environment configurations + +Env config files are held in `etc/env/` + +List configurations: + +``` +zk env +``` + +Switch between configurations: + +``` +zk env +``` + +Default confiruration is `dev.env`, which is generated automatically +from `dev.env.example` during `zk init` commandexecution. + +## Build and run server + prover locally for development + +Run server: + +``` +zk server +``` + +Server is configured using env files in `./etc/env` directory. After the first initialization, +file `./etc/env/dev.env`will be created. By default, this file is copied from the `./etc/env/dev.env.example` template. + +Server can produce block of different sizes, the list of available sizes is determined by +the`SUPPORTED_BLOCK_CHUNKS_SIZES` environment variable. Block sizes which will actually be produced by the server can be +configured using the `BLOCK_CHUNK_SIZES` environment variable. + +Note: for proof generation for large blocks requires a lot of resources and an average user machine is only capable +ofcreating proofs for the smallest block sizes. As an alternative, a dummy-prover may be used for development +(see`[development.md](https://hackmd.io/S7hTv1EwSpWu8VCReDmsBg)` for details). + +Run prover: + +``` +zk prover +``` + +Make sure you have environment variables set right, you can check it by running: `zk env`. You should +see `* dev` inoutput. + +## Troubleshooting + +### SSL error: certificate verify failed + +**Problem**. `zk init` fails with the following error: + +``` +Initializing download: https://universal-setup.ams3.digitaloceanspaces.com/setup_2%5E20.key +SSL error: certificate verify failed +``` + +**Solution**. Make sure that the version of `axel` on your computer is `2.17.10`. + +### rmSync is not a function + +**Problem**. `zk init` fails with the following error: + +``` +fs_1.default.rmSync is not a function +``` + +**Solution**. Make sure that the version of `node.js` installed on your computer is `14.14.0` or higher. diff --git a/docs/setup-dev.md b/docs/setup-dev.md new file mode 100644 index 0000000000..b753688198 --- /dev/null +++ b/docs/setup-dev.md @@ -0,0 +1,216 @@ +# Installing dependencies + +## `Docker` + +Install `docker`. It is recommended to follow the instructions from the +[official site](https://docs.docker.com/install/). + +Installing `docker` via `snap` or from the default repository can cause troubles. + +You need to install both `docker` and `docker-compose`. + +**Note:** On linux you may encounter the following error when you’ll try to work with `zksync`: + +``` +ERROR: Couldn't connect to Docker daemon - you might need to run `docker-machine start default`. +``` + +If so, you **do not need** to install `docker-machine`. Most probably, it means that your user is not added to +the`docker` group. You can check it as follows: + +```bash +docker-compose up # Should raise the same error. +sudo docker-compose up # Should start doing things. +``` + +If the first command fails, but the second succeeds, then you need to add your user to the `docker` group: + +```bash +sudo usermod -a -G docker your_user_name +``` + +After that, you should logout and login again (user groups are refreshed after the login). The problem should be +solvedat this step. + +If logging out does not help, restarting the computer should. + +## `Node` & `Yarn` + +1. Install `Node` (requires version 14.14.0 or higher). Since our team attempts to always use the latest LTS version + of`Node.js`, we suggest you to install [nvm](https://github.com/nvm-sh/nvm). It will allow you to + update `Node.js`version easily in the future. +2. Install `yarn`. Instructions can be found on the [official site](https://classic.yarnpkg.com/en/docs/install/). Check + if `yarn` is installed by running `yarn -v`.If you face any problems when installing `yarn`, it might be the case + that your package manager installed the wrong package.Make sure to thoroughly follow the instructions above on the + official website. It contains a lot of troubleshootingguides in it. +3. Run `yarn global add @vue/cli-service` + +## `Axel` + +Install `axel` for downloading keys: + +On mac: + +```bash +brew install axel +``` + +On debian-based linux: + +```bash +sudo apt-get install axel +``` + +Check the version of `axel` with the following command: + +``` +axel --version +``` + +Make sure the version is `2.17.10`. + +## `Rust` + +Install the latest `rust` version. + +Instructions can be found on the [official site](https://www.rust-lang.org/tools/install). + +Verify the `rust` installation: + +```bash +rustc --version +rustc 1.48.0 (7eac88abb 2020-11-16) +``` + +### `lld` + +Optionally, you may want to optimize the build time with the LLVM linker, `lld`. Make sure you have it installed and +append `"-C", "link-arg=-fuse-ld=lld"` to the `rustflags` in your `.cargo/config` file, so it looks like this: + +``` +[target.x86_64-unknown-linux-gnu] +rustflags = [ + "-C", "link-arg=-fuse-ld=lld", +] +``` + +**Warning:** This is only viable for linux since `lld` doesn’t work on mac. + +## PSQL + +Install `psql` CLI tool to interact with postgres. + +On debian-based linux: + +```bash +sudo apt-get install postgresql-client +``` + +## `Diesel` CLI + +Install `[diesel](https://diesel.rs/)` CLI (it is used for migrations management only): + +```bash +cargo install diesel_cli --no-default-features --features postgres +``` + +If at the install step you get the linkage errors, install the development version of `libpq`. + +On debian-based linux: + +```bash +sudo apt-get install libpq-dev +``` + +If you still see the errors, install the `build-essential` package. On debian-based linux: + +```bash +sudo apt install build-essential +``` + +## `sqlx` CLI + +Also, we need `[sqlx](https://github.com/launchbadge/sqlx)` CLI (it is used to generate database wrappers): + +```bash +cargo install --version=0.2.0 sqlx-cli +``` + +If you face an error `Could not find directory of OpenSSL installation`, then you should do the following. +Theinstruction is targeted on debian-based Linux, but generally, the steps are similar for all OS. + +- Install `libssl-dev`: + +``` +sudo apt install libssl-dev +``` + +- Install OpenSSL. Here is [the instruction for Ubuntu](https://www.spinup.com/installing-openssl-on-ubuntu/), but + thesteps should be similar for the debian-based Linux distros. +- Add `OPENSSL_DIR` variable to your environment. This would typically be `/usr/local/ssl`. You can do this by addingthe + following line to your shell profile file (e.g. `~/.bash_profile`): + +```bash +export OPENSSL_DIR=/usr/local/ssl +``` + +- Install `package-config`: + +```bash +sudo apt-get install -y pkg-config +``` + +## `solc` + +You have to install `solc` v0.5.16. Instructions can be found at +[readthedocs](https://solidity.readthedocs.io/en/v0.6.2/installing-solidity.html). + +The simplest option for linux is to use `snap`. + +For mac you can install it as follows: + +```bash +brew update +brew upgrade +brew tap ethereum/ethereum +brew install solidity@5 +``` + +If you're Arch user, download the archived version from [here](https://archive.archlinux.org/packages/s/solidity/) and +install it + +```bash +pacman -U solidity-0.5.14-1-x86_64.pkg.tar.xz +``` + +Finally, to prevent pacman from upgrading it, add this line to your /etc/pacman.conf + +``` +IgnorePkg = solidity +``` + +## drone cli + +drone cli used to create promotion jobs [described here](https://docs.drone.io/cli/install/). + +## `cmake` + +Required by `binaryen` to build C++ sources. In order to speed it up, you might want to install `clang` and `lld` too. + +```bash +sudo apt-get install cmake clang lld +``` + +## Environment + +Edit the lines below and add them to your shell profile file (e.g. `~/.bash_profile`): + +```bash +# Add path here: +export ZKSYNC_HOME=/path/to/zksync + +export PATH=$ZKSYNC_HOME/bin:$PATH + +# If you're like me, uncomment: +# cd $ZKSYNC_HOME +```