From 61f363947e0345e1aa7e30c9f062d9330e21ea8f Mon Sep 17 00:00:00 2001 From: Alfredo Garcia Date: Sat, 30 Jul 2022 17:20:43 -0300 Subject: [PATCH] Release Zebra to 1.0.0-beta.13 (#4765) * update zebra to 1.0.0-beta.13 * add changelog * update the release date * Update changelog for cryptographic batch PRs * Add `bitvec` to the changelog * Update CHANGELOG.md * Add zebra-network fix and tokio version bump * Recommend that users update their rust compiler * Update disk and network usage for recent large blocks * Fix spacing * Add disk and network usage update to README * apply size suggestions Co-authored-by: teor * add known performance issues * update changelog * change release date Co-authored-by: teor Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> --- CHANGELOG.md | 67 +++++++++++++++++++++++++++++++--- Cargo.lock | 24 ++++++------ README.md | 26 ++++++++++--- book/src/user/install.md | 2 +- tower-batch/Cargo.toml | 2 +- tower-fallback/Cargo.toml | 2 +- zebra-chain/Cargo.toml | 2 +- zebra-consensus/Cargo.toml | 2 +- zebra-network/Cargo.toml | 2 +- zebra-network/src/constants.rs | 2 +- zebra-node-services/Cargo.toml | 2 +- zebra-rpc/Cargo.toml | 2 +- zebra-script/Cargo.toml | 2 +- zebra-state/Cargo.toml | 2 +- zebra-test/Cargo.toml | 2 +- zebra-utils/Cargo.toml | 2 +- zebrad/Cargo.toml | 2 +- 17 files changed, 109 insertions(+), 36 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c0ff790c5d..efb2641103b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,20 +5,77 @@ All notable changes to Zebra are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org). -## Next Release (Draft) +## [Zebra 1.0.0-beta.13](https://github.com/ZcashFoundation/zebra/releases/tag/v1.0.0-beta.13) - 2022-07-29 -This release improves Zebra's sync and verification performance under heavy load. -(TODO - complete the summary.) +This release fixes multiple bugs in proof and signature verification, which were causing big performance issues near the blockchain tip. +It also improves Zebra's sync performance and reliability under heavy load. + +### Disk and Network Usage Changes + +Zebra now uses around 50 - 100 GB of disk space, because many large transactions were recently added to the block chain. (In the longer term, several hundred GB are likely to be needed.) + +When there are a lot of large user-generated transactions on the network, Zebra can upload or download 1 GB or more per day. ### Configuration Changes -- Split the checkpoint and full verification [`sync` concurrency options](https://doc.zebra.zfnd.org/zebrad/config/struct.SyncSection.html) (#4726): +- Split the checkpoint and full verification [`sync` concurrency options](https://doc.zebra.zfnd.org/zebrad/config/struct.SyncSection.html) (#4726, #4758): - Add a new `full_verify_concurrency_limit` - Rename `max_concurrent_block_requests` to `download_concurrency_limit` - Rename `lookahead_limit` to `checkpoint_verify_concurrency_limit` For backwards compatibility, the old names are still accepted as aliases. +- Add a new `parallel_cpu_threads` [`sync` concurrency option](https://doc.zebra.zfnd.org/zebrad/config/struct.SyncSection.html) (#4776). + This option sets the number of threads to use for CPU-bound tasks, such as proof and signature verification. + By default, Zebra uses all available CPU cores. + +### Rust Compiler Bug Fixes + +- The Rust team has recently [fixed compilation bugs](https://blog.rust-lang.org/2022/07/19/Rust-1.62.1.html) in function coercions of `impl Trait` return types, and `async fn` lifetimes. + We recommend that you update your Rust compiler to release 1.62.1, and re-compile Zebra. + +### Added + +- Add a `rayon` thread pool for CPU-bound tasks (#4776) +- Run multiple cryptographic batches concurrently within each verifier (#4776) +- Run deserialization of transaction in a rayon thread (#4801) +- Run CPU-intensive state updates in parallel rayon threads (#4802) +- Run CPU-intensive state reads in parallel rayon threads (#4805) +- Support Tiers and supported platforms per Tier doc (#4773) + +### Changed + +- Update column family names to match Zebra's database design (#4639) +- Update Zebra's mainnet and testnet checkpoints (#4777, #4833) +- Process more blocks and batch items concurrently, so there's a batch ready for each available CPU (#4776) +- Wrap note commitment trees in an `Arc`, to reduce CPU and memory usage (#4757) +- Increment `tokio` dependency from 1.19.2 to 1.20.0, to improve diagnostics (#4780) + +### Fixed + +- Only verify halo2 proofs once per transaction (#4752) +- Use a separate channel for each cryptographic batch, to avoid dropped batch results (#4750) +- Improve batch fairness and latency under heavy load (#4750, #4776) +- Run all verifier cryptography on blocking CPU-bound threads, using `tokio` and `rayon` (#4750, #4776) +- Use `tokio`'s `PollSemaphore`, instead of an outdated `Semaphore` impl (#4750) +- Check batch worker tasks for panics and task termination (#4750, #4777) +- Limit the length of the `reject` network message's `message` and `reason` fields (#4687) +- Change the `bitvec` dependency from 1.0.0 to 1.0.1, to fix a performance regression (#4769) +- Fix an occasional panic when a `zebra-network` connection closes (#4782) +- Stop panicking when the connection error slot is not set (#4770) +- When writing blocks to disk, don't block other async tasks (#4199) +- When sending headers to peers, only deserialize the header data from disk (#4792) +- Return errors from `send_periodic_heartbeats_with_shutdown_handle` (#4756) +- Make FindHeaders and FindHashes run concurrently with state updates (#4826) +- Stop reading redundant blocks for every FindHashes and FindHeaders request (#4825) +- Generate sapling point outside the method (#4799) + +#### CI + +- Workaround lightwalletd hangs by waiting until we're near the tip (#4763) +- Split out Canopy logs into a separate job (#4730) +- Make full sync go all the way to the tip (#4709) +- Split Docker logs into sprout, other checkpoints, and full validation (#4704) +- Add a Zebra cached state update test, fix lightwalletd tests (#4813) -(TODO - insert changelog here) ## [Zebra 1.0.0-beta.12](https://github.com/ZcashFoundation/zebra/releases/tag/v1.0.0-beta.12) - 2022-06-29 diff --git a/Cargo.lock b/Cargo.lock index bdb1a34bcbd..8dab1379d56 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5439,7 +5439,7 @@ dependencies = [ [[package]] name = "tower-batch" -version = "0.2.27" +version = "0.2.28" dependencies = [ "color-eyre", "ed25519-zebra", @@ -5462,7 +5462,7 @@ dependencies = [ [[package]] name = "tower-fallback" -version = "0.2.27" +version = "0.2.28" dependencies = [ "futures-core", "pin-project 0.4.29", @@ -6286,7 +6286,7 @@ dependencies = [ [[package]] name = "zebra-chain" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "aes", "bech32", @@ -6349,7 +6349,7 @@ version = "1.0.0-beta.0" [[package]] name = "zebra-consensus" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "bellman", "blake2b_simd 1.0.0", @@ -6392,7 +6392,7 @@ dependencies = [ [[package]] name = "zebra-network" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "arti-client", "bitflags", @@ -6430,14 +6430,14 @@ dependencies = [ [[package]] name = "zebra-node-services" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "zebra-chain", ] [[package]] name = "zebra-rpc" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "chrono", "futures", @@ -6466,7 +6466,7 @@ dependencies = [ [[package]] name = "zebra-script" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "displaydoc", "hex", @@ -6479,7 +6479,7 @@ dependencies = [ [[package]] name = "zebra-state" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "bincode", "chrono", @@ -6515,7 +6515,7 @@ dependencies = [ [[package]] name = "zebra-test" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "color-eyre", "futures", @@ -6540,7 +6540,7 @@ dependencies = [ [[package]] name = "zebra-utils" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "color-eyre", "hex", @@ -6555,7 +6555,7 @@ dependencies = [ [[package]] name = "zebrad" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" dependencies = [ "abscissa_core", "atty", diff --git a/README.md b/README.md index ed7106dca83..b16beef0ca2 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ for your platform: 2. Install Zebra's build dependencies: - **libclang:** the `libclang`, `libclang-dev`, `llvm`, or `llvm-dev` packages, depending on your package manager - **clang** or another C++ compiler: `g++`, `Xcode`, or `MSVC` -3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.12 zebrad` +3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.13 zebrad` 4. Run `zebrad start` (see [Running Zebra](https://zebra.zfnd.org/user/run.html) for more information) For more detailed instructions, refer to the [documentation](https://zebra.zfnd.org/user/install.html). @@ -95,7 +95,7 @@ cargo install --features= ... The recommended requirements for compiling and running `zebrad` are: - 4+ CPU cores - 16+ GB RAM -- 50GB+ available disk space for building binaries and storing finalized state +- 100 GB+ available disk space for building binaries and storing cached chain state - 100+ Mbps network connections We continuously test that our builds and tests pass on: @@ -147,8 +147,8 @@ If this is a problem for you, please [open a ticket.](https://github.com/ZcashFoundation/zebra/issues/new/choose) `zebrad`'s typical mainnet network usage is: -- Initial sync: 31 GB download -- Ongoing updates: 10-100 MB upload and download per day, depending on peer requests +- Initial sync: 40 GB download (in the longer term, several hundred GB are likely to be downloaded). +- Ongoing updates: 10 MB - 1 GB upload and download per day, depending on user-created transaction size, and peer requests Zebra also performs an initial sync every time its internal database version changes. @@ -182,9 +182,25 @@ So Zebra's state should always be valid, unless your OS or disk hardware is corr ## Known Issues There are a few bugs in Zebra that we're still working on fixing: -- [No Windows support #3801](https://github.com/ZcashFoundation/zebra/issues/3801) +- No Windows support [#3801](https://github.com/ZcashFoundation/zebra/issues/3801) - We used to test with Windows Server 2019, but not anymore; see issue for details +### Performance + +We are working on improving Zebra performance, the following are known issues: +- Send note commitment and history trees from the non-finalized state to the finalized state [#4824](https://github.com/ZcashFoundation/zebra/issues/4824) +- Speed up opening the database [#4822](https://github.com/ZcashFoundation/zebra/issues/4822) +- Revert note commitment and history trees when forking non-finalized chains [#4794](https://github.com/ZcashFoundation/zebra/issues/4794) +- Store only the first tree state in each identical series of tree states [#4784](https://github.com/ZcashFoundation/zebra/issues/4784) + +RPCs might also be slower than they used to be, we need to check: +- Revert deserializing state transactions in rayon threads [#4831](https://github.com/ZcashFoundation/zebra/issues/4831) + +Ongoing investigations: +- Find out which parts of CommitBlock/CommitFinalizedBlock are slow [#4823](https://github.com/ZcashFoundation/zebra/issues/4823) +- Mini-Epic: Stop tokio tasks running for a long time and blocking other tasks [#4747](https://github.com/ZcashFoundation/zebra/issues/4747) +- Investigate busiest tasks per tokio-console [#4583](https://github.com/ZcashFoundation/zebra/issues/4583) + ## Future Work Features: diff --git a/book/src/user/install.md b/book/src/user/install.md index 88fd68cdc94..7ce8fafbb8c 100644 --- a/book/src/user/install.md +++ b/book/src/user/install.md @@ -9,7 +9,7 @@ for your platform: 2. Install Zebra's build dependencies: - **libclang:** the `libclang`, `libclang-dev`, `llvm`, or `llvm-dev` packages, depending on your package manager - **clang** or another C++ compiler: `g++`, `Xcode`, or `MSVC` -3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.12 zebrad` +3. Run `cargo install --locked --git https://github.com/ZcashFoundation/zebra --tag v1.0.0-beta.13 zebrad` 4. Run `zebrad start` (see [Running Zebra](run.md) for more information) If you're interested in testing out `zebrad` please feel free, but keep in mind diff --git a/tower-batch/Cargo.toml b/tower-batch/Cargo.toml index 18c7ebe0b1b..915cd66b602 100644 --- a/tower-batch/Cargo.toml +++ b/tower-batch/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "tower-batch" -version = "0.2.27" +version = "0.2.28" authors = ["Zcash Foundation "] license = "MIT" edition = "2021" diff --git a/tower-fallback/Cargo.toml b/tower-fallback/Cargo.toml index d95771d7ed0..c2b7e8946ed 100644 --- a/tower-fallback/Cargo.toml +++ b/tower-fallback/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "tower-fallback" -version = "0.2.27" +version = "0.2.28" authors = ["Zcash Foundation "] license = "MIT" edition = "2021" diff --git a/zebra-chain/Cargo.toml b/zebra-chain/Cargo.toml index 2e15b23d2fd..9c310937894 100644 --- a/zebra-chain/Cargo.toml +++ b/zebra-chain/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "zebra-chain" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" edition = "2021" diff --git a/zebra-consensus/Cargo.toml b/zebra-consensus/Cargo.toml index 90c8cf73e81..5af30a29dc6 100644 --- a/zebra-consensus/Cargo.toml +++ b/zebra-consensus/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "zebra-consensus" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" edition = "2021" diff --git a/zebra-network/Cargo.toml b/zebra-network/Cargo.toml index cb3206c8b3d..41716cde3c8 100644 --- a/zebra-network/Cargo.toml +++ b/zebra-network/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "zebra-network" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" edition = "2021" diff --git a/zebra-network/src/constants.rs b/zebra-network/src/constants.rs index 18c4d331b43..9e58c6eb3d5 100644 --- a/zebra-network/src/constants.rs +++ b/zebra-network/src/constants.rs @@ -242,7 +242,7 @@ pub const TIMESTAMP_TRUNCATION_SECONDS: u32 = 30 * 60; /// [BIP 14]: https://github.com/bitcoin/bips/blob/master/bip-0014.mediawiki // // TODO: generate this from crate metadata (#2375) -pub const USER_AGENT: &str = "/Zebra:1.0.0-beta.12/"; +pub const USER_AGENT: &str = "/Zebra:1.0.0-beta.13/"; /// The Zcash network protocol version implemented by this crate, and advertised /// during connection setup. diff --git a/zebra-node-services/Cargo.toml b/zebra-node-services/Cargo.toml index 11574066f10..65bed6466ed 100644 --- a/zebra-node-services/Cargo.toml +++ b/zebra-node-services/Cargo.toml @@ -2,7 +2,7 @@ name = "zebra-node-services" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" edition = "2021" repository = "https://github.com/ZcashFoundation/zebra" diff --git a/zebra-rpc/Cargo.toml b/zebra-rpc/Cargo.toml index 1f44a2e1f64..14fb6815fc9 100644 --- a/zebra-rpc/Cargo.toml +++ b/zebra-rpc/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "zebra-rpc" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" edition = "2021" diff --git a/zebra-script/Cargo.toml b/zebra-script/Cargo.toml index 74009de967b..5ed62254bde 100644 --- a/zebra-script/Cargo.toml +++ b/zebra-script/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "zebra-script" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" edition = "2021" diff --git a/zebra-state/Cargo.toml b/zebra-state/Cargo.toml index 32b4f978e57..88212e191a0 100644 --- a/zebra-state/Cargo.toml +++ b/zebra-state/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "zebra-state" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" edition = "2021" diff --git a/zebra-test/Cargo.toml b/zebra-test/Cargo.toml index 2d4085a5411..960e99d4aa5 100644 --- a/zebra-test/Cargo.toml +++ b/zebra-test/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "zebra-test" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" edition = "2021" diff --git a/zebra-utils/Cargo.toml b/zebra-utils/Cargo.toml index aa924e215a7..a65077368bc 100644 --- a/zebra-utils/Cargo.toml +++ b/zebra-utils/Cargo.toml @@ -2,7 +2,7 @@ name = "zebra-utils" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" edition = "2021" # Prevent accidental publication of this utility crate. publish = false diff --git a/zebrad/Cargo.toml b/zebrad/Cargo.toml index 896671ec181..4f2eb0ccbba 100644 --- a/zebrad/Cargo.toml +++ b/zebrad/Cargo.toml @@ -2,7 +2,7 @@ name = "zebrad" authors = ["Zcash Foundation "] license = "MIT OR Apache-2.0" -version = "1.0.0-beta.12" +version = "1.0.0-beta.13" edition = "2021" # Zebra is only supported on the latest stable Rust version. Some earlier versions might work. # Zebra uses features introduced in Rust 1.58.