Splitting PR #44: BaseFold Profile and Cycle Reduction (#54)

* Dense Matrix

* Added hints

* Finished get_base_codeword_dimensions

* WIP mmcs

* WIP mmcs

* Update mmcs

* WIP mmcs

* Finished MMCS

* Added dot_product

* query_phase input

* WIP query_phase

* Fix bug in sorting

* WIP query phase

* WIP query phase

* WIP query phase

* WIP query phase

* New Sorting Impl

* WIP query phase

* WIP query phase

* WIP query phase

* Finished query_phase encoding

* Support serialized input

* Implement the naive encode small method

* Bug workaround

* Temp store: starting to use openvm mmcs instruction

* Temp store: index to bits

* Temp store: clean up mmcs

* Temp store: clear compilation errors

* Fix compilation error

* Fix hash variable reading bug

* Use dyn array for dimensions

* Fix multiplication between var and ext

* Fix mmcs reading

* Remove unnecessary witness stream

* Add doc for generating mmcs test data

* Try fixing mmcs

* Try fixing mmcs

* Try fixing mmcs

* Add comment

* Use the same poseidon2 constants as the test data

* Specify branch for test data gen

* MMCS test passes

* Rewrite fold coeff according to current basefold code

* Fix

* Merge e2e modification

* Fix compilation errors from merge

* Fix compilation errors from merge

* Fix compilation errors from merge

* (WIP) Connecting e2e with query phase

* WIP

* (WIP) transform ceno query phase verifier input to current

* (WIP) Fix query phase transform

* (WIP) Fix query phase transform

* (WIP) Fix query phase transform

* (WIP) Fix query phase transform

* Fix query phase transform

* Comment out connecting code temporarily

* Query phase compile successful

* (WIP) Debugging query phase

* Fix a bug in verifier query phase

* Read additional hints from binary file

* Fix: batch opening (#28)

* comment out

* wip

* hash read/write unit test passed

* wip2

* add inv_2 to input stream

* Fix memory out of bound problem

* update Cargo.lock

* Avoid providing two-adic generators inverses by hint

* Replace idx_bits by num2bits_f

* Replace idx_len by max_num_vars + rate log

* Change index bits to small endian

* Try fixing new index check

* Fix new index check

* Sub one from index len

* Identified the cause

* Fix index out of bound error

* Change comment

* Add native verify test

* Fix evals shape error

* Fix mmcs verify failure

* Remove some print lines

* fmt

* Supply all hints

* Fix new index compute

* Fix ext mmcs verify dimension

* Slice idx bits in ext mmcs verify

* Some small fixes

* right shift by hint

* Fix verifier_folding_coeffs_level

* Successfully run to first checkpoint

* Fails at last line

* Identified unimplemented function build_eq_x_r_vec_sequential_with_offset

* batch verifier query phase test passes

* Print the cycle count

---------

Co-authored-by: Yuncong Zhang <[email protected]>

* Switch ceno reliance

* Fix compilation errors due to out of date code

* Update test query phase batch

* Fix query opening proof

* Implement basefold proof variable

* Update query phase verifier input

* Preparing test data for query phase with updated code

* Implement basefold proof transform

* Prepare query phase verifier input

* Prepare query phase verifier input

* Fix final message access

* Switch ceno reliance to small field support

* Create basefold verifier function

* Check final message sizes

* Fix final message size

* Fix final message size

* Check query opening proof len

* Compute total number of polys

* Sample batch coeffs

* Compute max_num_var

* Write sumcheck messages and commits to transcript

* Write final message to transcript

* Complete the code for batch verifier

* Add verifier test

* Try to fix some compilation errors in e2e

* Connecting pcs with e2e

* Simplify BaseFold verifier (#34)

* [Upgrade] ZKVMProof Verifier Update (#31)

* Remove index reversal

* Add a cycle tracker

* Delete a loop

* Better casting

* Change verifier logic

* Finish opcdoe proof verification debugging

* Finish debugging table proof verification

* Debug verifier

* Finish debugging updated verifier

* Remove unnecessary table proof fields

* Remove unnecessary parsing

* Update Plonky3

* Migrate away from temporary build branch

* Switch ceno reliance

* Fix compilation errors due to out of date code

* Update test query phase batch

* Fix query opening proof

* Implement basefold proof variable

* Update query phase verifier input

* Preparing test data for query phase with updated code

* Implement basefold proof transform

* Prepare query phase verifier input

* Prepare query phase verifier input

* Fix final message access

* Switch ceno reliance to small field support

* basefold verifier for one matrix (#35)

* wip

* wip2

* wip3

* fix test

* fix

* fmt

* fri part of verifying basefold proof for 1 matrix passed

* sumcheck part 1

* sumcheck part 2

* sumcheck part 3

* cleanup

* more cleanups

---------

Co-authored-by: Ray Gao <[email protected]>
Co-authored-by: xkx <[email protected]>

* Fix some issues after merge

* Make compilation pass temporarily

* Make test pass before query phase

* add multiple matrices as inputs to the basefold's unit test

* fix

* unit test passed

* support matrices that have same height

* refactor tests

* Supply the permutation and make the random case pass

* Try fixing transcript inconsistency

* Use bin to dec le

* Add pow witness

* Basefold verifier passes for simple case

* Update dependency

* Basefold verifier passes decreasing and random batches

* update ceno dependencies

* comment out patch

* refactor

* the computation of max_num_var is simplified

* put perm to RoundVariable

* Change the way for computing batch

* Remove the unnecessary slice

* Simplify code

* Fix bug: the first mmcs batch verify passes

* Initialize all zeros

* remove debug routines

* rename

* clean

* cleanup

* ignore e2e test

* Complete Basefold verifier and Basefold e2e integration (#36)

* Switch ceno reliance

* Fix compilation errors due to out of date code

* Update test query phase batch

* Fix query opening proof

* Implement basefold proof variable

* Update query phase verifier input

* Preparing test data for query phase with updated code

* Implement basefold proof transform

* Prepare query phase verifier input

* Prepare query phase verifier input

* Fix final message access

* Switch ceno reliance to small field support

* Create basefold verifier function

* Check final message sizes

* Fix final message size

* Fix final message size

* Check query opening proof len

* Compute total number of polys

* Sample batch coeffs

* Compute max_num_var

* Write sumcheck messages and commits to transcript

* Write final message to transcript

* Complete the code for batch verifier

* Add verifier test

* Try to fix some compilation errors in e2e

* Connecting pcs with e2e

* Fix some issues after merge

* Make compilation pass temporarily

* Make test pass before query phase

* Supply the permutation and make the random case pass

* Try fixing transcript inconsistency

* Use bin to dec le

* Add pow witness

* Basefold verifier passes for simple case

* Update dependency

* Basefold verifier passes decreasing and random batches

* update ceno dependencies

* comment out patch

* refactor

* the computation of max_num_var is simplified

* put perm to RoundVariable

* remove debug routines

* rename

* clean

* cleanup

* ignore e2e test

---------

Co-authored-by: kunxian xia <[email protected]>

* Fix

* Fix one bug: alpha is batch coeffs 1 not 0

* Handle case when batch size is 1

* Allocate all zeros only once

* Construct test multiple rounds and test with fibonacci e2e data

* Add trace subscriber to profile cycles

* Profile query phase

* Profile query phase

* Some small optimizations

* Allocate opened values buffer outside query loop

* Compute low high values buffer outside loop

* Compute alpha offset outside loop

* Remove batch coeffs offset

* Compute all zero slices outside loop

* Remove round openings from loop

* Clean ups

* Compute height outside loop

* Compute permuted dimensions outside loop

* Merge two loops over openings

* Profile fri rounds

* Fix compilation errors after merge

* Dependencies moved to dev

* Fix unit tests

* chore remove unused dependency

* Remove opening height from round context: duplicate of log2_height

* Compute low and high values buffer in loop

* Provide some comments

* All zeros slice computed in loop

* Fix small issues

---------

Co-authored-by: Kunming Jiang <[email protected]>
Co-authored-by: kunxian xia <[email protected]>
Co-authored-by: Ray Gao <[email protected]>

Author: 4 people