Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support CSV output #119

Merged
merged 16 commits into from
Dec 4, 2024
Merged

feat: support CSV output #119

merged 16 commits into from
Dec 4, 2024

Conversation

sayon
Copy link
Contributor

@sayon sayon commented Nov 6, 2024

What ❔

  • Support for CSV output
  • Refactoring to enable generation of JSON in LNT-friendly format

Why ❔

  • CSV output is required so that we can more easily analyze the performance reports using Excel, R, or similar tools.
  • We are aiming at integrating with LNT.

Checklist

  • PR title corresponds to the body of PR.
  • Tests for the changes have been added / updated.
  • Documentation comments have been added / updated.
  • Code has been formatted via cargo fmt and checked with cargo clippy.

Sorry, something went wrong.

Copy link

github-actions bot commented Nov 6, 2024

╔═╡ Size (-%) ╞════════════════╡ All M3B3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Cycles (-%) ╞══════════════╡ All M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞════════════════╡ All M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞════════════════╡ All M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╚═══════════════════════════════════════════╝

╔═╡ Size (-%) ╞════════════════╡ All MzB3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Cycles (-%) ╞══════════════╡ All MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞════════════════╡ All MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞════════════════╡ All MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╚═══════════════════════════════════════════╝

╔═╡ Size (-%) ╞═════╡ EVMInterpreter M3B3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                                 NaN ║
╠═╡ Cycles (-%) ╞═══╡ EVMInterpreter M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞═════╡ EVMInterpreter M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞═════╡ EVMInterpreter M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs/gas ╞══════╡ EVMInterpreter M3B3 ╞═╣
║ ADD                                28.083 ║
║ MUL                                16.850 ║
║ SUB                                28.083 ║
║ DIV                                19.250 ║
║ SDIV                               36.050 ║
║ MOD                                19.250 ║
║ SMOD                               33.650 ║
║ ADDMOD                             18.156 ║
║ MULMOD                             16.656 ║
║ EXP                                 6.904 ║
║ SIGNEXTEND                         19.250 ║
║ LT                                 32.083 ║
║ GT                                 32.083 ║
║ SLT                                54.083 ║
║ SGT                                54.083 ║
║ EQ                                 32.083 ║
║ ISZERO                             27.750 ║
║ AND                                28.083 ║
║ OR                                 28.083 ║
║ XOR                                28.083 ║
║ NOT                                23.750 ║
║ BYTE                               38.083 ║
║ SHL                                34.083 ║
║ SHR                                34.083 ║
║ SAR                                52.083 ║
║ SGT                                54.083 ║
║ SHA3                               25.073 ║
║ ADDRESS                            41.812 ║
║ BALANCE                            20.924 ║
║ ORIGIN                           1351.625 ║
║ CALLER                             41.812 ║
║ CALLVALUE                          41.812 ║
║ CALLDATALOAD                       26.083 ║
║ CALLDATASIZE                       42.125 ║
║ CALLDATACOPY                       49.492 ║
║ CODESIZE                           45.625 ║
║ CODECOPY                           62.456 ║
║ GASPRICE                         1348.438 ║
║ EXTCODESIZE                         3.635 ║
║ EXTCODECOPY                         3.723 ║
║ RETURNDATASIZE                     43.500 ║
║ RETURNDATACOPY                     44.556 ║
║ EXTCODEHASH                         4.651 ║
║ BLOCKHASH                         238.819 ║
║ COINBASE                         1348.625 ║
║ TIMESTAMP                        1342.625 ║
║ NUMBER                           1342.625 ║
║ PREVRANDAO                       1342.625 ║
║ GASLIMIT                         1348.625 ║
║ CHAINID                          1336.625 ║
║ SELFBALANCE                       639.250 ║
║ BASEFEE                          1342.625 ║
║ POP                                39.125 ║
║ MLOAD                              43.667 ║
║ MSTORE                             55.248 ║
║ MSTORE8                            64.716 ║
║ SLOAD                              19.044 ║
║ SSTORE                              4.467 ║
║ JUMP                               15.667 ║
║ JUMPI                              15.636 ║
║ PC                                 42.312 ║
║ MSIZE                              48.812 ║
║ GAS                                42.312 ║
║ JUMPDEST                           59.625 ║
║ PUSH0                              42.312 ║
║ PUSH1                              37.958 ║
║ PUSH2                              43.375 ║
║ PUSH4                              46.208 ║
║ PUSH5                              47.625 ║
║ PUSH6                              49.042 ║
║ PUSH7                              50.458 ║
║ PUSH8                              51.875 ║
║ PUSH9                              53.292 ║
║ PUSH10                             54.708 ║
║ PUSH11                             56.125 ║
║ PUSH12                             57.542 ║
║ PUSH13                             58.958 ║
║ PUSH14                             60.375 ║
║ PUSH15                             61.792 ║
║ PUSH16                             63.208 ║
║ PUSH17                             64.625 ║
║ PUSH18                             66.042 ║
║ PUSH19                             67.458 ║
║ PUSH20                             68.875 ║
║ PUSH21                             70.292 ║
║ PUSH22                             71.708 ║
║ PUSH23                             73.125 ║
║ PUSH24                             74.542 ║
║ PUSH25                             75.958 ║
║ PUSH26                             77.375 ║
║ PUSH27                             78.792 ║
║ PUSH28                             80.208 ║
║ PUSH29                             81.625 ║
║ PUSH30                             83.042 ║
║ PUSH31                             84.458 ║
║ PUSH32                             85.875 ║
║ DUP1                               26.083 ║
║ DUP2                               32.417 ║
║ DUP3                               32.417 ║
║ DUP4                               32.417 ║
║ DUP5                               32.417 ║
║ DUP6                               32.417 ║
║ DUP7                               32.417 ║
║ DUP8                               32.417 ║
║ DUP9                               32.417 ║
║ DUP10                              32.417 ║
║ DUP11                              32.417 ║
║ DUP12                              32.417 ║
║ DUP13                              32.417 ║
║ DUP14                              32.417 ║
║ DUP15                              32.417 ║
║ DUP16                              32.417 ║
║ SWAP1                              32.417 ║
║ SWAP2                              32.417 ║
║ SWAP3                              32.417 ║
║ SWAP4                              32.417 ║
║ SWAP5                              32.417 ║
║ SWAP6                              32.417 ║
║ SWAP7                              32.417 ║
║ SWAP8                              32.417 ║
║ SWAP9                              32.417 ║
║ SWAP10                             32.417 ║
║ SWAP11                             32.417 ║
║ SWAP12                             32.417 ║
║ SWAP13                             32.417 ║
║ SWAP14                             32.417 ║
║ SWAP15                             32.417 ║
║ SWAP16                             32.417 ║
║ CALL                               34.125 ║
║ STATICCALL                         34.095 ║
║ DELEGATECALL                       33.175 ║
║ CREATE                              3.713 ║
║ CREATE2                             5.009 ║
║ RETURN                              1.000 ║
║ REVERT                              1.000 ║
╠═╡ Ergs/gas (-%) ╞═╡ EVMInterpreter M3B3 ╞═╣
╚═══════════════════════════════════════════╝

╔═╡ Size (-%) ╞═════╡ EVMInterpreter MzB3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                                 NaN ║
╠═╡ Cycles (-%) ╞═══╡ EVMInterpreter MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞═════╡ EVMInterpreter MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞═════╡ EVMInterpreter MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╚═══════════════════════════════════════════╝

╔═╡ Size (-%) ╞════════╡ Precompiles M3B3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Cycles (-%) ╞══════╡ Precompiles M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞════════╡ Precompiles M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞════════╡ Precompiles M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                                 NaN ║
╚═══════════════════════════════════════════╝

╔═╡ Size (-%) ╞════════╡ Precompiles MzB3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Cycles (-%) ╞══════╡ Precompiles MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞════════╡ Precompiles MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞════════╡ Precompiles MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                                 NaN ║
╚═══════════════════════════════════════════╝

╔═╡ Size (-%) ╞══════════╡ Real life M3B3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Cycles (-%) ╞════════╡ Real life M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞══════════╡ Real life M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞══════════╡ Real life M3B3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                                 NaN ║
╚═══════════════════════════════════════════╝

╔═╡ Size (-%) ╞══════════╡ Real life MzB3 ╞═╗
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Cycles (-%) ╞════════╡ Real life MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠═╡ Ergs (-%) ╞══════════╡ Real life MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                               0.000 ║
╠══╡ Gas (-%) ╞══════════╡ Real life MzB3 ╞═╣
║ Best                                0.000 ║
║ Worst                               0.000 ║
║ Total                                 NaN ║
╚═══════════════════════════════════════════╝

@sayon sayon force-pushed the iz-csv-support branch 2 times, most recently from 61c980a to 2b79711 Compare December 2, 2024 07:14
@sayon sayon requested a review from hedgar2017 December 2, 2024 14:25
@sayon sayon marked this pull request as ready for review December 3, 2024 11:06
Copy link
Collaborator

@hedgar2017 hedgar2017 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's an amazing piece of work, thank you!
I've left a few nits.

Have you tested all major use cases manually? I'd like to produce a benchmark of our LLVM EVM BE against solc. Is it able to compare reports such as Y+M3B3 with Y+?

@sayon
Copy link
Contributor Author

sayon commented Dec 3, 2024

Fixed the comments! Let's merge when we can to minimize future conflicts.

LNT will be able to compare Y+M3B3 and Y+. But for now we can already do that using the CSV output, which is uniform regardless of the target.

For example, if I execute the following:

cargo run --bin compiler-tester -- --target evm --load-system-contracts system-contracts-stable-build --benchmark-format csv  -v  --benchmark  Y_evm.csv  --path tests/solidity/simple/default.sol 

cargo run --bin compiler-tester --  --target eravm  --load-system-contracts system-contracts-stable-build  -v  --benchmark-format csv --benchmark  Y.csv  --path tests/solidity/simple/default.sol

then I can import both files to a spreadsheet for example to gdocs and do basic statistics there. Once we plug in another backend to generate JSON for LNT, it can be automated.

@hedgar2017 hedgar2017 enabled auto-merge (squash) December 4, 2024 05:06
@hedgar2017 hedgar2017 merged commit 59ff334 into main Dec 4, 2024
12 checks passed
@hedgar2017 hedgar2017 deleted the iz-csv-support branch December 4, 2024 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants