-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upstream simplified Optimism test #369
Conversation
The CI job with a new test got killed (https://github.com/runtimeverification/kontrol/actions/runs/7898491231/attempts/1?pr=369, and then timed out in 2 hours), and on my machine it also takes > 1 hour — I'll look into how it can be further minimized while still being illustrative of the performance changes. |
PortalTest.test_withdrawal_paused
PortalTest.test_withdrawal_paused
The following test (which is too simple to be illustrative of the performance difference between backend versions): /// @custom:kontrol-length-equals _withdrawalProof: 3,
/// @custom:kontrol-length-equals data: 32,
/// @custom:kontrol-length-equals _withdrawalProof[]: 32,
function test_withdrawal_paused(
Types.WithdrawalTransaction calldata _tx,
uint256 _l2OutputIndex,
Types.OutputRootProof calldata _outputRootProof,
bytes[] calldata _withdrawalProof
)
external
{
portalContract.pause();
vm.expectRevert();
portalContract.proveWithdrawalTransaction(_tx, _l2OutputIndex, _outputRootProof, _withdrawalProof);
}
} takes ~6 minutes to run locally on my machine and ~8 minutes — on CI when executed by a faster runner. It seems that we're getting very different execution time on booster tests depending on the runner, e.g.,
Making the test more challenging (and lemma-dependent) is needed to ensure that we can observe the difference in performance, however, it seems that it would also cause time outs in many CI runs. With @JuanCoRo, we've also realised that a cleanup of lemmas used to run this test is needed, and it seems that removing some of the lemmas improves the execution time w/o causing any issues — that's what I'm looking into now. Update: while updating lemmas file and the test, both @JuanCoRo and I are encountering the following issue with Kontrol hanging at the end of the execution while seemingly being unable to stop the server: #375. |
PortalTest.test_withdrawal_paused
13e044a
to
b2bdfeb
Compare
I've pushed an updated test as well as contracts (OptimismPortal, Types) it's calling. The test demonstrates ~10% performance diffs between Kontrol/backend versions taking
However, the test is the current format uses a different
|
Co-authored-by: Andrei Văcaru <[email protected]>
@palinatolmach not sure, but can we get this merged? One of the next tasks for Qian is to investigate loop invariant inference using the |
* Upstream `PortalTest.test_withdrawal_paused` with lemmas, update expected output * Set Version: 0.1.160 * Set Version: 0.1.161 * Simplify `test_withdrawal_paused` to speed it up * Experimental: temp increase CI timeout * Set Version: 0.1.165 * Set Version: 0.1.172 * Set Version: 0.1.177 * Set Version: 0.1.179 * Optimism lemmas cleanup * Update simplified OptimismTest corresponding to the copying loop lemma * Set Version: 0.1.180 * Set Version: 0.1.185 * Upstreamed simplified `PortalTest` * Cleanup updated `contracts.k` and `foundry.k` * Update output for `testFail_assume_true` w/simplifications, cosmetic changes in `foundry.k` * Set Version: 0.1.188 * Update output for `test_assume_false` * Update revert message in `Portal` Co-authored-by: Andrei Văcaru <[email protected]> * Set Version: 0.1.189 * Set Version: 0.1.190 * Set Version: 0.1.192 * Remove reflexivity lemma * Update bytecode for `Portal` * Set Version: 0.1.205 * Revert `lemmas.k` renaming * Formatting in tests * Set Version: 0.1.210 --------- Co-authored-by: devops <[email protected]> Co-authored-by: Andrei Văcaru <[email protected]>
Closes #366.
This PR upstreams a simplified Optimism test PortalTest that can be used for performance profiling:
It also adds a set of lemmas needed for this test (pausability-lemmas.k), and renames an existing file with lemmas for the
sumToN
test (sum-to-n-lemmas.k).The execution of this test on my machine takes: