Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement kevm.forgetBranch cheatcode #899

Merged
merged 28 commits into from
Jan 23, 2025

Conversation

anvacaru
Copy link
Contributor

@anvacaru anvacaru commented Dec 3, 2024

blocked on: #903
blocked on: runtimeverification/kontrol-cheatcodes#15
blocked on: runtimeverification/evm-semantics#2662
blocked on: runtimeverification/k#4700

Changes in this PR:

  • a new cheatcode forgetBranch(uint256,uint8,uint256):
    • when called, should remove a constraint from the set of path constraints.
    • It puts a #forget Int Int Int production at the top of the K Cell. The #forget rule is set as a cut-rule.
  • Creating a new class, FOUNDRYSemantics(KEVMSemantics), that is inheriting the KEVMSemantics.
  • Two new functions are added to FOUNDRYSemantics:
    • _check_forget_pattern(self, cterm: CTerm) -> bool: checks if the forget cut-rule is at the top of the K Cell
    • _exec_forget_custom_step(self, cterm: CTerm, cterm_symbolic: CTermSymbolic) -> KCFGExtendResult:
      1. reads the values from the #forget K production and reconstructs the constraint.
      2. It simplifies the constraint using the CTermSymbolic.simplify endpoint and a CTerm created from an empty configuration.
      3. If the simplified constraint is part of the current path constraints, it gets removed, and a new Step is returned; otherwise, None is returned.
  • overwrites can_make_custom_step and custom_step to check and run the forget pattern.

Usage example:

function test_forgetBranch(uint256 x) public {
    uint256 y;

    vm.assume(x > 200);
    kevm.forgetBranch(x, KontrolCheatsBase.ComparisonOperator.GreaterThan, 200);
    if(x > 200){
        y = 21;
    } else {
        y = 42;
    }
}

@anvacaru anvacaru changed the title draft replace_constraint cheatcode Implement kevm.forgetBranch cheatcode Dec 4, 2024
PetarMax pushed a commit to runtimeverification/k that referenced this pull request Dec 4, 2024
Needed for runtimeverification/kontrol#899 to
expose the `simplify` endpoint in the `custom_step` function.
src/kontrol/foundry.py Outdated Show resolved Hide resolved
src/kontrol/kdist/cheatcodes.md Outdated Show resolved Hide resolved
src/kontrol/foundry.py Outdated Show resolved Hide resolved
src/kontrol/foundry.py Outdated Show resolved Hide resolved
@anvacaru anvacaru marked this pull request as ready for review December 10, 2024 13:33
@ehildenb
Copy link
Member

I think this should be implemented as a custom kontrol forget-constraint ... command, or kontrol abstract-constraint ... command, not as a cheatcode. I think as a cheatcode, teh workflow for the verification engineers is worse actually, because tehy will have to

  • Run the test the first time, and see that there is a branch they want to forget.
  • Modify the code to insert the correc branch forgetting. This requires the rather clunky feeling system of "kinda specifying the condition to forget".
  • Rebuild, and rerun the entire proof from the beginning. You don't know if your cheatcode implementation worked until ti reaches the same point it reached before, and if it does not work (you got the condition wrong), you start over completely.

As a separate custom command, it's instead looks like:

  • Run the test the first time, and see that there is a branch to forget.
  • Prune the KCFG after the branching points (very fast).
  • Run the command to remove the constraints you care about, passing in exactly the unparsed text of the constraints you want ot remove (fast and precise, no guessing).
  • Inspect the KCFG to see that it's what you want.
  • Resume the proof from taht point.

In the second approach, iterating on removing the desired constraints is a very tight loop of commands that never even need to load a haskell backend up (inspecting kcfg, pruning nodes, and forgetting constraints), and you can resume the proof directly from the spot that you left off, instead of returning all the way to the beginning to even see if your change worked.

@Stevengre
Copy link
Contributor

I think we can have both. Custom command is a debugging method with quick response, cheatcode is a fixed proof for CI?

@PetarMax
Copy link
Contributor

After having read through, I think we should definitely have both, because the separate custom command allows for quick debugging and then the Solidity-level forgetting allows for a continuous proof.

@PetarMax
Copy link
Contributor

Here are my thoughts, starting from:

  • branch forgetting is an advanced technique that should be used only if there is no other recourse

From the point of understanding and debugging whether or not removing or abstracting a constraint will work, I 100% agree with @ehildenb that we should have a separate kontrol _______ command.

From the point of understanding what are the constraints that can be removed, those are all already present and clearly visible in the Solidity code, and using a cheatcode to remove them would ideally be straightforward. However, there are two snags:

  1. The less complicated one, in which the test does not have immediate access to the parts of the state on which the branching is happening. This one is handled by writing auxiliary functions that get to that part of the state and then using that information in the test. In the test that I needed it for, all of these functions except one were already in place, as reasoning about those parts of the state was needed by the test setup.
  2. The much more complicated one, caused by the backend simplifications, which will simplify any constraint that is part of the path constraints to true. This means, for example, that if a branching on a < b occurred and if we try to pass a < b directly to the forgetBranch cheatcode, the condition will be wrapped into a bool2Word, and will simplify to true. To the best of my knowledge, there is no way to either not wrap the condition into a bool2Word as all data is Word or disallow backend simplification at this specific point. This is why we have the unfortunate and truly UX-unpleasant forgetBranch(LHS, OP, RHS). Perhaps we could have two forgetBranch functions, one with just the condition and the other with it separated, and then do a pre-processing pass on the Solidity code that would transform the former into the latter.

When it comes to custom_step using simplify, I see custom_step is a free-for-all that takes us out of executing the semantics into Python, where we might as well use whatever is available to us. Having said that, it is much more complicated than the other custom steps that we've had so far.

@PetarMax PetarMax requested review from palinatolmach and ehildenb and removed request for PetarMax January 5, 2025 12:32
@anvacaru anvacaru requested a review from PetarMax January 23, 2025 10:58
Copy link
Contributor

@lucasmt lucasmt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cheatcode seems to be working on our tests, so I'm approving this and we can revisit it later to revise the implementation/add a command-line version.

@automergerpr-permission-manager automergerpr-permission-manager bot merged commit 7d496d3 into master Jan 23, 2025
12 checks passed
@automergerpr-permission-manager automergerpr-permission-manager bot deleted the abstraction-cheatcodes branch January 23, 2025 13:41
ACassimiro added a commit that referenced this pull request Jan 24, 2025
* Implement kevm.forgetBranch cheatcode (#899)

* draft FOUNDRYSemantics

* forgetBranch

* add mlEqualsTrue

* minor corrections

* add simplification step

* formatting

* add back not equal

* rename FOUNDRYSemantics to KontrolSemantics

* checking for negation as well

* correcting indentation

* expanding functionality

* heuristic simplifications

* further refinement

* refactoring _exec_forget_custom_step

* add show test

* fix test

* update expected output

---------

Co-authored-by: Petar Maksimovic <[email protected]>
Co-authored-by: Palina <[email protected]>
Co-authored-by: Petar Maksimović <[email protected]>

* Update dependency: deps/kevm_release (#940)

* deps/kevm_release: Set Version 1.0.780

* Sync Poetry files: kevm-pyk version 1.0.780

* flake.{nix,lock}: update Nix derivations

---------

Co-authored-by: devops <[email protected]>

* Build Custom Kontrol Versions using a workflow (#941)

* new file:   .github/workflows/kup-build-kontrol.yml
- Adding a new feature to build kontrol images using kup and --overrides

* Update .github/workflows/kup-build-kontrol.yml

Co-authored-by: Everett Hildenbrandt <[email protected]>

* modified:   .github/workflows/docker-push.yml
modified:   .github/workflows/kup-build-kontrol.yml
- Standardizing the naming between these two workflows to know which is
  for which
- Docker-push builds a custom version of kontrol with fixed dependencies
  already published and built with kontrol
- kup-build-kontrol.yml will build the existing kontrol code with a
  single new
  dependency
modified:   README.md
- Adding some readme instructions.

* renamed:    .github/workflows/docker-push.yml -> .github/workflows/kontrol-push-fixed-deps.yml
renamed:    .github/workflows/kup-build-kontrol.yml -> .github/workflows/kontrol-push-unfixed-deps.yml
modified:   README.md
- Add some background between the two kontrol build workflows

* modified:   README.md
-- Adding content to explain the two custom kontrol build workflows

* Update README.md

Co-authored-by: Anton Savienko <[email protected]>

* Filling in the 'get' placeholder with functionality to fetch appropriate
defaults when nothing is provided

* modified:   README.md
- Adding instructions to fetch hash and use the wrofklow with multiple
  dep options now available

* Update spelling and gramar

* modified:   .github/workflows/kontrol-push-unfixed-deps.yml
- Fixing reporting of final versions used to build kontrol.
- Removing dfining override for undefined inputs to the workflow

---------

Co-authored-by: Everett Hildenbrandt <[email protected]>
Co-authored-by: Anton Savienko <[email protected]>

---------

Co-authored-by: Andrei Văcaru <[email protected]>
Co-authored-by: Petar Maksimovic <[email protected]>
Co-authored-by: Palina <[email protected]>
Co-authored-by: Petar Maksimović <[email protected]>
Co-authored-by: rv-jenkins <[email protected]>
Co-authored-by: devops <[email protected]>
Co-authored-by: Freeman <[email protected]>
Co-authored-by: Everett Hildenbrandt <[email protected]>
Co-authored-by: Anton Savienko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants