Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix --early-exit #437

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

fix --early-exit #437

wants to merge 22 commits into from

Conversation

karmacoma-eth
Copy link
Collaborator

fixes #243

high-level notes:

  • the thread pool we were using during the assertion solving process did not allow us to interrupt work that was already started (thread_pool.shutdown(wait=False) was not what we needed it to be)
  • in order to be interruptible, we now run every solver instance as an external process (an actual subprocess.Popen object, wrapped for convenience in PopenFuture and scheduled with PopenExecutor)
  • when we call PopenExecutor.shutdown(wait=False), it does terminate/kill every process that has been submitted to that executor and is still running
  • an executor is scoped to a test function, so that a successful counterexample found for a path in a test function cancels the other solver instances for that test, but not for other tests that may be running concurrently
  • in order to support refinement, we still submit solve_end_to_end function calls to a thread pool, but that functions submits solve_low_level calls to the executor and blocks on the result. solve_low_level can be interrupted if the underlying process returns, times out, gets killed, etc -- this is the key architectural change
  • as a result, we no longer invoke the built-in z3 solve and every solver invocation is now an external solver invocation (hence --solver-command defaults to z3) and we need to parse the model values from the smtlib output. This means we no longer print Counterexample: see xyz.smt2

Additionally, to make the transition to this model easier I introduced a hierarchy of context objects:

  • ContractContext
  • FunctionContext
  • PathContext
  • each context has the associated args (HalmosConfig) with the relevant scoped overrides and intermediary values needed

@karmacoma-eth
Copy link
Collaborator Author

Things I checked:

  • works as expected without an explicit --solver-command (invokes an external z3 process)
  • works when passing an explicit --solver-command (caveat: bitwuzla and cvc require --produce-models, yices requires --smt2-model-format)
  • refinement works
  • timeouts work
  • we try to avoid leaving orphaned solvers behind, especially during timeouts
  • --early-exit really does print the first counterexample and cancels the other processes

Things I still need to check:

  • Windows support (I think --solver-command was never supported on Windows, but we now always require it)
  • ctrl-c behavior is likely wonky (it always was, but it should be fixable now, maybe as an additional improvement later)

@@ -0,0 +1,174 @@
import json
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no changes here, just moving build-related functionality out of __main__.py and into its own module

@@ -0,0 +1,160 @@
import io
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no changes here, just moving trace-related functionality out of __main__.py and into its own module

@@ -0,0 +1,217 @@
import concurrent.futures
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new module that provides an executor/futures abstraction layer built on top of subprocess.Popen

return SolverOutput.from_result(stdout, stderr, returncode, path_ctx)


def solve_end_to_end(ctx: PathContext) -> None:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used to be gen_model_from_sexpr

@dataclass(frozen=True)
class PathContext:
# id of this path
path_id: int
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

roughly corresponds to the old iteration idx


refined_ctx = ctx.refine()

if refined_ctx.query.smtlib != query.smtlib:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

related: #279

@@ -1411,6 +817,8 @@ def _main(_args=None) -> MainResult:
#

def on_exit(exitcode: int) -> MainResult:
ExecutorRegistry().shutdown_all()
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what's going on, this sounds like a safe thing to do (shut down external processes) but when I do this I can somewhat consistently get the following crash when triggering a ctrl-c at the right time:

halmos --root ~/projects/halmos-sandbox --function test_manyCexes --solver-command "yices-smt2 --smt2-model-format" --early-exit                                                         ✘ INT took 4s
[⠢] Compiling...
No files changed, compilation skipped

Running 1 tests for test/60_manyCexes.t.sol:Test60
setup: 0.01s (decode: 0.01s)
Generating SMT queries in /tmp/test_manyCexes-2459149073354744aa4b653b5fb94c9f
^C
Shutting down all executors
ASSERTION VIOLATION
File: ../src/ast/ast.cpp
Line: 406
UNEXPECTED CODE WAS REACHED.
Z3 4.12.6.0
Please file an issue with this message and more detail about how you encountered it at https://github.com/Z3Prover/z3/issues/new

(note that the interrupted assertion solver is yices, not z3 -- I don't understand why z3 is involved in this)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

--early-exit doesn't work as expected
1 participant