Support fetching witnesses via eth-tx-proof

[pgebheim]

Currently zero-bin allows fetching block trace data from a Erigon instance with the proper witness APIs as well as loading this trace data from a file and kicking off proofs.

To enable easier integration testing, as well as to give users a nice way to target chains which do not yet have the proper trace APIs, we would like to integrate the code from eth-tx-proof into zero-bin such that we can fetch information from an existing RPC node and kick off the proving process for a block or transaction.

[Nashtare]

@cpubot Will you be tackling this in the near future? Someone from my team or myself can take over if you're busy, just let me know! :)

[cpubot]

@Nashtare unsure -- but probably not. I'm on Agglayer stuff at the moment. @pgebheim might have a better sense

[pgebheim]

Yeah if someone from your side needs it @Nashtare feel free to grab this.

[frisitano]

I would be happy to pick this up if no one has already been assigned? In the future we plan to build a tracer integration for reth so I think this would be a good introductory issue to pick up. I would be doing this work on behalf of Fractal who intends to use the Type 1 EVM in the future.

Let me know what you think @Nashtare @pgebheim.

[Nashtare]

@frisitano Thank you for offering to tackle this, that'd be great! Note that eth-tx-proof is currently being refactored (including a lot of common logic with zero-bin) to support padding / withdrawals logic, see 0xPolygonZero/eth-tx-proof#15. Probably best to base off the work from this branch (should be merged soon).

[frisitano]

@Nashtare Sure thing, I've made a start on this.

[BGluth]

Ok, just wanting to let you know that the refactor PR mentioned above (#15) is merged back in.

[frisitano]

Thanks for the heads up @BGluth. I've got an initial implementation completed. I need to refactor and complete some integration tests for more cases but simple transfers and empty blocks are proving fine.

Here's the PR I'm working on: #51

[frisitano]

I now have #51 working on blocks produced by erigon devnet including smart contract interactions but for mainnet blocks I run into a KernelPanic error. I also experience similar KernelPanic errors when trying to prove mainnet blocks with eth-tx-proof. Do we have some specific test vectors / mainnet blocks that I can use to debug this issue?

registers: RegistersState { program_counter: 60095, is_kernel: true, stack_len: 3, stack_top: 2, is_stack_top_read: true, check_overflow: false, context: 0, gas_used: 59816 }
  �[2m2024-04-15T21:10:00.894079Z�[0m �[33m WARN�[0m �[1;33mevm_arithmetization::witness::transition�[0m�[33m: �[33mKernel panic at panic�[0m
    �[2;3mat�[0m /Users/f/dev/fractal/zk_evm/evm_arithmetization/src/witness/transition.rs:138

Error: Fatal operation error: KernelPanic in kernel at pc=panic, stack=[4, 2, 4], memory=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 195, 61, 143, 155, 72, 89, 183, 206, 130, 103, 197, 246, 235, 21, 38, 177, 110, 207, 8, 99, 224, 22, 211, 51, 246, 192, 177, 251, 253, 206, 59, 63, 34, 5, 39, 152, 214, 38, 1, 28, 241, 38, 102, 231, 194, 6, 63, 152, 28, 189, 32, 69]

[Nashtare]

@frisitano Which blocks are you trying to prove? The current zk_evm is Shanghai compatible, but does not support Cancun yet (it is implemented, and in process of debugging at the moment).

[frisitano]

@Nashtare I'm proving block number 17735424 which is a Shanghai block. I'm using an alchemy rpc to source the witness data for both eth-tx-proof and zero-bin, both of which are resulting in kernel panic. Do you have any suggestions of how I can debug the panic?

[Nashtare]

This is quite specific to the internals of the zk-evm arithmetization, so may be tough to dive into if you're not familiar with how the whole kernel logic behaves. I can try looking at it a bit more later today

[Nashtare]

@frisitano Just FYI, we're experiencing some bugs on eth-tx-proof side that @BGluth and I are investigating at the moment. zero-bin should be fine though, so the kernel panic may be coming from somewhere else. Would you mind sharing me the logs you obtained with it? (you may need to turn on RUST_LOG=evm_arithmetization=trace).

[frisitano]

@Nashtare Certainly. Please find the logs attached.

Command:

RUST_LOG=evm_arithmetization=trace ./target/release/leader --arithmetic 15..28 --byte-packing 9..28 --cpu 12..28 --keccak 14..28 --keccak-sponge 9..28 --logic 12..28 --memory 17..30  --runtime in-memory -n 1 native --rpc-url REDACTED --block-number 17735424 --proof-output-path native.json --checkpoint-block-number 17735423

zero-bin.logs.zip

[Nashtare]

I am confused, the transaction hash seems to be 0xc106edf40809d27dbaf88d7c305668f7fb2d03f236aeecf90481d328721b059c (this is the hash used during ECRECOVER signature verification, which passes successfully, but this doesn't appear to be in block 17735424 (or any block FWIW)... Are you targeting Ethereum mainnet?

This can be seen in this line (top of the stack is the txn hash)

Cycle 116868, ctx=0, pc=ecrecover, instruction=Dup(1), stack=[87308623151916798375216749502438337832795253495607303634714898271170639627676, 28, 56296191302052192061477373449743179050685448427087360131414109079395239272403, 3825636757570036313773666492659193980943799342832065023943496369727278541227, 60042, 45179, 0, 128, 2, 1]

EDIT: Nevermind, still looking into the txn failure.

[Nashtare]

The mismatch is actually surprising. The memory cell that should contain the cumulative gas used post txn execution, i.e. typically the gas used by this txn as it is the first one, is instead set with the txn gas limit. The gas used is correctly computed within the kernel.

Would you happen to have your GenerationInputs corresponding to this txn? More specifically, the value of inputs.gas_used_after.

[frisitano]

tx number before: 0
tx gas used after 365576

  �[2m2024-04-17T19:35:03.479761Z�[0m �[31mERROR�[0m �[1;31mpaladin::runtime�[0m�[31m: �[31mexecution error: Fatal { err: KernelPanic in kernel at pc=panic, stack=[1, 2, 1], memory=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 43, 112, 112, 18, 139, 144, 32, 231, 122, 76, 3, 111, 179, 111, 64, 224, 119, 249, 119, 106, 169, 109, 143, 94, 251, 152, 166, 125, 134, 54, 248, 24, 49, 229, 91, 22, 146, 213, 89, 153, 216, 199, 43, 213, 157, 248, 133, 179, 86, 123, 45, 119], strategy: Terminate }, �[1;31mrouting_key�[0m�[31m: 5f23c8b763784f6aae9bceec611b604f�[0m
    �[2;3mat�[0m /Users/f/.cargo/registry/src/index.crates.io-6f17d22bba15001f/paladin-core-0.4.2/src/runtime/mod.rs:727

Error: Fatal operation error: KernelPanic in kernel at pc=panic, stack=[1, 2, 1], memory=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 43, 112, 112, 18, 139, 144, 32, 231, 122, 76, 3, 111, 179, 111, 64, 224, 119, 249, 119, 106, 169, 109, 143, 94, 251, 152, 166, 125, 134, 54, 248, 24, 49, 229, 91, 22, 146, 213, 89, 153, 216, 199, 43, 213, 157, 248, 133, 179, 86, 123, 45, 119]

In regards to versions:
zero-bin - branched from commit hash 60f258d49106c2356536a90588820d9719cd24ea of develop
zk_evm - branched from HEAD of develop

[Nashtare]

Hmm interesting, tx gas used after should be 219,133 according to this. I'm curious about this parsing issue, eth-tx-proof gives me the valid value (0x357fd = 219133) and we never faced something like so on zero-bin side. 🤔

[frisitano]

Good catch. I can confirm the transaction receipt object returned by the rpc has the correct gas value that you quoted. I'll investigate and locate where this discrepancy arises and report back.

[frisitano]

@Nashtare the erroneous value was associated with bad mapping on the adapter I built. I've fixed it and that is no longer an issue.

However, now I am experiencing the following issue which I believe is related to state root consistency checks at the end of the kernel. See log extract below:

2024-04-18T14:42:52.693963Z TRACE evm_arithmetization::witness::transition: Cycle 546361, ctx=0, pc=49193, below mpt_hash, instruction=Push(3), stack=[0, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 32, 3724, 45259]    
2024-04-18T14:42:52.693968Z TRACE evm_arithmetization::witness::transition: Cycle 546362, ctx=0, pc=49197, below mpt_hash, instruction=Jumpi, stack=[49203, 0, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 32, 3724, 45259]    
2024-04-18T14:42:52.693974Z TRACE evm_arithmetization::witness::transition: Cycle 546363, ctx=0, pc=49198, below mpt_hash, instruction=Swap(1), stack=[6938084400579631288715715963269591884260219133622291964305563008489015499207, 32, 3724, 45259]    
2024-04-18T14:42:52.693979Z TRACE evm_arithmetization::witness::transition: Cycle 546364, ctx=0, pc=49199, below mpt_hash, instruction=Swap(2), stack=[3724, 32, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 45259]    
2024-04-18T14:42:52.693984Z TRACE evm_arithmetization::witness::transition: Cycle 546365, ctx=0, pc=49200, below mpt_hash, instruction=Swap(0), stack=[45259, 32, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 3724]    
2024-04-18T14:42:52.693989Z TRACE evm_arithmetization::witness::transition: Cycle 546366, ctx=0, pc=49201, below mpt_hash, instruction=Pop, stack=[32, 45259, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 3724]    
2024-04-18T14:42:52.693994Z TRACE evm_arithmetization::witness::transition: Cycle 546367, ctx=0, pc=49202, below mpt_hash, instruction=Jump, stack=[45259, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 3724]    
2024-04-18T14:42:52.694000Z TRACE evm_arithmetization::witness::transition: Cycle 546368, ctx=0, pc=45259, below check_state_trie, instruction=Push(5), stack=[6938084400579631288715715963269591884260219133622291964305563008489015499207, 3724]    
2024-04-18T14:42:52.694005Z TRACE evm_arithmetization::witness::transition: Cycle 546369, ctx=0, pc=45265, below check_state_trie, instruction=MloadGeneral, stack=[21474836490, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 3724]    
2024-04-18T14:42:52.694011Z TRACE evm_arithmetization::witness::transition: Cycle 546370, ctx=0, pc=45266, below check_state_trie, instruction=BinaryArithmetic(Sub), stack=[61333081217480535940718494550699021384310283736156928076268435607788798060026, 6938084400579631288715715963269591884260219133622291964305563008489015499207, 3724]    
2024-04-18T14:42:52.694017Z TRACE evm_arithmetization::witness::transition: Cycle 546371, ctx=0, pc=45267, below check_state_trie, instruction=Push(3), stack=[54394996816900904652002778587429429500050064602534636111962872599299782560819, 3724]    
2024-04-18T14:42:52.694022Z TRACE evm_arithmetization::witness::transition: Cycle 546372, ctx=0, pc=45271, below check_state_trie, instruction=Jumpi, stack=[60095, 54394996816900904652002778587429429500050064602534636111962872599299782560819, 3724]    
2024-04-18T14:42:52.694033Z  WARN evm_arithmetization::witness::transition: Kernel panic at panic    
Error: Fatal operation error: KernelPanic in kernel at pc=panic, stack=[3724], memory=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 63, 201, 26, 58, 253, 112, 57, 92, 212, 150, 198, 71, 213, 166, 204, 157, 75, 43, 127, 173, 104, 210, 92, 16, 189, 45, 210, 5, 86, 52, 23, 176, 8, 86, 15, 35, 139, 122, 149, 68, 114, 55, 252, 239, 85, 231, 33, 100, 5, 63, 193, 3]

I've compared the transaction traces for this transaction with the state diff here and the data looks consistent so I'm not sure why it would yield a different state root.

[frisitano]

I've attached the GenerationInputs for the transaction below:

inputs.json

[frisitano]

I fixed issues related to gas and storage slot deletions. I'm working through the transactions iteratively and fixing bugs as I go. Will keep this issue updated with progress.

[frisitano]

After some changes (I still need to push them) I have been able to successfully prove block 17735424. I'm currently using a branch from 'develop' for the prover but it looks like zero-bin currently uses v0.3.0. Are we happy to change zero-bin to use develop or how would we like to proceed? The required changes to the prover can be seen here - https://github.com/0xPolygonZero/zk_evm/pull/157/files.

[Nashtare]

Awesome!

As for zero-bin deps, we only use released versions on develop (note that @BGluth will be yanking today trace_decoder v0.3.0 and publish the fix). We usually test things on zero-bin@testing which is regularly updated with latest zk_evm@develop branch.

[Nashtare]

Done with #81