feat: improve block hashes fetching #62

vladimir-trifonov · 2024-04-19T11:59:26Z

feat: improve block hashes fetching

Issue: 0xPolygonZero/zk_evm#297

leader/src/cli.rs

leader/src/jerigon.rs

tools/prove_blocks.sh

leader/src/cli.rs

rpc/src/rpc.rs

Nashtare · 2024-04-23T20:28:45Z

tools/prove_blocks.sh

+if [ $retVal -ne 0 ]; then
+    # Some error occured.
+    echo "Blocks $1..=$2 errored. See ${OUT_LOG_PATH} for more details."
+    exit $retVal


I think we shouldn't abort early in case we're proving a range. The error log should be enough while we pursue proving other blocks.

I don't think it is feasible with the current implementation of jerigon_main since it exits when block proving fails.

Nashtare

LGTM!

vladimir-trifonov · 2024-04-25T08:21:25Z

@praetoriansentry simple_test.sh is failing for this PR, do you know what could be the reason... I try to run it locally and witness.json.bz2 has zero size.

praetoriansentry · 2024-04-25T12:13:10Z

@praetoriansentry simple_test.sh is failing for this PR, do you know what could be the reason... I try to run it locally and witness.json.bz2 has zero size.

I would guess there was some issue with pulling the witness temparily. I re-ran and it seems ok for now:

Nashtare · 2024-04-25T12:55:02Z

Curious about the double log Proof verified successfully 👀 (though orthogonal to this PR)

vladimir-trifonov · 2024-04-26T13:54:57Z

Curious about the double log Proof verified successfully 👀 (though orthogonal to this PR)

can't make it pass locally, maybe because of mac m1, and figure it out why is that :(

Nashtare · 2024-04-26T14:08:41Z

@vladimir-trifonov I think you just need to change nproc, it's not supported on Apple chips (cf my comment in #69)

vladimir-trifonov · 2024-05-07T06:40:12Z

@vladimir-trifonov I think you just need to change nproc, it's not supported on Apple chips (cf my comment in #69)

nproc returns 8 on my Apple M1 machine, so I think it works fine, but the simple_test.sh is still failing locally.

Nashtare · 2024-05-07T12:17:15Z

What's your error?

vladimir-trifonov · 2024-05-07T12:51:13Z

What's your error?

 ./simple_test.sh                        
Pulling sample witness

bunzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bunzip2: No such file or directory
        Input file = witness.json.bz2, output file = witness.json

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

bunzip2: Deleting output file witness.json, if it exists.
   Compiling rpc v0.1.0 (/Users/vladimirtrifonov/src/rnd/polygon-zkevm/zero-bin/rpc)
   Compiling leader v0.1.0 (/Users/vladimirtrifonov/src/rnd/polygon-zkevm/zero-bin/leader)
    Finished release [optimized] target(s) in 2m 47s
./simple_test.sh: line 63: witness.json: No such file or directory
Error: EOF while parsing a value at line 1 column 0

Stack backtrace:
   0: std::backtrace::Backtrace::capture
   1: verifier::main
   2: std::sys_common::backtrace::__rust_begin_short_backtrace
   3: _main
there was an issue with proof verification

it looks like it is mac specific, or the way it saves the data and reads it, I'm looking in it...

vladimir-trifonov · 2024-05-08T10:56:01Z

@Nashtare I managed to run it locally and this is the output:

2024-05-08T10:54:35.809263Z  INFO common::prover_state: initializing verifier state...
2024-05-08T10:54:35.809281Z  INFO common::prover_state: attempting to load preprocessed verifier circuit from disk...
2024-05-08T10:54:35.809677Z  INFO common::prover_state: successfully loaded preprocessed verifier circuit from disk
2024-05-08T10:54:35.814027Z  INFO verifier: Proof verified successfully!

INFO verifier: Proof verified successfully! exists just once...

Nashtare · 2024-05-08T11:12:20Z

INFO verifier: Proof verified successfully! exists just once...

What do you mean by this? The output looks fine to me

vladimir-trifonov · 2024-05-08T12:28:57Z

INFO verifier: Proof verified successfully! exists just once...

What do you mean by this? The output looks fine to me

I meant that because of:

Curious about the double log Proof verified successfully 👀 (though orthogonal to this PR)

I saw that it is doubled in the github output, but locally it is not.

Nashtare · 2024-05-08T12:32:00Z

Ah sorry I had forgotten about this old comment. Hmm yeah maybe logging is a bit messed up on the CI. Not a big deal probably!

vladimir-trifonov · 2024-05-31T08:15:51Z

@Nashtare @BGluth do you guys think this PR is still relevant, after looking into #78? cc: @atanmarko

atanmarko · 2024-05-31T08:18:25Z

@Nashtare @BGluth do you guys think this PR is still relevant, after looking into #78? cc: @atanmarko

CC: @muursh, @cpubot

Also taking into account #86 maybe we need more general approach to block ranges

atanmarko · 2024-06-13T18:16:29Z

Multi block proving is covered with #90. Block hash retrieval optimization needs to be ported to alloy

Nashtare · 2024-06-13T18:21:40Z

EDIT: I wrote this while you were creating the issue, all good then :)

@atanmarko The issue is different here. It was about reducing the number of RPC calls to get the 256 previous hashes needed for a block payload. This isn't tackled by #90, which calls block_prover_input() for every single block in the sequence, while once we fetched the initial 256 previous block hashes, we only need one more RPC call per following block, instead of the current 128. But this is a minor optimization.

atanmarko · 2024-06-13T19:08:11Z

EDIT: I wrote this while you were creating the issue, all good then :)

@atanmarko The issue is different here. It was about reducing the number of RPC calls to get the 256 previous hashes needed for a block payload. This isn't tackled by #90, which calls block_prover_input() for every single block in the sequence, while once we fetched the initial 256 previous block hashes, we only need one more RPC call per following block, instead of the current 128. But this is a minor optimization.

@Nashtare Right, this PR also introduced block ranges, that part has different implementation with #90. Hash retrial optimization is yet to be implemented(ported)

feat: improve block hashes fetching

13a6f30

vladimir-trifonov requested review from cpubot, muursh and Nashtare as code owners April 19, 2024 11:59

lastminutedev added 3 commits April 19, 2024 15:04

fix: fix clippy issues

15b6d20

fix: fix clippy issue

d36fe2e

fix: update log file filename

4bb38c2

BGluth reviewed Apr 19, 2024

View reviewed changes

leader/src/cli.rs Outdated Show resolved Hide resolved

lastminutedev added 5 commits April 22, 2024 11:04

fix: update block-numbers param

4825c8d

fix: fix docs and clippy issue

a553fb0

fix: fix docs

b209a0d

fix: fix prove_blocks.sh

6ffc72e

fix: fix prove_blocks.sh

2c295c0

vladimir-trifonov requested a review from BGluth April 22, 2024 08:27

BGluth approved these changes Apr 22, 2024

View reviewed changes

leader/src/jerigon.rs Outdated Show resolved Hide resolved

tools/prove_blocks.sh Show resolved Hide resolved

fix: use block_range

57c9031

vladimir-trifonov requested a review from BGluth April 23, 2024 06:53

BGluth approved these changes Apr 23, 2024

View reviewed changes

Nashtare reviewed Apr 23, 2024

View reviewed changes

lastminutedev added 2 commits April 24, 2024 13:35

fix: fix pr comments

f12ecc6

fix: fix pr comment

27d5280

vladimir-trifonov requested a review from Nashtare April 24, 2024 13:39

fix: fix clippy issue

6e62876

Nashtare approved these changes Apr 24, 2024

View reviewed changes

lastminutedev added 3 commits April 25, 2024 10:17

Merge branch 'develop' into improve_block_hashes_fetching

e3d57f2

fix: fix clippy issue

e703944

fix: fix merge issue

f74fa31

Nashtare mentioned this pull request May 28, 2024

chore: remove duplicate print in script and lighten circuit loading #85

Merged

cpubot mentioned this pull request May 30, 2024

Add support for block intervals #86

Closed

vgnom added this to the Cancun - Q2 2024 milestone Jun 11, 2024

atanmarko closed this Jun 13, 2024

atanmarko mentioned this pull request Jun 13, 2024

Port hash retrieval optimization to alloy 0xPolygonZero/zk_evm#282

Closed

atanmarko mentioned this pull request Jun 25, 2024

Cache prev_hashes input prover data 0xPolygonZero/zk_evm#325

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: improve block hashes fetching #62

feat: improve block hashes fetching #62

vladimir-trifonov commented Apr 19, 2024

Nashtare Apr 23, 2024

vladimir-trifonov Apr 24, 2024 •

edited

Loading

Nashtare left a comment

vladimir-trifonov commented Apr 25, 2024

praetoriansentry commented Apr 25, 2024

Nashtare commented Apr 25, 2024

vladimir-trifonov commented Apr 26, 2024

Nashtare commented Apr 26, 2024

vladimir-trifonov commented May 7, 2024

Nashtare commented May 7, 2024

vladimir-trifonov commented May 7, 2024

vladimir-trifonov commented May 8, 2024

Nashtare commented May 8, 2024

vladimir-trifonov commented May 8, 2024

Nashtare commented May 8, 2024

vladimir-trifonov commented May 31, 2024 •

edited

Loading

atanmarko commented May 31, 2024 •

edited

Loading

atanmarko commented Jun 13, 2024 •

edited

Loading

Nashtare commented Jun 13, 2024 •

edited

Loading

atanmarko commented Jun 13, 2024 •

edited

Loading

feat: improve block hashes fetching #62

feat: improve block hashes fetching #62

Conversation

vladimir-trifonov commented Apr 19, 2024

Nashtare Apr 23, 2024

Choose a reason for hiding this comment

vladimir-trifonov Apr 24, 2024 • edited Loading

Choose a reason for hiding this comment

Nashtare left a comment

Choose a reason for hiding this comment

vladimir-trifonov commented Apr 25, 2024

praetoriansentry commented Apr 25, 2024

Nashtare commented Apr 25, 2024

vladimir-trifonov commented Apr 26, 2024

Nashtare commented Apr 26, 2024

vladimir-trifonov commented May 7, 2024

Nashtare commented May 7, 2024

vladimir-trifonov commented May 7, 2024

vladimir-trifonov commented May 8, 2024

Nashtare commented May 8, 2024

vladimir-trifonov commented May 8, 2024

Nashtare commented May 8, 2024

vladimir-trifonov commented May 31, 2024 • edited Loading

atanmarko commented May 31, 2024 • edited Loading

atanmarko commented Jun 13, 2024 • edited Loading

Nashtare commented Jun 13, 2024 • edited Loading

atanmarko commented Jun 13, 2024 • edited Loading

vladimir-trifonov Apr 24, 2024 •

edited

Loading

vladimir-trifonov commented May 31, 2024 •

edited

Loading

atanmarko commented May 31, 2024 •

edited

Loading

atanmarko commented Jun 13, 2024 •

edited

Loading

Nashtare commented Jun 13, 2024 •

edited

Loading

atanmarko commented Jun 13, 2024 •

edited

Loading