[WIP] implement subtree-based SMT computations #341
Conversation
Qyriad
force-pushed
the
qyriad/parallel-construction
branch
from
0249e63 to
ebc975f
Compare
|
This should now be ready for review! |
There was a problem hiding this comment.
Thank you! Looks good! This is a pretty superficial review as I'm currently traveling.
I left a couple of comments inline. The main one is that this crate should work in no_std context, and so we should make rayon an optional dependency behind a feature flag. This would also imply that rather than having separate functions for parallel and sequential execution, we should have just single versions which would contain parallelized code when concurrent feature is turned on.
Also, could you share the overall results of these optimizations? (i.e., for constructing trees of 2^16 and 2^20 key-value pairs)?
|
Benchmark comparison for 2^16 (65,536) leaves on my Ryzen 7950X: Benchmark comparison for 2^20 (1,048,576) leaves: I also have a graph! The green line is the singlethreaded version and the purple line is the multithreaded version. |
This will be composed into depth-8-subtree-based computation of entire sparse Merkle trees.
This is intended for comparison with the benchmarks from the previous commit. This benchmark represents the theoretical perfect-efficiency performance we could possibly (but impractically) get for computing depth-8 sparse Merkle subtrees.
This commit ensures that `SparseMerkleTree::build_subtree()` can correctly compose into building an entire sparse Merkle tree, without yet getting into potential complications concurrency introduces.
Building on the previous commit, this commit implements a test proving that `SparseMerkleTree::build_subtree()` can be composed into itself not just concurrently, but in parallel, without issue.
This commit adds a new required method to the SparseMerkleTree trait, to allow generic construction from pre-computed parts. This will be used to add a generic version of `with_entries()` in a later commit.
What the previous few commits have been leading up to: SparseMerkleTree now has a function to construct the tree from existing data in parallel. This is significantly faster than the singlethreaded equivalent. Benchmarks incoming!
Qyriad
force-pushed
the
qyriad/parallel-construction
branch
from
ebc975f to
3f52ef3
Compare
Very cool! And also, AFAIK, Ryzen 7950X has only 16 cores - so, we are getting nearly perfect multi-threaded performance. |
krushimir
force-pushed
the
qyriad/parallel-construction
branch
from
58fa0a0 to
c9b4682
Compare
There was a problem hiding this comment.
Thank you! Still not a full review, but I left some more comments inline.
Also, what do you think about #341 (comment) ?
krushimir
force-pushed
the
qyriad/parallel-construction
branch
from
8b3c1bf to
6d93c0d
Compare
There was a problem hiding this comment.
Looks good! Thank you! I left a couple small comments inline. After these are addressed, we should be good to merge.
Also, we should update the bench command in the Makefile (i.e., here) to run with the concurrent feature enabled.
|
@krushimir - seems like there are some conflicts with the |
# Conflicts: # Cargo.toml
|
@krushimir - seems like some lint checks are failing. Could you fix these? (running |
|
Describe your changes
This is the working state for using the methods benchmarked in #334 to make the computations necessary to construct a new sparse Merkle tree from existing data, by splitting the work across fixed-size subtrees.
This is not ready to be merged, but does currently have passing tests for a singlethreaded version of the desired functionality.