Sync #166
Sync #166
Conversation
Make every request in batch, to save round trip times Fetch timestamp of blockheader to populate timestamp field in transaction Remove listunspent requests because we can compute it from our history
|
PS. Today I am seeing a big swing in esplora electrum server performance, previous results are just one run from yesterday (without considering address derivation time) |
|
I did a commit for almost any feedback for easier review, let me know if you prefer them squashed |
That's fine, I think we can keep them like this |
src/blockchain/utils.rs
Outdated
| for el in flattened { | ||
| // el.height = -1 means unconfirmed with unconfirmed parents | ||
| // el.height = 0 means unconfirmed with confirmed parents | ||
| // but we threat those tx the same |
|
|
||
| let mut wallet_chains = vec![ScriptType::Internal, ScriptType::External]; | ||
| // shuffling improve privacy, the server doesn't know my first request is from my internal or external addresses | ||
| wallet_chains.shuffle(&mut thread_rng()); |
There was a problem hiding this comment.
Would it be better to get all the script pubkeys first and shuffle those? Seems like that would mix internal and external better, but it might also mix different "chunks" of the script pubkeys together making the wallet easier to identify as a single cluster?
There was a problem hiding this comment.
I also had the same idea, it would be nice but it makes it very hard to stop when a gap large enough has been reached, because everything is mixed up. You would probably end up making many more requests than necessary
There was a problem hiding this comment.
yes, I think it would require more requests, especially in case where there are much more internal addresses than external (or viceversa)
There was a problem hiding this comment.
I opened #176 to keep track of this idea, we could potentially implement it with a flag that switches between a "fast" and a "private" sync algorithm. For now I'll merge this and the improvements can come with future PRs.
The new API endpoint is now deployed. https://github.com/Blockstream/esplora/blob/master/API.md#get-blockhashheader |
Update ToPublicKey API to take in context param
This PR contains:
check_tx_and_descendant. The rationale of the current algorithm is that internal address contains funds only in our transactions so you don't initially ask about them to the electrum server but check those in transactions retrieved from external addresses and only if present asks for spending tx. Doing so is problematic in case we have a deep change chain because these transactions must be asked serially. The algorithm in this PR simply asks for internal address as it's done for external so we can batch requests more, improving speed.listunspentcalls: closes Change electrum client to useget_historyinstead oflistunspent#140, spent information is in our history, so we can avoid to ask this endpoint.replexample can use esplora so that testing this blockchain source is easierResults:
Future improvements:
max_addressesin electrum-like syncing mechanism: change cache addresses mechanism to calculate the addresses every team previous stop_gap batch contains txs (max_addressescould still be useful for parallelism in blockchain source like compact filters)change_deep wallet:
single_many_utxos wallet: