Parallelize process_inboxes_and_force_validator_updates #3245

ndr-ds · 2025-02-04T20:55:34Z

Motivation

Will start paralellizing some parts of this existing flow

Proposal

Paralellize initial validator sync step

Test Plan

CI + ran locally

Release Plan

Nothing to do / These changes follow the usual release cycle.

ndr-ds · 2025-02-04T20:55:49Z

supply_fungible_tokens improvements #3247 : 2 dependent PRs (#3248 , #3249 )
make_benchmark_chains improvements #3246
Parallelize process_inboxes_and_force_validator_updates #3245 👈 (View in Graphite)
Remove max_in_flight from linera benchmark #3235
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

afck · 2025-02-05T09:30:54Z

linera-client/src/client_context.rs

+        let (listener, _listen_handle, notification_stream) = chain_client.listen().await?;
+        self.chain_listeners.spawn_task(listener);
+        Ok(notification_stream)
+    }


Do these need to be pub? And shouldn't they be in the #[cfg(feature = "benchmark")] impl below?

afck · 2025-02-05T09:31:30Z

linera-client/src/client_context.rs

+            result?
+        };
+        certificates.extend(new_certificates);
+        if maybe_timeout.is_none() {


Can this ever be Some in the benchmark? Aren't these all single-user chains, where we expect the first proposal to succeed?

Good point! They're all single super owner chains. This should simplify the code quite a bit :)

afck · 2025-02-05T09:33:34Z

linera-client/src/client_context.rs

+            })
+            .collect::<Vec<_>>();
+
+        let mut chain_client_and_notification_stream = Vec::new();


Suggested change

let mut chain_client_and_notification_stream = Vec::new();

let mut chain_clients_and_notification_streams = Vec::new();

ndr-ds · 2025-02-05T18:49:44Z

This one is significantly faster, 850ms in parallel vs 3500ms serially

linera-client/src/client_context.rs

afck · 2025-02-06T14:01:47Z

linera-client/src/client_context.rs

+                    .await
+                    .expect("Processing inbox should not fail!");
+                chain_client
+                    .update_validators(None)


I wonder if that's even necessary. process_inbox_without_prepare already updates the validators about the new certificates.

It's probably an unlikely case, but from what I looked, process_inbox_without_prepare will only execute a new block (and then update the validators) if we have pending message bundles. If that's not the case, then we won't update the validators.
update_validators in this case will only update anything if our committee has changed. So I guess in the case where we're running this against the testnet or something, and some validator gets added/removed, we should keep this around to make sure things will be properly updated in that case.

But if the committee changed, we do receive a message about it, so there will be a pending message bundle. (At least until we have event streams—but even if, I'd say it isn't the benchmark's task to update the validators about the admin chain.)

I see! Then maybe we don't need it. I'll remove it.

ndr-ds · 2025-02-06T18:28:12Z

Merge activity

Feb 6, 1:28 PM EST: A user started a stack merge that includes this pull request via Graphite.
Feb 6, 1:30 PM EST: Graphite rebased this pull request as part of a merge.
Feb 6, 1:30 PM EST: A user merged this pull request with Graphite.

ndr-ds mentioned this pull request Feb 4, 2025

Remove max_in_flight from linera benchmark #3235

Merged

This was referenced Feb 4, 2025

make_benchmark_chains improvements #3246

Merged

supply_fungible_tokens improvements #3247

Merged

Parallelize staging block execution #3248

Closed

Make generated block proposals cancel each other out #3249

Merged

ndr-ds requested review from afck, christos-h, jvff, ma2bd and Twey February 5, 2025 00:50

afck reviewed Feb 5, 2025

View reviewed changes

ndr-ds force-pushed the 02-03-remove_max_in_flight_from_linera_benchmark branch from d19808a to 34e94bf Compare February 5, 2025 20:07

ndr-ds force-pushed the 02-03-parallelize_process_inboxes_and_force_validator_updates branch from 2e762fc to 466a717 Compare February 5, 2025 20:07

ndr-ds requested a review from afck February 5, 2025 20:12

ndr-ds force-pushed the 02-03-parallelize_process_inboxes_and_force_validator_updates branch from 466a717 to 8af1570 Compare February 6, 2025 00:04

ndr-ds mentioned this pull request Feb 6, 2025

make_benchmark_block_proposals now returns BlockProposals #3264

Merged

ndr-ds force-pushed the 02-03-remove_max_in_flight_from_linera_benchmark branch from 34e94bf to 96d6a58 Compare February 6, 2025 03:47

ndr-ds force-pushed the 02-03-parallelize_process_inboxes_and_force_validator_updates branch from 8af1570 to f7cca92 Compare February 6, 2025 03:47

ndr-ds mentioned this pull request Feb 6, 2025

Introduce long running mode to linera benchmark #3265

Open

afck reviewed Feb 6, 2025

View reviewed changes

linera-client/src/client_context.rs Outdated Show resolved Hide resolved

linera-client/src/client_context.rs Outdated Show resolved Hide resolved

linera-client/src/client_context.rs Outdated Show resolved Hide resolved

ndr-ds force-pushed the 02-03-remove_max_in_flight_from_linera_benchmark branch from 96d6a58 to 179c093 Compare February 6, 2025 13:36

ndr-ds force-pushed the 02-03-parallelize_process_inboxes_and_force_validator_updates branch 2 times, most recently from 6797419 to 6e1bb0d Compare February 6, 2025 13:54

afck approved these changes Feb 6, 2025

View reviewed changes

ndr-ds force-pushed the 02-03-remove_max_in_flight_from_linera_benchmark branch from 179c093 to 4073be0 Compare February 6, 2025 17:31

ndr-ds force-pushed the 02-03-parallelize_process_inboxes_and_force_validator_updates branch from 6e1bb0d to 09aeee8 Compare February 6, 2025 17:31

ndr-ds changed the base branch from 02-03-remove_max_in_flight_from_linera_benchmark to graphite-base/3245 February 6, 2025 18:28

ndr-ds changed the base branch from graphite-base/3245 to main February 6, 2025 18:28

Parallelize process_inboxes_and_force_validator_updates

fdd3ba9

ndr-ds force-pushed the 02-03-parallelize_process_inboxes_and_force_validator_updates branch from 09aeee8 to fdd3ba9 Compare February 6, 2025 18:29

ndr-ds merged commit 62cac58 into main Feb 6, 2025
22 checks passed

ndr-ds deleted the 02-03-parallelize_process_inboxes_and_force_validator_updates branch February 6, 2025 18:30

This was referenced Feb 10, 2025

Flush DB after range delete and ignore range deletions on find_key_values_by_prefix_internal #3292

Draft

Unify run_benchmark and run_benchmark_long_running #3296

Open

[benchmark] Respect current Epoch and Committee on block proposal creation #3297

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize process_inboxes_and_force_validator_updates #3245

Parallelize process_inboxes_and_force_validator_updates #3245

ndr-ds commented Feb 4, 2025

ndr-ds commented Feb 4, 2025 •

edited

Loading

afck Feb 5, 2025

afck Feb 5, 2025

ndr-ds Feb 5, 2025

afck Feb 5, 2025

ndr-ds commented Feb 5, 2025

afck Feb 6, 2025

ndr-ds Feb 6, 2025

afck Feb 6, 2025

ndr-ds Feb 6, 2025

ndr-ds commented Feb 6, 2025 •

edited

Loading

	let mut chain_client_and_notification_stream = Vec::new();
	let mut chain_clients_and_notification_streams = Vec::new();

Parallelize process_inboxes_and_force_validator_updates #3245

Parallelize process_inboxes_and_force_validator_updates #3245

Conversation

ndr-ds commented Feb 4, 2025

Motivation

Proposal

Test Plan

Release Plan

ndr-ds commented Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndr-ds commented Feb 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndr-ds commented Feb 6, 2025 • edited Loading

Merge activity

ndr-ds commented Feb 4, 2025 •

edited

Loading

ndr-ds commented Feb 6, 2025 •

edited

Loading