feat(synth-bench): automate TPS measurement for wait_until: NONE
#12874
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #12874 +/- ##
==========================================
+ Coverage 70.25% 70.41% +0.15%
==========================================
Files 854 855 +1
Lines 174571 174826 +255
Branches 174571 174826 +255
==========================================
+ Hits 122646 123104 +458
+ Misses 46713 46483 -230
- Partials 5212 5239 +27
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| tokio = { version = "1.40.0", features = ["full"] } | ||
| serde = { version = "1.0", features = ["derive"] } | ||
| serde_json = "1.0" | ||
|
|
||
| [dev-dependencies] | ||
| indoc = "2" |
There was a problem hiding this comment.
It is somewhat unfortunate that synth-bm is not part of the overall workspace. Any reason why you chose to have this crate be entirely separate?
There was a problem hiding this comment.
IIRC it was the quickest way to get started and then the question of including er excluding it didn't get enough attention. I can include it once the outstanding PRs are merged #12887.
| impl MetricName { | ||
| fn report_name(self) -> &'static str { | ||
| match self { | ||
| MetricName::SuccessfulTransactions => "near_transaction_processed_successfully_total", |
There was a problem hiding this comment.
One important thing to keep in mind is that this metric is incremented when the transaction successfully converts into a receipt. It does not mean that the receipt is actually getting executed (it might even happen in a different block if the receipt gets delayed.) You will probably want a metric along the lines of near_action_called_count (allows determining the rate of actions based on action types even in mixed action workloads, or where receipts have multiple actions!) or receipts (I don't think we have a single metric for all of them together, but shouldn't be too hard to add up the delayed, local, incoming, etc. receipts.)
There was a problem hiding this comment.
Interesting! We've been using near_transaction_processed_successfully_total for that purpose since the ft benchmarking days...
Will look into near_action_called_count.
As described in #12806, the RPC response to transactions sent with
wait_until: NONEdoesn't tell much. In that case, measuring TPS in terms of successfully processed transactions was a manual process. This PR automates it and uses that functionality for benchmarking native transfers.Testing
/metricsendpoint.just benchmark_native_transfersprints TPS of successfully processed transactions.Closes #12806