Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework checkpoint memory usage estimation #4055

Merged
merged 1 commit into from
Aug 11, 2024
Merged

Rework checkpoint memory usage estimation #4055

merged 1 commit into from
Aug 11, 2024

Conversation

ray6080
Copy link
Contributor

@ray6080 ray6080 commented Aug 11, 2024

Description

  • rework checkpoint mem estimation to wal size + local storage size
  • add debug prefix to enable_multi_writes

add debug prefix to enable_multi_writes
Copy link

Benchmark Result

Master commit hash: f08f80d08b3164d36f3e593e058afe3b58e4c17b
Branch commit hash: 4977ad8f8078e3791faefab54ac879c326346c12

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 689.86 685.00 4.87 (0.71%)
aggregation q28 11848.71 11728.70 120.01 (1.02%)
filter q14 160.96 160.46 0.50 (0.31%)
filter q15 165.86 157.94 7.92 (5.02%)
filter q16 341.58 342.67 -1.09 (-0.32%)
filter q17 479.94 479.94 -0.00 (-0.00%)
filter q18 1941.60 1950.95 -9.35 (-0.48%)
fixed_size_expr_evaluator q07 575.53 572.21 3.32 (0.58%)
fixed_size_expr_evaluator q08 782.66 790.71 -8.05 (-1.02%)
fixed_size_expr_evaluator q09 786.27 779.88 6.39 (0.82%)
fixed_size_expr_evaluator q10 276.99 272.57 4.41 (1.62%)
fixed_size_expr_evaluator q11 271.89 267.46 4.43 (1.66%)
fixed_size_expr_evaluator q12 270.69 265.38 5.30 (2.00%)
fixed_size_expr_evaluator q13 1503.11 1510.47 -7.37 (-0.49%)
fixed_size_seq_scan q23 154.24 150.40 3.84 (2.55%)
join q31 12.75 12.59 0.16 (1.23%)
ldbc_snb_ic q35 747.97 773.12 -25.15 (-3.25%)
ldbc_snb_ic q36 51.63 51.10 0.54 (1.05%)
ldbc_snb_is q32 10.40 9.44 0.97 (10.23%)
ldbc_snb_is q33 19.22 17.31 1.91 (11.05%)
ldbc_snb_is q34 9.03 8.95 0.08 (0.85%)
multi-rel multi-rel-large-scan 2828.54 2874.51 -45.97 (-1.60%)
multi-rel multi-rel-lookup 62.53 60.86 1.66 (2.74%)
multi-rel multi-rel-small-scan 58.17 56.92 1.24 (2.18%)
order_by q25 168.32 165.08 3.24 (1.96%)
order_by q26 483.64 478.98 4.66 (0.97%)
order_by q27 1433.16 1437.19 -4.02 (-0.28%)
scan_after_filter q01 209.01 206.41 2.60 (1.26%)
scan_after_filter q02 198.89 195.41 3.48 (1.78%)
shortest_path_ldbc100 q39 154.41 158.50 -4.09 (-2.58%)
var_size_expr_evaluator q03 2095.41 2085.78 9.63 (0.46%)
var_size_expr_evaluator q04 2253.21 2273.79 -20.57 (-0.90%)
var_size_expr_evaluator q05 2624.63 2660.59 -35.96 (-1.35%)
var_size_expr_evaluator q06 1356.98 1362.80 -5.83 (-0.43%)
var_size_seq_scan q19 1490.24 1496.79 -6.55 (-0.44%)
var_size_seq_scan q20 3205.13 3191.11 14.01 (0.44%)
var_size_seq_scan q21 2449.44 2423.04 26.40 (1.09%)
var_size_seq_scan q22 136.50 135.94 0.56 (0.41%)

Copy link

codecov bot commented Aug 11, 2024

Codecov Report

Attention: Patch coverage is 80.85106% with 9 lines in your changes missing coverage. Please review.

Project coverage is 85.13%. Comparing base (f08f80d) to head (e532a1e).
Report is 1 commits behind head on master.

Files Patch % Lines
src/include/main/settings.h 66.66% 6 Missing ⚠️
src/storage/local_storage/local_rel_table.cpp 50.00% 2 Missing ⚠️
src/include/storage/storage_structure/disk_array.h 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4055      +/-   ##
==========================================
- Coverage   85.18%   85.13%   -0.05%     
==========================================
  Files        1297     1297              
  Lines       50626    50643      +17     
  Branches     6985     6986       +1     
==========================================
- Hits        43125    43116       -9     
- Misses       7362     7387      +25     
- Partials      139      140       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ray6080 ray6080 merged commit 6a65178 into master Aug 11, 2024
23 checks passed
@ray6080 ray6080 deleted the checkpoint-size branch August 11, 2024 17:19
ray6080 added a commit that referenced this pull request Aug 14, 2024
add debug prefix to enable_multi_writes
ted-wq-x pushed a commit to ted-wq-x/kuzu that referenced this pull request Nov 14, 2024
add debug prefix to enable_multi_writes

(cherry picked from commit 6a65178)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants