-
Notifications
You must be signed in to change notification settings - Fork 704
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add zfree_with_size to optimize sdsfree since we can get zmalloc_size from the header #453
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## unstable #453 +/- ##
============================================
- Coverage 70.07% 70.05% -0.03%
============================================
Files 110 110
Lines 59989 60085 +96
============================================
+ Hits 42037 42092 +55
- Misses 17952 17993 +41
|
I think the zmalloc_size is really expensive to update the |
@lipzhu the update of used_memory has 2 cost aspects, the zmalloc_size and also the fact it's an atomic operation.
|
@lipzhu good point about |
…to align the logic with sdsnewlen function. (#476) This patch try to correct the actual allocated size from allocator when call sdsRedize to align the logic with sdsnewlen function. Maybe the #453 optimization should depend on this. Signed-off-by: Lipeng Zhu <[email protected]>
…to align the logic with sdsnewlen function. (valkey-io#476) This patch try to correct the actual allocated size from allocator when call sdsRedize to align the logic with sdsnewlen function. Maybe the valkey-io#453 optimization should depend on this. Signed-off-by: Lipeng Zhu <[email protected]>
…to align the logic with sdsnewlen function. (valkey-io#476) This patch try to correct the actual allocated size from allocator when call sdsRedize to align the logic with sdsnewlen function. Maybe the valkey-io#453 optimization should depend on this. Signed-off-by: Lipeng Zhu <[email protected]> Signed-off-by: Samuel Adetunji <[email protected]>
zfree updates memory statistics. It gets the size of the buffer from jemalloc by calling zmalloc_size. This operation is costly. We can avoid it if we know the buffer size. For example, we can calculate size of sds from the data we have in its header. This commit introduces zfree_with_size function that accepts both pointer to a buffer, and its size. zfree is refactored to call zfree_with_size. sdsfree uses the new interface for all but SDS_TYPE_5. Signed-off-by: Vadym Khoptynets <[email protected]>
sdsalloc returns sds's length for SDS_TYPE_5. It's not correct for SDS_TYPE_5. This patch makes sdsAllocSize call zmalloc_size for SDS_TYPE_5. sdsalloc is a lower level function that continues to return length for SDS_TYPE_5. Signed-off-by: Vadym Khoptynets <[email protected]>
Instead of zmalloc_size use s_malloc_size. Signed-off-by: Vadym Khoptynets <[email protected]>
Change the doc string for `zfree_with_size` Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
Signed-off-by: Vadym Khoptynets <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Ok, I wasn't able to get as consistent of performance numbers. Diving into a perf analysis shows about a 20% reduction in CPU in sdsfree though. So it seems like there might be other bottlenecks, this is definitely a good optimization. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall it looks good to me, I have a question about an alternative, but it's not a blocker from me. If you still want to go with this approach, I think it's OK to merge.
Yes, let's go with this approach. |
Signed-off-by: Vadym Khoptynets <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, it is a good optimization.
Signed-off-by: Madelyn Olson <[email protected]>
Issue Introduced by #453. When we check the SDS _TYPE_5 allocation size we mistakenly used zmalloc_size which DOES take the PREFIX size into account when no malloc_size support. Later when we free we add the PREFIX_SIZE again which leads to negative memory accounting on some tests. Example test failure: https://github.com/valkey-io/valkey/actions/runs/9654170962/job/26627901497 Signed-off-by: ranshid <[email protected]>
**Motivation** Copy-on-write (COW) amplification refers to the issue where writing to a small object leads to the entire page being cloned, resulting in inefficient memory usage. This issue arises during the BGSAVE process, which can be particularly problematic on instances with limited memory. If the BGSAVE process could release unneeded memory, it could reduce memory consumption. To address this, the BGSAVE process calls the `madvise` function to signal the operating system to reclaim the buffer. However, this approach does not work for buffers smaller than a page (usually 4KiB). Even after multiple such calls, where a full page may be free, the operating system will not reclaim it. To solve this issue, we can call `zfree` directly. This allows the allocator (jemalloc) to handle the bookkeeping and release pages when buffers are no longer needed. This approach reduces copy-on-write events. **Benchmarks** To understand how usage of `zfree` affects BGSAVE and the memory consumption I ran 45 benchmarks that compares my clonewith the vanilla version. The benchmark has the following steps: 1. Start a new Valkey process 2. Fill the DB with data sequentially 3. Run a warmup to randomize the memory layout 4. Introduce fragmentation by deleting part of the keys 5. In parallel: 1. Trigger BGSAVE 2. Start 80/20 get/set load I played the following parameters to understand their influence: 1. Number of keys: 3M, 6M, and 12M. 2. Data size. While key themselves are of fixed length ~30 bytes, the value size is 120, 250, 500, 1000, and 2000 bytes. 3. Fragmentation. I delete 5%, 10%, and 15% of the original key range. I'm attaching a graph of BGSAVE process memory consumption. Instead of all benchmarks, I show the most representative runs IMO. <img width="1570" alt="3m-fixed" src="https://github.com/user-attachments/assets/3dbbc528-01c1-4821-a3c2-6be455e7f78a"> For 2000 bytes values peak memory usage is ~53% compared to vanilla. The peak happens at 57% BGSAVE progress. For 500 bytes values the peak is ~80% compared to vanilla. And happens at ~80% progress. For 120 bytes the difference is under 5%, and the patched version could even use more memory. ![500b-fixed](https://github.com/user-attachments/assets/b09451d3-4bce-4f33-b3db-2b5df2178ed2) For 12M keys, the peak is ~85% of the vanilla’s. Happens at ~70% mark. For 6M keys, the peak is ~87% of the vanilla’s. Happens at ~77% mark. For 3M keys, the peak is ~87% of the vanilla’s Happens at ~80% mark. **Changes** The PR contains 2 changes: 1. Static buffer for RDB comrpession. RDB compression leads to COW events even without any write load if we use `zfree`. It happens because the compression functions allocates a new buffer for each object. Together with freeing objects with `zfree` it leads to reusing of the memory shared with the main process. To deal with this problem, we use a pre-allocated constant 8K buffer for compression. If the object size is too big for this buffer, than we fall back to the ad hoc allocation behavior. 2. Freeing string objects instead of dismissing them Call to `zfree` is more expensive than direct call to `madvise`. But with #453 strings use the fast path – `zfree_with_size`. As a possible next step we can optimize `zfree` for other data types as well. --------- Signed-off-by: Vadym Khoptynets <[email protected]> Signed-off-by: ranshid <[email protected]> Co-authored-by: ranshid <[email protected]> Co-authored-by: Viktor Söderqvist <[email protected]>
Description
zfree updates memory statistics. It gets the size of the buffer from jemalloc by calling zmalloc_size. This operation is costly. We can avoid it if we know the buffer size. For example, we can calculate size of sds from the data we have in its header.
This commit introduces zfree_with_size function that accepts both pointer to a buffer, and its size. zfree is refactored to call zfree_with_size.
sdsfree uses the new interface for all but SDS_TYPE_5.
Benchmark
Dataset is 3 million strings. Each benchmark run uses its own value size (8192, 512, and 120). The benchmark is 100% write load for 5 minutes.