persist: inline small writes directly in consensus State #19383

danhhz · 2023-05-19T20:06:32Z

Previously, a compare_and_append operation always required writing the
data to Blob (s3) and then only putting a pointer to it in State (stored
in Consensus/crdb). The latencies of s3 puts are both considerably
higher and with a much fatter tail than crdb latencies. So, inline small
batch parts directly where we would have put the pointer.

The threshold for small is made a LD param, so we can tune it.
Skip keeping pushdown stats for these parts.
Represent them using the new LazyProto abstraction developed for part
stats. This keeps the inline parts as bytes::Bytes (which is in turn
just a slice of the encoded ProtoStateRollup/ProtoStateDiff), only
decoding them when they would have been "fetched". This considerably
lessens the cost of deserializing these in places that aren't
interested in the updates themselves (e.g. the controllers on envd).

Touches MaterializeInc/database-issues#7413

Motivation

This PR adds a known-desirable feature.

Tips for reviewer

Checklist

This PR has adequate test coverage / QA involvement has been duly considered.
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
This PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way) and therefore is tagged with a T-proto label.
If this PR will require changes to cloud orchestration, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
This PR includes the following user-facing behavior changes:

danhhz · 2024-04-03T20:13:40Z

There still a few small bits of cleanup (noticed I missed some WIPs in src/persist), but gonna mark this ready for review to get the ball rolling

shepherdlybot · 2024-04-03T20:15:15Z

Mitigations

Completing required mitigations increases Resilience Coverage.

(Required) Code Review 🔍 Detected
(Required) Feature Flag
(Required) Integration Test 🔍 Detected
(Required) Observability 🔍 Detected
(Required) QA Review 🔍 Detected
Unit Test

Risk Summary:

The pull request carries a high risk, with a score of 83, indicating a significant likelihood of introducing bugs, as PRs with similar characteristics have historically been 124% more likely to cause bugs compared to the repository baseline. This risk is informed by predictors such as the average line count and executable lines within files, and it's worth noting that three files modified are known hotspots for bugs. While the repository's observed bug trend is increasing, the predicted trend is decreasing.

Note: The risk score is not based on semantic analysis but on historical predictors of bug occurrence in the repository. The attributes above were deemed the strongest predictors based on that history. Predictors and the score may change as the PR evolves in code, time, and review activity.

Bug Hotspots:
What's This?

File	Percentile
../internal/metrics.rs	98
../operators/shard_source.rs	97
../src/cfg.rs	94

bkirwi

Nice! Overall feels quite solid, though it is Big, and I'll need another pass to get through the details. (I can se it doesn't seem that easy to split into multiple PRs, except for maybe the testing-macro bits...)

src/persist-client/src/batch.rs

src/persist-client/src/internal/state.rs

src/persist-client/src/batch.rs

src/persist-client/src/internal/compact.rs

danhhz · 2024-04-03T22:02:36Z

Nice! Overall feels quite solid, though it is Big, and I'll need another pass to get through the details. (I can se it doesn't seem that easy to split into multiple PRs, except for maybe the testing-macro bits...)

Commit number three is pretty hard to split up, yeah, but the first two commits could pretty easily be peeled off if that makes anything easier! The first one would be missing a good bit of context (e.g. your feedback about fn key), but might be fine? I also had at one point a version that added fn key without the option, so all the changes in it were much more mechanical. Happy to bring that back, if you're interested

bkirwi · 2024-04-04T14:38:30Z

Commit number three is pretty hard to split up, yeah, but the first two commits could pretty easily be peeled off if that makes anything easier! The first one would be missing a good bit of context, but might be fine?

I would prefer this if it's not a pain!

I also had at one point a version that added fn key without the option, so all the changes in it were much more mechanical. Happy to bring that back, if you're interested

I think I'm good with this one as-is, but thanks!

danhhz · 2024-04-04T14:56:41Z

👍 I started on this optimistically, so should be incoming shortly

danhhz · 2024-04-04T15:27:56Z

Okay, the prep PR is up in #26429! Note that I haven't yet addressed any of your initial comments here: as much as possible I kept the commits in that PR the same as they are here (modulo merge conflicts and a couple small tweaks to things like WIP comments). Will push updates as append-only commits to the other PR to make it easy to see the diffs from what you've already looked at.

danhhz

Okay, I think I've addressed the first round of comments.

I'll rebase this on top of the prep PR at your convenience. (Not blindly doing it now in case you had some pending comments, which github reviews would thoroughly barf on.)

src/persist-client/src/internal/state.rs

nrainer-materialize · 2024-04-05T16:41:04Z

Scalability Framework

Some insights of the scalability run in builds/7220 that was run with

    "persist_inline_update_max_bytes": "1048576",
    "persist_inline_update_threshold_bytes": "4096",

`UpdateWorkload`

There is a significant improvement regarding the UpdateWorkload (artifact plots/tps_UpdateWorkload.png). 👍

`InsertWorkload`

The InsertWorkload (artifact plots/tps_InsertWorkload.png), which runs INSERT INTO t1 DEFAULT VALUES; with t1 being a single int-column table, produced slightly worse results with high concurrencies:

Feature Benchmark

The feature benchmark of the same build does not show any significant changes. However, I think that this is because of the size of the changes.

`Insert` workload

Build

Insert                              | wallclock |           2.755 |           2.746 |      no       | 0.3 pct   more/slower
Insert                              | messages  |            None |            None |      no       | N/A
Insert                              | memory    |        1173.019 |        1166.344 |      no       | 0.6 pct   more/slower

Though, the insert operation (Insert in misc/python/materialize/feature_benchmark/scenarios/benchmark_main.py) involves reading from a cross product of the view ten containing 10 values:

INSERT INTO t1 SELECT (a1.f1 * 1) + (a2.f1 * 10) + (a3.f1 * 100) + (a4.f1 * 1000) + (a5.f1 * 10000) + (a6.f1 * 100000) FROM ten AS a1, ten AS a2, ten AS a3, ten AS a4, ten AS a5, ten AS a6

Furthermore, I could imaging that this exceeds the threshold for inline writes.

I will consider adding an additional thinner Insert scenario to the feature benchmark.

`Update` workload

The same applies to the Update workload in build 7220.

Update                              | wallclock |           2.886 |           2.901 |      no       | 0.5 pct   less/faster
Update                              | messages  |    33000223.000 |    33000223.000 |      no       |           same
Update                              | memory    |        1416.206 |        1420.021 |      no       | 0.3 pct   less/faster

I guess it touches as well too many rows (it updates a BIGINT column in 1'000'000 rows).

danhhz · 2024-04-05T17:01:54Z

The InsertWorkload (artifact plots/tps_InsertWorkload.png), which runs INSERT INTO t1 DEFAULT VALUES; with t1 being a single int-column table, produced slightly worse results with high concurrencies:

I noticed the same in a previous nightlies I kicked off a while ago, so it seems like a stable result. Adapter internally does batching of writes, so I wouldn't expect it to necessarily be better, but it sure is surprising that it's worse. No particular theories yet of what's going on there.

danhhz · 2024-04-05T17:11:22Z

I'll rebase this on top of the prep PR at your convenience. (Not blindly doing it now in case you had some pending comments, which github reviews would thoroughly barf on.)

@bkirwi Went ahead and pushed the rebase so we could pick up the fix for my backpressure bug. As much as possible I tried to keep everything that's changed since your first round of comments in the fixup commit (though had to resolve the merge conflicts with the other branch -- e.g. removing fn key --in the first commit).

danhhz · 2024-04-05T17:19:35Z

@nrainer-materialize It's possible that both of those feature benchmark scenarios are dominated by the cost of doing a read-write transaction, which is pretty inefficient in mz. Something that does a blind write would probably be more impacted by this. Looks like InsertBatch does, but I'm having trouble following how many values it inserts (and thus how big the write is)

bkirwi

Thinking about:

AFAIK this is the first time we've impose backpressure on writes of any kind. (In practice I suspect writers should all be able to handle some latency during compare-and-append, but it'll be interesting to watch. In theory I will want to think more about if/how this particular way of inflating state is worse than any other ones callers might invent...
For very small batches, flushing the batch out to Persist may actually increase the size of state. (Do we want some threshold under which an inline part can be appended no matter what?)

doc/developer/design/20240401_persist_inline_writes.md

bkirwi · 2024-04-05T18:53:02Z

src/persist-client/src/batch.rs

+    "persist_inline_update_max_bytes",
+    0,
+    "\
+    The (inclusive) maximum total size of inline writes in metadata before \


Config nits:

"Threshold" and "max" don't feel like they convey the distinction you're making here... maybe "single" and "total" or something?

It's mildly surprising that one of these is inclusive and one is exclusive. (My usual bias is to exclusive upper bounds, but consistency either way seems good.)

"Threshold" and "max" don't feel like they convey the distinction you're making here... maybe "single" and "total" or something?

Yeah, put these in early as placeholders and figured we'd hash it out in review. I like single and total. I also realize they should probably be "inline_write(s)" instead of "inline_updates" for discoverability since I've been pretty consistently calling the feature the former.

Done!

It's mildly surprising that one of these is inclusive and one is exclusive. (My usual bias is to exclusive upper bounds, but consistency either way seems good.)

Fair! I like exclusive here for "threshold" because then we don't need a separate sentinel for "off". Turns out they were both exclusive anyway and the doc was just wrong

src/persist-client/src/batch.rs

src/persist-client/src/fetch.rs

bkirwi · 2024-04-05T19:06:23Z

src/persist-client/src/internal/state.rs

@@ -176,12 +176,18 @@ pub struct HandleDebugState {
 #[serde(tag = "type")]
 pub enum BatchPart<T> {
    Hollow(HollowBatchPart<T>),
+    Inline {
+        updates: LazyInlineBatchPart,
+        key_lower: Vec<u8>,


I think it would probably be fine to omit the key_lower, in case that seems appealing! For tiny writes it might take up as much space as the write itself, and the main motivation for it (avoiding unnecessary fetches) doesn't really apply here.

Yeah, I'd love to remove this!

I remember adding it because something in iter.rs was failing an assert. That was early last week and a lot was in flux in the branch, so could be that I'm remembering incorrectly or that it's no longer an issue. I'll try it again and see what happens.

In the meantime, were you thinking that we hardcode vec![] as the key_lower for inline parts or that we'd decode it on demand and return the real key lower? The first seems tricky because of the case where we have a run where the last part is inline (and it needs to have a greater key_lower than the previous ones, correct? Maybe this is the assert I was hitting). The second is a bit unfortunate because the bytes are in there but prost won't let us get them out without allocs for key_offsets, val_offsets, ts, diff in the decoded proto, and I wasn't sure how hot a path key_lower is.

In the meantime, were you thinking [...]

The former!

The first seems tricky because of the case where we have a run where the last part is inline [...]

I had not been thinking about this case! It seems a bit pathological to have a run with N>0 100MiB parts and then 1 inline part. We could probably ban it, though I appreciate that might be more work than just threading the lower around.

and it needs to have a greater key_lower than the previous ones, correct?

That is certainly the case now; I don't know if anything relies on it, though it seems plausible that we're asserting on it somewhere. If so we could probably relax such an assert, assuming that seems like the easiest path...

Hardcode vec![] and disallow inline in a run if an earlier part is not inline sounds reasonable to me. I'll try typing it up and let you know how it goes

Pushed it up (without "disallow inline in a run if an earlier part is not inline", since it seemed to be working in bin/envd without), let's see what CI says

CI and nightlies seem happy with it! \o/

That works! (And should be easy enough to do the other half of it later if we decide it's important.)

bkirwi

Left a few questions, but on reflection I don't think any of them are blocking. Exciting stuff!

nrainer-materialize · 2024-04-08T06:08:21Z

Looks like InsertBatch does, but I'm having trouble following how many values it inserts (and thus how big the write is)

InsertBatch will insert 10^4 rows with a single, different integer value each.

danhhz

AFAIK this is the first time we've impose backpressure on writes of any kind. (In practice I suspect writers should all be able to handle some latency during compare-and-append, but it'll be interesting to watch. In theory I will want to think more about if/how this particular way of inflating state is worse than any other ones callers might invent...

In my mental model, because this short-circuits the cmd, the "backpressure" case is the same as our current behavior on main wrt talking to crdb/s3. There is an extra wasted encode to the proto columnar records, but feels like that would be a rounding error compared to the network stuff.

I do think it's "worse" than other ways of inflating state because this is someone using persist totally to spec, whereas e.g. registering a bunch of readers/writers is something we'd push back on as out of spec.

For very small batches, flushing the batch out to Persist may actually increase the size of state. (Do we want some threshold under which an inline part can be appended no matter what?)

Yeah, I think this is totally reasonable. Otoh, given the above, I'm inclined to what feels to me like the conservative option here and err on the side of keeping a hard limit on the total size of things inline in state for the initial rollout. We can always circle back here if we notice this be an issue, so left a TODO.

src/persist-client/src/fetch.rs

danhhz · 2024-04-08T17:59:38Z

src/persist-client/src/batch.rs

+    "persist_inline_update_max_bytes",
+    0,
+    "\
+    The (inclusive) maximum total size of inline writes in metadata before \


"Threshold" and "max" don't feel like they convey the distinction you're making here... maybe "single" and "total" or something?

Yeah, put these in early as placeholders and figured we'd hash it out in review. I like single and total. I also realize they should probably be "inline_write(s)" instead of "inline_updates" for discoverability since I've been pretty consistently calling the feature the former.

Done!

It's mildly surprising that one of these is inclusive and one is exclusive. (My usual bias is to exclusive upper bounds, but consistency either way seems good.)

Fair! I like exclusive here for "threshold" because then we don't need a separate sentinel for "off". Turns out they were both exclusive anyway and the doc was just wrong

src/persist-client/src/batch.rs

doc/developer/design/20240401_persist_inline_writes.md

danhhz · 2024-04-08T18:40:16Z

misc/python/materialize/mzcompose/__init__.py

@@ -87,6 +87,8 @@
    "enable_worker_core_affinity": "true",
    "persist_batch_delete_enabled": "true",
    "persist_fast_path_limit": "1000",
+    "persist_inline_update_max_bytes": "1048576",


Updated this to 4096 as well in an attempt to maximally stress backpressure in CI. This is not the config I'd expect us to run in prod, which is usually what we put here, but seems worth the discrepancy to me.

danhhz · 2024-04-08T19:31:14Z

src/persist-client/src/internal/state.rs

@@ -176,12 +176,18 @@ pub struct HandleDebugState {
 #[serde(tag = "type")]
 pub enum BatchPart<T> {
    Hollow(HollowBatchPart<T>),
+    Inline {
+        updates: LazyInlineBatchPart,
+        key_lower: Vec<u8>,


Hardcode vec![] and disallow inline in a run if an earlier part is not inline sounds reasonable to me. I'll try typing it up and let you know how it goes

danhhz · 2024-04-08T19:48:57Z

src/persist-client/src/iter.rs

+                // Inline parts are only written directly by the user and so may
+                // be unconsolidated.
+                //
+                // WIP: This makes me realize that having the version of the


Oh @bkirwi thoughts here?

It's a good question! Quick thoughts:

~~It's not useful for the specific thing that having the writer version has been useful for,~~ or the planned future use of garbage collection. (Edit: realized the first part of this is not quite true, though in practice if we did have issues with the sorting on inline parts I'm not sure we'd want to deal with it in the same way.)

The fact that these parts are meant to be short-lived makes it less important. (If we regret the encoding, it's pretty feasible to eg. rewrite all of them instead of keeping compatability.)

Seems pretty easy to add later.

So I guess while I think it's a good idea and we may need it sooner or later, I'd be tempted to punt for now. But it's a good callout!

bkirwi · 2024-04-08T21:09:16Z

Yeah, on board with all of the above. I think what you have here is the right place to start, and we'll learn more from getting this rolled out!

danhhz · 2024-04-09T20:56:50Z

Okay, modulo the unexpected regressions in the scalability benchmark at high concurrencies, I think this is otherwise good to go!

I spent about half a day investigating that yesterday, and was able to verify that my mental model of the crdb and s3 calls happening was correct, but wasn't able to quite explain it. Given that this is a measurable improvement in Insert performance at low concurrencies (the overwhelming common case for prod) as well as a notable improvement in update latencies and DDL at all concurrencies, I'm not sure the high concurrency Insert regression is a blocker.

It would be very good to at least understand what's going on, but I don't want this branch to rot, so if I can't get to the bottom of it by mid-day-ish tomorrow, I think it might be time to merge anyway.

danhhz · 2024-04-09T21:54:18Z

Aha! Aljoscha pointed me to the possibility that the writes in the scalability's InsertWorkload are all getting consolidated down and indeed that's the case.

WIP table writes 1712697143428
u1: [TimestamplessUpdate { row: Row{[Int32(1)]}, diff: 1 }]
WIP table writes 1712697143447
u1: [TimestamplessUpdate { row: Row{[Int32(1)]}, diff: 26 }]
WIP table writes 1712697143459
u1: [TimestamplessUpdate { row: Row{[Int32(1)]}, diff: 3 }]
WIP table writes 1712697143481
u1: [TimestamplessUpdate { row: Row{[Int32(1)]}, diff: 1 }]
environmentd: 2024-04-09T21:12:23.495725Z  INFO WIP table writes 1712697143493
u1: [TimestamplessUpdate { row: Row{[Int32(1)]}, diff: 32 }]

I've also run it with and without inline writes against my staging env so I could look at the persist graphs. It seems more and more like the high-concurrency InsertWorkload thing is an artifact of measurement, rather than a real result. Probably what's happening is something like getting worse batching in coord because the appends are faster.

bkirwi · 2024-04-09T22:02:17Z

src/persist-client/src/internal/state.rs

+            //
+            // The empty key might not be a tight lower bound, but it is a valid
+            // lower bound. If a caller is interested in a tighter lower bound,
+            // the data is inline.


bkirwi · 2024-04-09T22:05:20Z

src/persist-client/src/internal/state.rs

@@ -176,12 +176,18 @@ pub struct HandleDebugState {
 #[serde(tag = "type")]
 pub enum BatchPart<T> {
    Hollow(HollowBatchPart<T>),
+    Inline {
+        updates: LazyInlineBatchPart,
+        key_lower: Vec<u8>,


That works! (And should be easy enough to do the other half of it later if we decide it's important.)

nrainer-materialize · 2024-04-10T08:47:25Z

I've also run it with and without inline writes against my staging env so I could look at the persist graphs. It seems more and more like the high-concurrency InsertWorkload thing is an artifact of measurement, rather than a real result. Probably what's happening is something like getting worse batching in coord because the appends are faster.

Feel free to add an ignore entry for the scalability framework in _MIN_ANCESTOR_MZ_VERSION_PER_COMMIT_TO_ACCOUNT_FOR_SCALABILITY_REGRESSIONS in misc/python/materialize/version_ancestor_overrides.py. Happy to support you on that.

This allows us to efficiently use the `Bytes` that we get of out decoding a ProtoColumnarRecords (i.e. no copies).

Previously, a compare_and_append operation always required writing the data to Blob (s3) and then only putting a pointer to it in State (stored in Consensus/crdb). The latencies of s3 puts are both considerably higher and with a much fatter tail than crdb latencies. So, inline small batch parts directly where we would have put the pointer. - The threshold for small is made a LD param, so we can tune it. - Skip keeping pushdown stats for these parts. - Represent them using the new LazyProto abstraction developed for part stats. This keeps the inline parts as bytes::Bytes (which is in turn just a slice of the encoded ProtoStateRollup/ProtoStateDiff), only decoding them when they would have been "fetched". This considerably lessens the cost of deserializing these in places that aren't interested in the updates themselves (e.g. the controllers on envd).

danhhz · 2024-04-10T19:27:07Z

Okay CI and nightlies are happy, we have a plausible theory for the scalability bench results at high concurrencies, scalability bench shows improvements at low-concurrency-Insert/all-concurrency-Update/all-concurrency-DDL, and feature bench shows improvement in the new ManySmall{Insert,Update} scenarios. I'm going to go ahead and get this merged so we can try turning it on in staging next week.

TFTRs everyone!!

aljoscha · 2024-04-11T07:51:14Z

🙌

danhhz force-pushed the persist_inline branch from 11d3b32 to 1fec540 Compare June 23, 2023 23:19

danhhz changed the title ~~[DNM] persist: skunkworks impl of writing parts inline in consensus~~ persist: inline small writes directly in consensus State Jun 23, 2023

danhhz force-pushed the persist_inline branch from 1fec540 to 9a8207f Compare November 10, 2023 21:13

danhhz force-pushed the persist_inline branch 9 times, most recently from f37fd27 to a9ea316 Compare April 2, 2024 14:33

danhhz force-pushed the persist_inline branch 3 times, most recently from 0fc23e8 to b897d1c Compare April 3, 2024 14:29

danhhz requested a review from bkirwi April 3, 2024 20:13

danhhz marked this pull request as ready for review April 3, 2024 20:13

danhhz requested a review from a team April 3, 2024 20:13

danhhz requested a review from a team as a code owner April 3, 2024 20:13

bkirwi reviewed Apr 3, 2024

View reviewed changes

danhhz mentioned this pull request Apr 4, 2024

persist: preparation for inline writes #26429

Merged

5 tasks

danhhz force-pushed the persist_inline branch from cb4379c to 28793f4 Compare April 4, 2024 17:27

danhhz commented Apr 4, 2024

View reviewed changes

src/persist-client/src/internal/state.rs Outdated Show resolved Hide resolved

danhhz force-pushed the persist_inline branch from 28793f4 to 503e595 Compare April 5, 2024 17:07

bkirwi reviewed Apr 5, 2024

View reviewed changes

bkirwi approved these changes Apr 5, 2024

View reviewed changes

danhhz commented Apr 8, 2024

View reviewed changes

danhhz requested a review from benesch as a code owner April 8, 2024 19:33

danhhz commented Apr 8, 2024

View reviewed changes

danhhz force-pushed the persist_inline branch 2 times, most recently from b537906 to d1da4bd Compare April 8, 2024 21:52

bkirwi reviewed Apr 9, 2024

View reviewed changes

danhhz added 2 commits April 10, 2024 07:42

persist: use MaybeLgBytes for {key,val}_data on ColumnarRecords

6dee10f

This allows us to efficiently use the `Bytes` that we get of out decoding a ProtoColumnarRecords (i.e. no copies).

danhhz force-pushed the persist_inline branch from d1da4bd to 51cd30a Compare April 10, 2024 14:52

danhhz merged commit de92497 into MaterializeInc:main Apr 10, 2024
180 of 183 checks passed

danhhz deleted the persist_inline branch April 10, 2024 19:27

persist: inline small writes directly in consensus State #19383

persist: inline small writes directly in consensus State #19383

Conversation

danhhz commented May 19, 2023 • edited Loading

Motivation

Tips for reviewer

Checklist

danhhz commented Apr 3, 2024

shepherdlybot bot commented Apr 3, 2024 • edited Loading

Mitigations

bkirwi left a comment

Choose a reason for hiding this comment

danhhz commented Apr 3, 2024

bkirwi commented Apr 4, 2024

danhhz commented Apr 4, 2024

danhhz commented Apr 4, 2024

danhhz left a comment • edited Loading

Choose a reason for hiding this comment

nrainer-materialize commented Apr 5, 2024

Scalability Framework

UpdateWorkload

InsertWorkload

Feature Benchmark

Insert workload

Update workload

danhhz commented Apr 5, 2024 • edited Loading

danhhz commented Apr 5, 2024 • edited Loading

danhhz commented Apr 5, 2024

bkirwi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danhhz Apr 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkirwi left a comment

Choose a reason for hiding this comment

nrainer-materialize commented Apr 8, 2024

danhhz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkirwi Apr 8, 2024 • edited Loading

Choose a reason for hiding this comment

bkirwi commented Apr 8, 2024

danhhz commented Apr 9, 2024

danhhz commented Apr 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nrainer-materialize commented Apr 10, 2024

danhhz commented Apr 10, 2024

aljoscha commented Apr 11, 2024

danhhz commented May 19, 2023 •

edited

Loading

shepherdlybot bot commented Apr 3, 2024 •

edited

Loading

danhhz left a comment •

edited

Loading

`UpdateWorkload`

`InsertWorkload`

`Insert` workload

`Update` workload

danhhz commented Apr 5, 2024 •

edited

Loading

danhhz commented Apr 5, 2024 •

edited

Loading

danhhz Apr 8, 2024 •

edited

Loading

danhhz left a comment •

edited

Loading

bkirwi Apr 8, 2024 •

edited

Loading