compute servers: dependence on keydb

[williamstein]

Spanchat just sort of gave up on KeyDB, which we use, evidently: Snapchat/KeyDB#895

I just wanted to record this here, in case we're ever able to come up with a better approach to CloudFilesystem. Right now, keydb provides the best possible performance and does work well, and valkey doesn't for our use case. I've looked into other options "for fun" recently, but so far still nothing works.

[keithchew]

@williamstein similar to you,, I have done quite an extensive search for a keydb equivalent in a project I am working on, but nothing came up. In particular we use flash and modules in keydb. What are the missing features in your case, compared to say valkey/redis?

[williamstein]

What are the missing features in your case, compared to say valkey/redis?

We use KeyDB for exactly one thing, which is filesystem metadata for juicefs: https://juicefs.com/en/
It turns out that with a small patch I wrote to the inode generation algorithm, the (very minimal) multimaster replication implication in keydb is compatible with how JuiceFS uses redis. The result is a pretty amazing distributed scalable open source free filesystem (obviously, only useful if you're not writing to the same file from different places at the exact same time -- but great if you're writing to different files). If I switch to valkey, then I have to use a master server for all metadata writes, which means significantly increased latency for all but one of the nodes.

The multimaster replication in keydb seems like a relatively simple "trick", which was one of their first changes to their redis forks, so sometimes I wonder if it could be ported over to valkey...