Add multi-database support to cluster mode #1671
Conversation
This commit introduces multi-database support in cluster mode while maintaining backward compatibility and requiring no API changes. Key features include: - Database-agnostic hashing: The hashing algorithm is unchanged. Identical keys map to the same slot across all databases. No changes to slot calculation. This ensures consistency in key distribution and maintains compatibility with existing single-database setups. - Implementation is fully backward compatible with no API changes. - The core structure remains an array of databases, each containing a list of hashtables (one per slot). Cluster management commands are global commands, except for GETKEYSINSLOT and COUNTKEYSINSLOT, which run in selected-DB context. MIGRATE command operates a selected-db context. Please note that MIGRATE command parameter destination-db is used, when migrating keys they can be migrated to a different database in the target, like in non-cluster mode. Slot migration process changes when multiple databases are used: Iterate through all databases SELECT database keys = GETKEYSINSLOT MIGRATE source target keys Valkey-cli has been updated to support resharding across all databases. Signed-off-by: xbasel <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## unstable #1671 +/- ##
============================================
+ Coverage 70.97% 71.12% +0.14%
============================================
Files 121 123 +2
Lines 65238 65543 +305
============================================
+ Hits 46305 46619 +314
+ Misses 18933 18924 -9
|
| @@ -1728,12 +1714,6 @@ void swapMainDbWithTempDb(serverDb *tempDb) { | |||
| void swapdbCommand(client *c) { | |||
| int id1, id2; | |||
|
|
|||
| /* Not allowed in cluster mode: we have just DB 0 there. */ | |||
There was a problem hiding this comment.
Would that be enough for swapdb to work in cluster mode? What will happen in setup with 2 shards, each responsible for half of slots in db's?
There was a problem hiding this comment.
With this implementation SWAPDB must be executed in all primary nodes. There are three options:
- Allow SWAPDB and shift responsibility to the user – Risky, non-atomic, can cause temporary inconsistency and data corruption. Needs strong warnings.
- Keep SWAPDB disabled in cluster mode – Safest, avoids inconsistency.
- Make SWAPDB cluster-wide and atomic or – Complex, unclear feasibility.
I think option 2 is the safest bet. @JoBeR007 wdyt?
There was a problem hiding this comment.
Is SWAPDB replicated as a single command? Then it's atomic.
If it's risky, it's risky in standslone mode with replicas too, right?
I think we can allow it. Swapping the data can only be done in some non-realtime workloads anyway I think.
There was a problem hiding this comment.
I think risky because of replication and risky because of the need to execute SWAPDB on all primary nodes are unrelated just because as a user you can't control first, but user is the main risk in the second case.
I would keep SWAPDB disabled in cluster mode, if we decide to continue with this implementation
There was a problem hiding this comment.
In cluster mode, consistency is per slot.
There was a problem hiding this comment.
Is SWAPDB replicated as a single command? Then it's atomic.
If it's risky, it's risky in standslone mode with replicas too, right?
I think we can allow it. Swapping the data can only be done in some non-realtime workloads anyway I think.
I don’t think it’s very risky with standalone replicas. The only downside is if SWAPDB propagation to the replica takes time, a client might still access the wrong database. At least the client won’t be able to modify the wrong database, as they can only read.
In cluster mode, the same (logical) DB can be DB0 on one node and DB1 on another, but similar issues already exist today, FLUSHDB on one node doesn’t clear the entire DB since data exists in other slots/nodes. But as you said, consistency is per slot.
There was a problem hiding this comment.
Yes, FLUSHDB is very similar in this regard. If a failover happens just before this command has been propagated to replicas, it's a big thing, but it's no surprise I think. The client can use WAIT or check replication offset to make sure the FLUSHDB or SWAPDB was successful on the replicas.
soloestoy
requested review from
soloestoy
and removed request for
zuiderkwast
This commit introduces multi-database support in cluster mode while maintaining backward compatibility and requiring no API changes. Key features include:
Database-agnostic hashing: The hashing algorithm is unchanged. Identical keys map to the same slot across all databases. No changes to slot calculation. This ensures consistency in key distribution and maintains compatibility with existing single-database setups.
Implementation is fully backward compatible with no API changes.
The core structure remains an array of databases, each containing a list of hashtables (one per slot).
Cluster management commands are global commands, except for GETKEYSINSLOT and COUNTKEYSINSLOT, which run in selected-DB context.
MIGRATE command operates a selected-db context. Please note that MIGRATE command parameter destination-db is used, when migrating keys they can be migrated to a different database in the target, like in non-cluster mode.
Slot migration process changes when multiple databases are used:
Valkey-cli has been updated to support resharding across all databases.
#1319