Introduce RDMA transport (experimental) #55

pizhenwei · 2024-07-23T11:46:24Z

Hi,

Valkey Over RDMA has been supported as experimental feature since Valkey 8.0. Support RDMA transport for the client side.

There are commits in this PR:

Fully abstract IO framework
Fix timing issue on reconnect
Introduce RDMA transport (experimental)
cmake related support

These commits are already in order, they have separate function with detailed commit message, so no need to squash them into one.

michael-grunder

This is awesome. I've never used RDMA before so I plan on getting this building locally and playing around with it locally.

include/valkey/valkey_rdma.h

src/conn.c

src/rdma.c

pizhenwei · 2024-07-26T07:06:58Z

This is awesome. I've never used RDMA before so I plan on getting this building locally and playing around with it locally.

Hi,

If there is already a physical RDMA interface on your platform, build this branch and run command VALKEY_RDMA_MODULE=/path/to/valkey-rdma.so VALKEY_RDMA_ADDR=192.168.122.1 TEST_RDMA=1 ./test.sh.

There is also a benchmark guide in Redis style, I'll create valkey style once this repo gets ready. Enjoy the 2.5x performance!

It's also possible to setup a virtual RoCE device by command rdma link add rxe0 type rxe netdev $IFACENAME, or use this script to make it easy.

michael-grunder · 2024-07-29T17:52:54Z

cc @bjosv @zuiderkwast Does Anybody want to take a look before we merge. Unfortunately, I haven't been able to get SoftRoCE working at all, but have an RDMA-enabled NIC coming soon.

bjosv

This was new to me, but seems like a good feature!
Found some nits, and then I'll guess we need to add something to CI.
Maybe the plan was do do that after there is a (pre-)release of valkey with the feature?

Makefile

CMakeLists.txt

valkey_rdma-config.cmake.in

src/conn.c

include/valkey/valkey_rdma.h

src/rdma.c

- Hide valkeyNetRead & valkeyNetWrite Move valkeyContextDefaultFuncs into net.c, then valkeyNetRead & valkeyNetWrite could be declared as static functions. async.h is no longer needed by valkey.c, that makes valkey.c independent from networking IO specific operations. - Introduce .connect for valkeyContextFuncs Abstract .connect method, then new connection type has a chance to implement its own connect function. Finally, TCP/Unix/Userfd related functions have been removed from valkey.c. - Introduce .set_timeout method, setsockopt is only available for socket family. None-socket transport would fail on this. .set_timeout allows transport specific timeout setting function. Finally, the fully abstract IO framework is expected to support more transport. In especial, RDMA like none-socket transport. Signed-off-by: zhenwei pi <[email protected]>

If necessary private context gets freed first, NULL pointer would be accessed in .close method. The right and safe timing should be .close then .free_privctx. Signed-off-by: zhenwei pi <[email protected]>

Valkey Over RDMA[1] has been supported as experimental feature since Valkey 8.0. Support RDMA transport for the client side. RDMA is not a builtin feature, supported as module only, so we have to run test.sh with more argument @VALKEY_RDMA_MODULE and @VALKEY_RDMA_ADDR. An example to run test.sh: VALKEY_RDMA_MODULE=/path/to/valkey-rdma.so VALKEY_RDMA_ADDR=192.168.122.1 TEST_RDMA=1 ./test.sh ... Testing against RDMA connection (192.168.122.1:56379): valkey-io#138 Is able to deliver commands: PASSED valkey-io#139 Is a able to send commands verbatim: PASSED valkey-io#140 %s String interpolation works: PASSED valkey-io#141 %b String interpolation works: PASSED valkey-io#142 Binary reply length is correct: PASSED valkey-io#143 Can parse nil replies: PASSED valkey-io#144 Can parse integer replies: PASSED valkey-io#145 Can parse multi bulk replies: PASSED valkey-io#146 Can handle nested multi bulk replies: PASSED valkey-io#147 Send command by passing argc/argv: PASSED valkey-io#148 Can pass NULL to valkeyGetReply: PASSED valkey-io#149 RESP3 PUSH messages are handled out of band by default: PASSED valkey-io#150 We can set a custom RESP3 PUSH handler: PASSED valkey-io#151 We properly handle a NIL invalidation payload: PASSED valkey-io#152 With no handler, PUSH replies come in-band: PASSED valkey-io#153 With no PUSH handler, no replies are lost: PASSED valkey-io#154 We set a default RESP3 handler for valkeyContext: PASSED valkey-io#155 We don't set a default RESP3 push handler for valkeyAsyncContext: PASSED valkey-io#156 Our VALKEY_OPT_NO_PUSH_AUTOFREE flag works: PASSED valkey-io#157 We can use valkeyOptions to set a custom PUSH handler for valkeyContext: PASSED valkey-io#158 We can use valkeyOptions to set a custom PUSH handler for valkeyAsyncContext: PASSED valkey-io#159 We can use valkeyOptions to set privdata: PASSED valkey-io#160 Our privdata destructor fires when we free the context: PASSED valkey-io#161 Successfully completes a command when the timeout is not exceeded: PASSED valkey-io#162 Does not return a reply when the command times out: SKIPPED valkey-io#163 Reconnect properly reconnects after a timeout: PASSED #164 Reconnect properly uses owned parameters: PASSED #165 Returns I/O error when the connection is lost: PASSED #166 Returns I/O error on socket timeout: PASSED #167 Set error when an invalid timeout usec value is used during connect: PASSED #168 Set error when an invalid timeout sec value is used during connect: PASSED #169 Append format command: PASSED #170 Throughput: (1000x PING: 0.010s) (1000x LRANGE with 500 elements: 0.060s) (1000x INCRBY: 0.012s) (10000x PING (pipelined): 0.066s) (10000x LRANGE with 500 elements (pipelined): 0.523s) (10000x INCRBY (pipelined): 0.024s) ... Thanks to Michael Grunder for lots of review suggestions! Link[1]: valkey-io/valkey#477 Signed-off-by: zhenwei pi <[email protected]>

Signed-off-by: zhenwei pi <[email protected]>

pizhenwei · 2024-07-30T12:38:14Z

This was new to me, but seems like a good feature! Found some nits, and then I'll guess we need to add something to CI. Maybe the plan was do do that after there is a (pre-)release of valkey with the feature?

Sure, RDMA CI is in the plan.

The RDMA support is a module (shared library) of valkey-8.0:

if the library is built with valkey together, valkey-8.0 docker would be fine
otherwise, we need clone valkey-8.0 source code in libvalkey CI, and build valkey-8.0 with RDMA support.

There is a simple RDMA client in valkey, and valkey runs RDMA CI test.

And all the suggestions are appled, thanks!

pizhenwei · 2024-07-31T04:54:57Z

Hi @bjosv @michael-grunder

This PR requests no squash, if force squash is needed, please let me know.

Makefile

pizhenwei mentioned this pull request Jul 23, 2024

Fully abstract valkeyContextFuncs #51

Closed

pizhenwei force-pushed the feature-rdma branch from 83bc4a3 to 2921303 Compare July 24, 2024 01:53

michael-grunder requested changes Jul 25, 2024

View reviewed changes

pizhenwei force-pushed the feature-rdma branch 2 times, most recently from e519823 to aa40c41 Compare July 26, 2024 06:54

pizhenwei requested a review from michael-grunder July 26, 2024 07:49

bjosv reviewed Jul 30, 2024

View reviewed changes

pizhenwei added 4 commits July 30, 2024 20:09

Fix timing issue on reconnect

b526998

If necessary private context gets freed first, NULL pointer would be accessed in .close method. The right and safe timing should be .close then .free_privctx. Signed-off-by: zhenwei pi <[email protected]>

Add cmake support for RDMA

7e3f512

Signed-off-by: zhenwei pi <[email protected]>

pizhenwei force-pushed the feature-rdma branch from aa40c41 to 7e3f512 Compare July 30, 2024 12:30

michael-grunder approved these changes Jul 31, 2024

View reviewed changes

michael-grunder merged commit 35e0ec2 into valkey-io:main Aug 1, 2024
36 of 43 checks passed

pizhenwei deleted the feature-rdma branch August 1, 2024 10:54

bjosv reviewed Aug 1, 2024

View reviewed changes

Makefile Show resolved Hide resolved

bjosv added the experimental label Sep 3, 2024

michael-grunder mentioned this pull request Dec 4, 2024

RDMA support for valkey phpredis/phpredis#2594

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce RDMA transport (experimental) #55

Introduce RDMA transport (experimental) #55

pizhenwei commented Jul 23, 2024 •

edited

Loading

michael-grunder left a comment

pizhenwei commented Jul 26, 2024 •

edited

Loading

michael-grunder commented Jul 29, 2024

bjosv left a comment

pizhenwei commented Jul 30, 2024 •

edited

Loading

pizhenwei commented Jul 31, 2024

Introduce RDMA transport (experimental) #55

Introduce RDMA transport (experimental) #55

Conversation

pizhenwei commented Jul 23, 2024 • edited Loading

michael-grunder left a comment

Choose a reason for hiding this comment

pizhenwei commented Jul 26, 2024 • edited Loading

michael-grunder commented Jul 29, 2024

bjosv left a comment

Choose a reason for hiding this comment

pizhenwei commented Jul 30, 2024 • edited Loading

pizhenwei commented Jul 31, 2024

pizhenwei commented Jul 23, 2024 •

edited

Loading

pizhenwei commented Jul 26, 2024 •

edited

Loading

pizhenwei commented Jul 30, 2024 •

edited

Loading