Settings that enhance S3 performance on GET.

[unixsrv]

Configuration: 3 - (SCM, OM) 4 - (Datanode, S3G) with haproxy on top of S3G. NVME 6 disks per node are used, 6TB each, providing a throughput of 4 GB/s. In fact, we have a PUT speed (100 threads) of 4.5 GB/s and a GET speed (100 threads) of 2 GB/s. How can we improve the GET performance?

[unixsrv]

After assembling and updating from master branch, I got the following results:

Before the update:

        --- Total Results ---
Operation: get
Concurrency: 100
Total number of requests: 1000
Total number of unique objects: 1000
Failed requests: 0
Total elapsed time: 35.227772684s
Average request time: 5.885170619s
Minimum request time: 1.72112s
Maximum request time: 10.83071s
Nominal requests/s: 34.0
Actual requests/s: 28.4
Content throughput: 3633.496819 MB/s
Average Object Size: 1.34217728e+08

After the update from master branch:

        --- Total Results ---
Operation: get
Concurrency: 100
Total number of requests: 1000
Total number of unique objects: 1000
Failed requests: 0
Total elapsed time: 8.060409588s
Average request time: 701.705752ms
Minimum request time: 299.03ms
Maximum request time: 1.61175s
Nominal requests/s: 142.5
Actual requests/s: 124.1
Content throughput: 15880.086316 MB/s
Average Object Size: 1.34217728e+08
Total Object Size: 134217728000

Should we expect a release with a fix? Are there any timelines?

[ChenSammi]

What's your previous Ozone version? We just released Ozone 1.4.1.

[unixsrv]

Previous version was Ozone 1.4.1, in this version was slow GET. When update to version from master branch without change configuration perfomance increased about 5x.

[ChenSammi]

Ozone 2.0.0 for master branch is currently under discussion. It will happen next year.

[unixsrv]

Is there an understanding of what affects the performance, and is it possible to make these changes in version 1.4.1?

[ChenSammi]

We are continuously improving the performance. There are many new commits in master branch. It's not easy to say which among them contribute most to the performance improvement. One suggestion is pay attention to those commits which mainly focus on S3 gateway.

[julienlau]

If I understand well ozone-datanode + ozone-s3g + haproxy are colocated on the same servers ?
Is HAproxy configured in active-active mode ?
What is the network spec of the servers ?
I do not think your problem is related to network limits since you do not have the same performance for read and write.
Did you try with object size = 10MB ? less concurrency ?

I made my tests with minio/warp since it does not need the HAproxy layer : you can directly specificy all addresses of s3g instances and warp can make a round-robin.
please see this : #7501 for the snippet
You can also find some configuration file that may be useful.

[smileyboy2019]

The performance has changed after modification. When will it be released next and what is the version number