gateway: add one more hierarchy to upload tmp/multiupload dir to reduce txn conflicts

[polyrabbit]

In the current gateway implementation, the tmp/upload directory is a flat structure, which can easily lead to concurrency conflicts under high concurrency. These conflicts severely impact performance and prevent the gateway from scaling.

We can reduce conflicts in those large directories by adding a layer of hierarchy.

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 42 lines in your changes missing coverage. Please review.

Project coverage is 45.71%. Comparing base (5714660) to head (63bef7d).
Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/gateway/gateway.go	33.33%	38 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5549      +/-   ##
==========================================
- Coverage   50.57%   45.71%   -4.87%     
==========================================
  Files         166      164       -2     
  Lines       48152    47873     -279     
==========================================
- Hits        24353    21884    -2469     
- Misses      21076    23349    +2273     
+ Partials     2723     2640      -83     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[davies]

This is a behavior change, I think we should introduce a parameter to enable it.

[davies]

The CompleteMultipartUpload should also try to lookup parts in different dirs.

[polyrabbit]

OK, updated CompleteMultipartUpload to also look up in old flat dir when part is not found in new dir. I suppose this can made CompleteMultipartUpload smooth during upgrade, and we can avoid one extra flag.

Please review:
https://github.com/juicedata/juicefs/compare/21200b52e86135325cd705138e7a20e6ab0ecd74..4bbe479feefc402a9c50fe309a5a0a680cd69e9a