Splitting storage to folders

[sergey-shandar]

Firstly, these folders:

• roots/
• parts/

If we would like to reduce the number of files in the folder. We can splif further:

https://stackoverflow.com/questions/466521/how-many-files-can-i-put-in-a-directory

# of symbols	# of folders	# of files for one level	# of files for two levels
1	32	a/ 1_024	a/b/ 32768
2	1024	ab/ 1_048_576	ab/cd/ 1_073_741_824
3	32768	abc/ 1_073_741_824	abc/def/ 35_184_372_088_832

Git keeps 2 hex symbols. 256 folders.

I like two symbols ab/, two levels ab/cd/. With a limit of about 1024 files and folders in one folder, we can keep about 1 billion files which will occupy several TB of space.

Q: should we do it for roots as well?

Note: filenames can also be reduced.

Disadvantages:

• to synchronize two repositories, we need to copy the files recursively.
• if the file name is reduced, we cut 20 bits from the hash. It became 204 bits (41 characters) if someone just copied the file. However, the hash of the file can be easily restored, so it shouldn't be an issue. We can have an application that restores hashes and puts the files into the right folder. The algorithm can even find out if the file is a block or a part.

[sergey-shandar]

@o2alexanderfedin