replace all bytes amount-related usage of usize with u64 (and indexing into u64-sized files)

[laser]

Description

We should definitely use u64 any place where referring to a quantity of bytes. We should definitely use u64 when referring to an index into a structure that could have up to u64 indices.

usize is limited to 4294967295 (in decimal) on 32-bit architectures, which means ~4GiB if we're interpreting that number of as quantity of bytes. We know that sectors will eventually be much larger than 4GiB. In retrospect, the From<UnpaddedBytesAmount> for usize and From<PaddedBytesAmount> for usize impls were a mistake and should be removed.

Acceptance criteria

• remove From<*BytesAmount> for usize impls
• modify any place where we use usize for an "amount of bytes" such that it instead uses u64
• modify any place where we index into a u64-sized container (e.g. the call to sector_base::io::fr32::write_unpadded from get_unsealed_range) to use u64 for the type of the index

[steven004]

I'm taking this. It is expected to have a version mid. of next week.

[steven004]

@laser ， when I look into the system, I found that I need to understand the fr32.rs file first. In the fr32.rs file, there are lots of functions using usize as parameters' type instead of u64.

Is there an introduction of this file, regarding what it is for? I am thinking some of functions should be changed, but others may not be necessary.

[laser]

@steven004 - The fr32.rs file contains functions used to preprocess (bit pad) client data and to manipulate and inspect the resulting data.

What is preprocessing?

I'll quote the spec:

Preprocessing adds two zero bits after every 254 bits of original data, yielding a sequence of 32-byte blocks, each of which contains two zeroes in the most-significant bits, when interpreted as a little-endian number. That is, for each block, 0x11000000 & block[31] == 0.

Why preprocess?

We do this in order to produce 32-byte aligned sectors in which no 32-byte chunk overflows the field used for the proofs.

[porcuquine]

Although not directly applicable, I just want to point out that efficient replication of sectors larger than addressable RAM is not a design goal of the system. So, in practice, while we can avoid a class of problems by changing many usize types to u64 — this will be a somewhat illusory gain, since a system which requires that distinction won't be capable of replicating sectors large enough to exercise it.

It might be better just to detect this condition, report it as unsupported, and fail instead.

[steven004]

PR #641 is submitted for this issue.
This pr is not trying to solve all the conflict, but only some to avoid introducing new issues.

I did not touch fr32.rs since it is related to some low-level changes, and lots of changes should be done if we do that. Please advise if we should continue to change more places, or we just limit the program to run on 64-bit system.

[porcuquine]

I think #652 replaces this. Alternately, we can update this issue to reflect the plan.

Or, we can discuss here whether this is indeed the right approach.

@laser Thoughts?

@steven004 Thanks for all your efforts here: this is a complex and subtle issue which took some time to work through, and we wouldn't have if you had not pushed it forward.

cc: @dignifiedquire

[laser]

Yep, I too believe that 652 replaces this issue. There may be some follow up work to be done after 652 lands - but I’ll capture that work in a follow up issue.

…

On Tue, May 14, 2019, at 2:37 PM, porcuquine wrote: I think #652 <#652> replaces this. Alternately, we can update this issue to reflect the plan. Or, we can discuss here whether this is indeed the right approach. @laser <https://github.com/laser> Thoughts? @steven004 <https://github.com/steven004> Thanks for all your efforts here: this is a complex and subtle issue which took some time to work through, and we wouldn't have if you had not pushed it forward. cc: @dignifiedquire <https://github.com/dignifiedquire> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#622?email_source=notifications&email_token=AAGX6G5IMNDYFUQPROY57PTPVMBHVA5CNFSM4HG6JNFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVMMVRI#issuecomment-492358341>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGX6G27FVD25NUUTMK545DPVMBHVANCNFSM4HG6JNFA>.

[dignifiedquire]

@laser can we close this then?