feat(auto-balance): disk in pressure

[c3y1huang]

Which issue(s) this PR fixes:

Issue longhorn/longhorn#4105

What this PR does / why we need it:

Implement a new function to sync data contents by local files.

Special notes for your reviewer:

None

Additional documentation or context

longhorn/longhorn-engine#1003 (review)

[derekbit]

nit: Blocks is the block size here and a bit weird. We can rename it to block size in the future.

[derekbit]

Did a quick review.

What syncContent() does

1. Open the source file once and get a source file descriptor
2. Create the target file once and get a target file descriptor
3. Get data interval layout from the source file
4. Initiate multiple workers (numSyncWorkers) for concurrently reading from the source file and writing to the target file.

We don't have any synchronization in the implementation. The concurrent readers and writers on a single source file descriptor and a single target file descriptor can indeed introduce race conditions.

[c3y1huang]

Did a quick review.

What syncContent() does

1. Open the source file once and get a source file descriptor
2. Create the target file once and get a target file descriptor
3. Get data interval layout from the source file
4. Initiate multiple workers (numSyncWorkers) for concurrently reading from the source file and writing to the target file.

We don't have any synchronization in the implementation. The concurrent readers and writers on a single source file descriptor and a single target file descriptor can indeed introduce race conditions.

Wouldn't they be on different interval due to the fileIntervalChannel?

[derekbit]

Wouldn't they be on different interval due to the fileIntervalChannel?

Each worker seeks the file descriptor to a specific offset and reads or writes data, so multiple workers will race.

After discussing with @c3y1huang, we decided to use a single worker for now. We can improve it by using multiple workers in the future.

Using a single worker might not be a bad approach because:

• Sequential read and write operations usually perform well.
• We don't consume the entire disk IOPS.

cc @innobead