[core] Support upper bound in dynamic bucket mode

[liyubin117]

Purpose

In dynamic bucket mode, unlimited buckets lead to an unpredicable number of small files, which lead to stability problems. so we should support upper bound in dynamic bucket mode.

Linked issue: close #4942

Tests

HashBucketAssignerTest
SimpleHashBucketAssignerTest

• Use new feature, Produce 3 buckets after inserting 12 rows

• Not use new feature, Produce 6 buckets after inserting 12 rows

API and Format

Documentation

docs/layouts/shortcodes/generated/core_configuration.html

[liyubin117]

@JingsongLi CI passed, PTAL, thanks!

[JingsongLi]

I'm not sure if these modifications are effective, so let me give you my suggestion:
We only need to modify PartitionIndex.asign inside:

Only code:

// 3. create a new bucket
for (int i = 0; i < Short.MAX_VALUE; i++) {
    if (bucketFilter.test(i) && !totalBucket.contains(i)) {
        hash2Bucket.put(hash, (short) i);
        nonFullBucketInformation.put(i, 1L);
        totalBucket.add(i);
        return i;
    }
}

// 4. too many buckets, throw exception
@SuppressWarnings("OptionalGetWithoutIsPresent")
int maxBucket = totalBucket.stream().mapToInt(Integer::intValue).max().getAsInt();
throw new RuntimeException(
        String.format(
                "Too more bucket %s, you should increase target bucket row number %s.",
                maxBucket, targetBucketRowNumber));

New code:

// 3. create a new bucket
for (int i = 0; i < max_buckets; i++) {
    if (bucketFilter.test(i) && !totalBucket.contains(i)) {
        hash2Bucket.put(hash, (short) i);
        nonFullBucketInformation.put(i, 1L);
        totalBucket.add(i);
        return i;
    }
}

// 4. exceed max_buckets, just pick a bucket for record.
pick a min bucket (belongs to this task) to the record.

[liyubin117]

After offline discussion with @JingsongLi , we have reached a consesus: We can't just update PartitionIndex logic because it doesn't handle SimpleHashBucketAssigner; When the buckets are full, a random bucket is selected for writing.

[liyubin117]

@JingsongLi The feature has been completed as discussed, Looking forward your review !

[JingsongLi]

I think here should be: max bucket should be smaller than maxBucketsNum.

If it is a job restarted from scratch, each task is increasing, and the previous judgment may be problematic.

[liyubin117]

Wise consideration! done :)

[JingsongLi]

I cannot understand getMaxBucketsPerAssigner and getSpecifiedMaxBuckets. Why not just same to maxBucketId < maxBucketsNum - 1?

[liyubin117]

Good question, After offline discussion, we have reached a consesus, done :)