Adding Batching After Every Block

[eshwarprasadS]

The Problem:

Resolves #480, related to #424

Key Changes Implemented

Re-batch After Each Block:

After processing the dataset through a block, split it into batches based on the current batch_size (reuses ._split_dataset within class Pipeline)

Sequential Execution for Each Block:

Process batches sequentially within each block, and does not spawn any more threads of execution.

Recombine Batch Outputs:

Combine the processed batches back into a single dataset before sending it to the next block. (reuses concatenate_datasets())

[bbrowning]

Let's make sure we pull the jinja template change into a separate PR so that we can track/merge that fix separately from this batching change.

[eshwarprasadS]

Let's make sure we pull the jinja template change into a separate PR so that we can track/merge that fix separately from this batching change.

Got it, will revert that change, so it can be addressed separately.

[bbrowning]

@eshwarprasadS If you need to reproduce the ruff errors locally that happened during the CI run, see https://github.com/instructlab/sdg/blob/d59731f2eb4c1dec66549d0ce4bf031ead8d7088/CONTRIBUTING.md#running-code-formatting-and-linting-checks-locally on how to do that. Basically, tox -e ruff -- check if your local dev environment is setup properly.

[bbrowning]

Just a small question around how we check if batching is enabled, but otherwise the logic looks good to me and the new tests look great. Thanks for the thoroughness in fixing and testing this!

[nathan-weinberg]

@eshwarprasadS Please don't include references to RHEL AI Jira tickets in PRs - changed the title of this one

[bbrowning]

Looks good - thanks for getting this fix in!

[khaledsulayman]

LGTM!