[CI] Unpin Dask, adopt CUDA 12.8 and RAPIDS 24.12

[hcho3]

Related: dmlc/xgboost-devops#9, dmlc/xgboost-devops#11

Use latest Dask, CUDA 12.8, and RAPIDS 24.12.

Also it's now easy to update the image tag in all pipelines simultaneously.

[hcho3]

@trivialfis We have a problem here: Latest RAPIDS cannot be installed with Dask 2024.10.0, and yet our tests cannot pass with latest Dask. 😢

[jakirkham]

Thanks Hyunsu! 🙏

What are the failures we are seeing?

[hcho3]

What are the failures we are seeing?

See:

• Test fails with Dask 2024.11.0+ #10994
• https://github.com/dmlc/xgboost/actions/runs/13057002018/job/36432128685?pr=11194 (from [CI] Unpin Dask, adopt CUDA 12.8 and RAPIDS 24.12 #11194)

E distributed.client.FutureCancelledError: ('_argmax-16ef6ea6596412a85b175e90dab51ecb', 44) cancelled for reason: unknown.
/opt/miniforge/envs/gpu_test/lib/python3.10/site-packages/distributed/client.py:2427: FutureCancelledError

[trivialfis]

We have a problem here: Latest RAPIDS cannot be installed with Dask 2024.10.0, and yet our tests cannot pass with latest Das

I assume it's caused by dask dataframe. Can we disable some of the tests for now? Or use arrays in place of dataframes? I think it will take some time for dask to go over all the issues. It's unlikely we halt the ci for that.

[hcho3]

Yes, I'll probably skip the problematic test, as done in the 2.1 branch:

xgboost/tests/test_distributed/test_gpu_with_dask/test_gpu_with_dask.py

Lines 388 to 393 in 62e7923

	@pytest.mark.xfail(
	dask_version_ge110, reason="Test cannot pass with Dask 2024.11.0+"
	)
	@pytest.mark.skipif(**tm.no_cudf())
	@pytest.mark.parametrize("model", ["boosting"])
	def test_dask_classifier(self, model: str, local_cuda_client: Client) -> None:

[hcho3]

@trivialfis I change the CI scripts a bit to make it easy to test new CI images. Now we only have to change a single file ops/pipeline/get-image-tag.sh. See the updated instructions in https://xgboost--11194.org.readthedocs.build/en/11194/contrib/ci.html#making-changes-to-ci-containers.

TODO.

1. Wait until all the tests pass for this PR.
2. Merge Upgrade to CUDA 12.8 xgboost-devops#11.
3. Update this PR to change ops/pipeline/get-image-tag.sh to IMAGE_REPO=main.

[trivialfis]

Looks good to me, one minor question.

[trivialfis]

This is not in the stdlib, do we need to specify it in our testing environments?

[hcho3]

Currently, our CI already has packaging installed (due to it being a dependency of other packages we use). However, if we were to add a [test] suite for optional dependencies in pyproject.toml, we should have packaging in that suite.