512 sarus

[Lukas113]

closes #449
closed #189

For testing-purpose, this PR needs it's dev-dependencies, referenced in environment.yaml and meta.yaml. However, BEFORE merging into the default brach, they need to be adapted and the according builds of the Karabo-Feedstock must be built at more or less the same time.

This PR was initially intended to be much smaller. However, because I wasn't able to test my changes with the old setup, I also had to define a dev & test-setup for Karabo to not interfere with the live conda-wheel and docker-images.

This PR also includes changes in the Krabo-Feedstock, where a lot of dependencies were adjusted. The reason was originally because we had some open-mpi builds, which needed to be adapted. However, during the time I was working on this PR, we decided to remove the mpi-builds with the PR #521 from Karabo. This may seem like the work I was doing there is not necessary. However, I also needed to adapt the build-string setup to enable best-practice conda-builds, which depends usually on the python-version, package-hash and build-nr. Only then, the conda installer will be able to select the dependency wheels properly and can decide whether to download & install a new wheel or take it from the cache. The same goes for dev-builds. In addition, I loosened well-known and stable dependency-constraints. This had a positive side-effect that we are no more constrained as we used to be regarding upgrading dependencies, which solves the issue #449 .

This PR has the following changes in this repository:

• Intoducing dev- and test-setup to GitHub workflows
• Changed from open-mpi to mpich
• Added mpi4py tests
• Made codecov & pytest compatible with mpi-tests
• Loosened non-feedstock dependencies like numpy, dask or xarray.
• Included dev-builds of feedstock with different dependency-resolving to Karabo (need to be replaced before merging)
• Introducing dynamic versioning of Karabo using versioneer
• Changed packaging setup using pyproject.toml instead of setup.cfg -> got rid of requirements.txt
• Changed setup of user & dev- Dockerfile and made it theoretically ready for native mpich-hook (hook is disabled atm, because compilation takes too much time & memory) -> Details see Dockerfiles
• Adapted docs accordingly
• bugfix filter-sources for xarray>2023.2
• bugfix tests with cuda if cuda-is-available, but cuda-tests are disabled
• updated deprecated & buggy code in Karabo & it's dependencies for newer package-versions like numpy or xarray

[Lukas113]

Currently not working are Singularity containers due to permission issues & native mpi-hook of Sarus containers. As discussed in the meeting at 21-12-23, these will be fixed in another PR.

[Lukas113]

I've fixed the mpi-issue and have tested a simple example on cscs srun -N2 -n2 -C gpu -A sk05 sarus run --mpi ghcr.io/i4ds/karabo-pipeline:0.21.dev1 python -c 'print("ABCDEFG")' with and without the mpi-hook and it worked (output was 2x ABCDEFG). The only thing left to do is that karabo also works for singularity containers, which is work in progress.

Also all mpi-tests passed with the following command: srun -N2 -n2 -C gpu -A sk05 sarus run --mpi ghcr.io/i4ds/karabo-pipeline:0.21.dev1 pytest --only-mpi /opt/conda/envs/karabo/lib/python3.9/site-packages/karabo/test

[sfiruch]

👍 Mostly minor details, but also some things I'd think we should double-check.