512 sarus (#526)

* added sarus doc 🍔

* enhanced doc 🎏

* added conda-prefix in dockerfile to have correct set env-var in sarus container 👨

* bugfix gpu-testing 📒

* skip pinocchio-test because it seems not to work properly on each platform 😖

* integrated user-build testing in workflow 8️⃣

* updated dockerfile-dev 🎅

* removed test-user-package because testing is now happening before pushing the image 👠

* adapted docker-dev 🍹

* Revert "removed test-user-package because testing is now happening before pushing the image 👠"

This reverts commit a828dfd.

* updated dockerfile-dev 🔌

* changed dev-workflow for testing 💀

* removed workflow-run trigger of dev-image because out of memory 🚈

* added mpich std-loc in Dockerfile.user 🍞

* adapted docker-user ⛎

* minor bugfix dockerfile-user 📥

* updated environments-files 🎇

* updated mpi-version in dockerfile 🆎

* updated Dockerfile-user structure 👾

* added mpi-tests to karabo 🌱

* updated requirements.txt with new versions 🎼

* integrated mpi-tests into github workflow and codecov-report 🆖

* minor fix using pytest-cov instead of coverage in github workflow 🌊

* fixed coverage-files discovery (I think) 🏧

* adapted codecov workflow according to github.com/codecov/codecov-action 🎲

* adapted container-doc 🍗

* bugfix Dockerfile mpi-compilation needs python 🍚

* minor adjustments in Dockerfile.user 😉

* added mpi-supported h5py wheel as karabo dependency 🍟

* removed pinocchio from environment.yaml 🔘

* updated environment.yaml with mpich-deps 🈴

* updated conda-build files 🚟

* replaced np.object0 with according non-deprecated alias 🐳

* minor bugfix in pyproject.toml 🙇

* deleted test-pinocchio (wrongly merged) 🍭

* updated environment files 🐎

* imporved dev-setup 🔉

* removed requirements.txt 

* added ipykernel to dev-deps 🚙

* bugfix filter-sources for xarray>2023.2 🔦

* updated build-procedure ⛪

* implemented dynamic versioneering 😔

* removed remaining requirements.txt stuff 👜

* updated Dockerfile-user mpi-installation to be dependent on karabo-user installation version 🛀

* refactored dockerfiles 🐭

* added versioneer to meta.yaml build-stage 📚

* added versioneer to meta.yaml build-stage 🚕

* bugfix conda-build meta.yaml for versioneer 💩

* adapted conda-build to custom pkg-version 😤

* adapted Dockerfile.user to a testable (before release) setup 👸

* bugfix build-user-image.yml ➿

* bugfix build-user-image.yml 🚧

* updated Dockerfile-user 👏

* bugfix user-dockerfile 🕘

* bugfix Dockerfile-dev 🔖

* minor doc-update in dockerfile-dev 🚸

* updated description of build-user-image workflow inputs 🔫

* introduced venv in dockerfile-user to not f*** up base env 😌

* updated dev-img with venv 😓

* updated build-user-image workflow to be able to run on workflow-dispatch 🐬

* added file to build-and-export-docker action 📌

* bugfix build-args in build-user-image workflow 👨

* bugfix get-shallow git-repo from git-rev (not only branches or tags) 📷

* bugfix remote from https and not from ssh 🎮

* updated workflow-dispatch descriptions ‼️

* bugfix build-args passing to docker-build-push action 🆘

* throw exit-code in dockerfile-user if build not set correctly 🎵

* added dev-flag to conda-build 🌹

* added security to dev-builds in conda-build workflow 🛄

* bugfix dev-evaluation in conda-build.yml 🈂️

* outcommented mpich-compilation in Dockerfile 😥

* bugfix conda-build export of env-vars in same step ⛺

* added failing tests if PR is draft 🆖

* setup build-user-img, conda-build & test-user-package for dev-testing 🔍

* added df-h to build-user-image workflow 🎃

* made docker-build by myself ✂️

* made docker-build by myself entirely 🚥

* ensured to push on ghcr.io ✈️

* bugfix docker-push user-image workflow 💘

* bugfix docker-push user-image workflow 📢

* bugfix added img-name to push image 🚸

* bugfix registry docker push 🍎

* adapted docker-img address accordingly to ghcr.io 🎩

* adapted build-user-image to standard 🐗

* added pytest installation to docker-user image testing workflow 🎢

* bugfix pytest-call in user-image 💴

* added bash shell in docker-run in build-test-user-image 👥

* adapted ld-library-path to base-image 2️⃣

* made site-package-location identification in build-user-image more robust 📵

* defined entrypoint to Dockerfile-user ➡️

* bugfix calling tests in build-user-image 🐻

* readded conda-activate-karabo to .bashrc for interactive mode 💓

* bugfix removed unnecessary " at the end of docker-run 🚅

* set env-vars in test-user-image ♓

* minor changes 🎵

* adapted test-workflow to main-setup 💡

* adapted docker-img dev to karabo-venv 📄

* adapted mpi-doc ❇️

* addressed mypy-issues 6️⃣

* added type-ignore to __init__.py because mypy can't handle that 🍥

* adapted exclude-option in setup.cfg to hopefully ignore __init__.py on the runners 🐽

* adapted mypy-exclude-regex to exclude all __init__.py ⏰

* hopefully bugfix to ignore __init__.py by mypy 🍟

* trying editable install to avoid duplicate modules 😓

* added verbose-flag to pytest-testing of docker-image 📫

* removed pytest-mpi as dep and added --only-mpi flag handling in conftest.py 👪

* minor changes in pytest test-discovery 🎁

* removed mpi-pytest from codecov because of race-condition issue which doesn't seem to be solvable atm 👬

* bugfix compute filter-mask in filter-sky 👫

* ugly hotfix to initialize dask-mpi inside docker-container 🏯

* added mpi-tests to docker-user tests 💀

* bugfix only enter dask-initialize with mpi if mpirun 🐚

* bugfix conda-build inherited env-vars [skip ci] 👚

* bugfix conda-build inherited env-vars [skip ci] 🚑

* bugfix conda-build set output-vars [skip ci] 🌈

* removed notebook-test from user-img test because data-dirs are not part of package and therefore would cause an error [skip ci] 🕠

* bugfix casting to str-to-boolean for reproducable github-workflows 🔩

* added debugging logs to github workflows [skip ci] ☎️

* adapted meta.yaml to pin compatible numpy ❓

* updated conda-build to be closer to a best-practise build 😿

* fixed bluebild-version 🌅

* removed echos and improved build-user-image workflow 🔻

* removed build-string default to ensure that workflow-user knows what he/she does 🐟

* minor update in container.md ⏩

* bugfix check leading v in tag in conda-build workflow 🍙

* bugfix set dev-string check correclty in conda-build.yml 🚷

* added versioneer to dev-dep ⚪

* improved version definition security for conda-build workflows ♒

* minor imporvements in conda-build ⤵️

* hopefully bugfix of github boolean passing in reusable workflow see #1483 of github runners 🎈

* improved version definition security for conda-build workflows ↪️

* added little verbosity to conda-build workflow 📍

* hopefully bugfix to trigger build-docker by taking boolean values directly from input 👂

* bugfix build-user-image.yml 🛅

* minor improvements in build-user-image.yml 🎫

* replaced exit 2 with exit 1 in all bash scripts 3️⃣

* bugfix install environment in user Dockerfile correctly 💙

* removed weird leading v to github-workflows version-args 🍚

* updated codecov.yml to not fail if below target 🎰

* bugfix: added python interpreter and versioneer through conda for checking conda-build worklow inputs 🇷🇺

* added conda-prefix to python interpreter to hopefully get viable binary Ⓜ️

* added conda-prefix to python interpreter to hopefully get viable binary 

* removed dev-deps in environment.yaml and meta.yaml ⌛

* bugfix: removed bdsf dev-dep 💒

* addressed mypy-issues 🚝

* updated mypy-complaints chunks-dict issue in sky-model 🚞

* fixed build-nr of feedstock-deps 🐹

* addresses mypy attr-defined for matplotlib BLUE 😾

* renamed build-user-image do build-docker-image 🐚

* adapted readme-badges 🍰

* addressed pr-request to install mpich via apt 🐒

* adapted documentation and dockerfile-steps 🍶

* bugfix added -y to apt install in Dockerfile [skip ci] ↩️

* improved docs ♈

* changed Dockerfile setup to use andromeda user instead of root 😉

* minor update in dockerfile 🔛

* changed run-stages of dockerfile 🍲

* removed user-changeing because of singularity uid issues 🔫

* moved Dockerfile to root & removed docker-dir 🚌

* added bash-env to env-vars for singularity noninteractive-shell [skip ci] 🕣

* bugfix: correctly activate venv in docker & singularity container for interactive and non-interactive shells 💛

* fixed conda activate functionality for docker & singularity interactive and non-interactive shells 💉

* removed unnecessary sourcing in dockerfile [skip ci] 🔣

* added ldconfig after mpich-installation in dockerfile 👤

* minor changes in dockerfile [skip ci] 🐟

* outcommented mpi-hook from dockerfile because it's still error-prone 📢

* adapted container-doc ↩️

* bugfix: in test docker-image to enable --only-mpi custom flag for pytest 📓

* added karabo shared lib to ldconfig cache to enable native cscs-mpi-hook 🕡

* replaced weird file-handler root-dir setup with /tmp with honor of TMP, TMPDIR & TEMP 🌐

* changed tmp-dir-name setup to avoid collisions 🐂

* loosened mpich-version constraints because we no longer rely on apt to install mpich ◻️

* updated mpi-doc 😡

* minor bugfix for mpi-tests in ci 

* added scratch as a possible tmpdir in FileHandler ⏰

* bugfix get-tmp-dir 🐲

* addressed PR-requests 🎍

* addressed PR526 requests 🍀

* bugfix removed set libmamba-solver globally in installation-user.md 📦

.gitattributes

@@ -0,0 +1 @@
+karabo/_version.py export-subst

.github/workflows/build-docker-image.yml

@@ -0,0 +1,121 @@
+name: Build Docker Image
+
+on:
+  workflow_call:
+    inputs:
+      verstag:
+        type: string
+        required: true
+      latest:
+        type: boolean
+        required: false
+        default: false
+
+  workflow_dispatch:
+    inputs:
+      gitrev:
+        type: string
+        required: false
+        default: ""
+        description: "gitrev: commit-hash (full) | branch | tag if not current commit."
+      verstag:
+        type: string
+        required: true
+        description: "version: PEP440 version-tag of Karabo. DON't trigger build if you don't know what PEP440 is!"
+      latest:
+        type: boolean
+        required: false
+        default: false
+        description: "tag image as 'latest'?"
+      test:
+        type: boolean
+        required: false
+        default: false
+        description: "create env from environment.yaml instead of conda-wheel?"
+
+env:
+  REGISTRY: ghcr.io
+  IMG_NAME: karabo-pipeline
+
+jobs:
+  build-test-and-push-image:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to the Container registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Setup metadata img-name & img-tag
+        shell: bash -l {0}
+        run: |
+          if [[ ${{ github.event_name }} == "workflow_call" ]]; then
+            echo "gitrev=$GITHUB_SHA" >> "$GITHUB_ENV"
+            echo "build=user" >> "$GITHUB_ENV"
+          elif [[ ${{ github.event_name }} == "workflow_dispatch" ]]; then
+            if [[ "${{ inputs.gitrev }}" != "" ]]; then
+              echo "gitrev=${{ inputs.gitrev }}" >> "$GITHUB_ENV"
+            else
+              echo "gitrev=$GITHUB_SHA" >> "$GITHUB_ENV"
+            fi
+            if [[ "${{ inputs.test }}" == 'true' ]]; then
+              echo "build=test" >> "$GITHUB_ENV"
+            else
+              echo "build=user" >> "$GITHUB_ENV"
+            fi
+          else
+            echo "Invalid github-event!"
+            exit 1
+          fi
+          echo "latest=${{ inputs.latest }}" >> "$GITHUB_ENV"
+          echo "version=${{ inputs.verstag }}" >> "$GITHUB_ENV"
+          REPO_OWNER=${{ github.repository_owner }}
+          echo "IMG_ADDR=${{ env.REGISTRY }}/${REPO_OWNER@L}/${{ env.IMG_NAME }}" >> "$GITHUB_ENV"
+          DEV_STR="dev"
+          if [[ "${{ inputs.verstag }}" == *"$DEV_STR"* ]] && [[ "${{ inputs.latest }}" == 'true' ]]; then
+            echo "Invalid configuration of workflow-inputs!"
+            exit 1
+          fi
+
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ github.repository }}
+          tags: |
+            type=raw, enable=${{ env.latest }}, value=latest
+            type=raw, value=${{ env.version }}
+
+      - name: Docker build
+        shell: bash -l {0}
+        run: |
+          docker build \
+            --build-arg GIT_REV=${{ env.gitrev }} \
+            --build-arg BUILD=${{ env.build }} \
+            --build-arg KARABO_VERSION=${{ env.version }} \
+            -f Dockerfile \
+            -t ${{ env.IMG_ADDR }}:${{ env.version }} \
+            .
+          if [[ ${{ env.latest }} == 'true' ]]; then
+            docker tag ${{ env.IMG_ADDR }}:${{ env.version }} ${{ env.IMG_ADDR }}:latest
+          fi
+
+      - name: Test image
+        run: |  # karabo-sitepackage-location used for mpirun instead of --pyargs because --only-mpi is a custom-flag of karabo which lives in the site-packages
+          docker run --rm ${{ env.IMG_ADDR }}:${{ env.version }} bash -c \
+            'export IS_GITHUB_RUNNER=true RUN_GPU_TESTS=false RUN_NOTEBOOK_TESTS=false; pytest --pyargs karabo.test; SITE_PKGS=$(pip show karabo-pipeline | grep Location | sed "s/.*\(\/opt\/conda.*\).*/\1/"); mpirun -n 2 pytest --only-mpi $SITE_PKGS/karabo/test'
+      - name: Docker push
+        shell: bash -l {0}
+        run: |
+          docker push --all-tags ${{ env.IMG_ADDR }}

.github/workflows/build-docs.yml

@@ -20,7 +20,7 @@ jobs:
         run: |
           conda env create -n karabo_dev_env -f environment.yaml
           conda activate karabo_dev_env
-          pip install -r requirements.txt
+          pip install ".[dev]"
       - name: Build Docs
         shell: bash -l {0}
         run: |

.github/workflows/conda-build.yml

@@ -4,37 +4,127 @@ on:
   release:
     types: [published]
   workflow_dispatch:
+    inputs:
+      buildNumber:
+        type: string
+        required: true
+        description: "build-nr: anaconda.org build-nr (DON'T trigger build if you don't know how to set it)"
+      version:
+        type: string
+        required: true
+        description: "version: PEP440 package-version (DON'T trigger build if you don't know what PEP440 is)"
+      buildDocker:
+        type: boolean
+        required: false
+        default: false
+        description: "build docker-img if conda-build & tests succeeded?"
+      latestDocker:
+        type: boolean
+        required: false
+        default: false
+        description: "tag docker-img as latest (if `buildDocker` is enabled)"
+  workflow_call:
+    inputs:
+      buildNumber:
+        type: string
+        required: true
+      version:
+        type: string
+        required: true
+      buildDocker:
+        type: boolean
+        required: false
+        default: false
+      latestDocker:
+        type: boolean
+        required: false
+        default: false
 
 
 jobs:
   conda-build:
     runs-on: ubuntu-latest
     container: ghcr.io/i4ds/mambabuild-docker:latest
-
+    outputs:
+      karabo-version: ${{ steps.bcs.outputs.karabo_version }}
+      build-docker: ${{ steps.bcs.outputs.build_docker == 'true' }}
+      latest-docker: ${{ steps.bcs.outputs.latest_docker == 'true' }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Get Previous tag
         uses: actions-ecosystem/action-get-latest-tag@v1
         id: get-latest-tag
-      - name: Install conda build
+      - name: Build Conda
+        id: bcs
         shell: bash -l {0}
-        id: channels
         run: |
-          KARABO_TAG=${{ steps.get-latest-tag.outputs.tag }}
+          DEV_STR="dev"
+          if [[ ${{ github.event_name }} == 'release' ]]
+          then
+            echo "Conda build from release"
+            build="0"
+            KARABO_TAG=${{ steps.get-latest-tag.outputs.tag }}
+            if [[ ${KARABO_TAG:0:1} == "v" ]]; then
+              KARABO_VERSION="${KARABO_TAG:1}"
+            else
+              echo "invalid karabo-tag: has no leading v"
+              exit 1
+            fi
+            BUILD_DOCKER=true
+            LATEST_DOCKER=true
+          elif [[ ${{ github.event_name }} == 'workflow_dispatch' ]] || [[ ${{ github.event_name }} == 'workflow_call' ]]; then
+            echo "Conda build from dispatch | call"
+            build=${{ inputs.buildNumber }}
+            BUILD_DOCKER=${{ inputs.buildDocker }}
+            LATEST_DOCKER=${{ inputs.latestDocker }}
+            KARABO_VERSION=${{ inputs.version }}
+            conda install -c conda-forge python versioneer
+            VERSIONEER_VERSION=$(/opt/conda/bin/python -c 'import versioneer; print(versioneer.get_version())')
+            if [[ "$VERSIONEER_VERSION" == *"+"* ]]; then
+              if [[ "$KARABO_VERSION" != *"$DEV_STR"* ]]; then
+                echo "Dirty commit-version needs to be a PEP440 conform dev-version"
+                exit 1
+              fi
+            else
+              if [[ "$KARABO_VERSION" != "$VERSIONEER_VERSION" ]]; then
+                echo "Provided version $KARABO_VERSION doesn't match the actual version $VERSIONEER_VERSION"
+                exit 1
+              fi
+            fi
+          else
+            echo "Not specified github-event occured"
+            exit 1
+          fi
+          if [[ "$KARABO_VERSION" == *"$DEV_STR"* ]]; then
+            if [[ $LATEST_DOCKER == 'true' ]]; then
+              echo "Dev build docker image can not be tagged as latest!"
+              exit 1
+            fi
+            build="$(($build + 1000))"
+          fi
+
+          export KARABO_VERSION=$KARABO_VERSION build=$build
+          echo "karabo_version=$KARABO_VERSION" >> $GITHUB_OUTPUT
+          echo "build_docker=$BUILD_DOCKER" >> $GITHUB_OUTPUT
+          echo "latest_docker=$LATEST_DOCKER" >> $GITHUB_OUTPUT
+
           conda config --append channels i4ds
           conda config --append channels nvidia/label/cuda-11.7.0
           conda config --append channels conda-forge
-      - name: Build Conda
-        shell: bash -l {0}
-        run: |
+
           cd conda
-          KARABO_TAG=${{ steps.get-latest-tag.outputs.tag }}
-          export KARABO_VERSION="${KARABO_TAG:1}"
-          echo "__version__ = \"${KARABO_VERSION}\"" > karabo/version.py
           conda mambabuild .
       - name: Publish to Conda
         shell: bash -l {0}
         run: |
           conda activate base
           anaconda -t ${{ secrets.ANACONDA_SECRET }} upload /opt/conda/conda-bld/linux-64/karabo-pipeline-*.tar.bz2 --force
+  
+  test-build:
+    needs: conda-build
+    uses: ./.github/workflows/test-user-package.yml
+    with:
+      version: ${{ needs.conda-build.outputs.karabo-version }}
+      buildDocker: ${{ needs.conda-build.outputs.build-docker == 'true' }}
+      latestDocker: ${{ needs.conda-build.outputs.latest-docker == 'true' }}