535 singularity (#541)

* added sarus doc 🍔

* enhanced doc 🎏

* added conda-prefix in dockerfile to have correct set env-var in sarus container 👨

* bugfix gpu-testing 📒

* skip pinocchio-test because it seems not to work properly on each platform 😖

* integrated user-build testing in workflow 8️⃣

* updated dockerfile-dev 🎅

* removed test-user-package because testing is now happening before pushing the image 👠

* adapted docker-dev 🍹

* Revert "removed test-user-package because testing is now happening before pushing the image 👠"

This reverts commit a828dfd.

* updated dockerfile-dev 🔌

* changed dev-workflow for testing 💀

* removed workflow-run trigger of dev-image because out of memory 🚈

* added mpich std-loc in Dockerfile.user 🍞

* adapted docker-user ⛎

* minor bugfix dockerfile-user 📥

* updated environments-files 🎇

* updated mpi-version in dockerfile 🆎

* updated Dockerfile-user structure 👾

* added mpi-tests to karabo 🌱

* updated requirements.txt with new versions 🎼

* integrated mpi-tests into github workflow and codecov-report 🆖

* minor fix using pytest-cov instead of coverage in github workflow 🌊

* fixed coverage-files discovery (I think) 🏧

* adapted codecov workflow according to github.com/codecov/codecov-action 🎲

* adapted container-doc 🍗

* bugfix Dockerfile mpi-compilation needs python 🍚

* minor adjustments in Dockerfile.user 😉

* added mpi-supported h5py wheel as karabo dependency 🍟

* removed pinocchio from environment.yaml 🔘

* updated environment.yaml with mpich-deps 🈴

* updated conda-build files 🚟

* replaced np.object0 with according non-deprecated alias 🐳

* minor bugfix in pyproject.toml 🙇

* deleted test-pinocchio (wrongly merged) 🍭

* updated environment files 🐎

* imporved dev-setup 🔉

* removed requirements.txt 

* added ipykernel to dev-deps 🚙

* bugfix filter-sources for xarray>2023.2 🔦

* updated build-procedure ⛪

* implemented dynamic versioneering 😔

* removed remaining requirements.txt stuff 👜

* updated Dockerfile-user mpi-installation to be dependent on karabo-user installation version 🛀

* refactored dockerfiles 🐭

* added versioneer to meta.yaml build-stage 📚

* added versioneer to meta.yaml build-stage 🚕

* bugfix conda-build meta.yaml for versioneer 💩

* adapted conda-build to custom pkg-version 😤

* adapted Dockerfile.user to a testable (before release) setup 👸

* bugfix build-user-image.yml ➿

* bugfix build-user-image.yml 🚧

* updated Dockerfile-user 👏

* bugfix user-dockerfile 🕘

* bugfix Dockerfile-dev 🔖

* minor doc-update in dockerfile-dev 🚸

* updated description of build-user-image workflow inputs 🔫

* introduced venv in dockerfile-user to not f*** up base env 😌

* updated dev-img with venv 😓

* updated build-user-image workflow to be able to run on workflow-dispatch 🐬

* added file to build-and-export-docker action 📌

* bugfix build-args in build-user-image workflow 👨

* bugfix get-shallow git-repo from git-rev (not only branches or tags) 📷

* bugfix remote from https and not from ssh 🎮

* updated workflow-dispatch descriptions ‼️

* bugfix build-args passing to docker-build-push action 🆘

* throw exit-code in dockerfile-user if build not set correctly 🎵

* added dev-flag to conda-build 🌹

* added security to dev-builds in conda-build workflow 🛄

* bugfix dev-evaluation in conda-build.yml 🈂️

* outcommented mpich-compilation in Dockerfile 😥

* bugfix conda-build export of env-vars in same step ⛺

* added failing tests if PR is draft 🆖

* setup build-user-img, conda-build & test-user-package for dev-testing 🔍

* added df-h to build-user-image workflow 🎃

* made docker-build by myself ✂️

* made docker-build by myself entirely 🚥

* ensured to push on ghcr.io ✈️

* bugfix docker-push user-image workflow 💘

* bugfix docker-push user-image workflow 📢

* bugfix added img-name to push image 🚸

* bugfix registry docker push 🍎

* adapted docker-img address accordingly to ghcr.io 🎩

* adapted build-user-image to standard 🐗

* added pytest installation to docker-user image testing workflow 🎢

* bugfix pytest-call in user-image 💴

* added bash shell in docker-run in build-test-user-image 👥

* adapted ld-library-path to base-image 2️⃣

* made site-package-location identification in build-user-image more robust 📵

* defined entrypoint to Dockerfile-user ➡️

* bugfix calling tests in build-user-image 🐻

* readded conda-activate-karabo to .bashrc for interactive mode 💓

* bugfix removed unnecessary " at the end of docker-run 🚅

* set env-vars in test-user-image ♓

* minor changes 🎵

* adapted test-workflow to main-setup 💡

* adapted docker-img dev to karabo-venv 📄

* adapted mpi-doc ❇️

* addressed mypy-issues 6️⃣

* added type-ignore to __init__.py because mypy can't handle that 🍥

* adapted exclude-option in setup.cfg to hopefully ignore __init__.py on the runners 🐽

* adapted mypy-exclude-regex to exclude all __init__.py ⏰

* hopefully bugfix to ignore __init__.py by mypy 🍟

* trying editable install to avoid duplicate modules 😓

* added verbose-flag to pytest-testing of docker-image 📫

* removed pytest-mpi as dep and added --only-mpi flag handling in conftest.py 👪

* minor changes in pytest test-discovery 🎁

* removed mpi-pytest from codecov because of race-condition issue which doesn't seem to be solvable atm 👬

* bugfix compute filter-mask in filter-sky 👫

* ugly hotfix to initialize dask-mpi inside docker-container 🏯

* added mpi-tests to docker-user tests 💀

* bugfix only enter dask-initialize with mpi if mpirun 🐚

* bugfix conda-build inherited env-vars [skip ci] 👚

* bugfix conda-build inherited env-vars [skip ci] 🚑

* bugfix conda-build set output-vars [skip ci] 🌈

* removed notebook-test from user-img test because data-dirs are not part of package and therefore would cause an error [skip ci] 🕠

* bugfix casting to str-to-boolean for reproducable github-workflows 🔩

* added debugging logs to github workflows [skip ci] ☎️

* adapted meta.yaml to pin compatible numpy ❓

* updated conda-build to be closer to a best-practise build 😿

* fixed bluebild-version 🌅

* removed echos and improved build-user-image workflow 🔻

* removed build-string default to ensure that workflow-user knows what he/she does 🐟

* minor update in container.md ⏩

* bugfix check leading v in tag in conda-build workflow 🍙

* bugfix set dev-string check correclty in conda-build.yml 🚷

* added versioneer to dev-dep ⚪

* improved version definition security for conda-build workflows ♒

* minor imporvements in conda-build ⤵️

* hopefully bugfix of github boolean passing in reusable workflow see #1483 of github runners 🎈

* improved version definition security for conda-build workflows ↪️

* added little verbosity to conda-build workflow 📍

* hopefully bugfix to trigger build-docker by taking boolean values directly from input 👂

* bugfix build-user-image.yml 🛅

* minor improvements in build-user-image.yml 🎫

* replaced exit 2 with exit 1 in all bash scripts 3️⃣

* bugfix install environment in user Dockerfile correctly 💙

* removed weird leading v to github-workflows version-args 🍚

* updated codecov.yml to not fail if below target 🎰

* bugfix: added python interpreter and versioneer through conda for checking conda-build worklow inputs 🇷🇺

* added conda-prefix to python interpreter to hopefully get viable binary Ⓜ️

* added conda-prefix to python interpreter to hopefully get viable binary 

* removed dev-deps in environment.yaml and meta.yaml ⌛

* bugfix: removed bdsf dev-dep 💒

* addressed mypy-issues 🚝

* updated mypy-complaints chunks-dict issue in sky-model 🚞

* fixed build-nr of feedstock-deps 🐹

* addresses mypy attr-defined for matplotlib BLUE 😾

* renamed build-user-image do build-docker-image 🐚

* adapted readme-badges 🍰

* addressed pr-request to install mpich via apt 🐒

* adapted documentation and dockerfile-steps 🍶

* bugfix added -y to apt install in Dockerfile [skip ci] ↩️

* improved docs ♈

* changed Dockerfile setup to use andromeda user instead of root 😉

* minor update in dockerfile 🔛

* changed run-stages of dockerfile 🍲

* removed user-changeing because of singularity uid issues 🔫

* moved Dockerfile to root & removed docker-dir 🚌

* added bash-env to env-vars for singularity noninteractive-shell [skip ci] 🕣

* bugfix: correctly activate venv in docker & singularity container for interactive and non-interactive shells 💛

* fixed conda activate functionality for docker & singularity interactive and non-interactive shells 💉

* removed unnecessary sourcing in dockerfile [skip ci] 🔣

* added ldconfig after mpich-installation in dockerfile 👤

* minor changes in dockerfile [skip ci] 🐟

* outcommented mpi-hook from dockerfile because it's still error-prone 📢

* adapted container-doc ↩️

* bugfix: in test docker-image to enable --only-mpi custom flag for pytest 📓

* added karabo shared lib to ldconfig cache to enable native cscs-mpi-hook 🕡

* replaced weird file-handler root-dir setup with /tmp with honor of TMP, TMPDIR & TEMP 🌐

* changed tmp-dir-name setup to avoid collisions 🐂

* loosened mpich-version constraints because we no longer rely on apt to install mpich ◻️

* updated mpi-doc 😡

* minor bugfix for mpi-tests in ci 

* added scratch as a possible tmpdir in FileHandler ⏰

* bugfix get-tmp-dir 🐲

* added cachetools as dep 😇

* redesigned FileHandler for short- and long-term-memory caching 😡

* improved file-handler by getting unique tmp-dir per unique object 💥

* refactored image and imager to new filehandler-setup 😔

* adapted interferometer.py and telescope.py to new FileHandler setup ↪️

* adapted visibility and sourcedetection to new FileHandler setup 🙌

* enhanced FileHandler get-tmp-dir with subdir & mkdir option 🚘

* adapted Karabo to new FileHandler setup 🐍

* bugfix accessing .ltm and .stm through FileHandler 👅

* bugfixes FileHandler & according tests 📝

* changed Downloadobject-storage cache from site-packages to tmp 🌁

* removed weird KaraboResource pseudo-interface from repo ♥️

* removed FileHandler get-tmp-dir subdir option because seems unnecessary ⛽

* implemented seed-option to FileHandler.get-tmp-dir 🍢

* changed ltm & stm of FileHandler from static functions to lazy class-attributes 📄

* intermediate commit separating Dask & Slurm concerns in dask.py 🕔

* adapted other karabo-files to new DaskHandler setup 🚨

* readded accidentally removed plotting-util.py 👡

* bugfix dask-usage 🍀

* minor bugfix in test-dask 🚬

* refactored create_baseline_cut_telelescope to improved disk-caching & be less error-prone 😷

* bugfixes in Telescope.create-baseline-cut-telescope 🔑

* addressed mypy-issues 😐

* updated singularity-doc 🍓

* addressed PR-requests 🎍

* addressed PR526 requests 🍀

* bugfix removed set libmamba-solver globally in installation-user.md 📦

* made plot-function api more consistent 😰

* refactored DaskHandler to use just a single DaskHandler class for function-calling purpose 🎼

* bugfix removed fetch-dask-handler function from karabo 🎁

* addressed PR540 requests 🐻

* bugfix seeding ltm-memory 🎵

karabo/data/external_data.py

@@ -64,7 +64,7 @@ def get_object(self, remote_file_path: str, verbose: bool = True) -> str:
         else:
             purpose = None
         local_cache_dir = FileHandler().get_tmp_dir(
-            prefix="objects-download",
+            prefix="objects-download-",
             term="long",
             purpose=purpose,
         )

karabo/util/file_handler.py

@@ -272,6 +272,8 @@ def get_tmp_dir(
                 )
             if seed is not None:
                 seed = prefix + str(seed)
+            else:
+                seed = prefix
             dir_name = _get_rnd_str(k=10, seed=seed)
             dir_name = "".join((prefix, dir_name))
             dir_path = os.path.join(dir_path, dir_name)