Test environment for authenticated-HTTP data acces

[tcompa]

With @zonia3000, we prepared a testing environment for HTTP data access that requires BasicAuth (i.e. username and password) authentication.

1. Prepare the following Dockerfile:

FROM ghcr.io/fractal-analytics-platform/fractal-vizarr-viewer:stable

ENV CACHE_EXPIRATION_TIME=0
ENV FRACTAL_SERVER_URL=http://fake:8000
ENV ZARR_DATA_BASE_PATH=/zarr-files/
ENV AUTHORIZATION_SCHEME=testing-basic-auth

RUN apt install -y wget unzip

WORKDIR /zarr-files

RUN wget https://zenodo.org/records/10424292/files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1
RUN unzip 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1
RUN rm -rf __MACOSX 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1

WORKDIR /

2. Prepare the following build_and_run.sh script

#!/bin/bash

IMAGENAME=serve-zarr-behind-basic-auth

docker build . -t "$IMAGENAME"
docker run --network host -e TESTING_USERNAME=myusername -e TESTING_PASSWORD=mypassword "$IMAGENAME"

3. Run build_and_run.sh:

$ ./build_and_run.sh 

[+] Building 0.2s (10/10) FINISHED                                                                                                                                                                                                              docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 616B                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for ghcr.io/fractal-analytics-platform/fractal-vizarr-viewer:stable                                                                                                                                                          0.2s
 => [internal] load .dockerignore                                                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                                           0.0s
 => [1/7] FROM ghcr.io/fractal-analytics-platform/fractal-vizarr-viewer:stable@sha256:79f332cbc5f4e7a9cfa62dc28ab33a0a7026b60fdbcf0fc0f16e7e5a0c19cf3f                                                                                                    0.0s
 => CACHED [2/7] RUN apt install -y wget unzip                                                                                                                                                                                                            0.0s
 => CACHED [3/7] WORKDIR /zarr-files                                                                                                                                                                                                                      0.0s
 => CACHED [4/7] RUN wget https://zenodo.org/records/10424292/files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1                                                                                                                0.0s
 => CACHED [5/7] RUN unzip 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1                                                                                                                                                         0.0s
 => CACHED [6/7] RUN rm -rf __MACOSX 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1                                                                                                                                               0.0s
 => exporting to image                                                                                                                                                                                                                                    0.0s
 => => exporting layers                                                                                                                                                                                                                                   0.0s
 => => writing image sha256:a88314ced7a342f62d00ace861b10fc9d66e6381307626c3a7de1d1116780d6e                                                                                                                                                              0.0s
 => => naming to docker.io/library/serve-zarr-behind-basic-auth                                                                                                                                                                                           0.0s
2024-11-06 15:06:40,154 - WARN - Authorization scheme is set to "testing-basic-auth". Do not use in production!
2024-11-06 15:06:40,156 - INFO - fractal-vizarr-viewer is listening at http://localhost:3000/vizarr/

4. At this point, you can access a file if you use the myusername/mypassword credentials (as an example, try accessing http://localhost:3000/vizarr/data/zarr-files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/0/.zarray in the browser).

5. Prepare the following test_http_access.py script

import fsspec.implementations.http
import zarr
import aiohttp


url = (
    "http://localhost:3000/vizarr/data/zarr-files/"
    "20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/0"
)
print(f"Zarr array available at {url=}\n")

print("WITH CREDENTIALS:")
fs = fsspec.implementations.http.HTTPFileSystem(
    client_kwargs=dict(auth=aiohttp.BasicAuth("myusername", "mypassword"))
)
store = fs.get_mapper(url)
array_zarr = zarr.open_array(store)
print(f"{array_zarr=}\n")


print("WITHOUT CREDENTIALS")
fs = fsspec.implementations.http.HTTPFileSystem()
store = fs.get_mapper(url)
array_zarr = zarr.open_array(store)
print(f"{array_zarr=}")

6. Run test_http_access.py in an appropriate virtual environment:

python test_http_access.py

Zarr array available at url='http://localhost:3000/vizarr/data/zarr-files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/0'

WITH CREDENTIALS:
array_zarr=<zarr.core.Array (3, 1, 2160, 5120) uint16>

WITHOUT CREDENTIALS
Traceback (most recent call last):
  File "/home/tommaso/Fractal/http-zarr/open_remote_zarr_docker.py", line 24, in <module>
    array_zarr = zarr.open_array(store)
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/creation.py", line 649, in open_array
    init_array(
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/storage.py", line 455, in init_array
    _init_array_metadata(
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/storage.py", line 536, in _init_array_metadata
    shape = normalize_shape(shape) + dtype.shape
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/util.py", line 83, in normalize_shape
    raise TypeError("shape is None")
TypeError: shape is None

[tcompa]

cc @lorenzocerrone

In case it's useful, it is easy to add this to some GitHub Action.
The easiest would be to replace docker run with docker run --detach, and then you can directly use data in localhost:3000/vizarr/data/zarr-files/. And/or you can customize the Dockerfile to use whatever example dataset you have (e.g. something local, so that you won't have to fetch data from Zenodo during each build time).

[tcompa]

Also, for the record, there exist easier solutions (which are not based on fractal-vizarr-viewer).

E.g. you can mount a local zarr array via https://pypi.org/project/tiny-http-server/:

pip install -U tiny-http-server
tiny-http-server --auth myusername:mypassword
# and then head to localhost:8000

[lorenzocerrone]

Very cool!

For the moment, I am using a public ome-zarr on github. But yeah, the tiny-http-server seems to be a better solution for the normal CIs (plus, I can also test authentication).

The docker CI might be a bit overkill for the current state of ngio, but at some point it would make so much sense to use it for larger integration runs (maybe weekly scheduled actions).