DREs(Digital Research Environments ) for TU Delft and research groups

[jurra]

Common issues related to Research Software and Data workflows and computational environments

Disclaimer: I have no idea of how complicated it is to set up DREs, but I bet is quite complicated. These are comments more as a user than as a solution provider. Any thoughts from experienced colleagues in ICT to learn more on the subject are very welcome.

During the two rounds of DCC support I have seen some patterns of needs within research groups that range from data archiving to collaboration, environment management, infrastructural needs such as setting up servers and applications within the servers, and so on.

The most simple use case is setting up jupyter notebook environments to teach coding, the most advanced use cases may involve connecting environments or servers to storage solutions, but also doing continuous integrations and pipelines that connect research data produced in these environments with archives like Zenodo or 4TU. Setting up all these is extremely consuming, requires a lot of experience, and doesn't ensure that a full FAIR workflow is implemented.

The value proposition of DREs

An Infrastructure as a service that allows research groups to create workspaces. Such workspaces are secure cloud enviornments connected on demand to all kinds of services that include version control, package installations, connection to computing resources as well as storage solutions. Very similar to what Serkan did at Twente and the crib platform: https://crib.utwente.nl/

Here are some quotes of definitions from DREs provides I have found:

anDREa provides workspaces to safely work on data with your own tooling, the compute, storage and the infrastructure you need, in collaboration.

The Digital Research Environment (DRE) is a cloud based, globally available research environment where data is stored and organized securely and researchers can quickly generate workspaces to collaborate in and use the applications they love.

Observations and takeaways

• Customized cloud based environments managed by research groups or labs can be more organic than one solution fits all. For instance a Lab hosting their own git management system where they invite external collaborators is more flexible than inviting external collaborators to TU delft gitlab instance. (Research projects and domains dont have boundaries, they are cross institutional so the solutions should consider these aspects).

• Research Workspaces or environmets as a concept can be more suitable to address researchers needs than just providing virtual machines or servers. In the end the whole point is that these servers are used to setup such workspaces.

What do you think about DREs as a concept? Any ideas, experiences, comments?

References

[1]“8. AnDREa demo (Ralph van der Hoorn - Radboudumc.pdf.” Accessed: Jun. 01, 2021. [Online]. Available: https://www.surf.nl/files/2019-06/8.%20AnDREa%20demo%20%28Ralph%20van%20der%20Hoorn%20-%20Radboudumc.pdf
[2]“Digital Research Environment | Trusted Data Sharing Network | Digital Research Environment.” https://www.aridhia.com/digital-research-environment/ (accessed Jun. 01, 2021).
[3]“DRE Portal.” https://mydre.org/ (accessed Jun. 01, 2021).
[4]“Geospatial Computing Portal.” https://crib.utwente.nl/ (accessed Jun. 01, 2021).

[mwakok]

SURF is offering this as a service called Research Cloud. It links up to a SURF Research Drive and SURF HPC Cloud and allows you to collaborate with researchers accross different institutions.

The service itself is free, but using the underlying computing and storage infrastructure is not.

[mwakok]

This just popped up on my radar: https://2i2c.org/infrastructure/.

[jurra]

Thanks Maurits, i remembered long ago something like this was published in the slack channel of the datachampions and I was not able to find it

[jurra]

I also came accross the 2i2c documentation on getting started with their hubs.
https://pilot-hubs.2i2c.org/en/latest/reference/hubs.html

[jurra]

Following up on this subject of Digital Research Environments :

I have been reading a bit about Gitlab capabilities, particularly how it can integrate with Kubernetes. Why is this relevant?

1. Read on how Gitlab and Kubernetes work together: https://about.gitlab.com/solutions/kubernetes/

2. We have a gitlab instance at TU Delft, which means that we can activate/integrate different kind of services with Gitlab such as runners, gitlab pages, etc. The advantage of this is that once code has been changed in a repository new deployments of the code can take place continuously and in an integrated fashion.

3. If we want to scale up the deployment and management of applications for research software like Dask, Jupyterhubs and notebooks, etc, containers and kubernetes might be more suitable than traditional VMs. They are more flexible, scalable and managable than Vms. It is much more sustainable and efficient to run a jupyterhub on a container using kubernetes than assigning a Vm which resources might fall short or big in certain scenarios. I am talking mostly about standard applications that need to be deployed recurrently and in large quantities, this might also include static websites, which brings me to my next point, falling a bit out of DREs to some extent.

4. Gitlab comes with a special service called Gitlab Pages, it is a service that runs on a Go server and handles securily the deployment of static websites (html5), which is often requested by many research groups. Enabling and facilitating the generation of such websites following certain standards and best practices can become easier using something like gitlab, instead of selfhosted websites in Vms. There are many reasons why this could be more suitable, you would have one approach for all websites in terms of deployment, security and resource management.

The 4 points touch more on integrations and secure infrastructure to support researchers than just computational environments.

These are my opinions from the limited research I have been able to make, feedback from more experienced colleagues is more than welcome to contrast these opinions :)