forked from HSF/hsf.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[WIP] Proposal cvmfs+podman (HSF#643)
* draft for cvmfs+podman * update doc * move in correct directory * move in correct directory * rewords and expand the project * add mentors * adapt to new format * add redhat branding * finalize project * sort list of mentors * move in correct directory
- Loading branch information
Showing
7 changed files
with
144 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
--- | ||
title: "Red Hat" | ||
author: "Simone Mosciatti" | ||
layout: default | ||
organization: RedHat | ||
logo: REDHAT-logo.jpg | ||
description: | | ||
Red Hat offers a comprehensive software portfolio, covering 5 key technology areas: cloud, middleware, operating platform, storage, and virtualization, provided with a subscription model. | ||
--- | ||
|
||
{% include gsoc_proposal.ext %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
--- | ||
title: CernVM-FS | ||
project: CernVM-FS | ||
layout: default | ||
logo: cernvmfs-logo.png | ||
description: | | ||
The CernVM-File System (CernVM-FS) is a global, read-only POSIX file system that provides the universal namespace /cvmfs. | ||
It is based on content-addressable storage, Merkle trees, and HTTP data transport. CernVM-FS provides a mission | ||
critical infrastructure to small and large HEP collaborations. | ||
--- | ||
|
||
|
||
{% include gsoc_project.ext %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,116 @@ | ||
--- | ||
title: Podman CVMFS integration | ||
layout: default | ||
project: CernVM-FS | ||
year: 2020 | ||
organization: | ||
- CERN | ||
--- | ||
|
||
## Description | ||
|
||
[CernVM-FS (CVMFS)][cvmfs] is a globally-distributed filesystem used to efficiently distribute software to data centers and end-user workstations alike. | ||
|
||
[Podman][podman] is an utility to run and use containers. | ||
It provides the same command line interface than Docker but it runs without the need of a privileged daemon. | ||
These two characteristics make it extremely interesting for workload used in scientific data-centers. | ||
|
||
It has been shown that only a small portion of all the files in a container images is necessary to run the image itself. | ||
This is even more accentuated in scientific container images since they usually include complex software stacks comprising hundreds of thousands of files, and often not all the files are needed for each task. | ||
Our goal is to merge the lazy load capabilities of CVMFS with the container workflow allowed by podman, to quickly load big scientific container images while maintaining the isolation and convenience of containers. | ||
|
||
There is already an [integration for Docker][docker-graphdriver], and another [for __containerd__ (kubernetes)][remote-containerd] is about ready. | ||
All those implementations are based on the filesystem structure generated by [DUCC][ducc]. | ||
|
||
A CernVM-FS file system hosting container images and layers shows the following structure. | ||
From a distribution point of view, in CernVM-FS a layer (or image) is a directory containing the unpacked files rather than a single tarball. | ||
|
||
|
||
``` | ||
/cvmfs/unpacked.cern.ch/ | ||
│ | ||
├─ .layers | ||
│ ├── 00 | ||
│ │ ├── 001dba6e0b44ff57a26d944d9a307ef39927e4882b45eb9d3c9257d754ef7d56 | ||
│ │ │ └── layerfs | ||
│ │ │ ├── etc | ||
│ │ │ ├── home | ||
│ │ │ └── opt | ||
│ │ └── 008deed8f79c35003fb8808e37c39245e244cd6af7498e5b7874ac7e186c7307 | ||
│ │ └── layerfs | ||
│ │ └── code | ||
│ └─ ... many more ... | ||
└─ .flat | ||
├── 02 | ||
│ ├── 0212054c85a9b966aa4f9c08048686603c7d0583067b759d14633070fcea30a1 | ||
│ │ ├── bin | ||
│ │ ├── dev | ||
│ │ ├── etc | ||
│ │ ├── home | ||
│ │ ├── lib | ||
│ │ └── var | ||
│ └── 027998886ae41faa55490baeb6b5e37f4295375ac5dcae5bcf3fe91f141687c2 | ||
│ ├── bin | ||
│ ├── boot | ||
│ ├── dev | ||
│ ├── etc | ||
│ ├── home | ||
│ ├── lib | ||
│ ├── lib64 | ||
│ ├── lost+found | ||
│ ├── media | ||
│ ├── pool | ||
│ ├── root | ||
│ ├── sbin | ||
│ ├── tmp | ||
│ ├── usr | ||
│ └── var | ||
└─ ... many more ... | ||
``` | ||
|
||
The `.layer` directory store the content of the layers unpacked in an ordinary directory and the `.flat` directory stores the content of a whole container images, with each layer unpacked one of top of each other. | ||
|
||
|
||
The project will be mentored from both CERN and Red Hat. | ||
|
||
## Task Ideas | ||
|
||
1. Enhance [DUCC][ducc] to automatically create filesystem structure in CVMFS that can be used by podman | ||
2. Allow podman to understand the filesystem structure already present in CVMFS to load container images | ||
|
||
## Expected Result | ||
|
||
Allow podman to run container images directly from CVMFS or any other file system that host directories with the unpacked layer contents. | ||
|
||
## Evaluation Task | ||
|
||
Interested students can contact me ([Simone Mosciatti][simo]) directly for an evaluation task, it requires basic understanding of containers and FUSE filesystem. | ||
|
||
## Requirements | ||
|
||
The code-base will mostly be in Go(lang), hence it is necessary to know the language. It is also important to have a basic understanding of Linux. | ||
|
||
## Mentors | ||
|
||
* [Simone Mosciatti (CERN)][simo] | ||
* [Giuseppe Scrivano (RedHat)](mailto:[email protected]) | ||
* [Jakob Blomer (CERN)](mailto:[email protected]@cern.ch) | ||
|
||
## Links | ||
|
||
* [CernVM-FS][cvmfs] | ||
* [CVMFS Github][cvmfs-repo] | ||
* [podman][podman] | ||
* [podman Github][podman-repo] | ||
* [podman storage][podman-storage] | ||
|
||
[cvmfs]: http://cernvm.cern.ch/portal/filesystem | ||
[cvmfs-repo]: https://github.com/cvmfs/cvmfs | ||
[podman]: https://podman.io/ | ||
[podman-repo]: https://podman.io/ | ||
[podman-storage]: https://github.com/containers/storage | ||
[simo]: mailto:[email protected] | ||
[ducc]: https://github.com/cvmfs/cvmfs/tree/devel/ducc | ||
[docker-graphdriver]: https://cvmfs.readthedocs.io/en/stable/cpt-graphdriver.html | ||
[remote-containerd]: https://github.com/ktock/stargz-snapshotter/pull/27 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,6 +10,7 @@ layout: plain | |
* Vasco Chibante Barroso [[email protected]](mailto:[email protected]) CERN | ||
* Bertrand Bellenot [[email protected]](mailto:[email protected]) CERN | ||
* Thomas Beermann [[email protected]](mailto:[email protected]) University of Wuppertal | ||
* Jakob Blomer [[email protected]](mailto:[email protected]) CERN | ||
* Ken Bloom [[email protected]](mailto:[email protected]) University of Nebraska-Lincoln | ||
* Brian Bockelman [[email protected]](mailto:[email protected]) Morgridge Institute for Research | ||
* Andy Buckley [[email protected]](mailto:[email protected]) UGlasgow | ||
|
@@ -22,10 +23,12 @@ layout: plain | |
* David Lange [[email protected]](mailto:[email protected]) Princeton University | ||
* Mario Lassnig [[email protected]](mailto:[email protected]) CERN | ||
* Federica Legger [[email protected]](mailto:[email protected]) Universita e INFN Torino | ||
* Simone Mosciatti [[email protected]](mailto:[email protected]) CERN | ||
* Teo Mrnjavac [[email protected]](mailto:[email protected]) CERN | ||
* Alexander Penev [[email protected]](mailto:[email protected]) University of Plovdiv "Paisii Hilendarski" | ||
* Jim Pivarski [[email protected]](mailto:[email protected]) Princeton University | ||
* Alex Richards [[email protected]](mailto:[email protected]) Imperial College London | ||
* Giuseppe Scrivano [[email protected]](mailto:[email protected]) RedHat | ||
* Oksana Shadura [[email protected]](mailto:[email protected]) University of Nebraska-Lincoln | ||
* Mark Smith [[email protected]](mailto:[email protected]) Imperial College London | ||
* Vassil Vassilev [[email protected]](mailto:[email protected]) Princeton University |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.