Skip to content

Commit

Permalink
[WIP] Proposal cvmfs+podman (HSF#643)
Browse files Browse the repository at this point in the history
* draft for cvmfs+podman

* update doc

* move in correct directory

* move in correct directory

* rewords and expand the project

* add mentors

* adapt to new format

* add redhat branding

* finalize project

* sort list of mentors

* move in correct directory
  • Loading branch information
siscia authored Jan 31, 2020
1 parent 592fc91 commit a084472
Show file tree
Hide file tree
Showing 7 changed files with 144 additions and 1 deletion.
11 changes: 11 additions & 0 deletions _gsocorgs/2020/redhat.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
title: "Red Hat"
author: "Simone Mosciatti"
layout: default
organization: RedHat
logo: REDHAT-logo.jpg
description: |
Red Hat offers a comprehensive software portfolio, covering 5 key technology areas: cloud, middleware, operating platform, storage, and virtualization, provided with a subscription model.
---

{% include gsoc_proposal.ext %}
2 changes: 1 addition & 1 deletion _gsocprojects/2019/project_CernVM-FS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: CernVM-FS
project: CernVM-FS
layout: default
logo: cernvmfs-logo.png
logo: CERNVM-FS-logo.png
description: |
The CernVM-File System (CernVM-FS) is a global, read-only POSIX file system that provides the universal namespace /cvmfs.
It is based on content-addressable storage, Merkle trees, and HTTP data transport. CernVM-FS provides a mission
Expand Down
13 changes: 13 additions & 0 deletions _gsocprojects/2020/project_CernVM-FS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
title: CernVM-FS
project: CernVM-FS
layout: default
logo: cernvmfs-logo.png
description: |
The CernVM-File System (CernVM-FS) is a global, read-only POSIX file system that provides the universal namespace /cvmfs.
It is based on content-addressable storage, Merkle trees, and HTTP data transport. CernVM-FS provides a mission
critical infrastructure to small and large HEP collaborations.
---


{% include gsoc_project.ext %}
116 changes: 116 additions & 0 deletions _gsocproposals/2020/proposal_CVMFSPodmanIntegration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
---
title: Podman CVMFS integration
layout: default
project: CernVM-FS
year: 2020
organization:
- CERN
---

## Description

[CernVM-FS (CVMFS)][cvmfs] is a globally-distributed filesystem used to efficiently distribute software to data centers and end-user workstations alike.

[Podman][podman] is an utility to run and use containers.
It provides the same command line interface than Docker but it runs without the need of a privileged daemon.
These two characteristics make it extremely interesting for workload used in scientific data-centers.

It has been shown that only a small portion of all the files in a container images is necessary to run the image itself.
This is even more accentuated in scientific container images since they usually include complex software stacks comprising hundreds of thousands of files, and often not all the files are needed for each task.
Our goal is to merge the lazy load capabilities of CVMFS with the container workflow allowed by podman, to quickly load big scientific container images while maintaining the isolation and convenience of containers.

There is already an [integration for Docker][docker-graphdriver], and another [for __containerd__ (kubernetes)][remote-containerd] is about ready.
All those implementations are based on the filesystem structure generated by [DUCC][ducc].

A CernVM-FS file system hosting container images and layers shows the following structure.
From a distribution point of view, in CernVM-FS a layer (or image) is a directory containing the unpacked files rather than a single tarball.


```
/cvmfs/unpacked.cern.ch/
│ 
├─ .layers
│ ├── 00
│ │   ├── 001dba6e0b44ff57a26d944d9a307ef39927e4882b45eb9d3c9257d754ef7d56
│ │   │   └── layerfs
│ │   │   ├── etc
│ │   │   ├── home
│ │   │   └── opt
│ │   └── 008deed8f79c35003fb8808e37c39245e244cd6af7498e5b7874ac7e186c7307
│ │      └── layerfs
│ │      └── code
│ └─ ... many more ...
└─ .flat
├── 02
│   ├── 0212054c85a9b966aa4f9c08048686603c7d0583067b759d14633070fcea30a1
│   │ ├── bin
│   │ ├── dev
│   │ ├── etc
│   │ ├── home
│   │ ├── lib
│   │ └── var
│   └── 027998886ae41faa55490baeb6b5e37f4295375ac5dcae5bcf3fe91f141687c2
│   ├── bin
│   ├── boot
│   ├── dev
│   ├── etc
│   ├── home
│   ├── lib
│   ├── lib64
│   ├── lost+found
│   ├── media
│   ├── pool
│   ├── root
│   ├── sbin
│   ├── tmp
│   ├── usr
│   └── var
└─ ... many more ...
```

The `.layer` directory store the content of the layers unpacked in an ordinary directory and the `.flat` directory stores the content of a whole container images, with each layer unpacked one of top of each other.


The project will be mentored from both CERN and Red Hat.

## Task Ideas

1. Enhance [DUCC][ducc] to automatically create filesystem structure in CVMFS that can be used by podman
2. Allow podman to understand the filesystem structure already present in CVMFS to load container images

## Expected Result

Allow podman to run container images directly from CVMFS or any other file system that host directories with the unpacked layer contents.

## Evaluation Task

Interested students can contact me ([Simone Mosciatti][simo]) directly for an evaluation task, it requires basic understanding of containers and FUSE filesystem.

## Requirements

The code-base will mostly be in Go(lang), hence it is necessary to know the language. It is also important to have a basic understanding of Linux.

## Mentors

* [Simone Mosciatti (CERN)][simo]
* [Giuseppe Scrivano (RedHat)](mailto:[email protected])
* [Jakob Blomer (CERN)](mailto:[email protected]@cern.ch)

## Links

* [CernVM-FS][cvmfs]
* [CVMFS Github][cvmfs-repo]
* [podman][podman]
* [podman Github][podman-repo]
* [podman storage][podman-storage]

[cvmfs]: http://cernvm.cern.ch/portal/filesystem
[cvmfs-repo]: https://github.com/cvmfs/cvmfs
[podman]: https://podman.io/
[podman-repo]: https://podman.io/
[podman-storage]: https://github.com/containers/storage
[simo]: mailto:[email protected]
[ducc]: https://github.com/cvmfs/cvmfs/tree/devel/ducc
[docker-graphdriver]: https://cvmfs.readthedocs.io/en/stable/cpt-graphdriver.html
[remote-containerd]: https://github.com/ktock/stargz-snapshotter/pull/27
3 changes: 3 additions & 0 deletions gsoc/2020/mentors.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ layout: plain
* Vasco Chibante Barroso [[email protected]](mailto:[email protected]) CERN
* Bertrand Bellenot [[email protected]](mailto:[email protected]) CERN
* Thomas Beermann [[email protected]](mailto:[email protected]) University of Wuppertal
* Jakob Blomer [[email protected]](mailto:[email protected]) CERN
* Ken Bloom [[email protected]](mailto:[email protected]) University of Nebraska-Lincoln
* Brian Bockelman [[email protected]](mailto:[email protected]) Morgridge Institute for Research
* Andy Buckley [[email protected]](mailto:[email protected]) UGlasgow
Expand All @@ -22,10 +23,12 @@ layout: plain
* David Lange [[email protected]](mailto:[email protected]) Princeton University
* Mario Lassnig [[email protected]](mailto:[email protected]) CERN
* Federica Legger [[email protected]](mailto:[email protected]) Universita e INFN Torino
* Simone Mosciatti [[email protected]](mailto:[email protected]) CERN
* Teo Mrnjavac [[email protected]](mailto:[email protected]) CERN
* Alexander Penev [[email protected]](mailto:[email protected]) University of Plovdiv "Paisii Hilendarski"
* Jim Pivarski [[email protected]](mailto:[email protected]) Princeton University
* Alex Richards [[email protected]](mailto:[email protected]) Imperial College London
* Giuseppe Scrivano [[email protected]](mailto:[email protected]) RedHat
* Oksana Shadura [[email protected]](mailto:[email protected]) University of Nebraska-Lincoln
* Mark Smith [[email protected]](mailto:[email protected]) Imperial College London
* Vassil Vassilev [[email protected]](mailto:[email protected]) Princeton University
Binary file added images/CERNVM-FS-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/REDHAT-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit a084472

Please sign in to comment.