feat: fuse-overlayfs configuration (#2685)

* feat: fuse-overlayfs configuration

Signed-off-by: dkwon17 <[email protected]>

* Update

Signed-off-by: dkwon17 <[email protected]>

* Indentation

Signed-off-by: dkwon17 <[email protected]>

* Update

Signed-off-by: dkwon17 <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update butane version

Signed-off-by: dkwon17 <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/administration-guide/pages/configuring-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/accessing-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/accessing-fuse.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/enabling-overlay-with-a-configmap.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/using-the-fuse-overlay-storage-driver.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/enabling-overlay-with-a-configmap.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/enabling-overlay-with-a-configmap.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/using-the-fuse-overlay-storage-driver.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/enabling-overlay-with-a-configmap.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update modules/end-user-guide/pages/enabling-overlay-with-a-configmap.adoc

Co-authored-by: Jana Vrbkova <[email protected]>

* Update prerequisites

Signed-off-by: dkwon17 <[email protected]>

* Fix indentation formatting

Signed-off-by: dkwon17 <[email protected]>

* Mention Buildah

Signed-off-by: dkwon17 <[email protected]>

* Fix build

Signed-off-by: dkwon17 <[email protected]>

---------

Signed-off-by: dkwon17 <[email protected]>
Co-authored-by: Jana Vrbkova <[email protected]>

modules/administration-guide/nav.adoc

@@ -80,6 +80,7 @@
 *** xref:configuring-cluster-roles-for-users.adoc[]
 *** xref:configuring-advanced-authorization.adoc[]
 *** xref:removing-user-data-in-compliance-with-the-gdpr.adoc[]
+** xref:configuring-fuse.adoc[]
 * xref:managing-ide-extensions.adoc[]
 ** xref:extensions-for-microsoft-visual-studio-code-open-source.adoc[]
 * xref:managing-workloads-using-the-che-server-api.adoc[]

modules/administration-guide/pages/configuring-fuse.adoc

@@ -0,0 +1,144 @@
+:_content-type: PROCEDURE
+:description: Configuring fuse-overlayfs
+:keywords: administration-guide, configuring, fuse
+:navtitle: Configuring fuse-overlayfs
+:page-aliases:
+
+[id="configuring-fuse-overlayfs"]
+= Configuring fuse-overlayfs
+
+By default, the Universal Developer Image (UDI) contains Podman and Buildah which you can use to build and push container images within a workspace.
+However, Podman and Buildah in the UDI are configured to use the `vfs` storage driver which does not provide copy-on-write support.
+For more efficient image management, use the fuse-overlayfs storage driver which supports copy-on-write in rootless environments.
+
+
+== Enabling container access to /dev/fuse for OpenShift
+
+To use fuse-overlayfs, you must make `/dev/fuse` accessible to workspace containers first.
+
+[WARNING]
+====
+Creating `MachineConfig` resources on an OpenShift cluster is a potentially dangerous task, as you are making advanced, system-level changes to the cluster.
+
+View the link:https://docs.openshift.com/container-platform/{ocp4-ver}/post_installation_configuration/machine-configuration-tasks.html#machine-config-overview-post-install-machine-configuration-tasks[MachineConfig documentation] for more details and possible risks.
+
+====
+
+.Prerequisites
+
+* The link:https://docs.openshift.com/container-platform/{ocp4-ver}/installing/install_config/installing-customizing.html#installation-special-config-butane-install_installing-customizing[Butane] tool (`butane`) is installed in the operating system you are using.
+
+* An active `{orch-cli}` session with administrative permissions to the destination OpenShift cluster. See {orch-cli-link}.
+
+.Procedure
+
+. Set the environment variable based on the type of your OpenShift cluster: a single node cluster, or a multi node cluster with separate control plane and worker nodes.
++
+* For a single node cluster, set:
++
+[subs="+quotes,+attributes,+macros"]
+----
+$ NODE_ROLE=master
+----
++
+* For a multi node cluster, set:
+[subs="+quotes,+attributes,+macros"]
++
+----
+$ NODE_ROLE=worker
+----
+
+. Set the environment variable for the OpenShift Butane config version. This variable is the major and minor version of the OpenShift cluster. For example, `4.12.0`, `4.13.0`, or `4.14.0`.
++
+[subs="+quotes,+attributes,+macros"]
+----
+$ VERSION=4.12.0
+----
++
+. Create a `MachineConfig` resource that creates a drop-in CRI-O configuration file named `99-podman-fuse` in the `NODE_ROLE` nodes. This configuration file makes access to the `/dev/fuse` device possible for certain pods.
++
+[source,subs="+quotes,+macros"]
+----
+cat << EOF | butane | oc apply -f -
+variant: openshift
+version: ${VERSION}
+metadata:
+  labels:
+    machineconfiguration.openshift.io/role: ${NODE_ROLE}
+  name: 99-podman-dev-fuse-${NODE_ROLE}
+storage:
+  files:
+  - path: /etc/crio/crio.conf.d/99-podman-fuse <1>
+    mode: 0644
+    overwrite: true
+    contents: <2>
+      inline: |
+        [crio.runtime.workloads.podman-fuse] <3>
+        activation_annotation = "io.openshift.podman-fuse" <4>
+        allowed_annotations = [
+          "io.kubernetes.cri-o.Devices" <5>
+        ]
+        [crio.runtime]
+        allowed_devices = ["/dev/fuse"] <6>
+EOF
+----
+<1> The absolute file path to the new drop-in configuration file for CRI-O.
+<2> The content of the new drop-in configuration file.
+<3> Define a `podman-fuse` workload.
+<4> The pod annotation that activates the `podman-fuse` workload settings.
+<5> List of annotations the `podman-fuse` workload is allowed to process.
+<6> List of devices on the host that a user can specify with the `io.kubernetes.cri-o.Devices` annotation.
++
+. After applying the `MachineConfig` resource, scheduling will be temporarily disabled for each node with the `worker` role as changes are applied. View the nodes' statuses.
++
+[subs="+quotes,+attributes,+macros"]
+----
+$ oc get nodes
+----
++
+Example output:
++
+[subs="+quotes,+attributes,+macros"]
+----
+NAME                           STATUS                     ROLES    AGE   VERSION
+ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.27.9
+ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.27.9
+ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.27.9
+ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.27.9
+ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.27.9
+ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.27.9
+----
++
+. Once all nodes with the `worker` role have a status `Ready`, `/dev/fuse` will be available to any pod with the following annotations.
++
+[source,yaml,subs="+quotes,+attributes"]
+----
+io.openshift.podman-fuse: ''
+io.kubernetes.cri-o.Devices: /dev/fuse
+----
+
+.Verification steps
+
+. Get the name of a node with a `worker` role:
++
+[subs="+attributes,+quotes"]
+----
+$ oc get nodes
+----
+
+. Open an `oc debug` session to a worker node.
++
+[subs="+attributes,+quotes"]
+----
+$ oc debug node/__<nodename>__
+----
+
+. Verify that a new CRI-O config file named `99-podman-fuse` exists.
++
+[subs="+attributes,+quotes"]
+----
+sh-4.4# stat /host/etc/crio/crio.conf.d/99-podman-fuse
+----
+
+== Using fuse-overlayfs for Podman and Buildah within a workspace
+Users can follow xref:end-user-guide:using-the-fuse-overlay-storage-driver.adoc[] to update existing workspaces to use the fuse-overlayfs storage driver for Podman and Buildah.

modules/end-user-guide/nav.adoc

@@ -15,6 +15,9 @@
 ** xref:starting-a-workspace-from-a-raw-devfile-url.adoc[]
 ** xref:basic-actions-you-can-perform-on-a-workspace.adoc[]
 ** xref:authenticating-to-a-git-server-from-a-workspace.adoc[]
+** xref:using-the-fuse-overlay-storage-driver.adoc[]
+*** xref:accessing-fuse.adoc[]
+*** xref:enabling-overlay-with-a-configmap.adoc[]
 * xref:using-che-in-team-workflow.adoc[]
 ** xref:first-time-contributors.adoc[]
 ** xref:benefits-of-pull-requests-review-in-che.adoc[]

modules/end-user-guide/pages/accessing-fuse.adoc

@@ -0,0 +1,37 @@
+:_content-type: PROCEDURE
+:description: Accessing /dev/fuse from your workspaces.
+:keywords: user-guide, fuse, overlay, device, /dev/fuse, pod, overrides
+:navtitle: Accessing /dev/fuse
+:page-aliases:
+
+[id="accessing-fuse"]
+= Accessing /dev/fuse from a workspace
+
+You must have access to `/dev/fuse` to use fuse-overlayfs. This section describes how to make `/dev/fuse` accessible to workspace containers.
+
+.Prerequisites
+
+* The administrator has enabled access to `/dev/fuse` by following xref:administration-guide:configuring-fuse.adoc[].
+* Determine a workspace to use fuse-overlayfs with.
+
+.Procedure
+
+. Use the `pod-overrides` attribute to add the required annotations defined in xref:administration-guide:configuring-fuse.adoc[] to the workspace. The `pod-overrides` attribute allows merging certain fields in the workspace pod's `spec`.
++
+[subs="+quotes,+attributes,+macros"]
+----
+$ {orch-cli} patch devworkspace __<DevWorkspace_name>__ \
+  --patch '{"spec":{"template":{"attributes":{"pod-overrides":{"metadata":{"annotations":{"io.kubernetes.cri-o.Devices":"/dev/fuse","io.openshift.podman-fuse":""}}}}}}}' \
+  --type=merge
+----
+
+.Verification steps
+
+. Start the workspace and verify that `/dev/fuse` is available in the workspace container.
++
+[subs="+attributes,+quotes"]
+----
+$ stat /dev/fuse 
+----
+
+After completing this procedure, follow the steps in xref:enabling-overlay-with-a-configmap.adoc[] to use fuse-overlayfs for Podman.

modules/end-user-guide/pages/enabling-overlay-with-a-configmap.adoc

@@ -0,0 +1,93 @@
+:_content-type: PROCEDURE
+:description: You can enable fuse-overlayfs storage driver for Podman
+:keywords: overlay, configmap, podman, storage, driver, fuse-overlayfs
+:navtitle: Enabling fuse-overlayfs with a ConfigMap
+:page-aliases:
+
+[id="enabling-fuse-overlay-with-a-configmap"]
+= Enabling fuse-overlayfs with a Configmap
+
+You can define the storage driver for Podman and Buildah in the `~/.config/containers/storage.conf` file. Here are the default contents of the `/home/user/.config/containers/storage.conf` file in the UDI container:
+
+.`storage.conf`
+[source]
+----
+[storage]
+driver = "vfs"
+----
+
+To use fuse-overlayfs, `storage.conf` can be set to the following:
+
+.`storage.conf`
+[source]
+----
+[storage]
+driver = "overlay"
+
+[storage.options.overlay]
+mount_program="/usr/bin/fuse-overlayfs" <1>
+----
+<1> The absolute path to the `fuse-overlayfs` binary. The `/usr/bin/fuse-overlayfs` path is the default for the UDI.
+
+You can do this manually after starting a workspace. Another option is to build a new image based on the UDI with changes to `storage.conf` and use the new image for workspaces.
+
+Otherwise, you can update the `/home/user/.config/containers/storage.conf` for all workspaces in your {orch-namespace} by creating a ConfigMap that mounts the updated file. See xref:mounting-configmaps.adoc[].
+
+.Prerequisites
+
+* The administrator has enabled access to `/dev/fuse` by following xref:administration-guide:configuring-fuse.adoc[].
+
+* A workspace with the required annotations are set by following xref:accessing-fuse.adoc[]
+
+
+[NOTE]
+====
+Since ConfigMaps mounted by following xref:mounting-configmaps.adoc[this guide] mounts the ConfigMap's data to all workspaces, following this procedure will set the storage driver to fuse-overlayfs for all workspaces. Ensure that your workspaces contain the required annotations to use fuse-overlayfs by following xref:accessing-fuse.adoc[].
+====
+
+.Procedure
+
+. Apply a ConfigMap that mounts a `/home/user/.config/containers/storage.conf` file in your {orch-namespace}.
++
+[source,yaml,subs="+quotes,+attributes,+macros"]
+----
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: fuse-overlay
+  labels:
+    controller.devfile.io/mount-to-devworkspace: 'true'
+    controller.devfile.io/watch-configmap: 'true'
+  annotations:
+    controller.devfile.io/mount-as: file
+    controller.devfile.io/mount-path: /home/user/.config/containers
+data:
+  storage.conf: |
+    [storage]
+    driver = "overlay"
+
+    [storage.options.overlay]
+    mount_program="/usr/bin/fuse-overlayfs"
+----
+
+.Verification steps
+
+. Start the workspace containing the required annotations and verify that the storage driver is `overlay`.
++
+[subs="+attributes,+quotes"]
+----
+$ podman info | grep overlay
+----
+
++
+Example output:
++
+[subs="+attributes,+quotes"]
+----
+graphDriverName: overlay
+  overlay.mount_program:
+    Executable: /usr/bin/fuse-overlayfs
+    Package: fuse-overlayfs-1.12-1.module+el8.9.0+20326+387084d0.x86_64
+      fuse-overlayfs: version 1.12
+  Backing Filesystem: overlayfs
+----

modules/end-user-guide/pages/getting-started-with-che.adoc

@@ -14,3 +14,4 @@ If your organization is already running a {prod-short} instance, you can get sta
 * xref:starting-a-workspace-from-a-raw-devfile-url.adoc[]
 * xref:basic-actions-you-can-perform-on-a-workspace.adoc[]
 * xref:authenticating-to-a-git-server-from-a-workspace.adoc[]
+* xref:using-the-fuse-overlay-storage-driver.adoc[]

modules/end-user-guide/pages/using-the-fuse-overlay-storage-driver.adoc

@@ -0,0 +1,25 @@
+:_content-type: CONCEPT
+:description: Using the fuse-overlayfs storage driver for Podman and Buildah
+:keywords: fuse, overlay, fuse-overlayfsm, podman, buildah, storage, driver
+:navtitle: Using the fuse-overlayfs storage driver for Podman and Buildah
+:page-aliases:
+
+[id="using-the-fuse-overlay-storage-driver-for-podman-and-buildah"]
+= Using the fuse-overlayfs storage driver for Podman and Buildah
+
+By default, newly created workspaces that do not specify a devfile will use the Universal Developer Image (UDI).
+The UDI contains common development tools and dependencies commonly used by developers.
+
+Podman and Buildah are included in the UDI, allowing developers to build and push container images from their workspace.
+
+By default, Podman and Buildah in the UDI are configured to use the `vfs` storage driver.
+For more efficient image management, use the fuse-overlayfs storage driver which supports copy-on-write in rootless environments.
+
+You must meet the following requirements to fuse-overlayfs in a workspace:
+
+. The administrator has configured `/dev/fuse` access on the cluster by following xref:administration-guide:configuring-fuse.adoc[].
+. The workspace has the necessary annotations for using the `/dev/fuse` device. See xref:accessing-fuse.adoc[].
+. The `storage.conf` file in the workspace container has been configured to use fuse-overlayfs. See xref:enabling-overlay-with-a-configmap.adoc[].
+
+.Additional resources
+* link:https://github.com/devfile/developer-images[Universal Developer Image]