Add kubernetes documentation

Signed-off-by: Kern Walster <[email protected]>

README.md

@@ -50,6 +50,11 @@ for more details.
 - [Debug and Useful Commands](docs/debug.md): accessing logs/metrics and debugging common errors.
 - [Glossary](docs/glossary.md): glossary we use in the project.
 
+### Integration-specific documentation
+
+- [SOCI on Kubernetes](docs/kubernetes.md): an overview of how to use SOCI on Kubernetes in general
+- [SOCI on Amazon Elastic Kubernetes Service (EKS)](docs/eks.md): a walk through for setting up SOCI on Amazon EKS
+
 ## Project Origin
 
 There a few different lazy loading projects in the containerd snapshotter community.  This project began as a

docs/eks.md

@@ -0,0 +1,312 @@
+# SOCI on Amazon Elastic Kubernetes Service (EKS)
+
+## Overview
+
+The [SOCI on Kubernetes](./kubernetes.md) documentation explains how to configure SOCI to work with Kubernetes, however translating that documentation into a working set up can be challenging. This doc bridges that gap by offering a copy-pasteable walkthrough for setting up a new Amazon EKS cluster that launches containers with SOCI.
+
+This guides will create a launch template to launch nodes configrued with SOCI, an EKS cluster using the defaults from [eksctl](https://eksctl.io), and a managed node group that uses the launch template to create nodes. For the sake of simplicity, we will confirm that everything worked by launching a pre-indexed tensorflow image (public.ecr.aws/soci-workshop-examples/tensorflow_gpu:latest) which sleeps forever.
+
+In this guide, we chose to use the [CRI Credentials](./registry-authentication.md#kubernetes-cri-credentials) mechanism for getting private registry credentials to the SOCI snpashotter. Please consult [the documentation](./registry-authentication.md) to confirm that this will work for your usecase if you plan to use this setup beyond this example.
+
+## Prerequisites
+
+This guide requires the following tools. Please follow the links to find installtion instructions.
+
+1. **[eksctl](https://eksctl.io/installation/)** - used for creating an EKS cluster and managed nodegroup
+1. **[kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)** - used for creating a deployment on the EKS cluster
+1. **[aws CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) with credentials configured** - used to create launch template for use in the EKS managed nodegroup
+
+## Setup
+### Step 1: Configuration
+
+First, we will set some configuration variables. You can update these to match your preferrered settings with the following conditions:
+
+`ARCH` can be x86_64 or arm64  
+`INSTANCE_TYPE` should match the architecture (e.g. `t4g.large` for `arm64`)  
+`AMI_ID` should be an AL2023-based [EKS optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html). This setup relies on [`nodeadm`](https://awslabs.github.io/amazon-eks-ami/nodeadm/) to configure containerd and the Kublet which is not available in AL2-based AMIs.  
+
+```
+AWS_REGION=us-west-2
+CLUSTER_NAME=soci
+KUBERNETES_VERSION=1.30
+ARCH=x86_64
+INSTANCE_TYPE=t3.large
+AMI_ID=$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${KUBERNETES_VERSION}/amazon-linux-2023/${ARCH}/standard/recommended/image_id  --region $AWS_REGION --query "Parameter.Value" --output text)
+```
+
+### Step 2: Create an EKS cluster 
+
+Next we will create a cluster using eksctl. This step will create a cluster and the necessary resources for EKS to function, but importantly will not create any nodegroups. We will create nodegroups later using a custom launch template that installs and configures the SOCI snapshotter.
+
+```
+eksctl create cluster --without-nodegroup -f - <<EOF
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+metadata:
+  name: $CLUSTER_NAME
+  region: $AWS_REGION
+  version: "$KUBERNETES_VERSION"
+EOF
+```
+
+### Step 3: Create Node configuration file
+
+This config will allow our node to join the cluster with updated containerd/kubelet config to support SOCI. 
+
+```
+CLUSTER_ENDPOINT=$(aws eks describe-cluster \
+  --name $CLUSTER_NAME \
+  --region $AWS_REGION \
+  --output text \
+  --query 'cluster.endpoint')
+CLUSTER_CERTIFICATE_AUTHORITY=$(aws eks describe-cluster \
+  --name $CLUSTER_NAME --region $AWS_REGION \
+  --output text \
+  --query 'cluster.certificateAuthority.data')
+CLUSTER_CIDR=$(aws eks describe-cluster \
+  --name $CLUSTER_NAME \
+  --region $AWS_REGION \
+  --output text \
+  --query 'cluster.kubernetesNetworkConfig.serviceIpv4Cidr')
+
+cat <<EOF > node_config.yaml
+apiVersion: node.eks.aws/v1alpha1
+kind: NodeConfig
+spec:
+  cluster:
+    name: $CLUSTER_NAME
+    apiServerEndpoint: $CLUSTER_ENDPOINT
+    certificateAuthority: $CLUSTER_CERTIFICATE_AUTHORITY
+    cidr: $CLUSTER_CIDR
+  kubelet:
+    config:
+      imageServiceEndpoint: unix:///run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock
+  containerd: 
+    config: |
+      [proxy_plugins.soci]
+        type = "snapshot"
+        address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"
+        [proxy_plugins.soci.exports]
+          root = "/var/lib/soci-snapshotter-grpc"
+      [plugins."io.containerd.grpc.v1.cri".containerd]
+        snapshotter = "soci"
+        # This line is required for containerd to send information about how to lazily load the image to the snapshotter
+        disable_snapshot_annotations = false
+EOF
+```
+
+
+### Step 4: Create SOCI install script
+
+This will create an script that will be run on instance boot to install and configure the SOCI snapshotter.
+
+```
+cat <<'EOF_SCRIPT' >install_soci.sh
+#!/bin/bash
+# Set environment variables
+ARCH=$(uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)
+VERSION=0.7.0
+ARCHIVE=soci-snapshotter-$VERSION-linux-$ARCH.tar.gz
+
+pushd /tmp
+# Download, verify, and install the soci-snapshotter
+curl --silent --location --fail --output $ARCHIVE https://github.com/awslabs/soci-snapshotter/releases/download/v$VERSION/$ARCHIVE
+curl --silent --location --fail --output $ARCHIVE.sha256sum https://github.com/awslabs/soci-snapshotter/releases/download/v$VERSION/$ARCHIVE.sha256sum
+sha256sum ./$ARCHIVE.sha256sum
+tar xzvf ./$ARCHIVE -C /usr/local/bin soci-snapshotter-grpc
+rm ./$ARCHIVE
+rm ./$ARCHIVE.sha256sum
+
+# Configure the SOCI snapshotter for CRI credentials
+mkdir -p /etc/soci-snapshotter-grpc
+cat <<EOF >/etc/soci-snapshotter-grpc/config.toml
+[cri_keychain]
+# This tells the soci-snapshotter to act as a proxy ImageService
+# and to cache credentials from requests to pull images.
+enable_keychain = true
+# This tells the soci-snapshotter where containerd's ImageService is located.
+image_service_path = "/run/containerd/containerd.sock"
+EOF
+
+# Start the soci-snapshotter
+curl --silent --location --fail --output /etc/systemd/system/soci-snapshotter.service https://raw.githubusercontent.com/awslabs/soci-snapshotter/v$VERSION/soci-snapshotter.service
+systemctl daemon-reload
+systemctl enable --now soci-snapshotter
+
+popd
+EOF_SCRIPT
+```
+
+
+### Step 5: Create EC2 Userdata
+
+This step will combine the node configuration and SOCI installation script from the previous steps into a MIME multipart archive that works as EC2 user data. 
+
+
+```
+cat <<EOF > userdata.txt
+MIME-Version: 1.0
+Content-Type: multipart/mixed; boundary="BOUNDARY"
+
+--BOUNDARY
+Content-Type: application/node.eks.aws
+
+---
+$(cat node_config.yaml)
+
+--BOUNDARY
+Content-Type: text/x-shellscript; charset="us-ascii"
+
+$(cat install_soci.sh)
+
+--BOUNDARY--
+EOF
+```
+
+### Step 6: Create a node launch template
+
+Next, we will create a launch template that uses our chosen AMI, instance type, and userdata. This will be used to launch new nodes for our EKS cluster.
+
+```
+cat <<EOF > launch_template_data.json
+{
+  "ImageId": "${AMI_ID}",
+  "InstanceType": "${INSTANCE_TYPE}",
+  "UserData": "$(cat userdata.txt | base64 --wrap=0)"
+}
+EOF
+
+aws ec2 create-launch-template \
+  --launch-template-name soci-eks-node \
+  --launch-template-data file://launch_template_data.json \
+  --region $AWS_REGION
+
+LAUNCH_TEMPLATE_ID=$(aws ec2 describe-launch-templates \
+  --launch-template-name soci-eks-node \
+  --region $AWS_REGION \
+  --query "LaunchTemplates[0].LaunchTemplateId" \
+  --output text)
+```
+
+### Step 7: Create an EKS managed nodegroup
+
+As the final setup for our cluster, we will use the launch template to create a managed nodegroup. After this step, we will have nodes in our cluster configured to launch containers with the SOCI snapshotter.
+
+```
+eksctl create nodegroup -f - <<EOF
+---
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+
+metadata:
+  name: "$CLUSTER_NAME"
+  region: "$AWS_REGION"
+
+managedNodeGroups:
+  - name: "${CLUSTER_NAME}-ng-1"
+    launchTemplate:
+      id: "$LAUNCH_TEMPLATE_ID"
+EOF
+```
+
+## Launching a Pod
+### Step 1: Configure kubectl
+
+Here, we update `kubeconfig` so that `kubectl` can communicate with our EKS cluster
+
+```
+aws eks update-kubeconfig --region $AWS_REGION --name "$CLUSTER_NAME"
+```
+
+### Step 2: Create a deployment
+
+And finally we can create a deployment using SOCI. 
+
+For this demo, we will use the a pre-indexed tensorflow-gpu image (~3GB) that will just sleep forever. This represents the best case scenario for SOCI where approximately none of the image is needed to start the container. 
+
+```
+kubectl apply -f - <<EOF
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: soci-sample-deployment
+  labels:
+    app: soci
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: soci
+  template:
+    metadata:
+      labels:
+        app: soci
+    spec:
+      containers:
+      - name: soci-container
+        image: public.ecr.aws/soci-workshop-examples/tensorflow_gpu:latest
+        command: ["sleep"]
+        args: ["inf"]
+EOF
+```
+
+## Verification
+
+Containerd snapshotters are not directly visible to Kubernetes so they do not appear in any UI. As an approximation, we can use the shortened image pull time to infer that SOCI was used. To be sure, we can inspect the node to verify that the SOCI filesystems were created. 
+
+For this, we will look up the node where the pod was scheduled, map that to an EC2 instance, and use AWS Systems Manager (SSM) to look up the SOCI filesystems with `findmnt`
+
+```
+NODE_NAME=$(kubectl get pods --selector app=soci --output jsonpath="{.items[0].spec.nodeName}")
+INSTANCE_ID=$(aws ec2 describe-instances \
+  --region $AWS_REGION \
+  --filter "Name=private-dns-name,Values=$NODE_NAME" \
+  --query "Reservations[0].Instances[0].InstanceId" \
+  --output text)
+SSM_COMMAND_ID=$(aws ssm send-command \
+    --instance-ids $INSTANCE_ID \
+    --document-name "AWS-RunShellScript" \
+    --comment "Get SOCI mounts" \
+    --parameters commands='findmnt --source soci' \
+    --query "Command.CommandId"\
+    --output text \
+    --region $AWS_REGION)
+aws ssm list-command-invocations \
+    --command-id $SSM_COMMAND_ID \
+    --region $AWS_REGION \
+    --details \
+    --query "CommandInvocations[*].CommandPlugins[*].Output" \
+    --output text
+```
+
+If everything worked, we should expect an output like:
+```
+TARGET                                                     SOURCE FSTYPE         OPTIONS
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/27/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/28/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/29/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/30/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/31/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/32/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/33/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/34/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+/var/lib/soci-snapshotter-grpc/snapshotter/snapshots/35/fs soci   fuse.rawBridge rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=131072
+```
+
+##  Clean up (Optional)
+
+Run these clean up commands to delete the files, Amazon EKS cluster, and launch template created by this guide.
+
+```
+rm launch_template_data.json userdata.txt install_soci.sh node_config.yaml
+eksctl delete cluster --name $CLUSTER_NAME --disable-nodegroup-eviction --region $AWS_REGION
+aws ec2 delete-launch-template --launch-template-id $LAUNCH_TEMPLATE_ID --region $AWS_REGION
+```
+
+## Next Steps
+
+This guide showed you how to set up SOCI on an Amazon EKS cluster. Check out the following documentation to get a better understanding of the components of this set up and to learn how to index your own container images.
+
+1. [SOCI on Kubernetes Documentation](./kubernetes.md) - general SOCI on Kubernetes information
+1. [Registry Authentication](./registry-authentication.md) - tradoffs for each authentication mechanism
+1. [Getting Started](./getting-started.md) - instructions for indexing your own images

docs/kubernetes.md

@@ -0,0 +1,70 @@
+# SOCI on Kubernetes
+
+This document explains how to configure SOCI on Kubernetes. For a hands on example, see [SOCI on Amazon Elastic Kubernetes Service (EKS)](./eks.md).
+
+> **Note**
+>
+> SOCI on Kubernetes has a few rough edges that you should consider before
+> using it for important workloads. See [Limitations](#limitations).  
+> We welcome your feedback and suggestions for improving the Kubernetes experience.
+
+## Configuration
+
+SOCI on kubernetes requires two pieces of configuration:
+1) [Containerd Configuration](#containerd-configuration) to launch containers with SOCI
+2) [Registry Authentication Configuration](#registry-authentication-configuration) so that SOCI can pull images from non-public container registries
+
+### Containerd configuration
+
+To configure containerd to launch containers with SOCI, add the following snippet to the containerd config. The config is located at `/etc/containerd/config.toml` by default. 
+
+```
+[proxy_plugins.soci]
+type = "snapshot"
+address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"
+[proxy_plugins.soci.exports]
+  root = "/var/lib/soci-snapshotter-grpc"
+
+[plugins."io.containerd.grpc.v1.cri".containerd]
+  snapshotter = "soci"
+  # This line is required for containerd to send information about how to lazily load the image to the snapshotter
+  disable_snapshot_annotations = false
+```
+
+> **NOTE**
+>
+> Your config might already have the 
+> `[plugins."io.containerd.grpc.v1.cri".containerd]` section in which case you should add the `snapshotter` and `disable_snapshot_annotations` lines to the existing section rather than defining a new one.
+
+Breaking it down line-by-line:  
+`[proxy_plugins.soci]` makes containerd aware of the SOCI plugin  
+`type = "snapshot"` tells containerd that the SOCI plugin is a snapshotter and implements the snapshotter API  
+`address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"` tells containerd where to connect to the SOCI snapshotter 
+`[proxy_plugins.soci.exports]` defines a set of metadata  
+`    root = "/var/lib/soci-snapshotter-grpc"` defines the root data directory for the SOCI snapshotter. Kubernetes uses this to calculate disk utilization, enforce storage limits, and trigger garbage collection.
+
+`[plugins."io.containerd.grpc.v1.cri".containerd]` defines kubernetes-specific configuration  
+`  snapshotter = "soci"` tells containerd to use SOCI by default. This name must match the proxy_plugin name. (this is required. See [SOCI must be configured at launch](#soci-must-be-configured-at-launch))  
+`  disable_snapshot_annotations = false` tells containerd to send lazy loading information to the SOCI snapshotter  
+
+### Registry Authentication Configuration
+
+The SOCI snapshotter laizly pulls image content outside of the normal image pull context. As a result, it must be independently configured to receive credentials to access non-public contaienr registries.
+
+There are several mechanisms to configure SOCI to access non-public container registries with different tradeoffs. See the [registry authentication documentation](./registry-authentication.md) for full information. Choosing a specific mechanism requires evaluating which set of tradeoffs best suits a particular use-case.
+
+If you are looking to quickly evaluate SOCI on Kubernetes or you are unsure what the tradeoffs look like in practice, [Kubernetes CRI Credentials](./registry-authentication.md#kubernetes-cri-credentials) will work with widest range of use-cases. You should verify that the tradeoffs are appropriate for your use-case before using this for important workloads.
+
+## Limitations
+
+1. SOCI must be used for all containers on the node
+    Kubernetes has its own view of images that is not containerd snapshotter-aware. For example, If an image has been pulled with the default OverlayFS and then a pod is scheduled with SOCI, Kubernetes will not pull the image with the new snapshotter. The pod launch will fail because the image is not found in the SOCI snapshotter.
+1. SOCI must be configured at node launch
+    Related to the previous limitation, if SOCI is not configured at launch time, then the pause container will be pulled with the default OverlayFS snapshotter. As a result, SOCI pods will not launch because the pause container is not found in the SOCI snapshotter.
+
+## Known Bugs
+
+1. **containerd <1.7.16 does not enforce storage limits or run garbage collection**  
+    A bug in containerd caused the Kubelet to calculate SOCI snapshotter disk utilization incorrectly which broke kubernetes garbage collection and prevented enforcement of storage limits.
+
+