From 11eb004ecdf3908f4af2592df2cbc9bea64e38ce Mon Sep 17 00:00:00 2001 From: Zeid Adabel Date: Tue, 13 Feb 2024 16:10:32 +0100 Subject: [PATCH] Improved clarity of readme --- README.md | 4 ++-- sailfish-k8s/README.md | 7 ++----- sailfish-k8s/sailfish/overlay/README.md | 10 +++------- 3 files changed, 7 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index a0000d0..8b8d914 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Introduction This is a HPC Framework built by Ortec Finance that works natively on kubernetes. -Sailfish uses two RedHat supported operators to function: an AMQ Broker to capture the jobs and work items, and KEDA, an autoscaler that listens to these queues and matches the amount of containers deployed to process the jobs and work items. This project also uses Knative to submit the initial job to the queue. +Sailfish uses two RedHat supported operators to function: an AMQ Broker to capture the jobs and work items, and KEDA, an autoscaler that listens to these queues and matches the amount of containers deployed to process the jobs and work items. This project also uses Knative to submit the initial job to the queue and Prometheus to communicate Queue sizes for KEDA to trigger on. This enables Sailfish to complete distributed computations on container level, leveraging the Public Cloud providers flexbility on provisioning Virtual Machines. @@ -26,7 +26,7 @@ If you wish to just spin up Sailfish, a python demo app is included in the repos # Build and Test You may simply `oc apply` to your namespaces on a kubernetes Cluster to deploy Sailfish, also follow the Prerequiste paragraph in the `sailfish-k8s` README to setup your cluster with the proper configuration. -The demo `sailfish-py` is stored in DockerHub, the dockerfiles are in the same folder if you wish to push the Run manager and Runner Images to your image registry! Changes to the image will be picked up immediately with newly created Jobs. +The demo `sailfish-py` is stored in DockerHub, the dockerfiles are in the same folder if you wish to rebuild and push the Run manager and Runner Images to your image registry! Changes to the image will be picked up immediately with newly created Jobs. # Architecture ![A high level overview of the flow of Sailfish](Sailfish-Architecture-and-flow.png) diff --git a/sailfish-k8s/README.md b/sailfish-k8s/README.md index d6a7474..c2b5464 100644 --- a/sailfish-k8s/README.md +++ b/sailfish-k8s/README.md @@ -1,8 +1,6 @@ # Sailfish HPC K8s - This folder contains all the kubernetes resources that is required for the Sailfish Framework. - # Cluster Configuration ## 1. Prometheus @@ -38,10 +36,9 @@ We recommend to deploy seperate machinesets for Sailfish, we've provided an exam ``` Change these parameters to fit your cluster. - ## 4. Deploying Sailfish -To deploy all components necessary to run sailfish, do `oc apply -k sailfish-k8s/argocd`. This will deploy all prerequisites and deploy an environment called `sailfish` that runs the demo workloads. - +To deploy all components necessary to run sailfish, do `oc apply -k sailfish-k8s/argocd` in your ArgoCD namespace. This will deploy all prerequisites and deploy an environment called `sailfish` that runs the demo workloads. +If you want to deploy your own workloads, read the `README.md` file under `/sailfish-k8s/sailfish/overlay` # Configuring Sailfish with your workloads Sailfish is made to listen to a Prometheus metric. The workloads will scale down to zero when there is no traffic. diff --git a/sailfish-k8s/sailfish/overlay/README.md b/sailfish-k8s/sailfish/overlay/README.md index f9242d9..dfdb25a 100644 --- a/sailfish-k8s/sailfish/overlay/README.md +++ b/sailfish-k8s/sailfish/overlay/README.md @@ -1,16 +1,12 @@ # Configuring Sailfish Prometheus Trigger -There are three files in this Sailfish configuration that need to have your custom input! +There are two files in this Sailfish configuration that need to have your custom input! +Use this overlay as an example to setup yours! To prevent duplication, you can reference to this repo using a (kustomize remote ref)[https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md] - run-manager-job.yaml > In this file you need to add your container that will manage the Job Queue. The container is supposed to take one job and split it into multiple tasks - runner-job.yaml > These tasks are picked up by the container that will be configured in this yaml. - - kustomization.yaml - > Make sure that the namespace fields point at where the broker is deployed. +In your `kustomization.yaml` you can define which components(features) you'd like to enable, you can use (kustomize remote ref)[https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md] for this as well. -# Configuring your Namespace -- Make sure there is a secret called sailfish-broker-credentials-secret -Inside this secret add AMQ_USER and AMQ_PASSWORD. The values can be anything, but make sure it is a secure password. (Both the Application and Sailfish Broker must use this secret) -- The KEDA `TriggerAuthentication` uses the bearer token of a `ServiceAccount` named Sailfish, you can find all the resources regarding permissions in the `base/prometheus-trigger/permissions` folder