Skip to content

Commit

Permalink
Merge pull request #21 from Ortec-Finance/fix-readmes
Browse files Browse the repository at this point in the history
  • Loading branch information
ZeidH authored Feb 15, 2024
2 parents fa11202 + 11eb004 commit 6611abc
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 14 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Introduction
This is a HPC Framework built by Ortec Finance that works natively on kubernetes.

Sailfish uses two RedHat supported operators to function: an AMQ Broker to capture the jobs and work items, and KEDA, an autoscaler that listens to these queues and matches the amount of containers deployed to process the jobs and work items. This project also uses Knative to submit the initial job to the queue.
Sailfish uses two RedHat supported operators to function: an AMQ Broker to capture the jobs and work items, and KEDA, an autoscaler that listens to these queues and matches the amount of containers deployed to process the jobs and work items. This project also uses Knative to submit the initial job to the queue and Prometheus to communicate Queue sizes for KEDA to trigger on.

This enables Sailfish to complete distributed computations on container level, leveraging the Public Cloud providers flexbility on provisioning Virtual Machines.

Expand All @@ -26,7 +26,7 @@ If you wish to just spin up Sailfish, a python demo app is included in the repos
# Build and Test
You may simply `oc apply` to your namespaces on a kubernetes Cluster to deploy Sailfish, also follow the Prerequiste paragraph in the `sailfish-k8s` README to setup your cluster with the proper configuration.

The demo `sailfish-py` is stored in DockerHub, the dockerfiles are in the same folder if you wish to push the Run manager and Runner Images to your image registry! Changes to the image will be picked up immediately with newly created Jobs.
The demo `sailfish-py` is stored in DockerHub, the dockerfiles are in the same folder if you wish to rebuild and push the Run manager and Runner Images to your image registry! Changes to the image will be picked up immediately with newly created Jobs.

# Architecture
![A high level overview of the flow of Sailfish](Sailfish-Architecture-and-flow.png)
7 changes: 2 additions & 5 deletions sailfish-k8s/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# Sailfish HPC K8s

This folder contains all the kubernetes resources that is required for the Sailfish Framework.


# Cluster Configuration

## 1. Prometheus
Expand Down Expand Up @@ -38,10 +36,9 @@ We recommend to deploy seperate machinesets for Sailfish, we've provided an exam
```
Change these parameters to fit your cluster.


## 4. Deploying Sailfish
To deploy all components necessary to run sailfish, do `oc apply -k sailfish-k8s/argocd`. This will deploy all prerequisites and deploy an environment called `sailfish` that runs the demo workloads.

To deploy all components necessary to run sailfish, do `oc apply -k sailfish-k8s/argocd` in your ArgoCD namespace. This will deploy all prerequisites and deploy an environment called `sailfish` that runs the demo workloads.
If you want to deploy your own workloads, read the `README.md` file under `/sailfish-k8s/sailfish/overlay`

# Configuring Sailfish with your workloads
Sailfish is made to listen to a Prometheus metric. The workloads will scale down to zero when there is no traffic.
Expand Down
10 changes: 3 additions & 7 deletions sailfish-k8s/sailfish/overlay/README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,12 @@
# Configuring Sailfish Prometheus Trigger

There are three files in this Sailfish configuration that need to have your custom input!
There are two files in this Sailfish configuration that need to have your custom input!
Use this overlay as an example to setup yours! To prevent duplication, you can reference to this repo using a (kustomize remote ref)[https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md]

- run-manager-job.yaml
> In this file you need to add your container that will manage the Job Queue. The container is supposed to take one job and split it into multiple tasks
- runner-job.yaml
> These tasks are picked up by the container that will be configured in this yaml.
- kustomization.yaml
> Make sure that the namespace fields point at where the broker is deployed.
In your `kustomization.yaml` you can define which components(features) you'd like to enable, you can use (kustomize remote ref)[https://github.com/kubernetes-sigs/kustomize/blob/master/examples/remoteBuild.md] for this as well.

# Configuring your Namespace
- Make sure there is a secret called sailfish-broker-credentials-secret
Inside this secret add AMQ_USER and AMQ_PASSWORD. The values can be anything, but make sure it is a secure password. (Both the Application and Sailfish Broker must use this secret)
- The KEDA `TriggerAuthentication` uses the bearer token of a `ServiceAccount` named Sailfish, you can find all the resources regarding permissions in the `base/prometheus-trigger/permissions` folder

0 comments on commit 6611abc

Please sign in to comment.