KubeIT is designed to be a framework for the iterative distribution of bioinformatic analyses. It exposes a small dedicated API, that is used to simplify workflow scheduling and data handling. KubeIT uses the cloud native workflow management engine Argo workflows.
- Kubernetes (> v1.19.2)
- Access to S3 (Credentials with Secret and Access key)
- kubectl (install guide)
- Create a namespace. This namespace must be specified in all following kubectl command with
-n.
kubectl create namespace kubeit
- Create a secret called
s3secretcontaining your S3 Credentials (replaceYOUR-S3-KEY&YOUR-S3-SECRET).
kubectl create secret generic s3secret --from-literal='AWS_ACCESS_KEY_ID=YOUR-S3-KEY' --from-literal='AWS_SECRET_ACCESS_KEY=YOUR-S3-SECRET' -n kubeit
- Create a secret with KubeITs access token: This token is used for authorization (replace
YOUR-TOKEN). You can create a new Token on Linux systems with the openssl command:openssl rand -base64 20
kubectl create secret generic kubeit-token --from-literal='TOKEN=YOUR-TOKEN' -n kubeit
- Install Argo. For more information see: Argo quickstart. Make sure to use the "namspace-install".
kubectl apply -n kubeit -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/namespace-install.yaml
- Pre-configure KubeIT and Argo configmaps. Download the following file configmaps.yaml . Edit both configmap entries to specify your S3 endpoint, S3 Region and the desired S3 Bucket for data storage. (Caveat: the first configmap must contain your S3 endpoint without procotol (https://). Apply the configmap via:
kubectl apply -f confimaps.yaml -n kubeit
- Install KubeIT and the default WorkflowTemplates. The first contains the KubeIT deployment and the KubeIT service account and service. The second contains the WorkflowTemplates for the example usecase.
kubectl apply -n kubeit -f https://raw.githubusercontent.com/KubeITerator/KubeIT/master/deployment.yaml
kubectl apply -n kubeit -f https://raw.githubusercontent.com/KubeITerator/KubeIT/master/default-settings/templates/templates.yaml
- Configure ingress for external access: This last step highly depends on your individual network installation (see here for mor information).
Step 5. creates a service with the name
kubeit-service. Use this service for configuring external access.
The KubeIT backend is designed to be used via the KubeIT CLI. If the API is accessed directly see API Documentation.
A short guideline for the creation of new WorkflowTemplates can be found here. Guidelines for the creation of new schemes can be found here. If you want to create your own custom splitting logic you can reference this