Skip to content

Commit

Permalink
Merge branch 'aiap15ds-update' into 'main'
Browse files Browse the repository at this point in the history
OCP Cluster Testing for AIAP 15 MLOps Week

Closes #2

See merge request mlops/kapitan-hull!1
  • Loading branch information
Syakyr committed Dec 6, 2023
2 parents 21be902 + 99beca5 commit 0cb30e8
Show file tree
Hide file tree
Showing 30 changed files with 1,031 additions and 518 deletions.
43 changes: 43 additions & 0 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
image:
name: python:3.10

stages:
- docs
- update

pages:
stage: docs
before_script:
- pip install -r mkdocs-requirements.txt
script:
- mkdocs build --strict --verbose --site-dir public
artifacts:
paths:
- public
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH

update:onprem-runai:
stage: update
tags:
- on-prem
#variables:
# REPO_URL_ONPREM_RUNAI: url-in-settings
# REPO_USERNAME_ONPREM_RUNAI: username-in-settings
# REPO_PASSWORD_ONPREM_RUNAI: password-in-settings
before_script:
- pip install cookiecutter
- mkdir -p /tmp/kapitan-hull && cd $_
- git config --global user.name "Kapitan Hull Bot"
- git config --global user.email "[email protected]"
script:
- cookiecutter --replay-file $CI_PROJECT_DIR/cookiecutter-onprem-runai.json $CI_PROJECT_DIR
- git clone https://$REPO_USERNAME_ONPREM_RUNAI:$REPO_PASSWORD_ONPREM_RUNAI@$REPO_URL_ONPREM_RUNAI git-repo
- cd git-repo
- git checkout -B $CI_COMMIT_BRANCH
- git pull origin $CI_COMMIT_BRANCH || echo "Nothing to pull"
- cd ../kapitan-hull-onprem-runai-test
- cp -rv ../git-repo/.git .
- git add .
- git commit -m "$CI_COMMIT_MESSAGE" || echo "Nothing to commit"
- git push origin $CI_COMMIT_BRANCH || echo "Nothing to push"
51 changes: 29 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# AI Singapore's Cookiecutter Template for End-to-end ML Projects (On-prem | Run:ai)
# Kapitan Hull

![AI Singapore's Kapitan Hull EPTG Onprem Run:ai Banner](./assets/kapitan-hull-eptg-onprem-runai-banner.png)

## Table of Contents

- [AI Singapore's Cookiecutter Template for End-to-end ML Projects (On-prem | Run:ai)](#ai-singapores-cookiecutter-template-for-end-to-end-ml-projects-on-prem--runai)
- [Kapitan Hull)](#kapitan-hull)
- [Table of Contents](#table-of-contents)
- [Preface](#preface)
- [Usage](#usage)
Expand All @@ -13,12 +13,16 @@

## Preface

This repository contains the
[`cookiecutter`](https://cookiecutter.readthedocs.io/en/stable/)
template for generating a repository that provides boilerplates touching
on the differing components of an end-to-end ML project. This template
is dedicated for AI Singapore's on-premise environment, and where
Run:ai is used as the MLOps platform.
This repository contains the [`cookiecutter`][ccutter] template for
generating a repository that provides boilerplates touching on the
differing components of an end-to-end ML project.

For now, this template is dedicated for AI Singapore's on-premise
environment, and where Run:ai is used as the MLOps platform. Other
platforms and orchestrators would be integrated to this repository in
the near future.

[ccutter]: https://cookiecutter.readthedocs.io/en/stable/

## Usage

Expand All @@ -35,22 +39,25 @@ You will then be prompted to provide inputs.These inputs will be used to
populate different parts of the repository to be generated by
`cookiecutter`.

> **Note**: Your cookiecutter version must be at least ***2.2.0***.
### Input Parameters

| Parameter | Detail | Default | Regex Reference |
|------------------------------- |-------------------------------------------------------------------------------------------------------------------------------- |-------------------------------------------------------------------------------------------- |-------------------------------------------------------------------------------------------------------------------- |
| `project_name` | Name of project that will be the header for the `README.md`. Input to start with alphabet. Only whitespace as separators. | Name of project (not name of repository). Use whitespace instead of underscores or hyphens. | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L8) |
| `description` | A short description of the project that will be populated in `README.md`. Max of 72 characters. | A short description of the project. | NIL |
| `repo_name` | Name of the repository folder. Input to start with alphabet characters. No whitespaces or underscores are allowed. | `project_name` where whitespaces and underscores are replaced with hyphens. | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L13) |
| `src_package_name` | Name of the source code's package under `src`. Input to start with alphabet characters. No whitespaces or hyphens are allowed. | `repo_name` where hyphens are replaced with underscores. | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L16) |
| `src_package_name_short` | The alias for the source code's package. Input to start with alphabet characters. No whitespaces or hyphens are allowed. | `src_package_name` | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L19) |
| `runai_proj_name` | The RunAI namespace used by the project - the project name that you're sending jobs to. | The RunAI namespace used by the project. | NIL
| `harbor_registry_project_path` | Path of the Harbor registry repository for your container images to be located under. Cannot end with a slash character. | Path of the project's container registry on Harbor. | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L22) |
| `author_name` | Your alias or project team's name. Relatively arbitrary. No hyphens are allowed. | Your alias or project team's name. | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L25) |
| Parameter | Detail | Default | Regex Reference |
|------------------------- |--------------------------------------------------------------------------------------------------------------------------------- |---------------------------------------------------------------------------- |-------------------------------------------------------------------------------------------------------------------- |
| `project_name` | Name of project that will be the header for the `README.md`. Input to start with alphabet. Only whitespace as separators. | My Project | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L8) |
| `description` | A short description of the project that will be populated in `README.md`. Max of 72 characters. | A short description of the project. | NIL |
| `repo_name` | Name of the repository folder. Input to start with alphabet characters. No whitespaces or underscores are allowed. | `project_name` where whitespaces and underscores are replaced with hyphens. | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L13) |
| `src_package_name` | Name of the source code's package under `src`. Input to start with alphabet characters. No whitespaces or hyphens are allowed. | `repo_name` where hyphens are replaced with underscores. | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L16) |
| `src_package_name_short` | The alias for the source code's package. Input to start with alphabet characters. No whitespaces or hyphens are allowed. | `src_package_name` | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L19) |
| `platform` | The platform the project is running on. (Choose between "on-premise" or "Google Cloud Platform") | `onprem` or `gcp` | NIL |
| `orchestrator` | The orchestrator the project is using. (Choose between "Run:AI", "Polyaxon" or "No orchestrator") | `runai` or `polyaxon` or `none` | NIL |
| `proj_name` | The project name used in by the repository. If you're using Run:AI, this will be the Run:AI project name used by the repository. | `sample-project` | NIL |
| `registry_project_path` | Path of the registry repository for your container images to be located under. Cannot end with a slash character. | `registry.domain.tld/sample-project/my-project` | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L22) |
| `author_name` | Your alias or project team's name. Relatively arbitrary. No hyphens are allowed. | `AISG` | [Link](https://github.com/aisingapore/ml-project-cookiecutter-onprem-runai/blob/main/hooks/pre_gen_project.py#L25) |

### Version Control

Following the creation of your repository,
initialise it with Git, push it to a
remote, and follow its
`README.md` document for a full guide on its usage.
Following the creation of your repository, initialise it with Git, push
it to a remote, and follow its `README.md` document for a full guide on
its usage.
14 changes: 14 additions & 0 deletions cookiecutter-gcp-runai.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"cookiecutter": {
"project_name": "Kapitan Hull GCP RunAI Test",
"description": "Testing Grounds for Kapitan Hull on GCP using RunAI.",
"repo_name": "kapitan-hull-gcp-runai-test",
"src_package_name": "kapitan_hull_gcp_runai_test",
"src_package_name_short": "khgr_test",
"platform": "gcp",
"orchestrator": "runai",
"proj_name": "mlops-test",
"registry_project_path": "registry.aisingapore.net/mlops/kapitan-hull-gcp-runai",
"author_name": "mlops"
}
}
14 changes: 14 additions & 0 deletions cookiecutter-onprem-runai.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"cookiecutter": {
"project_name": "Kapitan Hull Onprem RunAI Test",
"description": "Testing Grounds for Kapitan Hull on premise using RunAI.",
"repo_name": "kapitan-hull-onprem-runai-test",
"src_package_name": "kapitan_hull_onprem_runai_test",
"src_package_name_short": "khor_test",
"platform": "onprem",
"orchestrator": "runai",
"proj_name": "mlops-test",
"registry_project_path": "registry.aisingapore.net/mlops/kapitan-hull-onprem-runai",
"author_name": "mlops"
}
}
31 changes: 27 additions & 4 deletions cookiecutter.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,33 @@
{
"project_name": "Name of project (not name of repository). Use whitespace instead of underscores or hyphens.",
"project_name": "My Project",
"description": "A short description of the project.",
"repo_name": "{{ cookiecutter.project_name.lower().replace(' ', '-').replace('_', '-') }}",
"src_package_name": "{{ cookiecutter.repo_name.replace('-', '_') }}",
"src_package_name_short": "{{ cookiecutter.src_package_name }}",
"runai_proj_name": "The RunAI namespace used by the project.",
"harbor_registry_project_path": "Path of the project's container registry on Harbor.",
"author_name": "Your alias or project team's name."
"platform": ["onprem", "gcp"],
"orchestrator": ["runai", "polyaxon", "none"],
"proj_name": "sample-project",
"registry_project_path": "registry.domain.tld/sample-project/my-project",
"author_name": "AISG",
"__prompts__": {
"project_name": "Name of project that will be the header for the README.md. Input to start with alphabet. Only whitespace as separators.",
"description": "A short description of the project that will be populated in README.md. Max of 72 characters.",
"repo_name": "Name of repository. Name of the repository folder. Input to start with alphabet characters. No whitespaces or underscores are allowed.",
"src_package_name": "Name of the source code's package under the src folder. Input to start with alphabet characters. No whitespaces or hyphens are allowed.",
"src_package_name_short": "The alias for the source code's package. Input to start with alphabet characters. No whitespaces or hyphens are allowed.",
"platform": {
"__prompt__": "The platform the project is running on.",
"onprem": "On premise",
"gcp": "Google Cloud Platform (not implemented yet)"
},
"orchestrator": {
"__prompt__": "The orchestrator the project is using.",
"runai": "Run:AI",
"polyaxon": "Polyaxon (not implemented yet)",
"noorch": "No orchestrator (not implemented yet)"
},
"proj_name": "The project name used in by the repository. If you're using Run:AI, this will be the Run:AI project name used by the repository.",
"registry_project_path": "Path of the registry repository for your container images to be located under. Cannot end with a slash character.",
"author_name": "Your alias or project team's name. Relatively arbitrary. No hyphens are allowed."
}
}
6 changes: 3 additions & 3 deletions hooks/pre_gen_project.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@
"src_package_name_short": {
"user_input": "{{cookiecutter.src_package_name_short}}",
"regex": r"^[a-z](?:_?[a-z0-9]+)*$"},
"harbor_registry_project_path": {
"user_input": "{{cookiecutter.harbor_registry_project_path}}",
"registry_project_path": {
"user_input": "{{cookiecutter.registry_project_path}}",
"regex": r"^registry\.aisingapore\.net[\-\/\w\d]+(?:[a-z0-9]+)$"},
"author_name": {
"user_input": "{{cookiecutter.author_name}}",
Expand Down Expand Up @@ -52,7 +52,7 @@ def check_input_regex(cookie_input_key, cookie_input_val):
ERROR_MSG_LIST.append("ERROR: %s - '%s' is not a valid Python package name."
% (cookie_input_key, cookie_input_val["user_input"]))

if cookie_input_key == "harbor_registry_project_path":
if cookie_input_key == "registry_project_path":
ERROR_MSG_LIST.append("ERROR: %s - '%s' is not a valid Harbor path."
% (cookie_input_key, cookie_input_val["user_input"]))

Expand Down
12 changes: 8 additions & 4 deletions {{cookiecutter.repo_name}}/.gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,16 @@ build:data-prep-image:
/kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/docker/{{cookiecutter.repo_name}}-data-prep.Dockerfile"
--destination "{{cookiecutter.harbor_registry_project_path}}/data-prep:${CI_COMMIT_SHORT_SHA}"
--destination "{{cookiecutter.registry_project_path}}/data-prep:${CI_COMMIT_SHORT_SHA}"
rules:
- if: $CI_MERGE_REQUEST_IID
changes:
- docker/{{cookiecutter.repo_name}}-data-prep.Dockerfile
- src/**/*
- conf/**/*
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- if: $CI_PIPELINE_SOURCE == "web" && $BUILD_ALL
- if: $CI_PIPELINE_SOURCE == "web" && $BUILD_DATAPREP

build:model-training-image:
stage: build
Expand All @@ -67,14 +69,16 @@ build:model-training-image:
/kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/docker/{{cookiecutter.repo_name}}-model-training.Dockerfile"
--destination "{{cookiecutter.harbor_registry_project_path}}/model-training:${CI_COMMIT_SHORT_SHA}"
--destination "{{cookiecutter.registry_project_path}}/model-training:${CI_COMMIT_SHORT_SHA}"
rules:
- if: $CI_MERGE_REQUEST_IID
changes:
- docker/{{cookiecutter.repo_name}}-model-training.Dockerfile
- src/**/*
- conf/**/*
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
- if: $CI_PIPELINE_SOURCE == "web" && $BUILD_ALL
- if: $CI_PIPELINE_SOURCE == "web" && $BUILD_MODEL

build:retag-images:
stage: build
Expand All @@ -83,8 +87,8 @@ build:retag-images:
entrypoint: [""]
script:
- cat $HARBOR_ROBOT_CREDS_JSON > /root/.docker/config.json
- crane tag {{cookiecutter.harbor_registry_project_path}}/data-prep:${CI_COMMIT_SHORT_SHA} ${CI_COMMIT_TAG}
- crane tag {{cookiecutter.harbor_registry_project_path}}/model-training:${CI_COMMIT_SHORT_SHA} ${CI_COMMIT_TAG}
- crane tag {{cookiecutter.registry_project_path}}/data-prep:${CI_COMMIT_SHORT_SHA} ${CI_COMMIT_TAG}
- crane tag {{cookiecutter.registry_project_path}}/model-training:${CI_COMMIT_SHORT_SHA} ${CI_COMMIT_TAG}
rules:
- if: $CI_COMMIT_TAG && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Expand Down
Loading

0 comments on commit 0cb30e8

Please sign in to comment.