Skip to content

Commit

Permalink
Merge pull request #46 from boozallen/24-migrate-downstream-helm-char…
Browse files Browse the repository at this point in the history 
…t-names

 #24 Create Baton migration for Helm chart name changes
  • Loading branch information
@chang-annie
chang-annie authored May 7, 2024
2 parents 9cfbed9 + 0ab1fa5 commit 067fddc
Show file tree
Hide file tree
Showing 34 changed files with 431 additions and 85 deletions.
48 changes: 39 additions & 9 deletions DRAFT_RELEASE_NOTES.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Major Additions

* Python modules were renamed to reflect aiSSEMBLE. These include the following.

| Old Python Module | New Python Module |
| --------------------------------------- | ------------------------------------------------- |
| --------------------------------------- |---------------------------------------------------|
| foundation-core-python | aissemble-core-python |
| foundation-model-training-api | aissemble-foundation-model-training-api |
| foundation-versioning-service | aissemble-foundation-versioning-service |
Expand All @@ -15,7 +16,7 @@
| foundation-transform-core-python | aissemble-Foundation-transform-core-python |
| extensions-model-training-api-sagemaker | aissemble-extensions-model-training-api-sagemaker |
| extensions-data-delivery-spark-py | aissemble-extensions-data-delivery-spark-py |
| extensions-encryption-valut-python | aissemble-extensions-encryption-valut-python |
| extensions-encryption-valut-python | aissemble-extensions-encryption-vault-python |
| extensions-transform-spark-python | aissemble-extensions-transform-spark-python |
| test-data-delivery-pyspark-model | aissemble-test-data-delivery-pyspark-model |
| test-data-delivery-pyspark-model-basic | aissemble-test-data-delivery-pyspark-model-basic |
Expand All @@ -24,6 +25,34 @@
| machine-learning-training-base | aissemble-machine-learning-training-base |
| machine-learning-sagemaker-training | aissemble-machine-learning-sagemaker-training |

* Helm Charts and their relevant modules have been renamed to the following:

| Old Module Name | New Helm Chart and Module Name |
|-----------------------------------------|-----------------------------------------|
| extensions-helm-airflow | aissemble-airflow-chart |
| extensions-helm-data-access | aissemble-data-access-chart |
| extensions-helm-elasticsearch | aissemble-elasticsearch-chart |
| extensions-helm-elasticsearch-operator | aissemble-elasticsearch-operator-chart |
| extensions-helm-fastapi | aissemble-fastapi-chart |
| extensions-helm-hive-metastore-db | aissemble-hive-metastore-db-chart |
| extensions-helm-hive-metastore-service | aissemble-hive-metastore-service-chart |
| extensions-helm-inference | aissemble-inference-chart |
| extensions-helm-jenkins | aissemble-jenkins-chart |
| extensions-helm-kafka | aissemble-kafka-chart |
| extensions-helm-keycloak | aissemble-keycloak-chart |
| extensions-helm-lineage-http-consumer | aissemble-lineage-http-consumer-chart |
| extensions-helm-localstack | aissemble-localstack-chart |
| extensions-helm-metadata | aissemble-metadata-chart |
| extensions-helm-mlflow | aissemble-mlflow-chart |
| extensions-helm-pipeline-invocation | aissemble-pipeline-invocation-chart |
| extensions-helm-pipeline-invocation-lib | aissemble-pipeline-invocation-lib-chart |
| extensions-helm-policy-decision-point | aissemble-policy-decision-point-chart |
| extensions-helm-quarkus | aissemble-quarkus-chart |
| extensions-helm-sealed-secrets | aissemble-sealed-secrets-chart |
| extensions-helm-spark-application | aissemble-spark-application-chart |
| extensions-helm-spark-operator | aissemble-spark-operator-chart |
| extensions-helm-vault | aissemble-vault-chart |
| extensions-helm-versioning | aissemble-versioning-chart |

## OpenLineage Namespace Conventions
Conventions for setting namespaces when leveraging `Data Lineage` has been updated to better follow [OpenLineage's guidelines](https://openlineage.io/docs/spec/naming/). Moving forward, namespaces should be defined in the `data-lineage.properties` file, such that Jobs are tied to pipelines and Datasets are tied to data sources. This is a departure from the old pattern of one single namespace property (`data.lineage.namespace`) being leveraged for an entire project. Refer to the [GitHub docs](https://boozallen.github.io/aissemble/current-dev/lineage-medatada-capture-overview.html#_configuration) for updated guidance. Usage of the `data.lineage.namespace` property in a project's `data-lineage.properties` file will be supported as a fallback but should not be used in practice.
Expand Down Expand Up @@ -68,13 +97,14 @@ The following steps will upgrade your project to 1.7. These instructions consist
## Automatic Upgrades
To reduce burden of upgrading aiSSEMBLE, the Baton project is used to automate the migration of some files to the new version. These migrations run automatically when you build your project, and are included by default when you update the `build-parent` version in your root POM. Below is a description of all of the Baton migrations that are included with this version of aiSSEMBLE.

| Migration Name | Description |
| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| upgrade-tiltfile-aissemble-version-migration | Updates the aiSSEMBLE version within your project's Tiltfile |
| upgrade-v2-chart-files-aissemble-version-migration | Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-mlflow-v2-external-s3-migration | Update the mlflow V2 deployment (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments |
| upgrade-foundation-extension-python-package-migration | Updates the pyproject.toml files within your projects pipelines folder (<YOUR_PROJECT>-pipelines) to use the updated aiSSEMBLE foundation and extension Python packages with the latest naming convention | |
| Migration Name | Description |
| -------------------------------------------------- |----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| upgrade-tiltfile-aissemble-version-migration | Updates the aiSSEMBLE version within your project's Tiltfile |
| upgrade-v2-chart-files-aissemble-version-migration | Updates the helm chart dependencies within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-v1-chart-files-aissemble-version-migration | Updates the docker image tags within your project's deployment resources (<YOUR_PROJECT>-deploy/src/main/resources/apps/) to use the latest version of the aiSSEMBLE |
| upgrade-mlflow-v2-external-s3-migration | Update the mlflow V2 deployment (if present) in your project to utilize Localstack for local development and SealedSecrets for remote deployments |
| upgrade-foundation-extension-python-package-migration | Updates the pyproject.toml files within your projects pipelines folder (<YOUR_PROJECT>-pipelines) to use the updated aiSSEMBLE foundation and extension Python packages with the latest naming convention |
|upgrade-helm-chart-files-names-migration | Updates the Chart.yaml and values*.yaml files within your project's deploy folder (<YOUR_PROJECT>-deploy) to use the new Helm chart naming convention (aissemble-<chart-name>-chart) |

To deactivate any of these migrations, add the following configuration to the `baton-maven-plugin` within your root `pom.xml`:

Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-helm/aissemble-data-access-chart/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ All properties must be prefixed with the key `aissemble-quarkus-chart` to overri
If you are migrating from the v1 version of the data access chart, use the tables below to apply any existing customizations from the old chart to the new v2 chart.

## Property Location
Old properties would have to be prepended with aissemble-data-access.aissemble-quarkus, e.g.: app.name => aissemble-data-access.aissemble-quarkus.app.name
Old properties would have to be prepended with aissemble-data-access-chart.aissemble-quarkus-chart, e.g.: app.name => aissemble-data-access-chart.aissemble-quarkus-chart.app.name

If any properties are set to the default value, we recommend removing them from your values file entirely.

Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-helm/aissemble-elasticsearch-chart/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ helm install elasticsearch ghcr.io/boozallen/aissemble-elasticsearch-chart --ver
**Note**: *the version should match the aiSSEMBLE project version.*

# aiSSEMBLE Custom Properties
The following properties are unique to the aissemble-elasticsearch chart and extend the functionality of the base ECK chart.
The following properties are unique to the `aissemble-elasticsearch-chart` chart and extend the functionality of the base ECK chart.
| Property | Description | Required Override | Default |
|----------------------------|--------------------------------------|-------------------|------------|
| basicAuth.enabled | Enable basic http auth | No | true |
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-helm/aissemble-fastapi-chart/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ helm install fastapi ghcr.io/boozallen/aissemble-fastapi-chart --version <AISSEM
| service.type | The service type | No | ClusterIP |

# Developing with aissemble-fastapi
When leveraging deployments that utilize the aissemble-fastapi Helm charts as a dependency, it may be necessary to manually modify the deployment's Dockerfile of the downstream project so that the python file defining the FastAPI business logic is copied into the Docker container. For example:
When leveraging deployments that utilize the `aissemble-fastapi-chart` Helm charts as a dependency, it may be necessary to manually modify the deployment's Dockerfile of the downstream project so that the python file defining the FastAPI business logic is copied into the Docker container. For example:

```Docker
COPY ./target/scripts/main.py /app/main.py
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-helm/aissemble-fastapi-chart/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
app:
# Sets labels for app.kubernetes.io/name Default is Chart.Name
# Sets labels for app.kubernetes.io/name Default is Chart.Name (aissemble-fastapi-chart)
name: ''
# Sets the labels for app.kubernetes.io/version. Default is Chart.AppVersion
version: ''
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-helm/aissemble-inference-chart/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ helm install inference ghcr.io/boozallen/aissemble-inference-chart --version <AI
# Properties
| Property | Description | Required Override | Default |
|-----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|-------------------|--------------------------------------------------------------------------------------------------------|
| app.name | Sets label for app.kubernetes.io/name | No | Chart.Name (aissemble-inference) |
| app.name | Sets label for app.kubernetes.io/name | No | Chart.Name (aissemble-inference-chart) |
| app.version | Sets label for app.kubernetes.io/version | No | Chart.AppVersion (aiSSEMBLE project version) |
| deployment.image.repo | The image repository | No | NB: OSS: update with default aissemble docker repository |
| deployment.image.name | The image name | Yes | boozallen/aissemble-fastapi |
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-helm/aissemble-inference-chart/values.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
app:
# Sets labels for app.kubernetes.io/name Default is Chart.Name (aissemble-inference)
# Sets labels for app.kubernetes.io/name Default is Chart.Name (aissemble-inference-chart)
name: ''
# Sets the labels for app.kubernetes.io/version. Default is Chart.AppVersion
version: ''
Expand Down
2 changes: 1 addition & 1 deletion extensions/extensions-helm/aissemble-kafka-chart/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Several properties that used to be free-form name/value pairs within the `env` b
In the table below, the notation `env[KEY]` refers the `env` list item whose `name` value was `key`.


**Note**: *all new property locations include the prefix `aissemble-kafka.kafka`*
**Note**: *all new property locations include the prefix `aissemble-kafka-chart.kafka`*

| Old Property Location | New Property Location | Additional Notes |
|-------------------------------------------|-----------------------------|----------------------------------------------------------------------------------------------------------------------------|
Expand Down
6 changes: 3 additions & 3 deletions extensions/extensions-helm/aissemble-lineage-http-consumer-chart/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ helm install lineage-http-consumer ghcr.io/boozallen/aissemble-lineage-http-cons
**Note**: *the version should match the aiSSEMBLE project version.*

# Properties
The following properties are overridden from the base [aissemble-quarkus chart](../aissemble-quarkus-chart/README.md).
Overriding inherited options requires prepending `aissemble-lineage-http-consumer.aissemble-quarkus`, following the standards for working with [Helm subcharts](https://helm.sh/docs/chart_template_guide/subcharts_and_globals/).
The following properties are overridden from the base [aissemble-quarkus-chart chart](../aissemble-quarkus-chart/README.md).
Overriding inherited options requires prepending `aissemble-lineage-http-consumer-chart.aissemble-quarkus-chart`, following the standards for working with [Helm subcharts](https://helm.sh/docs/chart_template_guide/subcharts_and_globals/).

| Property | Description | Required Override | Default |
|-----------------------|---------------------------------------|-------------------|------------------------------------------------|
| app.name | Sets label for app.kubernetes.io/name | No | aissemble-lineage-http-consumer |
| app.name | Sets label for app.kubernetes.io/name | No | aissemble-lineage-http-consumer-chart |
| deployment.image.name | The image name | No | boozallen/aissemble-data-lineage-http-consumer |

# Quarkus Configuration
Expand Down
Loading

0 comments on commit 067fddc

Please sign in to comment.