Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug GitHub Action to create new GCP Composer Environment #50

Draft
wants to merge 12 commits into
base: master
Choose a base branch
from
Draft

Debug GitHub Action to create new GCP Composer Environment #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
cee8aaa
fix broken quotes
andrewphilipsmith Jul 21, 2021
61cc5ad
test local variable syntax
andrewphilipsmith Jul 21, 2021
f90f6e0
Update new_airflow_environment.yml
andrewphilipsmith Jul 21, 2021
e0d93a2
Update new_airflow_environment.yml
andrewphilipsmith Jul 21, 2021
8a640c6
Update new_airflow_environment.yml
andrewphilipsmith Jul 21, 2021
ba89116
fix syntax for local vars
andrewphilipsmith Jul 21, 2021
837f760
consistent var names!
andrewphilipsmith Jul 21, 2021
4848ec5
Update new_airflow_environment.yml
andrewphilipsmith Jul 21, 2021
112eee6
Switch to really creating new env (not just query existing ones)
andrewphilipsmith Jul 21, 2021
f27dc33
Add `zone` switch to create composer cmd
andrewphilipsmith Jul 21, 2021
3122ee6
Include env vars in composer create cmd
andrewphilipsmith Jul 22, 2021
42595d1
Update CHANGELOG.md
andrewphilipsmith Jul 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .github/workflows/new_airflow_environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: New_Airflow_Environment
name: Create New GCP Composer Environment (Airflow instance)

on:
workflow_dispatch:
Expand All @@ -20,24 +20,24 @@ jobs:
project_id: ${{ secrets.GCS_PROJECT }}
export_default_credentials: true
- name: Create New Environment
# run: gcloud composer environments create ${{ github.event.inputs.newComposerEnvName }} --location=europe-west2 --project=datapipeline-295515 --machine-type=n1-standard-2
run: gcloud composer environments describe ${{ github.event.inputs.newComposerEnvName }} --location=europe-west2 --project=datapipeline-295515
run: gcloud composer environments create ${{ github.event.inputs.newComposerEnvName }} --location=europe-west2 --project ${{ secrets.GCLOUD_PROJECT }} --machine-type=n1-standard-2 --zone=europe-west2-a --env-variables ENVIRONMENT=PRODUCTION --env-variables GCP=TRUE
# run: gcloud composer environments describe ${{ github.event.inputs.newComposerEnvName }} --location=europe-west2 --project=datapipeline-295515
- name: Get URL of new DAG bucket full path
id: dag_path
run: |
dagpath=$(gcloud composer environments describe --location=europe-west2 --project=datapipeline-295515 ${{ github.event.inputs.newComposerEnvName }} --flatten=config.dagGcsPrefix --format=object)
echo ""::debug::$dagpath"
echo "::set-output name=dag_url::$dagpath"
dagpath=$(gcloud composer environments describe ${{ github.event.inputs.newComposerEnvName }} --location=europe-west2 --project= ${{ secrets.GCLOUD_PROJECT }} --flatten=config.dagGcsPrefix --format=object)
echo "::debug::${dagpath}"
echo "::set-output name=dag_url::${dagpath}"
- name: Get URL of new DAG bucket root
id: bucket_root
run: |
url=$(python -c "import re; print(re.search('(gs:.+\/)(dags$)', r'${{ steps.dag_path.outputs.dag_url }}').group(1))")
echo "::set-output name=bucket_url::$url"
url=$(python -c "import re; print(re.search('gs:\/\/(.+)(\/dags$)', r'${{ steps.dag_path.outputs.dag_url }}').group(1))")
echo "::set-output name=bucket_url::${url}"
- name: Update DAG bucket secret
run: echo ${{ steps.bucket_root.outputs.dag_url }}
run: echo ${{ steps.bucket_root.outputs.bucket_url }}
# uses: hmanzur/[email protected]
# with:
# name: 'GCLOUD_BUCKET_PATH'
# value: ${{ steps.bucket_root.outputs.dag_url }}
# value: ${{ steps.bucket_root.outputs.bucket_url }}
# repository: ${{ env.GITHUB_REPOSITORY }}
# token: ${{ secrets.REPO_ACCESS_TOKEN }}
5 changes: 3 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@ Please maintain both this file AND the README.md file.
## [Unreleased]

### Added

* initial GitHub Actions for Continuous Integration (CI)
- Changed to GitHub Actions:
* initial GitHub Actions for Continuous Integration (CI)
* A new GCP Composer Environment can now be creatd using GitHub Actions

### Changed
### Deprecated
Expand Down