Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test] Cast from bool to string #39

Closed
wants to merge 38 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
e4c1aef
Add dbt profile to the project
wcchang1115 May 16, 2024
3aaa254
Merge pull request #2 from DataRecce/dbt/profiles
wcchang1115 May 16, 2024
2c2dc03
Merge branch 'opensource-observer:main' into dev
kentwelcome May 16, 2024
5ec9600
Add recce staging env preparation workflow
wcchang1115 May 20, 2024
8f3bf78
Refine workflow
wcchang1115 May 20, 2024
c80542c
Merge pull request #5 from DataRecce/ci/staging-daily-job
wcchang1115 May 20, 2024
4cf3ac5
Add recce ci workflow
wcchang1115 May 16, 2024
29254a0
Add triggering event
wcchang1115 May 20, 2024
fe1d438
[Feature] DRC-443 Setup dev-container to integrate with GitHub Codespace
kentwelcome May 16, 2024
9b75cd3
[Feature] Execute recce by bash script
kentwelcome May 21, 2024
7cdf571
Merge pull request #17 from DataRecce/feature/drc-430-setup-devcontai…
kentwelcome May 21, 2024
f9768ce
Add condition to staging env preparation workflow
wcchang1115 May 22, 2024
d5598be
Merge pull request #28 from DataRecce/ci/update-staging-daily-job
wcchang1115 May 22, 2024
247461e
Merge pull request #1 from DataRecce/dbt/artifacts_by_user
wcchang1115 May 23, 2024
8027ae6
[Fix] Skip dbt build when recce state file is found
kentwelcome May 23, 2024
78b0572
Ad prsync
popcornylu May 21, 2024
8bfc679
Add prsync
popcornylu May 21, 2024
4ad2383
Add prsync
popcornylu May 21, 2024
fa2fcf3
Add prsync
popcornylu May 22, 2024
db3d54d
Add workflow
popcornylu May 22, 2024
d9b9163
Add git username
popcornylu May 22, 2024
719e453
Add workflow permission
popcornylu May 22, 2024
f53fce4
Add PAT
popcornylu May 23, 2024
1a131dc
Update the ci
popcornylu May 23, 2024
3f6b2fe
Merge pull request #20 from DataRecce/feature/drc-444-implement-the-g…
popcornylu May 23, 2024
3452a85
Merge pull request #31 from DataRecce/feature/drc-446-enhancement-loa…
kentwelcome May 23, 2024
4be70f2
Add manual trigger
popcornylu May 23, 2024
65f86eb
Merge pull request #33 from DataRecce/chore/add-manual-dispatch-for-ci
popcornylu May 23, 2024
ad428aa
Add manual trigger
popcornylu May 23, 2024
00696a3
Merge pull request #34 from DataRecce/chore/add-manual-dispatch-for-ci
popcornylu May 23, 2024
2ade5f0
Dont put the workflow file in the prsync
popcornylu May 23, 2024
8bd2fb6
Merge pull request #35 from DataRecce/chore/add-manual-dispatch-for-ci
popcornylu May 23, 2024
e7b7bd3
Add dummy workflows
popcornylu May 23, 2024
c70e0a9
test permission
popcornylu May 23, 2024
b8d3ca9
Merge pull request #36 from DataRecce/chore/fix-the-workflow-permissi…
popcornylu May 23, 2024
e529c37
Test permmision issue
popcornylu May 23, 2024
9336f2f
Cast from bool to string
wcchang1115 May 24, 2024
f76ec9b
Adjust trusted user criteria
wcchang1115 May 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
FROM mcr.microsoft.com/vscode/devcontainers/python:3.11

# Add GitHub CLI
RUN (type -p wget >/dev/null || (sudo apt update && sudo apt-get install wget -y)) \
&& sudo mkdir -p -m 755 /etc/apt/keyrings \
&& wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
&& sudo chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null

RUN apt-get update && apt-get install -y python3-dev gh && rm -rf /var/lib/apt/lists/*

ARG USER_UID=1000
ARG USER_GID=$USER_UID

RUN if [ "$USER_GID" != "1000" ] || [ "$USER_UID" != "1000" ]; then groupmod --gid $USER_GID vscode && usermod --uid $USER_UID --gid $USER_GID vscode; fi

RUN pip3 install --upgrade pip
RUN pip3 install pipx
RUN pipx install poetry

# Downloading gcloud package
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz

# Installing the package
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh

# Adding the package path to local
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin

# Install osos python dependencies
COPY . /tmp/oso
RUN cd /tmp/oso && pip3 install .
RUN pip3 install recce-nightly
22 changes: 22 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
"name": "Recce CodeSpace",
"image": "infuseai/oso-dev-container:3.11",
"containerEnv": {
"DBT_GOOGLE_PROJECT": "infuseai-dev",
"DBT_GOOGLE_DATASET": "oso_playground",
"DBT_GOOGLE_DEV_DATASET": "oso_playground_dev",
"DBT_GOOGLE_KEYFILE": "/home/vscode/.config/gcloud/google-service-account.json",
"RECCE_CI_WORKFLOW_NAME": "OSO Recce CI",
"RECCE_DAILY_CI_WORKFLOW_NAME": "OSO Recce Staging CI"
},
"customizations": {
"vscode": {
"settings": {
"terminal.integrated.shell.linux": "/bin/bash",
"python.terminal.activateEnvironment": true
}
},
"extensions": []
},
"forwardPorts": [8000]
}
3 changes: 3 additions & 0 deletions .devcontainer/docker-build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#! /bin/bash

docker buildx build --platform linux/amd64 -t infuseai/oso-dev-container:3.11 .. -f Dockerfile
30 changes: 30 additions & 0 deletions .devcontainer/github_codespace_env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#! /bin/bash

if [ "${CODESPACES}" == "true" ]; then
# Set the default git repository if running in GitHub Codespaces
echo "Setting the default git repository to $GITHUB_REPOSITORY"
gh repo set-default $GITHUB_REPOSITORY

current_branch=$(git branch --show-current)
# Check if the current branch is under a pull request
if gh pr view > /dev/null ; then
# Check if the Recce state file is downloaded
run_id=$(gh run list -b ${current_branch} -s success --limit 1 -w "${RECCE_CI_WORKFLOW_NAME}" --json databaseId | jq .[].databaseId)
if [ -z "$run_id" ]; then
echo "No successful Recce run found for the current branch."
else
echo "Downloading the Recce state file for the last successful run."
gh run download $run_id --dir .recce
echo "The Recce state file is downloaded to '.recce/recce_state_file/recce_state.json'."
fi
fi

# Check daily staging artifact files
default_branch=$(gh repo view --json defaultBranchRef --jq .defaultBranchRef.name)
daily_artifact_workflow_id=$(gh run list -w "${RECCE_DAILY_CI_WORKFLOW_NAME}" --status success -b dev --limit 1 --json databaseId | jq .[].databaseId)
gh run download $daily_artifact_workflow_id --dir .recce
if [ -d ".recce/dbt-artifacts" ]; then
mv .recce/dbt-artifacts target-base
echo "The daily staging artifact files are downloaded to 'target-base'."
fi
fi
12 changes: 12 additions & 0 deletions .devcontainer/launch_recce_server.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#! /bin/bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

if [ -f "${DIR}/../.recce/recce_state_file/recce_state.json" ]; then
echo "Launching the Recce server in review mode. The Recce state file is found."
cp ${DIR}/../.recce/recce_state_file/recce_state.json recce_state.json
recce server --review recce_state.json
else
echo "Launching the Recce server."
recce server
fi
14 changes: 14 additions & 0 deletions .devcontainer/prepare_dbt_manifest.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#! /bin/bash

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

# Check if the dbt's target path is existed
if [ -d "${DIR}/../recce_state.json" ]; then
echo "The Recce state file is found. Skip preparing the dbt manifest."
exit 0
fi

echo "Preparing the dbt manifest."
pushd ${DIR}/../
dbt deps && dbt build && dbt docs generate
popd
61 changes: 61 additions & 0 deletions .devcontainer/setup_required_env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#! /bin/bash

GREEN="\033[0;32m"
YELLOW="\033[1;33m"
ENDCOLOR="\033[0m"

function show_env_hint() {
echo -e "[${YELLOW}Required ENV${ENDCOLOR}] $1 is not set."
cat << EOF

Please set the following environment variables in your GitHub Codespaces Secrets.
You can set the secret in your GitHub Codespaces Secrets by going to:
GitHub Personal Account -> Settings -> Codespaces -> Codespaces secrets

Then add the secret with the name $1 and the value of the secret.
After adding the secret, please restart the Codespaces to apply the changes.
EOF
}

# Check the ENV variables should be provided by the user
if [ -z "$DBT_GOOGLE_PROJECT" ]; then
show_env_hint "DBT_GOOGLE_PROJECT"
exit 1
fi

if [ -z "$DBT_GOOGLE_DATASET" ]; then
show_env_hint "DBT_GOOGLE_DATASET"
exit 1
fi

if [ -z "$DBT_GOOGLE_DEV_DATASET" ]; then
show_env_hint "DBT_GOOGLE_DEV_DATASET"
exit 1
fi

mkdir -p $HOME/.config/gcloud
DBT_GOOGLE_KEYFILE=$HOME/.config/gcloud/google-service-account.json

# Setup dbt profiles.yml
if [ "$DBT_PROFILES_YML_CONTENT" != '' ]; then
echo "$DBT_PROFILES_YML_CONTENT" > $HOME/.dbt/profiles.yml
echo "dbt profiles.yml is saved to $HOME/.dbt/profiles.yml"
fi


# Check if the user is already logged in
if [ -z "$GOOGLE_CLOUD_SERVICE_ACCOUNT_KEY_CONTENT" ]; then
# Change to use OAuth2 to login
if [ -f "${DBT_GOOGLE_KEYFILE}" ]; then
echo "User is already logged in Google cloud"
gcloud auth list
exit 0
else
echo -e "[${GREEN}Action${ENDCOLOR}] Please login to Google cloud to continue."
gcloud auth application-default login
fi
else
# Use the service account key to login
echo "$GOOGLE_CLOUD_SERVICE_ACCOUNT_KEY_CONTENT" > ${DBT_GOOGLE_KEYFILE}
echo "Google cloud service account key is saved to ${DBT_GOOGLE_KEYFILE}"
fi
115 changes: 115 additions & 0 deletions .github/workflows/recce-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
name: OSO Recce CI

on:
workflow_dispatch:
pull_request:
branches: [main, dev]
paths:
- warehouse/dbt/**

env:
# dbt env variables used in your dbt profiles.yml
DBT_PROFILES_DIR: ./
DBT_GOOGLE_PROJECT: ${{ vars.DBT_GOOGLE_PROJECT }}
DBT_GOOGLE_DATASET: ${{ vars.DBT_GOOGLE_DATASET }}
DBT_GOOGLE_KEYFILE: /tmp/google/google-service-account.json
KEYFILE_CONTENTS: ${{ secrets.KEYFILE_CONTENTS }}

jobs:
check-pull-request:
name: Check pull request by Recce CI
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.12.x"

- name: Install dependencies
run: |
pipx install poetry==1.7.1
poetry install
poetry run which dbt

- name: Install Recce
run: poetry run pip install recce-nightly

- name: Add packages.yml file
run: |
echo '${{ vars.PACKAGES_YAML }}' > packages.yml

- name: Prep Google keyfile
run: |
mkdir -p "$(dirname $DBT_GOOGLE_KEYFILE)"
echo "$KEYFILE_CONTENTS" > $DBT_GOOGLE_KEYFILE

- name: Prepare dbt Base environment
run: |
run_id=$(gh run list --workflow "OSO Recce Staging CI" --repo DataRecce/oso --status success --limit 1 --json databaseId --jq '.[0].databaseId')
gh run download $run_id --repo DataRecce/oso
mv dbt-artifacts target-base
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Set PR Schema
run: echo "DBT_GOOGLE_DEV_DATASET=OSO_PR_${{ github.event.pull_request.number }}" >> $GITHUB_ENV

- name: Prepare dbt Current environment
run: |
source $(poetry env info --path)/bin/activate
dbt deps
dbt build --target ${{ env.DBT_CURRENT_TARGET}}
dbt docs generate --target ${{ env.DBT_CURRENT_TARGET}}
env:
DBT_CURRENT_TARGET: "dev"

- name: Run Recce CI
run: poetry run recce run

- name: Archive Recce State File
uses: actions/upload-artifact@v4
id: recce-artifact-uploader
with:
name: recce-state-file
path: recce_state.json

- name: Prepare Recce Summary
id: recce-summary
run: |
source $(poetry env info --path)/bin/activate
recce summary recce_state.json > recce_summary.md
cat recce_summary.md >> $GITHUB_STEP_SUMMARY
echo '${{ env.NEXT_STEP_MESSAGE }}' >> recce_summary.md

# Handle the case when the recce summary is too long to be displayed in the GitHub PR comment
if [[ `wc -c recce_summary.md | awk '{print $1}'` -ge '65535' ]]; then
echo '# Recce Summary
The recce summary is too long to be displayed in the GitHub PR comment.
Please check the summary detail in the [Job Summary](${{github.server_url}}/${{github.repository}}/actions/runs/${{github.run_id}}) page.
${{ env.NEXT_STEP_MESSAGE }}' > recce_summary.md
fi

env:
ARTIFACT_URL: ${{ steps.recce-artifact-uploader.outputs.artifact-url }}
NEXT_STEP_MESSAGE: |
## Next Steps
If you want to check more detail inforamtion about the recce result, please download the [artifact](${{ steps.recce-artifact-uploader.outputs.artifact-url }}) file and open it by [Recce](https://pypi.org/project/recce/) CLI.

### How to check the recce result
```bash
# Unzip the downloaded artifact file
tar -xf recce-state-file.zip

# Launch the recce server based on the state file
recce server --review recce_state.json

# Open the recce server http://localhost:8000 by your browser
```

- name: Comment on pull request
uses: thollander/actions-comment-pull-request@v2
with:
filePath: recce_summary.md
39 changes: 39 additions & 0 deletions .github/workflows/recce-prsync.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# This workflow use the Github PR Syncer https://github.com/dataRecce/github-pr-syncer/
# to sync the PRs from the upstream repository
name: OSO Recce PR Sync

on:
workflow_dispatch:
schedule:
- cron: '30 17 * * *' # run at 1:30 AM (UTC + 8) everyday

jobs:
prsync:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.x'
cache: 'pip'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install git+https://github.com/DataRecce/github-pr-syncer.git

- name: Set up Git
run: |
git config --global user.name "prsync[bot]"
git config --global user.email "prsync[bot]@users.noreply.github.com"

- name: Run pysync
env:
# GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GH_TOKEN_FOR_ADVANCED_PERMISSIONS }}
run: |
prsync 'DataRecce/oso'
Loading
Loading