-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Uncluster SAR release from tag release, and place it along ESF terraform dependencies #809
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| --- | ||
| # IMPORTANT: | ||
| # If you change the name of this file, you will have to update | ||
| # https://github.com/elastic/oblt-infra/blob/main/conf/resources/repos/elastic-serverless-forwarder/01-aws-oidc-github.tf | ||
| # to include the current one! | ||
|
|
||
| # Workflow to push zip with dependencies to S3 bucket every time the ESF version is updated | ||
| # (we need this for ESF terraform), and to publish the new SAR version | ||
| name: releases-production | ||
|
|
||
| on: | ||
| workflow_run: | ||
| workflows: [create-tag] | ||
| types: | ||
| - completed | ||
|
|
||
| permissions: | ||
| id-token: write # This is required for requesting the JWT | ||
| contents: read # This is required for actions/checkout | ||
|
|
||
| env: | ||
| AWS_REGION: "eu-central-1" | ||
| AWS_ACCOUNT_ID: "267093732750" # account 'elastic-observability-prod' | ||
|
|
||
| jobs: | ||
|
|
||
| get-esf-version: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
|
|
||
| outputs: | ||
| version: ${{ steps.get-version.outputs.version }} | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Get version number | ||
| id: get-version | ||
| shell: bash | ||
| run: | | ||
| version=$(grep -oE '[0-9]+\.[0-9]+\.[0-9]+(\-[a-zA-Z]+[0-9]+)?' share/version.py) | ||
| echo "version=${version}" >> $GITHUB_OUTPUT | ||
| echo "::notice::ESF version is ${version}." | ||
|
|
||
|
|
||
| build-and-upload-dependencies: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| needs: get-esf-version | ||
|
|
||
| env: | ||
| BUCKET_NAME: "esf-dependencies" | ||
|
|
||
| steps: | ||
| # See https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-create-dependencies | ||
|
|
||
| - uses: actions/checkout@v4 | ||
| with: | ||
| ref: 'lambda-v${{ needs.get-esf-version.outputs.version }}' | ||
|
|
||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.9' | ||
| cache: 'pip' # caching pip dependencies | ||
|
|
||
| - name: Install requirements in a directory and zip it. | ||
| shell: bash | ||
| run: | | ||
| pip3 install -r requirements.txt -t ./dependencies | ||
| cd dependencies && zip -r ../lambda-v${{ needs.get-esf-version.outputs.version }}.zip . | ||
|
|
||
| - name: Place handlers in the zip file. | ||
| shell: bash | ||
| run: | | ||
| zip -r ./lambda-v${{ needs.get-esf-version.outputs.version }}.zip main_aws.py | ||
| zip -r ./lambda-v${{ needs.get-esf-version.outputs.version }}.zip handlers | ||
| zip -r ./lambda-v${{ needs.get-esf-version.outputs.version }}.zip share | ||
| zip -r ./lambda-v${{ needs.get-esf-version.outputs.version }}.zip storage | ||
| zip -r ./lambda-v${{ needs.get-esf-version.outputs.version }}.zip shippers | ||
|
|
||
| - name: Configure AWS credentials | ||
constanca-m marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| uses: elastic/oblt-actions/aws/auth@v1 | ||
| with: | ||
| aws-account-id: "${{ env.AWS_ACCOUNT_ID }}" | ||
| aws-region: "${{ env.AWS_REGION }}" | ||
|
|
||
| - name: Copy file to s3 | ||
| run: | | ||
| aws s3 cp ./lambda-v${{ needs.get-esf-version.outputs.version }}.zip s3://${{ env.BUCKET_NAME }}/ | ||
|
|
||
|
|
||
| release-sar: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| needs: get-esf-version | ||
|
|
||
| env: | ||
| BUCKET_NAME: "elastic-serverless-forwarder" | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| ref: 'lambda-v${{ needs.get-esf-version.outputs.version }}' | ||
|
|
||
| - uses: elastic/oblt-actions/aws/auth@v1 | ||
| with: | ||
| aws-account-id: "${{ env.AWS_ACCOUNT_ID }}" | ||
| aws-region: "${{ env.AWS_REGION }}" | ||
|
|
||
| - uses: aws-actions/setup-sam@2360ef6d90015369947b45b496193ab9976a9b04 # v2 | ||
| with: | ||
| use-installer: true | ||
| token: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Build and package | ||
| run: | | ||
| .internal/aws/scripts/dist.sh \ | ||
| elastic-serverless-forwarder \ | ||
| ${{ needs.get-esf-version.outputs.version }} \ | ||
| ${{ env.BUCKET_NAME }} \ | ||
| ${{ env.AWS_ACCOUNT_ID }} \ | ||
| ${{ env.AWS_REGION }} \ | ||
| "Elastic" | ||
This file was deleted.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will require a change in the infra part, so the terraform definition for the OIDC is configured for the
releases-productionThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't it be better to create a role in the AWS account? This way the team could change the permissions if necessary. So:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand your question about flexibility, but we are not using that approach anymore for various security reasons:
In other words, there should be a separation of concerns when running things in the CI. I'll ping you to the original request to do the separation of concerns when accessing cloud accounts in the CI.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I understand, and it makes sense. In that case, for this PR to work:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I found this guide, but I don't understand how the job will ever know about this
*.tffile. And would the role assumed scope to the job? Can we have 2 different roles, one being used for each job, for the same workflow?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are two parts:
id-token: writeand https://github.com/elastic/oblt-actions/tree/main/aws/auth*.tffiles2)needs to be applied before1)can run, for such, it's required to list all the required permissions so it can be configured accordingly. Those*.tffiles are the ones saying what's the permissions for the given GitHub workflow. To apply tho*.tffiles, that's only possible by a few people for the reasons I mentioned above.Let me ping @reakaleek; he did all the heavy lifting and can answer your questions better than I can.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @constanca-m,
You can find our official documentation about our keyless setup at https://github.com/elastic/observability-robots/blob/main/docs/teams/ci/keyless/github-actions/amazon-web-services.md
Unfortunately, we don't differentiate between jobs in a workflow. We only separate between workflows. Essentially, we set up OIDC for a workflow. For each workflow that is defined, a role is created which can be assumed by the workflow.
In the linked documentation, you can find how we are attaching policies to a given role.
Hence, in this case we would attach the permissions needed for both jobs to a single role.
If you think it's crucial to give each job in a workflow different permissions, then let's discuss, and we can try to figure out how to extend the existing functionality to also accomodate this need.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the detailed answer. Since this is just the permissions to execute the workflow, I don't think there is an issue with granting both jobs permissions to just one role.
I have opened https://github.com/elastic/oblt-infra/pull/194 to fix this issue.