Automate deployments to SAR

[girodav]

Create a Github Workflow that automatically release Elastic Serverless Forwarder to SAR, when a new version tag is pushed

Relevant doc: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#running-your-workflow-only-when-a-push-of-specific-tags-occurs

[v1v]

I think we can help with this. We have configured OIDC access to AWS accounts in GitHub actions (see https://github.com/elastic/observability-robots/blob/main/docs/teams/ci/keyless/README.md), so there is no need to use any secrets but configure the automation.

If you can provide how you deploy to SAR currently and the steps you do now, then we can figure out the bits and pieces that are required.

Thanks

[zmoog]

Hey @v1v, thanks! I'm sharing this with the team.

[bturquet]

@v1v that would be good news if you could help us on that. Would it be possible to have this automation for both "regular" SAR and GovCloud SAR ? We are struggling a lot to have the proper permissions for the latest.

@zmoog @kaiyan-sheng would it make sense to move this issue to our current iteration to make sure that we could provide Victor the relevant information ?

[zmoog]

would it make sense to move this issue to our current iteration to make sure that we could provide Victor the relevant information ?

Yep, definitely. We already have the information Victor needs; I'll pick them all and share them here.

[zmoog]

@v1v, I have a public note zmoog/public-notes#15 that describes how to publish ESF on SAR as a private application (a good option for testing).

The process for publishing ESF as a public application remains the same. We need to use the same .internal/aws/scripts/dist.sh script with different parameters.

The parameters to change are (at least):

• AWS account (we possibly need different account IDs for the non-GovCloud public version)
• AWS region (we published ESF on eu-central-1 first, so we need to keep using it)

We may need to repeat the publishing with different settings for AWS GovCloud (US): @kaiyan-sheng probably knows more than me.

I see we already have a GitHub workflow to upload the .zip file for the Terraform installation method at https://github.com/elastic/elastic-serverless-forwarder/blob/main/.github/workflows/upload-dependencies.yml.

We can probably change the like https://github.com/elastic/elastic-serverless-forwarder/blob/196240566f7f59bad7bc952063cc80b563b5a3a2/.internal/aws/scripts/dist.sh#L73-79 since we are already building a .zip file in an S3 bucket for the Terraform installation method.

The release workflow should publish the artifacts on Terraform and SAR channels, right?

[zmoog]

Thanks to @girodav for double-checking with me and integrating this info before posting ❤️

[v1v]

Hi folks,

I'm now back from PTO and we just started a new iteration, aka sprint, so I plan to work on this in the upcoming days, I'll keep you posted.

[zmoog]

I'm now back from PTO and we just started a new iteration, aka sprint, so I plan to work on this in the upcoming days, I'll keep you posted.

Hey @v1v, thank you for taking this task! Let us know if you need anything!

[v1v]

I've just created #758

I need some help with a few things:

• the expected parameters when running .internal/aws/scripts/dist.sh for a release.
• when to run the SAR deployment for a release. I assumed I could use the release workflow and include a new job to run for the given git tag.

Would it be possible to have this automation for both "regular" SAR and GovCloud SAR ? We are struggling a lot to have the proper permissions for the latest.

I'm unfamiliar with GovCloud SAR, so it will probably be postponed in a follow-up if we can even automate it.

I see we already have a GitHub workflow to upload the .zip file for the Terraform installation method at main/.github/workflows/upload-dependencies.yml.

How does it work at the moment? IIUC, upload-dependencies can accidentally publish a version since the checkout is not fixed to the tag but main, so someone could accidentally push a new commit while the release workflow runs.

Maybe it can be combined with the release workflow so we can ensure the sha commit when the release runs is the one used for generating the artifacts in the s3 bucket.

We can probably change the like 1962405/.internal/aws/scripts/dist.sh#L73-79 since we are already building a .zip file in an S3 bucket for the Terraform installation method.
The release workflow should publish the artifacts on Terraform and SAR channels, right?

Can you contribute to my above-mentioned PR with the required changes please?

[v1v]

Status update

I enabled the OIDC targeting the elastic-observablity dev AWS account and create my s3 bucket for testing my changes.

Then I created a branch based on my PR but removing some bits and pieces that were not needed, and I finally ran:

• https://github.com/elastic/elastic-serverless-forwarder/actions/runs/10286873394/job/28470615564

It worked fine after discovering the required permissions to publish the application with sam publish and

• https://us-east-1.console.aws.amazon.com/serverlessrepo/home?region=us-east-1#/published-applications/arn:aws:serverlessrepo:us-east-1:627286350134:applications~elastic-serverless-forwarder

Therefore, the only missing bits and pieces are the questions I asked above.

Cheers

[zmoog]

Would it be possible to have this automation for both "regular" SAR and GovCloud SAR ? We are struggling a lot to have the proper permissions for the latest.

I'm unfamiliar with GovCloud SAR, so it will probably be postponed in a follow-up if we can even automate it.

Yeah, we discussed this during a team meeting and agreed it's better to focus on automating SAR first and then evaluate the gap later.

[v1v]

Thanks @zmoog

Do you happen to know who can help me with some of the questions I have (see #279 (comment))? If it helps, we can syncup in a zoom call

Thanks

[zmoog]

Do you happen to know who can help me with some of the questions I have (see #279 (comment))?

Yeah, I can help with this.

I'm sorry for the delay. I started working on this but got interrupted, so I'm resuming today!

If it helps, we can syncup in a zoom call

Thanks! Let's see how far I can go. I won't hesitate to contact you for a call if it helps!

[zmoog]

I need some help with a few things:

• the expected parameters when running .internal/aws/scripts/dist.sh for a release.

I have a public note zmoog/public-notes#15 describing the publishing process of ESF on SAR as a private application. IIRC, the process is identical, and we need to use a specific AWS account and region for publishing it as a public application.

Here's the command line format:

.internal/aws/scripts/dist.sh \
    sar-app-name \
    version-in-sar
    s3-bucket-for-sar \
    account-id \
    sar-region \
    "author name"

Here is the parameter info I remember or found in my notes. I'll update this table as I get more details.

Parameter	Notes	Example values	Public release value
sar-app-name	The SAR application name.	elastic-serverless-forwarder or elastic-serverless-forwarder-<YOUR NAME>	Probably the name is elastic-serverless-forwarder — @constanca-m, you was probably the last person to release ESF, did you use this app name?
version-in-sar	The SAR application version.	Possible values are 1.8.0.	IIRC, the latest version can be found in the share/version.py file.
s3-bucket-for-sar	S3 bucket that the publishing process uses to store artifacts.	zmoog-elastic-serverless-forwarder-artifacts
account-id	AWS account ID where to publish the SAR application.
sar-region		eu-west-1	eu-central-1 — @constanca-m, did you use this region, right?
author name	The SAR author name	Elastic	Elastic

[zmoog]

• when to run the SAR deployment for a release. I assumed I could use the release workflow and include a new job to run for the given git tag.

Yep, I would extend the current workflow for releasing the Terraform artifacts with a job for SAR!

[zmoog]

I see we already have a GitHub workflow to upload the .zip file for the Terraform installation method at main/.github/workflows/upload-dependencies.yml.

How does it work at the moment?

@constanca-m, please correct me if I'm wrong. This workflow packages ESF as a release .zip file. It is designed to deploy ESF using the Terraform module at https://github.com/elastic/terraform-elastic-esf and is unrelated to SAR.

IIUC, upload-dependencies can accidentally publish a version since the checkout is not fixed to the tag but main, so someone could accidentally push a new commit while the release workflow runs.

Oh, good point! We need to fix this then!

Maybe it can be combined with the release workflow so we can ensure the sha commit when the release runs is the one used for generating the artifacts in the s3 bucket.

Sounds great.

[zmoog]

We can probably change the like 1962405/.internal/aws/scripts/dist.sh#L73-79 since we are already building a .zip file in an S3 bucket for the Terraform installation method.
The release workflow should publish the artifacts on Terraform and SAR channels, right?

Can you contribute to my above-mentioned PR with the required changes please?

I need to check whether we can use the exact .zip file for SAR and Terraform or if they differ.

[constanca-m]

Yes to your questions, I followed the script the last time I updated SAR like this:

AWS_PROFILE=elastic-observability-prod .internal/aws/scripts/[dist.sh](http://dist.sh/) elastic-serverless-forwarder 1.14.0 elastic-serverless-forwarder 267093732750 eu-central-1

[v1v]

Status update

#758 is ready for review. I kept what we discussed in #279 (comment); I mentioned in the PR that it might be done in a follow-up. I prefer to keep it simple for now and iterate in small PRs.

[v1v]

Hi folks,

Can I get some more eyes for #758? I don't know if it can be merged as is or if you have any concerns, but I need you to validate if the implementation is what you had in mind. In addition, if the arguments are the expected ones when using .internal/aws/scripts/dist.sh

Thanks

[v1v]

Status update

PR has been migrated so that you can proceed with your regular release -🤞 the automation should do the rest.

As we said, I didn't enable any support for GovCloud. We don't have access, and in my opinion, that's something to be done, likely in a follow-up.

If nothing else is needed, I'll close this task in our project board. Please ping me or Robots if you need any clarifications or help.

Cheers

[zmoog]

Thanks @v1v, great work! We are going to use the release process today or tomorrow to release a bug fix.

We can work on GovCloud in the next iteration.