feat: keel deploy command

[jonbretman]

Keel Deploy

This adds keel deploy to the CLI for deploying a Keel app to any AWS account (self-hosting).

Key things to review

deploy.Build

The Build function in the deploy package creates the required assets needed for deploying and puts them into the .build directory. There is no real use-case for users to do a manual but it is exposed under the command keel deploy build`, mostly for debugging purposes so if something is going wrong you can have a look at the build files without needing to deploy.

deploy.Run

The Run function in the deploy package either creates/updates all the necessary resources in AWS or it deletes all resources. These two actions correspond to the Pulumi up and deploy commands. For our CLI they are exposed as keel deploy up and keel deploy remove.

IaC

The IaC code is all in deploy/program.go. I think it's fairly straight forward and I've tried to comment the bits that might be less obvious.

Lambda handlers

This code is in deploy/lambdas/runtime and deploy/lambdas/functions. Most of what you see here was taken (and slightly modified) from the backend repo. The big difference here is rather than having separate packages for each Lambda (api, subscribers, jobs) we just have one and it uses a different handler function depending on an env var. Given they all do the same thing on boot and it's only the handler function that changes I think this is a nice approach.

Integration tests

I have updated the testing package to use both deploy.Build and the handlers from the deploy/lambdas/... package. This means by running the integration tests (or using keel test) we are testing much of the code that is actually used in the deployed environments, rather than having test-specific code.

One interesting aspect of how this works is mocking AWS API calls. This is done with a little mock server in testing/aws.go, which knows how to handle the few API calls the runtime handler makes on boot (for example to get secrets) as well as things like SQS and S3 usage. This works by configuring the runtime Lambda code to use a custom endpoint when constructing AWS clients.

Manual testing

First you need to build the runtime Lambda binary, which you can do by running this command in the repo:

$ goreleaser build --snapshot --clean --id runtime-lambda

Then you can do a deploy by running the CLI:

$ go run ./cmd/keel/main.go deploy up --env <env> --runtime-binary ./dist/runtime-lambda_linux_amd64_v1/runtime-lambda -d ../path/to/some/keel/app

The Keel app must have a keelconfig.<env>.yaml file with a deploy section in it. A minimal valid config looks like this:

deploy:
  projectName: my-project
  region: eu-west-2

Other than you just need to be authenticated into the AWS account you want to deploy to. A first deploy with RDS will take around 5 minutes and updates around 2 minutes.

To destroy all resources in AWS you run the same command but replace up with remove.

Key features

• You can view aggregated logs for all Lambdas using the keel deploy logs command. This is fairly basic at the moment but even so it's probably nicer than digging around in Cloudwatch logs.
• You can set/get/list/remove secrets for deployed environments using keel deploy secrets set etc... Note this is a different command to the one for setting local secrets - keel secrets.
• Out of the box keel deploy will use Amazon RDS for the database but you can use any publicly available Postgres database by setting the deploy.database.provider value in config to "external" and setting a DATABASE_URL secret.
• You can enable OTEL by setting the deploy.telemetry.collector value in config to an OpenTelemetry collector config file. If you set this then the collector layer will be included.

Docs

A docs PR is on it's way...

[davenewza]

Ooh!