Add CI for gitian-builder repo?

[josephbisch]

Currently it is a pain to test PRs locally with all the various possible host-guest combinations. Even just considering the popular ones (Debian Jessie and Stretch hosts with Ubuntu as a guest), it can be difficult. Plus building with LXC vs KVM. Sometimes a contributor may just test with one, because that is what they use, and not properly test with the other.

Should we consider CI that will do a Gitian build of a small test project? At least with the more popular configurations? I don't think Travis CI would work for this, but maybe there is some other service that would?

Thoughts?

[devrandom]

That would be ideal. I am up for handling any service fees (e.g. AWS).

[josephbisch]

I'm still learning AWS, but I guess having SNS (based on a PR) trigger Lambda to launch an EC2 instance would be most cost effective. Like this article: https://aws.amazon.com/blogs/compute/dynamic-github-actions-with-aws-lambda/.

So then the EC2 instance(s) can be up for only as long as they are necessary. And of course add rate limiting and limit the number of instances running at any one time.

So I guess we wouldn't even need something like Jenkins. The AWS infrastructure would handle launching builds as necessary in that case.

Is that what you were thinking of?

[devrandom]

Yes, although having to integrate it with github might be a pain. I wonder if there's an off-the-shelf solution.

[josephbisch]

I found 2 possible solutions for this.

LambCI is interesting, because it just uses Lambda without launching any EC2 instances, however it isn't sufficient because of the 5 minute limitation Lambda imposes, among other limitations. The base VM creation and build probably would take longer than 5 minutes. However they also have a version that uses ECS (https://github.com/lambci/ecs) that might work for us. I'm not sure though, looks like ECS uses Docker and I don't know how that will play with KVM for Gitian.

The other one is this thing. It doesn't look like it is standalone from Buildkite though. Buildkite seems to cost money, but their pricing page mentions something about being free for open source projects.

So I couldn't find anything standalone that just uses EC2.

[josephbisch]

It looks like CodePipeline supports GitHub as a source, but I am not sure if the build results can be sent to GitHub. Plus it looks like it would require leaving a Jenkins instance up and running on EC2, which would be inefficient.

At this point, LambCI with ECS looks like the best solution to me. With privileged containers on ECS, we should be able to do what we need to do wrt KVM and LXC usage.

I can go ahead getting LambCI set up and make sure it does fit our requirements if you want. Unless you wanted to put it under your AWS account for billing purposes and since LambCI needs access to an API key to access the repo for publishing status updates.

[devrandom]

I'm happy to cover the costs. Also, I think I can create a specific API key for this and send it to you. Or if you want, you can get started with your account and switch to mine later. Up to you.

[josephbisch]

Why don't you create the API key and send it to me then. It doesn't look like there is much advantage to starting with my account and then switching later.

[josephbisch]

ping @devrandom

Looks pretty simple to setup LambCI, since they provide CloudFormation templates. Should essentially involve supplying a GitHub API key when creating the CloudFormation stack and then adding a script for the build.

[devrandom]

Looking into it now.

[devrandom]

Sent, let me know if you got it.

[josephbisch]

I got the instructions you sent. I'll try following them tomorrow.