Improving the usability of nightly build outputs

[zaneselvans]

We have a couple of other repos that are downloading nightly build outputs and the code that goes with them (from dev) but when the nightly builds break these projects also break — since their DB is then older than the code as the DB is from the most recent successful build, but the code is from the current dev branch. These repos include at least CCAI Entity Matching and RMI Energy Communities. Are there others @cmgosnell @katie-lamb?

Now that we're merging dev into main whenever the nightly builds succeed, it seems like a better pattern might be to associate the publicly available build outputs with main -- since then folks will always reliably have data and code that are in sync. We should also automate (or at least semi-automate) the practice of merging dev into main when the builds succeed, and take precautions to ensure that no additional commits can be added between when the build is kicked off, and the branch is merged.

Maybe what we really want to do is:

• Automatically create a nightly-build-YYYY-MM-DD branch off of dev and create a draft PR from that branch into main
• Run a nightly build / deployment using code checked out from the nightly build branch.
• If the build is successful, automatically merge the PR into main and push the build outputs to their various externally visible locations — Datasette, GCS, and the AWS Open Data bucket — referring to main or latest or some other label that’s not dev in those locations.
• Delete the now merged nightly build branch.

That way the "default" outputs would be as fresh as possible while still definitely being associated with a successful build, and there would be a reliable correspondence between the code on the main branch and the currently public nightly builds.

If the build fails what do we do? It could be that a database has been created, but it didn’t pass the data validations & tests, in which case it would be useful to make that database easily accessible for debugging the failure. Maybe in this case the ouputs get sent to the easily accessible to us, but less obvious to the rest of the world gs://intake.catalyst.coop/dev bucket. But it could also be that the construction of the DB failed, so maybe having it only sometimes moved to dev would be confusing for us even internally.

[zaneselvans]

This feels like it must be a very common deployment pattern. I guess the nightly-build-YYYY-MM-DD branch would become a kind of staging environment, where no changes are allowed and we run all the checks we have to try and verify that everything is working, and if everything looks good, we deploy it publicly.

[jdangerx]

I think this is a great idea! Typically I've heard of this pattern as a "release branch." We're basically moving towards continuous deployment, modulo the extremely long runtime of our end-to-end testing.

This avoids having to worry about new stuff landing in dev after the nightly build kicks off, too.

If I'm reading this correctly, your plan is:

1. git checkout dev && git checkout -b release-XXX and run the nightly build process. output goes to the internal nightly-build-outputs.catalyst.coop GCS bucket under release-XXX/
2. if nightly build is successful:

• merge release-XXX into main, --ff-only to make sure it's really the same code
• copy outputs to intake.catalyst.coop/main, to avoid having to re-run nightly build on main

3. clean up

We could do the two parts of step 2 in either order. If we merge first, there will be a few minutes where main code is pointing at old main data until the copy is finally complete. If we upload first, there will be a few minutes where main code is pointing at newer data than it knows what to do with.

I think that's... fine... for now. If we really want no downtime for this changeover, we could do this:

1. make release branch, run nightly build
2. put output at intake.catalyst.coop/
3. refer directly to $CURRENT_GIT_HEAD_HASH downstream - maybe make a CLI tool to make this a little less painful
4. OR, instead of ^, have a little server that resolves the /main url to the actual git ref. which seems like overkill?

[zaneselvans]

I guess I'm not that worried about a couple of minutes of downtime or things being out of sync in the wee hours (US time) every night. It'll be a big improvement from the current situation! If we get to where the time it takes for the update is actually impacting something I guess that will mean it's getting used.

Should we merge main into dev before creating the release branch and kicking off the nightly build? There are often these no-diff merge commits that show up which I don't really understand but make main look like it's technically ahead of dev even though there's no code difference.

I guess we probably want to to automate both the copy-to-outputs and merge-on-build-success and skip creating a PR so it all happens at once, hopefully in the middle of the (US) night.

[jdangerx]

Yeah, I'm not worried about the downtime either.

In terms of the funky no-diff merge commits - this happens because dev doesn't always contain the head of main in its history when we're merging branches into main. If we want to get rid of those, we will need to make sure that the main history is always a subset of dev's. Since this is not now the case, we will need to do a bit of a forced alignment before proceeding:

1. On dev: git rebase -i main, or git reset origin/main --hard (since right now there is no difference between dev and main, code-wise - just history-wise)
2. When we create a release branch off of dev, that will also contain main in its history
3. On main: git merge release-XXX --ff-only - this ensures that we won't create a new merge commit, but just move the main pointer to the commit that release-XXX points at.

If we ever land a commit on main that doesn't then land in dev (e.g. a merge commit) then we'll have to rebase dev onto main again and force-push. Which is kind of annoying but not unrecoverable :)

Agreed also that we want to make the merge happen automatically once the nightly builds pass. This will probably involve shoving some github credentials into the GCP VM. I think the docker image does have git installed (for our setuptools-scm stuff), so we should be able to script that git juggling so long as we have creds.

[jdangerx]

Forking a conversation here from Slack:

@jdangerx: We’ve gotten to a point where dev doesn’t merge cleanly into main on GH. Seems like we need to merge main into dev again…

It would be nice to have main and dev stay on the same timeline, instead of being on different timelines that happen to have landed on the same codebase. We could do this by:

• making sure that we always fast-forward when we merge dev into main - this ensures that main is always somewhere behind dev on the same timeline.
  • this can get funky if we make a release branch off of dev and then forget to --ff-only when we merge into main, or if we make a commit to the release branch that doesn’t exist on dev.
  • there’s no GH-native way to do this, but there’s a github action that can do this for us…
• eliminating dev altogether and just branching directly off of main - this means we only have one Canonical Timeline to manage.
  • we’d have to set up some sort of release tagging situation so downstream folks aren’t pinned to something that moves super fast: weekly/monthly builds that correspond to e.g. v20230501 and also stable/nightly builds that correspond to stable/nightly tags that get updated manually/on nightly build success might work well
    Solutions that do not really work:
• flipping the “require linear history” switch on the main branch in GitHub - this means we can only squash/rebase into main, which keeps main linear and easy to follow but does not fix the problem of “dev and main look the same but have totally divergent histories”

@rousik: I would favor the idea of trunk-based development and elimination of dev branch altogether. This may not be an entirely trivial change, but I think that in the long run it leads to simpler process and favors/forces more attention to testing/validation procedures to ensure safer software development. In essence, if you want more automation and things like nightly data releases, you need to invest in safeguards to ensure that these (likely automated) processes are reasonably safe and at that point, you can basically run these on a continuous basis and ensure continuous health of main, instead of trying to verify these things when dev is merged to main.

@jdangerx: I have a Q2 task of doing some automated git stuff on nightly-build success - it will probably be easier to do the “push some tags & move some other tags around” workflow than the “juggle branches and make sure everything is properly --ff-only” workflow

[jdangerx]

To expand on the "move refs, no dev" approach, I am imagining:

1. nightly build kicks off on a schedule on main
2. when nightly build succeeds:
3. on whatever release cadence we want (monthly?): git tag v<date> && git push <tagname>
4. every time: git checkout nightly && git merge <git ref> --ff-only && git push
5. We move the stable branch manually by fast-forwarding it along to various tags as we see fit.

Then nightly, stable, and v<date> are:

• never pushed to by a human
• always at or behind main
• always associated with outputs on Intake

I guess we basically only push to dev and feature branches as humans, and the "automate merges from dev to main" pitch just does this:

• do some git juggling to make sure that dev and main have the same git history
• think of dev as the trunk, and then update main instead of nightly

Which will get us most of the value of "only use main" - we could even tell github that dev is the real Main, and make sure everyone knows that main is not actually to be developed off of. But I think at that point, why not just use less surprising branch names?

[jdangerx]

One of the issues blocking adoption of trunk-based development with tagged releases was the library nature of our code. Since we seem to be moving away from supporting the library use-case in #2979 , I want to re-propose the following:

1. Use main as the canonical branch - new branches come off of main, PRs get merged back in. Minimize usage of feature branches, since we can usually hide new/in-progress functionality from users simply by not distributing the data outputs.
2. Every night, we tag whatever is on main as nightly-build-xxx and then run the nightly build process on it. If all checks succeed, publish nightly-build-xxx outputs and re-deploy Datasette.

[jdangerx]

Maybe a quick diagram is a little easier to understand re: the various git refs:

The gray tags refer to specific versions (vX are manual versions, nightly-X are nightly builds); the colorful nightly/stable/main tags are branches that move around.

[jdangerx]

The docs issue you raised is a good argument, I think, for explicitly having a branch that points at the latest passing nightly builds.

If we ever get the CI to a fast enough point that we don't need to run nightly builds and instead we could run commit-ly builds, then we can run everything off of main and "just" fix the build whenever main is broken - but I think we can cross that bridge when we get to it.

[zaneselvans]

Ah okay, so in this scenario:

• main is the bleeding edge of potential brokenness, where all we know for sure is that the CI on the PR passed, and shouldn't be used by anyone not involved in ongoing development.
• nightly-2023-10-30 is some random nightly build that may or may not have actually worked, but it's definitely the code that went into that nightly build attempt.
• nightly always corresponds to a build that succeeded, and is responsible for the data currently available via Datasette and the freshest outputs on AWS.
• stable is also definitely a build that passed, and which will also stick around for a long time for future reference in AWS, and via an archive on Zenodo that's got a DOI and a manual tag like v2023-10-30, and there may be many such previous tags, corresponding to past manual tags + archives.

And in the docs, we'd probably want the nightly tag (or maybe the stable) tag to be the default landing page, with the previous versions that had ever been marked stable around for future reference in case someone is working with an older archive (as well as having the current nightly tag be available in the docs).

That sounds like a good setup to me.

[jdangerx]

Yes, exactly! If we were to move to this, then, we'd need to:

• on a nightly schedule, create a nightly-YYYY-MM-DD tag on main and push it, triggering the build-deploy-pudl script
• in build-deploy-pudl:
  • on success for any tag starting with nightly-:
    • push to AWS & GCS
    • update nightly branch ref
    • redeploy datasette
  • on success for tag starting with v:
    • push to AWS & GCS
    • update stable branch ref
    • make sure Zenodo archives the repo
• on push to nightly or stable branch ref: RTD build

We could chunk this work into separate PRs:

• push stuff to nightly namespace in cloud storage buckets + update the nightly branch when a nightly-XXXX tag is pushed
• push stuff to stable namespace in cloud storage buckets + update the stable branch when a vXXX tag is pushed
• redeploy datasette on nightly pushes
• only archive vXXX tags on Zenodo
• write an action to trigger RTD build on push to nightly/stable
• write an action to create nightly tags automatically
• force main to look like dev, then remove dev

[zaneselvans]

I think we can configure RTD to build the docs for the nightly and stable without needing to set up a separate action.

• At some point hopefully before the end of the year we should automate the creation an archive of the v202X-XX-XX data release on Zenodo whenever stable is updated.
• We probably also want to look at the branch protection rules on dev and main and make sure main has the protections we want.
• Need to change which branch dependabot runs on.
• Update the nightly build / direct link sections of the documentation.
• Update the cloud bucket address that Kaggle pulls the nightly builds down from.