Dockerfiles are janky

[afuller-TT]

Currently the docker image to use is:

1. Not versioned - we may pick up the wrong image
2. May be regenerated needlessly
3. Wastes gh api calls, which are limited
4. Unable to be used at the job scope

Proposed solution:

• A new job that:

  • runs on cloud (do not consume a custom runner)
  • computes the sha1 of ingredient files for the docker image
  • queries the container registry for an image tagged with that sha1
  • Exports both values
  • For split containers (eg: build vs test), then export the pair for each container.

• The build-docker is run IFF a container is missing

  • New images are tagged according to the ingredient's SHA1

• All downstream jobs read the output of the initial Docker job and may run a container at the job scope.

Footnotes:

1. is because we use 'branchname" or "latest" for main. This results in a race condition where:

• Two consecutive jobs on a branch trigger. The second one changes the Dockerfile
• The first one doesn't get through the pipeline fully before the second one rebuilds + reuploads the image
• The remainder of job 1's pipeline now pick up the image not intended for that commit

2. build-docker-artifact tests for changes of certain files against main. Once a branch contains such a change, then all subsequent commits on that branch will flow down the path of 'generate a new docker image'
3. build-docker-artifact does the above test by using a gh api call
4. generate-docker-tag is invoked at the start of every job, so the job itself doesn't know the container to use until after it has begun executing steps