Move dependencies needed in CI on prod images to separate requirements file.

[KevinMind]

Fixes: mozilla/addons#15191
Fixes: mozilla/addons#15066

Description

This PR solves several problems. In a nutshell the changes here enable independent control of the following features when running addons-server:

1. which DOCKER_TARGET to use during runtime as opposed to at build time
2. installing the CI relevant dependencies used for linting/testing
3. mounting the local host files into the web container

With these changes, a user can decide if they want to use a local or pulled image, and separately decide which set of dependencies should be installed and whether to mount their local source files to the container.

These changes, in combination, enable several important workflows, for example:

• running the production image without mounted source files (real prod mode) (via 1 and 3)
• running tests on the production image (via 1 and 2)
• making the development image build really fast (via 1 and 2)

Additionally this PR:

• mounts the /deps directory to the host, re-enabling tracking of dependency files via an IDE
• makes locales available in dev mode on demand
• collapses the web/worker containers to remove race conditions during startup
• forces the build-docker action to build the production image (ensuring all pushed images are prod images)

This PR adds a lot of CI testing to ensure this feature set is maintained for the long term. By running the container in a variety of different setups we can validate the expected behavior of a given environment and ensure any code changes don't break these requirements in the future.

Context

Some examples of how you could use make up with these changes.

1. Running a near perfect production environment, but still able to run tests

make up DOCKER_VERSION=latest DOCKER_TARGET=production INCLUDE_CI_DEPS=True

This image pulls the latest tag (literally production image) and then installs the CI dependencies in the ./deps mount, allowing us to run pytest and linting on a production image (that doesn't have these dependencies natively)

2. Run a development version of a production image (built in CI)

This pulls a specific digest (say from a PR CI) and let's the user run a development version that will install all the dev dependencies for debuging and additionally mount local source files enabling modifications to the source code to debug an issue in CI.

make up DOCKER_DIGEST=sha256@sdflkjsdflkjsdf DOCKER_TARGET=development

3. Run a really fast dev image

make up DOCKER_VERSION=local DOCKER_TARGET=development

This is the default action of make up and will build the image locally to the development target. The development stage includes almost nothing but the base os level dependencies and environment variables needed to build the rest of the image. This makes the build lightning fast (around 1 second)

Make up will then install the dependencies from a cached bind mount (also really fast) and will include locales and assets on demand (more fast, more good)

Running this should be relatively fast as it removes most of the time intensive tasks from the docker build and let's the containers do as much work asynchronously as possible.

Testing

TODO:

• test generating locales at runtime
• test data load/dump
• test make up CLEAN
• test docker_compose_run
• test COMPOSE_FILE
• test failing web/worker fails make up

Verify make up fails when web doesn't start.

• Modify (one at a time) the web and worker items in the Procfile
• run make up
• Expect it to fail as web is unhealthy

Verify site-static files are available in "prod" mode only

• run make up DOCKER_TARGET=development
• in a web shell expect /data/olympia/site-static to be empty

Verify CI dependencies available only with flag set

• run make up INSTALL_CI_DEPS=True
• in a web shell run ./scripts/check_pip_packages.sh ci
• If you set the flag to false, it should fail with missing dependencies

First create an env in prod mode

make up DOCKER_TARGET=production DOCKER_VERSION=local

Expect no ci or dev dependencies.

now run:

make update_deps_production INSTALL_CI_DEPS=true

Now expect production and ci deps to be installed

FInally install dev deps

make update_deps_production INSTALL_CI_DEPS=true

You can trigger these deps to be installed automatically if you set the argument on make up

make up DOCKER_TARGET=production INCLUDE_CI_DEPS=true

This will install prod and ci deps in the container.

Checklist

• Add #ISSUENUM at the top of your PR to an existing open issue in the mozilla/addons repository.
• Successfully verified the change locally.
• The change is covered by automated tests, or otherwise indicated why doing so is unnecessary/impossible.
• Add before and after screenshots (Only for changes that impact the UI).
• Add or update relevant docs reflecting the changes made.

[KevinMind]

Closed in favor of #22911