Add capability of iterating locally on constraint files for CI image (a…

…pache#30002)

Whenever you want to upgrade a set of dependencies (for example all
google packages) you might deal with a set of dependencies that
are conflicting or difficult to upgrade all-at-once. And there might
be multiple people working on it in parallel. This brings in the
necessity of building the CI image of breeze with a different set
of constraints than the ones in "main" branch of Airflow and share
it with others working with you.

This PR adds the capability of storing a temporary set of constraints
in the repository (during PR iteration) and use this constraint
file to build the CI image, rather than constraints from the
repository. It also provides appropriate README.md file explaining
how to iterate and regenerate such constraint files - even
multiple times when iterating.

.dockerignore

@@ -57,8 +57,8 @@
 
 # This folder is for you if you want to add any packages to the docker context when you build your own
 # docker image. most of other files and any new folder you add will be excluded by default
-# if you need other types of files - please add the extensions here.
 !docker-context-files
+!constraints
 
 # Avoid triggering context change on README change (new companies using Airflow)
 # So please do not uncomment this line ;)

Dockerfile.ci

@@ -1268,6 +1268,7 @@ COPY setup.py ${AIRFLOW_SOURCES}/setup.py
 COPY setup.cfg ${AIRFLOW_SOURCES}/setup.cfg
 COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/
 COPY generated/provider_dependencies.json ${AIRFLOW_SOURCES}/generated/
+COPY constraints/* ${AIRFLOW_SOURCES}/constraints/
 
 COPY --from=scripts install_airflow.sh /scripts/docker/
 

constraints/README.md

@@ -0,0 +1,55 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+ -->
+
+Sometimes you might want to update dependencies of airflow in bulk, when working on related set of
+packages/dependencies. This might lead to excessive time needed to rebuild the image with new versions
+of dependencies (due to `pip` backtracking) and necessity to rebuild the image multiple times
+and solving conflicting dependencies.
+
+For development use only, you can store your own version of constraint files used during CI image build
+here and refer to them during building of the image using `--airflow-constraints-location constraints/constraints.txt`
+This allows you to iterate on dependencies without having to run `--upgrade-to-newer-dependencies` flag continuously.
+
+Typical workflow in this case is:
+
+* download and copy the constraint file to the folder (for example via
+[The GitHub Raw Link](https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-3.7.txt)
+* modify the constraint file in "constraints" folder
+* build the image using this command
+
+```bash
+breeze ci-image build --python 3.7 --airflow-constraints-location constraints/constraints-3.7txt
+```
+
+You can continue iterating and updating the constraint file (and rebuilding the image)
+while iterating, you can also manually add/update the dependencies and after you are done,
+you can regenerate the set of constraints based on your currently installed packages, using this command:
+
+```bash
+pip freeze | sort | \
+    grep -v "apache_airflow" | \
+    grep -v "apache-airflow==" | \
+    grep -v "@" | \
+    grep -v "/opt/airflow" > /opt/airflow/constraints/constraints-3.7.txt
+```
+
+If you are working with others on updating the dependencies, you can also commit the constraint
+file to the repository and then anyone else working on the same branch/fork, can use it for local
+image building (but the file should be removed before the branch you are working on is merged to
+the main branch).

dev/breeze/src/airflow_breeze/commands/ci_image_commands.py

@@ -35,6 +35,7 @@
     option_additional_extras,
     option_additional_pip_install_flags,
     option_additional_python_deps,
+    option_airflow_constraints_location,
     option_airflow_constraints_mode_ci,
     option_airflow_constraints_reference_build,
     option_answer,
@@ -188,6 +189,7 @@ def start_building(params: BuildCiParams):
 @option_dev_apt_deps
 @option_force_build
 @option_python_image
+@option_airflow_constraints_location
 @option_airflow_constraints_mode_ci
 @option_airflow_constraints_reference_build
 @option_tag_as_latest

dev/breeze/src/airflow_breeze/commands/ci_image_commands_config.py

@@ -55,6 +55,7 @@
             "options": [
                 "--builder",
                 "--install-providers-from-sources",
+                "--airflow-constraints-location",
                 "--airflow-constraints-mode",
                 "--airflow-constraints-reference",
                 "--python-image",

dev/breeze/src/airflow_breeze/commands/production_image_commands.py

@@ -37,6 +37,7 @@
     option_additional_runtime_apt_command,
     option_additional_runtime_apt_deps,
     option_additional_runtime_apt_env,
+    option_airflow_constraints_location,
     option_airflow_constraints_mode_prod,
     option_airflow_constraints_reference_build,
     option_builder,
@@ -168,6 +169,7 @@ def prod_image():
 @option_prepare_buildx_cache
 @option_push
 @option_empty_image
+@option_airflow_constraints_location
 @option_airflow_constraints_mode_prod
 @click.option(
     "--installation-method",

dev/breeze/src/airflow_breeze/commands/production_image_commands_config.py

@@ -56,6 +56,7 @@
                 "--builder",
                 "--install-providers-from-sources",
                 "--airflow-extras",
+                "--airflow-constraints-location",
                 "--airflow-constraints-mode",
                 "--airflow-constraints-reference",
                 "--python-image",

dev/breeze/src/airflow_breeze/utils/common_options.py

@@ -474,6 +474,14 @@ def _set_default_from_parent(ctx: click.core.Context, option: click.core.Option,
     default=DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH,
     envvar="AIRFLOW_CONSTRAINTS_REFERENCE",
 )
+option_airflow_constraints_location = click.option(
+    "--airflow-constraints-location",
+    type=str,
+    default="",
+    help="If specified, it is used instead of calculating reference to the constraint file. "
+    "It could be full remote URL to the location file, or local file placed in `docker-context-files` "
+    "(in this case it has to start with /opt/airflow/docker-context-files).",
+)
 option_airflow_constraints_reference_build = click.option(
     "--airflow-constraints-reference",
     default=DEFAULT_AIRFLOW_CONSTRAINTS_BRANCH,

dev/breeze/src/airflow_breeze/utils/docker_command_utils.py

@@ -86,6 +86,7 @@
     ("NOTICE", "/opt/airflow/NOTICE"),
     ("RELEASE_NOTES.rst", "/opt/airflow/RELEASE_NOTES.rst"),
     ("airflow", "/opt/airflow/airflow"),
+    ("constraints", "/opt/airflow/constraints"),
     ("provider_packages", "/opt/airflow/provider_packages"),
     ("dags", "/opt/airflow/dags"),
     ("dev", "/opt/airflow/dev"),

docker-context-files/.README.md

@@ -21,12 +21,12 @@ This folder is part of the Docker context.
 
 Most of other folders in Airflow are not part of the context in order to make the context smaller.
 
-The Production [Dockerfile](../Dockerfile) and the CI one [Dockerfile.ci](../Dockerfile.ci) copies
-the [docker-context-files](.) folder to the image context - in case of production image it copies it to
-the build segment, co content of the folder is available in the `/docker-context-file` folder inside
-the build image. You can store constraint files and wheel
+The Production [Dockerfile](../Dockerfile) image copies the [docker-context-files](.) folder to the
+image context forthe build segment, so content of the folder is available in the `/docker-context-files`
+folder inside the build image. You can store constraint files and wheel
 packages there that you want to install as PYPI packages and refer to those packages using
-`--constraint-location` flag for constraints or by using `--install-packages-from-context` flag.
+`--airflow-constraints-location /docker-context-files/constraints.txt` flag for
+constraints or by using `--install-packages-from-context` flag.
 
 By default, the content of this folder is .gitignored so that any binaries and files you put here are only
 used for local builds and not committed to the repository.

images/breeze/output-commands-hash.txt

@@ -10,10 +10,10 @@ ci:get-workflow-info:01ee34c33ad62fa5dc33e0ac8773223f
 ci:resource-check:1d4fe47dff9fc64ac1648ec4beb2d85c
 ci:selective-check:3a085894f24cb909812fbc8253a21e13
 ci:9884bf4cef3e70270068de89b9d5df2f
-ci-image:build:f8d0fafe7246ca7ac412df8c520908a3
+ci-image:build:3ffe4dd24ae7090415543e27d8504955
 ci-image:pull:c16c6e57c748bfe9b365b4ffafb18472
 ci-image:verify:aee88f55e8837028d19316356e29b009
-ci-image:0c727b84a764e09c54e461b30ad0410b
+ci-image:d53ee5067c46a6d8f3b21d0098c56ca2
 cleanup:231de69d5f47ba29c883164e4575e310
 compile-www-assets:c8a8c4f002f7246d0541897fc7c70313
 exec:42bbd3c1659128b0341ae118c3482da2
@@ -31,10 +31,10 @@ k8s:status:2b8aa0b9597a1790c184c7b776ebd9c3
 k8s:tests:dc44e907de908d1c4f11cf5d99915da1
 k8s:upload-k8s-image:b012457bb6a98524859ecaf5746b4439
 k8s:a04e7db8fb49c16a0633ac32f14463ec
-prod-image:build:b2c87adc30320ea70d1b32faf1500959
+prod-image:build:5d0fcf0f0b1f4a8d13378253214162d6
 prod-image:pull:e3c89dd908fc44adf6e159c2950ebdd0
 prod-image:verify:31bc5efada1d70a0a31990025db1a093
-prod-image:eb1ef0cf6e139d01ceb26f09ca3deaaa
+prod-image:2ac8e1e6b4afb259f00390ad0e90a06f
 release-management:create-minor-branch:6a01066dce15e09fb269a8385626657c
 release-management:generate-constraints:ae30d6ad49a1b2c15b61cb29080fd957
 release-management:generate-issue-content-providers:f3c00ba74e3afc054fe29b65156740ac