Introduce dependency test suite

The plan is to build out this test suite to test against the AWS
CLI's dependencies to help facilitate dependency upgrades.

To start, this test suite contains the following new test cases
to better monitor the overall dependency closure of the awscli
package:

* Assert expected packages in runtime closure. This will alert
  us if a dependency introduces a new transitive depenency
  to the AWS CLI closure.

* Assert expected unbounded dependencies in runtime closure.
  Specifically these are dependencies that do not have a version
  ceiling. This will alert us if a new unbounded dependency is
  introduced into the AWS CLI runtime dependency closure.

* Assert expected packages used in building the awscli package
  and any of its runtime dependencies. Specifically, this
  targets the use case of a user installing the awscli
  (and all of its dependencies) from sdist as it will capture
  which build packages are pulled into the isolated virtual
  environment when building the package.

* Assert expected unbounded dependencies in building the awscli
  package and any of its runtime dependencies. This helps monitor
  out of all of the build packages used, which ones may have no
  version ceiling when installed into an isolated virutal environment

See additional implementation notes below:

* These tests were broken into a separate test suite (i.e. instead
  of adding them to the unit and functional test suite) to allow
  more granularity when running them. Specifically, it is useful for:

  1. Avoiding the main unit and functional CI test suite from failing
     if a dependency changes from underneath of us (e.g. a new build
     dependency is added that we cannot control).

  2. For individuals that package the awscli, they generally will not
     want to run this test suite as it is fairly specific to how pip
     installs dependencies.

  3. The tests currently run between 3-5 minutes, which would significantly
     slow down the unit/functional test runs. Having it in a seperate directory
     allows you to only run the dependency test suite if you are making a change
     in dependencies.

* To determine the runtime dependency closure, the Package and DependencyClosure
  utilities traverse the dist-info METADATA files of the packages installed in
  the current site packages to build the runtime graph. This approach was chosen
  because:

  1. Since pip already installed the package, this logic avoids having to
     reconstruct the logic of how pip decides to resolve dependencies to figure
     out how to traverse the runtime graph. Any custom logic may deviate from
     how pip behaves which is what most users will be using to install the awscli
     as a Python package
  2. It's faster. The runtime closure test cases do not require downloading or
     installing any additional packages.

* To determine the build dependency closure of each package in the awscli runtime
  closure, the Package and DependencyClosure utilities leverage the "build" package
  to invoke PEP517 hooks to collect which packages are needed to build both the sdist
  and wheel. Relying on PEP517 hooks is really the only standard that can be relied on
  to derive the build dependencies as they can be dynamically generated as part of
  invoking a build backend. The "build" package was chosen to invoke the PEP517 hooks
  because:

  1. It has helpful higher level utilities to capture the results from the hooks if
     provided a virtualenv and project path. This would have been a significant amount
     of scaffolding to produce our own PEP517 hook invoker utilities.
  2. It has compatibility logic on how to handle projects that do not include a
     pyproject.toml and automatically can invoke the correct build backend (e.g.
     falls back to the legacy setuptools backend when needed) when a build backend
     is not declared.

.github/workflows/run-dep-tests.yml

@@ -0,0 +1,27 @@
+name: Run dependency tests
+
+on:
+  push:
+  pull_request:
+    branches-ignore: [ master ]
+
+jobs:
+  build:
+
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        os: [ubuntu-latest, macOS-latest, windows-latest]
+
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: python scripts/ci/install
+    - name: Run tests
+      run: python scripts/ci/run-dep-tests

requirements-dev-lock.txt

@@ -7,6 +7,10 @@
 atomicwrites==1.4.1 \
     --hash=sha256:81b2c9071a49367a7f770170e5eec8cb66567cfbbc8c73d20ce5ca4a8d71cf11
     # via -r requirements-dev.txt
+build==1.0.3 \
+    --hash=sha256:538aab1b64f9828977f84bc63ae570b060a8ed1be419e7870b8b4fc5e6ea553b \
+    --hash=sha256:589bf99a67df7c9cf07ec0ac0e5e2ea5d4b37ac63301c4986d1acb126aa83f8f
+    # via -r requirements-dev.txt
 colorama==0.4.4 \
     --hash=sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b \
     --hash=sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2
@@ -79,22 +83,29 @@ exceptiongroup==1.1.3 \
     --hash=sha256:097acd85d473d75af5bb98e41b61ff7fe35efe6675e4f9370ec6ec5126d160e9 \
     --hash=sha256:343280667a4585d195ca1cf9cef84a4e178c4b6cf2274caef9859782b567d5e3
     # via pytest
+importlib-metadata==7.0.1 \
+    --hash=sha256:4805911c3a4ec7c3966410053e9ec6a1fecd629117df5adee56dfc9432a1081e \
+    --hash=sha256:f238736bb06590ae52ac1fab06a3a9ef1d8dce2b7a35b5ab329371d6c8f5d2cc
+    # via build
 iniconfig==1.1.1 \
     --hash=sha256:011e24c64b7f47f6ebd835bb12a743f2fbe9a26d4cecaa7f53bc4f35ee9da8b3 \
     --hash=sha256:bc3af051d7d14b2ee5ef9969666def0cd1a000e121eaea580d4a313df4b37f32
     # via pytest
-packaging==21.3 \
-    --hash=sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb \
-    --hash=sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522
-    # via pytest
+packaging==23.2 \
+    --hash=sha256:048fb0e9405036518eaaf48a55953c750c11e1a1b68e0dd1a9d62ed0c092cfc5 \
+    --hash=sha256:8c491190033a9af7e1d931d0b5dacc2ef47509b34dd0de67ed209b5203fc88c7
+    # via
+    #   -r requirements-dev.txt
+    #   build
+    #   pytest
 pluggy==1.0.0 \
     --hash=sha256:4224373bacce55f955a878bf9cfa763c1e360858e330072059e10bad68531159 \
     --hash=sha256:74134bbf457f031a36d68416e1509f34bd5ccc019f0bcc952c7b909d06b37bd3
     # via pytest
-pyparsing==3.0.9 \
-    --hash=sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb \
-    --hash=sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc
-    # via packaging
+pyproject-hooks==1.0.0 \
+    --hash=sha256:283c11acd6b928d2f6a7c73fa0d01cb2bdc5f07c57a2eeb6e83d5e56b97976f8 \
+    --hash=sha256:f271b298b97f5955d53fb12b72c1fb1948c22c1a6b70b315c54cedaca0264ef5
+    # via build
 pytest==7.4.0 \
     --hash=sha256:78bf16451a2eb8c7a2ea98e32dc119fd2aa758f1d5d66dbf0a59d69a3969df32 \
     --hash=sha256:b4bf8c45bd59934ed84001ad51e11b4ee40d40a1229d2c79f9c592b0a3f6bd8a
@@ -109,9 +120,15 @@ tomli==2.0.1 \
     --hash=sha256:939de3e7a6161af0c887ef91b7d41a53e7c5a1ca976325f429cb46ea9bc30ecc \
     --hash=sha256:de526c12914f0c550d15924c62d72abc48d6fe7364aa87328337a31007fe8a4f
     # via
+    #   build
     #   coverage
+    #   pyproject-hooks
     #   pytest
 wheel==0.38.1 \
     --hash=sha256:7a95f9a8dc0924ef318bd55b616112c70903192f524d120acc614f59547a9e1f \
     --hash=sha256:ea041edf63f4ccba53ad6e035427997b3bb10ee88a4cd014ae82aeb9eea77bb9
     # via -r requirements-dev.txt
+zipp==3.17.0 \
+    --hash=sha256:0e923e726174922dce09c53c59ad483ff7bbb8e572e00c7f7c46b88556409f31 \
+    --hash=sha256:84e64a1c28cf7e91ed2078bb8cc8c259cb19b76942096c8d7b84947690cabaf0
+    # via importlib-metadata

requirements-dev.txt

@@ -7,3 +7,7 @@ pytest==7.4.0
 pytest-cov==4.1.0
 atomicwrites>=1.0 # Windows requirement
 colorama>0.3.0 # Windows requirement
+
+# Dependency test specific deps
+packaging==23.2
+build==1.0.3

scripts/ci/run-dep-tests

@@ -0,0 +1,35 @@
+#!/usr/bin/env python
+# Don't run tests from the root repo dir.
+# We want to ensure we're importing from the installed
+# binary package not from the CWD.
+
+import os
+import sys
+from contextlib import contextmanager
+from subprocess import check_call
+
+_dname = os.path.dirname
+
+REPO_ROOT = _dname(_dname(_dname(os.path.abspath(__file__))))
+
+
+@contextmanager
+def cd(path):
+    """Change directory while inside context manager."""
+    cwd = os.getcwd()
+    try:
+        os.chdir(path)
+        yield
+    finally:
+        os.chdir(cwd)
+
+
+def run(command):
+    env = os.environ.copy()
+    env['TESTS_REMOVE_REPO_ROOT_FROM_PATH'] = 'true'
+    return check_call(command, shell=True, env=env)
+
+
+if __name__ == "__main__":
+    with cd(os.path.join(REPO_ROOT, "tests")):
+        run(f"{sys.executable} {REPO_ROOT}/scripts/ci/run-tests dependencies")

tests/dependencies/__init__.py

@@ -0,0 +1,12 @@
+# Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+#     http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.