Force PIP not to upgrade any Debian-installed Python packages.

[gouttegd]

This PR amends the update-constraints.sh script to automatically extract from the filesystem the list of Python packages that have already been installed by Debian, and that PIP should not attempt to upgrade to any later version.

In addition to the normal constraints.txt file, it generates an additional pip-constraints.txt file, to be used to force PIP not to do anything with the system-managed packages.

This has the side-effect of ensuring that we never upgrade to any version of setuptools more recent than 72 (known to cause issues with some packages needed by the ODK), since the version provided by Debian is 68.1.2.

[matentzn]

I didnt review this line but for the rest it looks good!

If the build passes the first 1 minute, maybe its fine.

[gouttegd]

I didnt review this line but for the rest it looks good!

The /usr/lib/python3/dist-packages directory looks like this:

$ ls /usr/lib/python3/dist-packages
__pycache__		  gyp-0.1.egg-info    psycopg2			  six-1.16.0.egg-info
_distutils_hack		  pip		      psycopg2-2.9.9.egg-info	  six.py
distutils-precedence.pth  pip-24.0.dist-info  setuptools		  wheel
gyp			  pkg_resources       setuptools-68.1.2.egg-info  wheel-0.42.0.dist-info

where what is of interest for us are the *.egg-info or *.dist-info directories containing informations about the installed packages. That line simply transforms a pathname like /usr/lib/python3/dist-packages/setuptools-68.1.2.egg-info into setuptools==68.1.2.

[matentzn]

Thank you for the clarification!

[gouttegd]

Of note, with the approach used in this PR we may run into two problems in the future:

(A) A build may break if Debian/Ubuntu updates ones of the Python packages whose version is fixed in pip-constraints, because then the version that will actually be installed at the time the ODK is built will no longer be the same than the version that was present at the time the constraints were updated.

Presumably this should rarely happen, if ever. Once a Debian version has been released, the only package updates are normally security-related updates, and it’s probably unlikely that packages such as python3-setuptools or python3-wheels should be concerned.

If/when that happens, we will need to update the pip-constraints.txt manually so that it reflects the actual versions installed by the system after the package updates.

(B) We will run into a much bigger problem the next time we upgrade the base image (say, when we upgrade to Ubuntun 26.04), if by then the Ubuntu package for setup tools has been updated to >=72. We can only hope that by then, all the current packages that are broken by setuptools >= 72 will have gotten the memo that they need to fix their packages (something they should have done since… 2017).

[matentzn]

Sounds horrible, but maybe warrants a comment right there in the code as well?

[gouttegd]

For (A), I am thinking about updating again both the Dockerfile and the update-constraints.sh files to (1) reduce the likelihood of the problem occurring and (2) give us a clear error message if the problem does occur.

The idea is:

(1) Add a apt-get step in the update-constraints.sh file to (re-)install python3-dev and python3-pip3 in the image before attempting to calculate the constraints, so that, if the Ubuntu Python packages have been updated, we will get a pip-constraints.txt file that reflects the new version. Incidentally, this will also allow that script to be used to seed a completely new constraints.txt file when we upgrade the base system.

(2) In the Dockerfile, check the list of Ubuntu-installed Python packages, and if differs from the pip-constraints.txt list, display a warning that pip-constraints.txt is out-of-date.

As for (B), I don’t even want to think about it. When that happens (and it will happen eventually: if not with Ubuntu 26.04 then with Ubuntu 28.04), we will be screwed anyway. Most likely the only solution at the time will be to isolate the Python packages that are still broken with setuptools>=72 into a separate environment.