Firedrake install problem with Python 3.10

[andrewlhicks]

Hello, I am currently trying to install Firedrake on my Windows machine using WSL (this is a different machine from the other problem I posted earlier this week). I installed Python 3.10 and used that to run the firedrake-install script, and everything goes fine until I get to the Coffee installation part. I did take a look at another GitHub discussion where pip was broken (here) and I followed those instructions to fix pip. However, Coffee won't install even with this fix. It gives me:

Installing pip dependencies for COFFEE
Traceback (most recent call last):
  File "/home/andrewh/firedrake-install", line 1804, in <module>
    pip_requirements(p, compiler_env)
  File "/home/andrewh/firedrake-install", line 937, in pip_requirements
    run_pip_install(["--no-deps", "-r", "%s/requirements.txt" % package])
  File "/home/andrewh/firedrake-install", line 906, in run_pip_install
    check_call(pipinstall + pipargs)
  File "/home/andrewh/firedrake-install", line 592, in check_call
    log.debug(subprocess.check_output(arguments, stderr=subprocess.STDOUT, env=os.environ).decode())
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/andrewh/firedrake/bin/python', '-m', 'pip', 'install', '--no-build-isolation', '--no-binary', 'mpi4py,randomgen,islpy,numpy', '-vvv', '--no-deps', '-r', 'COFFEE/requirements.txt']' returned non-zero exit status 1.

I'll attach the logs here:
firedrake-install.log
configure.log

[JDBetteridge]

SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
error: subprocess-exited-with-error

Seems like a strange numpy build error, could you try building numpy in the firedrake venv by running pip install --no-binary numpy numpy. If that doesn't error we should check that it is built against the right BLAS libraries beofre running firedrake-update to complete installation.

[andrewlhicks]

This too seems to not do anything, as it says that python-dev was already installed. So the pip command fails yet again.

[ksagiyam]

You probably need sudo apt install python3.10-dev.

[andrewlhicks]

Alright, I ended up doing sudo apt install python3.10-dev, however this was only able to be completed after fixing the ModuleNotFoundError: No module named 'apt_pkg' bug that I had mentioned above. The solution to that bug as well as the explanation for why it may have appeared can be found at this link. This, however, is largely a tangent from the main goal here.

Getting back to the main problem: After successfully running sudo apt-get install python3.10-dev, I finally was able to run pip install --no-binary numpy with no issues now. How would we go about checking that it is built against the right BLAS libraries before running firedrake-update to complete the installation?

[JDBetteridge]

Run ldd $VIRTUAL_ENV/lib/python3.10/site-packages/numpy/linalg/lapack_lite.*.so and if the BLAS distribution matches the one you want to use (probably /usr/lib/libopenblas.so on Ubuntu) then numpy is installed and ready. If you want to use another BLAS follow the instructions here, but note that this must match what you passed to firedrake-install originally.

You can complete your firedrake install by running firedrake-update inside the Firedrake virtual environment.

[andrewlhicks]

The BLAS distribution seemed to indicate OpenBLAS and everything seemed normal so I decided to go ahead and run firedrake-update. The installation completed correctly and with no issues that I am aware of. I will post a summary of how this issue was resolved. The one thing that remains mysterious to me is how this got broken to the point where I had to repair it manually.

[andrewlhicks]

After going back and forth with @JDBetteridge and @ksagiyam, I came up with the following solution:

First, I found that trying to manually install numpy via pip install --no-binary numpy gave me the same error as my original problem, so it was likely that the issue was coming from the installation of numpy. I ran sudo apt-get install python3.10-dev to try to fix the problem. This gave me an error, so I ran sudo apt-get update which gave me the same exact error, which is:

Traceback (most recent call last):
  File "/usr/lib/command-not-found", line 28, in <module>
    from CommandNotFound import CommandNotFound
  File "/usr/lib/python3/dist-packages/CommandNotFound/CommandNotFound.py", line 19, in <module>
    from CommandNotFound.db.db import SqliteDatabase
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/db.py", line 5, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

After searching the web and coming across this page, I checked and I realized that the directory /usr/lib/python3/dist-packages was missing the file apt_pkg.so, so I simply ran

cd /usr/lib/python3/dist-packages
sudo cp apt_pkg.cpython-36m-x86_64-linux-gnu.so apt_pkg.so

and this allowed me to successfully run sudo apt-get update.

Next, I ran sudo apt-get install python3.10-dev and this now worked successfully. Then I once again ran pip install --no-binary numpy numpy and this time it completed without error. After this, I ran ldd $VIRTUAL_ENV/lib/python3.10/site-packages/numpy/linalg/lapack_lite.*.so and saw that OpenBLAS was indeed being used. Finally, it was at this point that I felt comfortable to run firedrake-update and complete the installation of Firedrake, which was successful and posed no further issues.

In summary: my Python 3 installation was somehow missing the file /usr/lib/python3/dist-packages/apt_pkg.so which caused sudo apt-get update and sudo apt-get install python3.10-dev to fail. But after replacing this file and rerunning sudo apt-get install python3.10-dev then reinstalling numpy, I was able to use firedrake-update to complete the installation of Firedrake successfully.