Use pyinstaller

[pdelboca]

• fixes pip timeout during initialisation in a corporate/public sector environment with HTTP proxy #359

---

This PR refactors the application to use PyInstaller to handle the backend FastAPI server when building the application.

Main Changes:

• Add PyInstaller as a dependency
• Remove old logic to handle Python Virtual Environments
• Simplify runServer logic to only one function and one module.
• Updates Python to 3.11 (required for PyInstaller)
• Refactor relative imports from FastAPI server (required for PyInstaller)

TODO:

• With this changes, the loading.html splash screen do not work properly since we need to migrate the logic of waiting to the server from the main application to the launch script. This will be better to do in Improve the launch & onboarding experience for first time, and subsequent users #445 .

[cloudflare-workers-and-pages]

Deploying opendataeditor with    Cloudflare Pages

Latest commit:	68a7438
Status:	✅  Deploy successful!
Preview URL:	https://5403905b.opendataeditor.pages.dev
Branch Preview URL:	https://use-pyinstaller.opendataeditor.pages.dev

View logs

[guergana]

Test OS: Ubuntu

Test case 1: installing the Ubuntu artifacts from this PR at https://github.com/okfn/opendataeditor/actions/runs/9710357555

Hello, @pdelboca I am testing this in Ubuntu with the file generated in the distribution files in the PR, and while I dont get any errors when opening the app after installation without access to the Internet, I can't open any files in the app:

This is what i see, the files/folders I try to add are not displayed either on the left menu or the right content panel.

Test case 2: Making a build locally

I can't make a local build because I am missing some dependencies. I have run make install and afterwards make build and got these errors:

opendataeditorgutz@C306:~/Desktop/Things/opendataeditor/opendataeditor$ make build
hatch run build
[assets] Copying 'desktop/assets' to 'build'
[server] Creating executable file for FastAPI server
246 WARNING: Failed to collect submodules for 'frictionless.vendors.wkt' because importing 'frictionless.vendors.wkt' raised: ModuleNotFoundError: No module named 'grako'
46563 WARNING: Hidden import "pysqlite2" not found!
46564 WARNING: Hidden import "MySQLdb" not found!
Traceback (most recent call last):
  File "/home/gutz/.local/share/hatch/env/virtual/.pythons/3.10/python/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,

I have googled the issue and I have found a couple links like this one and this one suggesting to change pysqlite2 to sqlite3

to remove the Hidden import "MySQLdb" not found! warning I had to go through the following steps

• sudo apt-get -y install clang
• Then followed the instructions to build MySQLdb from here: https://github.com/PyMySQL/mysqlclient/blob/main/README.md#linux
  I couldn't finish this step because of an error with the clang command: clang error unknown target cpu 'x86-64-v3'. I found an explanation to the issue but I couldn't make it work because the version of clang that gets installed is version 10, and to install clang-12 or clang-13, one needs to install it as a separate package and it's still not picked up by the other installer that expects the package to be clang.

So I also failed to build it manually for my particular cpu version.

Test OS: Windows

Version from artifacts

The version from the PR artifacts installs well in Windows 10 inside VirtualBox and works fine.

[guergana]

Great work. It works on Windows 10 for me but not in Ubuntu. I left a comment explaining my findings.

[guergana]

Given that the results for Windows and Ubuntu were very different, we should test this as well natively in MacOS, can we organize a call for this @romicolman ? 🙏 (We test it together and we make some comments for Patricio based in our findings)

[roll]

Amazing! It works for me on Ubuntu 🎉

I confirm that something weird is happing for the project selector (it was select and select for me as well initially but then it showed folders)

Also can be reported here - frictionlessdata/frictionless-py#1600. Interestingly, what have happened to pyinstaller that fixes the setup compared to the year ago. The setup is pretty simple so I guess I was doing the same but it had a lot of dependency problems then

PS.
@guergana
Probably we might not need to fix these warnings. It seems to be working anyway, as I have them as well

[pdelboca]

@guergana I see that you are using Python 3.10. This PR migrates to Python 3.11 since it became a dependency for PyInstaller (it didn't worked for me on Python 3.10).

Could you update your virtualenv to use 3.11 and try building the application again?

@guergana and @roll the weird behaviour of showing select and then the project is due that we are starting the FastAPI process in the desktop module and then launching the application without waiting for the FastAPI server to finish loading. This is mentioned in #454 , I didn't wanted to move the logic of pooling the server in this PR to not polute it, but I can do it and then reduce the scope of #454 .

[pdelboca]

@guergana regarding your Test Case 1. Is there any log in your machine that could help me debug what's going on?

[romicolman]

Given that the results for Windows and Ubuntu were very different, we should test this as well natively in MacOS, can we organize a call for this @romicolman ? 🙏 (We test it together and we make some comments for Patricio based in our findings)

Yes, I have just messaged you to coordinate so I can test it

[guergana]

@guergana regarding your Test Case 1. Is there any log in your machine that could help me debug what's going on?

Hello @pdelboca I tested again with the new changes and at first the app wouldn't install at all. Then I upgraded my local version to Python 3.11 and the app was installed. I think this is less than ideal, because there is no notification at all that this is the reason why the installation is failing. This probably has to more with my ubuntu file installer than with the app itself perhaps?

After upgrading to python 3.11 I could open the app but I still see the select select. I tried importing a file and opening a new project and I got the file prompt but the left menu didn't load any files and had a loader running for some very long minutes (5 to10) and then I got a message saying that the app couldn't connect to the server and the app crashed.

I don't see any logs from the second run I did, but from the first run I got these logs only.

[2024-07-13 16:31:50.664] [info]  # Start application
[2024-07-13 16:31:50.670] [info]  ## Start client
[2024-07-13 16:31:50.737] [info]  Opening loading.html
[2024-07-13 16:31:50.740] [info]  ## Start server
[2024-07-13 16:31:50.741] [info]  [FastAPI Server] Running backend server on port 4040
[2024-07-13 16:31:50.748] [info]  Opening index.html
[2024-07-13 16:31:50.758] [error] [FastAPI Server stderr] [10447] Failed to load Python shared library '/opt/Open Data Editor/resources/server/__main__/_internal/libpython3.11.so.1.0': dlopen: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /opt/Open Data Editor/resources/server/__main__/_internal/libpython3.11.so.1.0)
[2024-07-13 16:31:50.759] [info]  [FastAPI Server] { message: 'child process exited with code 255' }

I almost broke my Ubuntu installation by replacing the Python version. The base Python version in my system should be 3.8.

I can imagine other users having the same issue I am having... Can we somehow forsee what will be the amount of users of the ODE using Ubuntu?

[pdelboca]

@guergana thanks for the detailed input. Let's clarify some points.

We have 2 scenarios mixing up: developer experience and user experience.

User experience

We need to test whether the distribution files that our pipelines create can be installed in Ubuntu and if the application works. (Same for Windows and Mac)

This scenario is usually tested with the following actions:

1. Clean up any Open Data Editor app installed in your machine
2. Download the distribution files file created in our Github Actions.
3. Install it in your machine (Either with a Software Manager or running dpkg -i opendateditor-linux-1.0.0.deb)
4. ODE should be installed correctly
5. ODE should run correctly

This scenario tests that the desktop application can be installed and it words. ODE is a Desktop application so it shouldn't be affected by the local Python versions.

Developer experience

This is what happens when we, as a developers, build and run the application in our machines.

This scenario usually relates to our day to day work:

1. Checkout the repository/branch
2. make install
3. make build
4. make dist
5. Install the *.deb file generated by the make dist command
6. Run the application.

User vs Developers.

So back to your questions:

Hello @pdelboca I tested again with the new changes and at first the app wouldn't install at all. Then I upgraded my local version to Python 3.11 and the app was installed. I think this is less than ideal, because there is no notification at all that this is the reason why the installation is failing. This probably has to more with my ubuntu file installer than with the app itself perhaps?

I should have been more explicit in the description of the PR. To be able to use Pyinstaller we need to move to Python 3.11. Ideally hatch should take care of this under the hood, with the extra requirement that @roll mentioned of having to install python3.11-dev.

I almost broke my Ubuntu installation by replacing the Python version. The base Python version in my system should be 3.8.

There shouldn't be needed to update the base Python Distribution of your Ubuntu for this. Ubuntu can have several python's installed (I have 3.9, 3.10 and 3.11 in my machine). It is true that there is no notification why the installation is failing, but this PR includes changes in the documentation and the config files to make it explicity that Python 3.11 is now required to develop.

I can imagine other users having the same issue I am having... Can we somehow forsee what will be the amount of users of the ODE using Ubuntu?

As mentioned before, we should distinguish end users with developers. End Users shouldn't care about the Ubuntu Version or the Python version since ODE should install all the dependencies required as any normal Desktop application. If installing the *.deb package does not work in Ubuntu then there is something wrong in our build process.

Let's have a call to clarify further this scenarios. But in a nutshell I have one follow-up questions for you: Does the error your are discribing happens with the application you build (developer experience) or with the distribution file you downloaded (end user experience)?

[guergana]

@guergana thanks for the detailed input. Let's clarify some points.

We have 2 scenarios mixing up: developer experience and user experience.

User experience

We need to test whether the distribution files that our pipelines create can be installed in Ubuntu and if the application works. (Same for Windows and Mac)

This scenario is usually tested with the following actions:

1. Clean up any Open Data Editor app installed in your machine
2. Download the distribution files file created in our Github Actions.
3. Install it in your machine (Either with a Software Manager or running dpkg -i opendateditor-linux-1.0.0.deb)
4. ODE should be installed correctly
5. ODE should run correctly

This scenario tests that the desktop application can be installed and it words. ODE is a Desktop application so it shouldn't be affected by the local Python versions.

Developer experience

This is what happens when we, as a developers, build and run the application in our machines.

This scenario usually relates to our day to day work:

1. Checkout the repository/branch
2. make install
3. make build
4. make dist
5. Install the *.deb file generated by the make dist command
6. Run the application.

User vs Developers.

So back to your questions:

Hello @pdelboca I tested again with the new changes and at first the app wouldn't install at all. Then I upgraded my local version to Python 3.11 and the app was installed. I think this is less than ideal, because there is no notification at all that this is the reason why the installation is failing. This probably has to more with my ubuntu file installer than with the app itself perhaps?

I should have been more explicit in the description of the PR. To be able to use Pyinstaller we need to move to Python 3.11. Ideally hatch should take care of this under the hood, with the extra requirement that @roll mentioned of having to install python3.11-dev.

I almost broke my Ubuntu installation by replacing the Python version. The base Python version in my system should be 3.8.

There shouldn't be needed to update the base Python Distribution of your Ubuntu for this. Ubuntu can have several python's installed (I have 3.9, 3.10 and 3.11 in my machine). It is true that there is no notification why the installation is failing, but this PR includes changes in the documentation and the config files to make it explicity that Python 3.11 is now required to develop.

I can imagine other users having the same issue I am having... Can we somehow forsee what will be the amount of users of the ODE using Ubuntu?

As mentioned before, we should distinguish end users with developers. End Users shouldn't care about the Ubuntu Version or the Python version since ODE should install all the dependencies required as any normal Desktop application. If installing the *.deb package does not work in Ubuntu then there is something wrong in our build process.

Let's have a call to clarify further this scenarios. But in a nutshell I have one follow-up questions for you: Does the error your are discribing happens with the application you build (developer experience) or with the distribution file you downloaded (end user experience)?

@pdelboca Ah, sorry for not being clear. I installed the *.deb distribution file generated as an artifact in GitHub. I did it with the end user experience approach and I updated my python and managed to install the app after changing the python version.

[pdelboca]

@guergana I installed the application in a new VM running Ubuntu 20.04 and I was able to reproduce the error.

I was reading a little bit about it and I got to this section of pyinstaller documentation: https://pyinstaller.org/en/stable/usage.html?highlight=glibc#making-gnu-linux-apps-forward-compatible

Which suggest to always build the application in the oldest version of the OS we want to support:

The solution is to always build your app on the oldest version of GNU/Linux you mean to support. It should continue to work with the libc found on newer versions.

I will try to build our application using Ubuntu 20.04 to see if it fixes the error.

[guergana]

@guergana I installed the application in a new VM running Ubuntu 20.04 and I was able to reproduce the error.

I was reading a little bit about it and I got to this section of pyinstaller documentation: https://pyinstaller.org/en/stable/usage.html?highlight=glibc#making-gnu-linux-apps-forward-compatible

Which suggest to always build the application in the oldest version of the OS we want to support:

The solution is to always build your app on the oldest version of GNU/Linux you mean to support. It should continue to work with the libc found on newer versions.

I will try to build our application using Ubuntu 20.04 to see if it fixes the error.

@pdelboca It works!!!! amazing find! 💯 i see here that version 20.04 was released on 2020 so I guess that's enough support for now? Maybe we can mention that this is the minimum version we support from ubuntu somewhere in the docs as well. My local version is 20.04.06 and the installation file generated as an artifact works like a charm ✨

For the record, I tested again with the .deb file generated with the PR I didn't build the file locally.

[guergana]

great work @pdelboca !

[roll]

🎉