Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for PyQt6 (and PySide6) #575

Merged
merged 25 commits into from
Aug 15, 2023
Merged

Support for PyQt6 (and PySide6) #575

merged 25 commits into from
Aug 15, 2023

Conversation

karlch
Copy link
Owner

@karlch karlch commented Jan 8, 2023

Initial attempt at supporting PyQt6 and potentially PySide6 by writing a Qt wrapper module vimiv.qt which must be used instead of importing from PyQt* / PySide6 directly.

This went surprisingly smooth, but still has a few rough edges, and some are breaking before being able to merge:

PyQt6:

  • Update documentation
  • Fix pylint no-member errors
  • Fix mypy

PySide6 additionally:

  • Fix end2end tests and add to CI

I also noticed performance issues upon startup with PyQt6 on my local laptop. These do not seem to affect the CI speed, and are also not relevant after startup, which puzzles me. Namely, the instantiation of any QWidget seems to be really slow, __init__ of the main window takes several hundred ms instead of less than 30 with another wrapper.

If anyone, @jcjgraf 😉, would like to test this, that would be super helpful.

Going forward, I believe it makes sense to merge this after releasing v0.9.0, but explicitly calling support for PyQt6 and especially PySide6 experimental. If we don't merge soon-ish, there will continue to be a bunch of merge conflicts for obvious reasons. This would probably mean we need a v0.10.0 with this in, before going to v1.0. However, we could still release both of these versions with PySide6 if issues are too large, but PySide6 is the "official" Qt for python now, and the performance issues are not there for me.

fixes #318

@karlch karlch marked this pull request as draft January 8, 2023 13:33
@jcjgraf
Copy link
Contributor

jcjgraf commented Jan 8, 2023

Wow, you are amazing. I hope there is some kind of tool that helped you with the transition and that you did you have to to everything by hand 🙈

My general thoughts on the transition to PyQt6 (I have not yet had a close look at your code and the changes); Is there a strong necessity for supporting both version at the same time? I.e. are there some distros out there that do not support PyQt6 yet? Or are there some unforeseen issues we may get with PyQt6? Or do we want to maintain PyQt5 support only for people who do not yet want to update for some personal reason? While having a smooth transition with a transition period from PyQt5 to PyQt6 would be favorable, I am thinking about the economy of this approach. If it induces a multiple-hours overhead, compared to directly updating, I would argue that, having the limited manpower we have, it is not really feasible.

Concerning the performance issues; What kind of profiling tool did you use?

I will certainly look into it, but not right now... First I would like to get #467 done 😊

@karlch
Copy link
Owner Author

karlch commented Jan 8, 2023

This has been done before, e.g. by qutebrowser, so I didn't have to do too much thinking 😂

I do think it makes sense to have the wrapper module in any case, no matter what we end up supporting, as we can e.g. also use it for PyQt6 vs. PySide6 and it really helps with experimenting. And, given most of it just works, I would leave it in until PyQt5 is basically dead. It probably makes sense to drop support for PyQt < 5.15 rather soon though, as those have some extra issues that require manual workarounds.

The super-fancy vimiv.utils.debug module with the timed decorator / profile context manager 😄 nothing crazy, but as the difference was > 10x, and noticeable by feeling, I don't think the details really matter.

Thanks, and no rush, the other open PRs are definitely more important atm 😊

karlch added 18 commits July 5, 2023 18:21
In addition to PyQt5, the qt wrapper now also allows importing from
PyQt6. The user can pick which PyQt version to use using an environment
variable. Currently PyQt6 is still completely broken.
Tests run, application seems to run fine, issues with Svg known.
We can use type(QObject) to get its metaclass instead.
We still had some hard-coded lazy imports to PyQt5, which, obviously,
don't mix well with PyQt6.
end2end tests unfortunately still broken, but we do get a startup. Due
to the broken tests, unsure what else is still not working.
Fuzzy completion does not work with QRegularExpression for PyQt < 5.15.
Still a lot of complaints due to no-member which need to be sorted out.
karlch added 7 commits July 5, 2023 19:01
* Remove hard-coded PyQt5 imports
* Adapt flags
Catches and fixes a few more issues :)
* Allow # type: ignore lines to be long.
* Fix import order and unused import.
* Do not use pylint for member check of PyQt / PySide objects. The error
  is still caught by mypy.
There are still multiple unsolved issues. We leave it in explicitly
though, so further testing and integration remains simple.
@karlch
Copy link
Owner Author

karlch commented Jul 12, 2023

Since mypy, lint and tests now do what they should, the PyQt5 / PyQt6 part is now essentially merge-able and definitely ready for review 🎉.

I disabled auto-detecting PySide6 for now as there is a lot more work to do.

As this is always a hassle to rebase (will probably just merge master into here or do a final merge), I would prefer to have this in sooner rather than later, but not in the already extremely blown up v0.9.0. However, I think we can release v0.9.0 after #650 is in, what do you think @jcjgraf ?

After the release, I would like to prioritize getting this into master.

In case bugs come up, we can always release a follow up v0.9.X from the v0.9.0 tag. One example would be #652, which may end up being quite some effort.

@karlch karlch marked this pull request as ready for review July 12, 2023 20:34
@karlch
Copy link
Owner Author

karlch commented Jul 13, 2023

Another note: replacing the api.objreg.register decorator with a regular call within the __init__ fixes the performance issues with PyQt6. Absolutely no idea why, and that is quite unfortunate, as it is pretty deep in the api and also used by plugins.

Details:

# slow
@api.objreg.register
def __init__(self):
    ...

# fast
def __init__(self):
    api.objreg.register(self)

As this is a pretty major issue, I think changing the behaviour, while keeping backward compatibility and logging a deprecation warning, is fine.


def _select_wrapper():
"""Select a python Qt wrapper version based on environment variable and fallback."""
env_var = "VIMIV_QT_WRAPPER"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you introduce a custom env variable instead of relying on the standardized QT_SELECT=5/6?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would we differentiate between PyQt6 and PySide6?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just take whichever is available. I doubt that users have both installed. And if so, why should they care which one is used?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I care for testing both without changing the source code all the time 😄 I do think it is nice to expose this option in any case, but we could split it into QT_SELECT and USE_PYSIDE, if you think this has some benefit? Personally, I find one easier than two though.

Copy link
Contributor

@jcjgraf jcjgraf Jul 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, for development/testing/debugging this would certainly make sense to explicitly choose between QT and Side. I just think that if there is a standard, it would make sense to follow it. What about having only QT_SELECT but allow not only the values 5/6 but also the explicit options PyQt5/PyQt6/PySide6?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this compromise 👍 So our QT_SELECT would then have:

  • 5 -> PyQt5
  • 6 -> PyQt6 or PySide6 with a preference that we set
  • all three explicit options as currently done by VIMIV_QT_WRAPPER

@jcjgraf
Copy link
Contributor

jcjgraf commented Jul 15, 2023

Thanks a lot for your hard work on this. I had a glimpse at the new QT module and that makes sense so far. To me it seems like there are no major changes between QT5 and QT6, except a few methods renames. Or did I miss anything?

The rough roadmap makes totally sense to me. Everything for v0.9.0 is done, except for #325. But I do not think that is relevant for that milestone anymore. So I guess we can make a new release, and after that focus on getting QT6 in.

The issue with api.objreg.register is really unfortunate. But at least you figured out that this is the cause of the performance degradation. Maybe we can find a way to mitigate/fix the issue while still keeping the decorator.

@karlch
Copy link
Owner Author

karlch commented Jul 15, 2023

Thanks for checking! 😊 Yes, except for a few renames / deprecations, and the annoying new enum access, nothing major implementation-wise.

Sounds good 👍 I will try to release the new version over the week-end then.

I see two (maybe three) options:

  1. Deprecate the decorator and use as described above.
  2. Go back to the "old-style" objreg where we store everything in a dictionary again (objreg._registry[cls] = component) instead of directly in the class via cls.instance = component.

Personally I am leaning towards 1. as it actually also makes the tests a bit cleaner (no need to hack-mock decorators). But from a "plugin-writer" perspective 2. is definitely easier, sure.

EDIT: Forgot the third option, somehow figuring out how to get an instance without using the objreg, no idea for this, so not really an option ATM.

@karlch karlch added this to the v0.10.0 milestone Jul 15, 2023
@jcjgraf
Copy link
Contributor

jcjgraf commented Jul 16, 2023

I will try to release the new version over the week-end then.

Awesome, let me know if I can help with anything!

I see two (maybe three) options

I need to have a close look at it and investigate a bit. Do you know how other projects deal with that?

@karlch
Copy link
Owner Author

karlch commented Jul 16, 2023

Release is out, haven't heard any complaints yet 😄

I remember qutebrowser having an objreg, probably with string names though instead of our version with one per-class. Our main "issue" is that we register commands using the decorator during a time when there are no instances. I really like this syntax though as you can put the command registry right where the function is defined ...

The way it is usually done is probably something along:

  • Always have the instances where you need them code-wise
  • Have a central place to register commands / keybindings

@jcjgraf
Copy link
Contributor

jcjgraf commented Jul 20, 2023

Thanks and congrats for a new release! That one took quite some time 😁

I have rebased the branch for the current master in my fork. I have also changed the env variable to QT_SELECT as discussed above. I won't have any further time to work on this this week. In case you have time, you may continue where I left off 😉

@karlch
Copy link
Owner Author

karlch commented Jul 25, 2023

Thanks, I will take a look once I find time for more changes.

However, I have also merged some stuff locally without pushing into another branch and played around with the objreg, so not sure if it is easier for me to continue off your rebase (except for picking some QT_SELECT parts).

EDIT: I pushed this to qt6-merge for now, this includes the objreg change, deprecating the decorator syntax. This is still up for discussion.

New "issue" is that the PyQt6 install actually seems to "support" pdfs. I tested this with a quick function, and we do get a preview of the first page as discussed ages ago in #525. I would add pdfs to the ignore part for now, and re-consider this in a new PR if this is fine with you, as I am not sure what we want here.

@jcjgraf
Copy link
Contributor

jcjgraf commented Aug 4, 2023

except for picking some QT_SELECT parts

I have also created a testcase for it (same commit). I am not sure if you have seen that. Nothing amazing, but better then nothing, I guess.

I remember qutebrowser having an objreg, probably with string names though instead of our version with one per-class.

Right, but they do not seem to be completely happy abut it (not related to our issue though...).

I have investigated into the objreg weirdness. I have condensed the issue down to the following MVE:

import time

from PyQt5.QtCore import QObject as QObject5
from PyQt6.QtCore import QObject as QObject6


def timed(func):
    def inner(*args, **kwargs):
        start = time.time()
        return_value = func(*args, **kwargs)
        elapsed_in_ms = (time.time() - start) * 1000
        print(f"{func.__qualname__}: took {elapsed_in_ms:.3f} ms")
        return return_value

    return inner


def register_decorator(func):
    def inner(component) -> None:
        cls = component.__class__
        cls.instance = component
        func(component)
    return inner


def register_function(component):
    cls = component.__class__
    cls.instance = component


class DecoQt5(QObject5):
    @timed
    @register_decorator
    def __init__(self) -> None:
        super().__init__()
        print("QT5 decorator")


class FuncQt5(QObject5):
    @timed
    def __init__(self) -> None:
        super().__init__()
        print("QT5 function")
        register_function(self)


class DecoQt6(QObject6):
    @timed
    @register_decorator
    def __init__(self) -> None:
        super().__init__()
        print("QT6 decorator")


class FuncQt6(QObject6):
    @timed
    def __init__(self) -> None:
        super().__init__()
        print("QT6 function")
        register_function(self)


a = DecoQt5()
b = FuncQt5()
c = DecoQt6()
d = FuncQt6()

assert hasattr(DecoQt5, "instance")
assert hasattr(FuncQt5, "instance")
assert hasattr(DecoQt6, "instance")
assert hasattr(FuncQt6, "instance")

The big difference between the register_decorator (i.e. the old register decorator) and register_function (i.e. the new register function) is that the former stores the component before the superclass (QObject in this case) is initialized, while the later does it after the initialisation. For FuncQt6, if one swaps the super().__init__() and register_function(self) in there is a delay too.

So, our issue is not the decorator, but the execution order. I think we can just move the func(component) before the assignment to cls.instance in the decorator and we are good, aren't we? Like:

diff --git a/vimiv/api/objreg.py b/vimiv/api/objreg.py
index cb974c3a..8dc350b4 100644
--- a/vimiv/api/objreg.py
+++ b/vimiv/api/objreg.py
@@ -57,10 +57,10 @@ def register(component_init: Callable) -> Callable:
         Args:
             component: Corresponds to self.
         """
         cls = component.__class__
         _logger.debug("Registering '%s.%s'", cls.__module__, cls.__qualname__)
+        component_init(component, *args, **kwargs)
         cls.instance = component
-        component_init(component, *args, **kwargs)

     return inside

New "issue" is that the PyQt6 install actually seems to "support" pdfs. I tested this with a quick function, and we do get a preview of the first page as discussed ages ago in #525. I would add pdfs to the ignore part for now, and re-consider this in a new PR if this is fine with you, as I am not sure what we want here.

Which part of QT is supporting PDF? I only found this, but that is specific to PySide6. As long as the they are not detected by utils.imageheader as valid images, they are not further considered. So, why explicitly ignoring them?
Anyways, I would certainly not consider PDF support in this PR.

@karlch
Copy link
Owner Author

karlch commented Aug 5, 2023

I did see it, but the comment "The test case is failing, as it should skip tests, based on installed QT binding." made me think this is a problem for future us 😄

Yep, I actually remember seeing this issue ages ago and switching from names to types, to at some point actually just having .instance which is pretty close to storing one in the module and using .get(). We have it a bit easier, as we don't need multiple instances of the same class.

That is an excellent catch, thanks for digging!! 😊 I reverted the function commit and simply changed the order in the qt6-merge branch.

What I mean with "supports" is that QImageReader.supportedImageFormats actually includes pdf and fails the tests, i.e.

        """Tests if all formats readable by QT are also detected."""
>       assert name in NOT_DETECT_FORMATS or name in DETECT_FORMATS, "Missing check"
E       AssertionError: Missing check
E       assert ('pdf' in ['svgz', 'wbmp'] or 'pdf' in ['jpg', 'png', 'gif', 'jp2', 'webp', 'tiff', ...])

Fixed by adding the pdf into the NOT_DETECT_FORMATS in 0513132, this is what I meant with ignore.

@jcjgraf
Copy link
Contributor

jcjgraf commented Aug 5, 2023

What I mean with "supports" is that QImageReader.supportedImageFormats actually includes pdf and fails the tests

That is weird, I checked their docs, and PDFs should not be supported. Have you installed any PDF QT module? Does it also happen in the CI or only locally?

Do you know what is going wrong here? I have regenerated the .venv.

$ tox -e mypy                                                                                          12:40:46
.pkg: _optional_hooks> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_editable> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_wheel> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
mypy: install_package> python -I -m pip install --force-reinstall --no-deps /home/jeanclaude/Documents/Programming/vimiv-qt/.tox/.tmp/package/210/vimiv-0.9.0.tar.gz
mypy: commands_pre[0]> .tox/mypy/bin/python scripts/maybe_build_cextension.py
mypy: commands[0]> pytest
Error importing a valid Qt python wrapper:
No valid Qt wrapper found. Valid options: PyQt5, PyQt6, PySide6

Please install / configure a valid wrapper to run vimiv.
mypy: exit 2 (0.56 seconds) /home/jeanclaude/Documents/Programming/vimiv-qt> pytest pid=20083
.pkg: _exit> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  mypy: FAIL code 2 (5.12=setup[4.50]+cmd[0.06,0.56] seconds)
  evaluation failed :( (5.32 seconds)

@karlch
Copy link
Owner Author

karlch commented Aug 5, 2023

Both in CI and locally, I mean, tox runs a virtualenv anyway. An example action is this one.

Yep, mypy now needs a flag, could probably consider a default, but currently only mypy-pyqt5 or mypy-pqt6 run.

@jcjgraf
Copy link
Contributor

jcjgraf commented Aug 7, 2023

Yep, mypy now needs a flag

Ah sure, silly me 🙈

QImageReader.supportedImageFormats actually includes pdf

True, same for me.

Is there anything else I can help with concerning this PR?

@karlch
Copy link
Owner Author

karlch commented Aug 14, 2023

Perfect, not really, in case you have no further comments, I would go ahead and finalise documentation and merge into master 😊

@jcjgraf
Copy link
Contributor

jcjgraf commented Aug 14, 2023

No, from my side everything is good 👍

@karlch karlch merged commit ee28b6f into master Aug 15, 2023
@karlch karlch deleted the qt6 branch August 15, 2023 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for (Py)Qt 6
2 participants