diff --git a/CHANGELOG.md b/CHANGELOG.md index 83370937..10e4d4b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,62 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). +## [0.10.3] 2022-05-30 + +### Features +- General + - new parameter `--suggestions`: forces loading software suggestions after the initialization process [#260](https://github.com/vinifmor/bauh/issues/260) + - allowing custom suggestions / curated software to be mapped by Linux distributions (more on [README.md](https://github.com/vinifmor/bauh#suggestions)) [#260](https://github.com/vinifmor/bauh/issues/260) + +### Improvements +- Arch + - suggestions available for repository packages (and the associated caching expiration property `suggestions_exp`. Default: 24 hours) + +- General + - preventing command injection through the search mechanism [#266](https://github.com/vinifmor/bauh/issues/266) + - code refactoring + +- UI + - manage window minimum width related to the table columns + - table columns width for maximized window + - settings window size rules moved to stylesheet files + - enforcing maximum width and height for the management window based on the primary screen resolution [#261](https://github.com/vinifmor/bauh/issues/261) + - some columns of the management window now have their widths limit based on a percentage of the primary screen's width: + - name limit: 15% + - description limit: 18% + - publisher limit: 12% + - version: the limit for displaying both the installed and latest versions is 22% (otherwise just the latest version will be displayed) + - auto-resizing the management panel when filters are applied + +- Settings + - new property to disable SSL checking when downloading files (disabled by default) + +

+ +

+ + - the default value for `suggestions.by_type` is now `15`. + +### Fixes +- Arch + - conflict resolution: removing hard dependencies that would be satisfied with the inclusion of the new package [#268](https://github.com/vinifmor/bauh/issues/268) + - e.g: `pipewire-pulse` conflicts with `pulseaudio`. `pulseaudio-alsa` (a dependency of pulseaudio) should not be removed, since `pipewire-pulse` provides `pulseaudio` + - AUR: + - build: error raised when the temporary directory does not exist (when changing the CPUs governors) + - date parsing when checking for updates + - not caching the 'LastModified' field of installed AUR dependencies (could lead to wrong display updates) + +- Flatpak + - not all selected runtime partials to upgrade are actually requested to be upgraded + +- Web + - not reading from the cached suggestions file after the first request + - not detecting some generated apps as installed + +- UI + - double suggestions loading call when no app is returned + + ## [0.10.2] 2022-04-16 ### Improvements - Arch diff --git a/README.md b/README.md index a2e6954c..307c1189 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ [![GitHub release](https://img.shields.io/github/release/vinifmor/bauh.svg?label=Release)](https://github.com/vinifmor/bauh/releases/) [![PyPI](https://img.shields.io/pypi/v/bauh?label=PyPI)](https://pypi.org/project/bauh) [![AUR](https://img.shields.io/aur/version/bauh?label=AUR)](https://aur.archlinux.org/packages/bauh) [![AUR-staging](https://img.shields.io/aur/version/bauh-staging?label=AUR-staging)](https://aur.archlinux.org/packages/bauh-staging) [![License](https://img.shields.io/github/license/vinifmor/bauh?label=License)](https://github.com/vinifmor/bauh/blob/master/LICENSE) [![kofi](https://img.shields.io/badge/Ko--Fi-Donate-orange?style=flat&logo=ko-fi)](https://ko-fi.com/vinifmor) [![Follow on Twitter](https://img.shields.io/twitter/follow/bauh4linux?style=social&label=Twitter)](https://twitter.com/bauh4linux) -**bauh** (ba-oo), formerly known as **fpakman**, is a graphical interface for managing your Linux software (packages/applications). It currently supports the following formats: AppImage, ArchLinux packages (including AUR), Debian packages, Flatpak, Snap and Web applications. +**bauh** (ba-oo), formerly known as **fpakman**, is a graphical interface for managing your Linux software (packages/applications). It currently supports the following formats: AppImage, Debian and Arch Linux packages (including AUR), Flatpak, Snap and Web applications. Key features -- A management panel where you can: search, install, uninstall, upgrade, downgrade and launch you applications (and more...) +- A management panel where you can: search, install, uninstall, upgrade, downgrade and launch your applications - Tray mode: it launches attached to the system tray and publishes notifications when there are software updates available - System backup: it integrates with [Timeshift](https://github.com/teejee2008/timeshift) to provide a simple and safe backup process before applying changes to your system - Custom themes: it's possible to customize the tool's style/appearance. More at [Custom themes](#custom_themes) @@ -32,6 +32,7 @@ Key features - [Native Web applications](#type_web) 7. [General settings](#settings) - [Forbidden packaging formats](#forbidden_gems) + - [Custom suggestions / curated software](#suggestions) 8. [Directory structure, caching and logs](#dirs) 9. [Custom themes](#custom_themes) 10. [Tray icons](#tray_icons) @@ -213,7 +214,6 @@ bauh is officially distributed through [PyPi](https://pypi.org/project/bauh) and - Downloaded database files are stored at `~/.cache/bauh/appimage` (or `/var/cache/bauh/appimage` for **root**) as **apps.db** and **releases.db** - Databases are updated during the initialization process if they are considered outdated - The configuration file is located at `~/.config/bauh/appimage.yml` (or `/etc/bauh/appimage.yml` for **root**) and it allows the following customizations: -- Applications with ignored updates are defined at `~/.config/bauh/appimage/updates_ignored.txt` (or `/etc/bauh/appimage/updates_ignored.txt` for **root**) ``` database: @@ -222,6 +222,9 @@ suggestions: expiration: 24 # defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. Default: 24. ``` +- Applications with ignored updates are defined at `~/.config/bauh/appimage/updates_ignored.txt` (or `/etc/bauh/appimage/updates_ignored.txt` for **root**) +- Cached package suggestions: `~/.cache/bauh/web/suggestions.txt` (or `/var/cache/bauh/web/suggestions.yml` for **root**) + ##### Arch packages/AUR @@ -287,7 +290,9 @@ suggest_optdep_uninstall: false # if the optional dependencies associated with categories_exp: 24 # It defines the expiration time (in HOURS) of the packages categories mapping file stored in disc. Use 0 so that it is always updated during initialization. aur_rebuild_detector: true # it checks if packages built with old library versions require to be rebuilt. If a package needs to be rebuilt, it will be marked for update ('rebuild-detector' must be installed). Default: true. prefer_repository_provider: true # when there is just one repository provider for a given a dependency and several from AUR, it will be automatically picked. +suggestions_exp: 24 # it defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. ``` +- Cached package suggestions: `~/.cache/bauh/arch/suggestions.txt` (or `/var/cache/bauh/arch/suggestions.yml` for **root**) ##### Debian packages - Basic actions supported: **search**, **install**, **uninstall**, **upgrade** @@ -299,7 +304,7 @@ prefer_repository_provider: true # when there is just one repository provider f - **purge**: removes the packages and all related configuration files - Files: - runnable applications index: `~/.cache/bauh/debian/apps_idx.json` (or `/var/cache/bauh/debian/apps_idx.json` for **root**) - - package suggestions: `~/.cache/bauh/debian/suggestions.txt` (or `/var/cache/bauh/debian/suggestions.txt` for **root**) + - cached package suggestions: `~/.cache/bauh/debian/suggestions.txt` (or `/var/cache/bauh/debian/suggestions.txt` for **root**) - configuration: `~/.config/bauh/debian.yml` or `/etc/bauh/debian.yml` - `index_apps.exp`: time period (**in minutes**) in which the installed applications cache is considered up-to-date during startup (default: `1440` -> 24 hours) - `sync_pkgs.time`: time period (**in minutes**) in which the packages synchronization must be done on startup (default: `1440` -> 24 hours) @@ -317,7 +322,6 @@ installation_level: null # defines a default installation level: "user" or "syst - Custom actions supported: - **Full update**: it completely updates the Flatpak apps and components. Useful if you are having issues with runtime updates. - #### Snap - Make sure **snapd** is properly installed and enabled on your system: https://snapcraft.io/docs/installing-snapd @@ -386,7 +390,7 @@ environment: suggestions: cache_exp: 24 # defines the period (in HOURS) in which suggestions stored on the disk are considered up to date during the initialization process. Use 0 so that they are always updated. Default: 24. ``` - +- Cached package suggestions: `~/.cache/bauh/web/suggestions.txt` (or `/var/cache/bauh/web/suggestions.yml` for **root**) #### General settings @@ -398,6 +402,7 @@ You can change some application settings via environment variables or arguments - `--reset`: it cleans all configurations and cached data stored in the HOME directory. - `--logs`: it enables logs (for debugging purposes). - `--offline`: it assumes the internet connection is off. +- `--suggestions`: it forces loading software suggestions after the initialization process. ##### Configuration file (**~/.config/bauh/config.yml**) @@ -407,6 +412,7 @@ download: icons: true # allows bauh to download the applications icons when they are not saved on the disk multithreaded: true # allows bauh to use a multithreaded download client installed on the system to download applications source files faster multithreaded_client: null # defines the multi-threaded download tool to be used. If null, the default installed tool will be used (priority: aria2 > axel). Possible tools/values: aria2, axel + check_ssl: true # if the security certificate (SSL) should be checked before downloading files. gems: null # defines the enabled applications types managed by bauh (a null value means "all available") locale: null # defines a different translation for bauh (a null value will retrieve the system's default locale) store_root_password: true # if the root password should be asked only once @@ -414,7 +420,7 @@ memory_cache: data_expiration: 3600 # the interval in SECONDS that data cached in memory will live icon_expiration: 300 # the interval in SECONDS that icons cached in memory will live suggestions: - by_type: 10 # the maximum number of application suggestions that must be retrieved per type + by_type: 20 # the maximum number of application suggestions that must be retrieved per type enabled: true # if suggestions must be displayed when no application is installed system: notifications: true # if system popup should be displayed for some events. e.g: when there are updates, bauh will display a system popup @@ -460,6 +466,30 @@ appimage # flatpak # 'sharps' can be used to ignore a given line (comment) ``` +##### Custom suggestions / curated software +- The software suggestions are download from [bauh-files](https://github.com/vinifmor/bauh-files) by default + - [appimage](https://github.com/vinifmor/bauh-files/blob/master/appimage/suggestions.txt) + - [arch](https://github.com/vinifmor/bauh-files/blob/master/appimage/suggestions.txt) + - [debian](https://github.com/vinifmor/bauh-files/blob/master/debian/suggestions_v1.txt) + - [flatpak](https://github.com/vinifmor/bauh-files/blob/master/flatpak/suggestions.txt) + - [snap](https://github.com/vinifmor/bauh-files/blob/master/snap/suggestions.txt) + - [web](https://github.com/vinifmor/bauh-files/blob/master/web/env/v2/suggestions.yml) + +- Most of the files follow the pattern: `{priority_number}=${id or name}` + - Priority numbers: 0 (LOW), 1 (MEDIUM), 2 (HIGH), 3 (TOP) + - The priority number is used to sort the retrieved suggestions + +- If Linux distributions want to provide their custom suggestions files: + - Create the file `/etc/bauh/suggestions.conf` + - The content is basically a mapping for each gem to a url or local file (absolute path). + - Example: + ``` + arch=https://mydomain.com/arch/suggestions.txt # remote file + appimage=/etc/bauh/appimage/suggestions.txt # local file (absolute path) + # snap = my mapping # comments with a '#' are allowed + ``` + - If a given gem name is omitted, its suggestions will be downloaded from the default location. + #### Directory structure, caching and logs - `~/.config/bauh` (or `/etc/bauh` for **root**): stores configuration files - `~/.cache/bauh` (or `/var/cache/bauh` for **root**): stores data about your installed applications, databases, indexes, etc. Files are stored here to provide a faster initialization and data recovery. diff --git a/bauh/__init__.py b/bauh/__init__.py index 6d149908..206851b5 100644 --- a/bauh/__init__.py +++ b/bauh/__init__.py @@ -1,4 +1,4 @@ -__version__ = '0.10.2' +__version__ = '0.10.3' __app_name__ = 'bauh' import os diff --git a/bauh/api/abstract/context.py b/bauh/api/abstract/context.py index 2b41a891..3fe3f901 100644 --- a/bauh/api/abstract/context.py +++ b/bauh/api/abstract/context.py @@ -1,5 +1,6 @@ import logging import sys +from typing import Optional, Dict from bauh.api.abstract.cache import MemoryCacheFactory from bauh.api.abstract.disk import DiskCacheLoaderFactory @@ -15,7 +16,7 @@ def __init__(self, download_icons: bool, http_client: HttpClient, app_root_dir: cache_factory: MemoryCacheFactory, disk_loader_factory: DiskCacheLoaderFactory, logger: logging.Logger, file_downloader: FileDownloader, distro: str, app_name: str, app_version: str, internet_checker: InternetChecker, root_user: bool, screen_width: int = -1, - screen_height: int = -1): + screen_height: int = -1, suggestions_mapping: Optional[Dict[str, str]] = None): """ :param download_icons: if packages icons should be downloaded :param http_client: a shared instance of http client @@ -31,6 +32,7 @@ def __init__(self, download_icons: bool, http_client: HttpClient, app_root_dir: :param internet_checker :param screen_width :param screen_height + :param suggestions_mapping """ self.download_icons = download_icons self.http_client = http_client @@ -51,6 +53,7 @@ def __init__(self, download_icons: bool, http_client: HttpClient, app_root_dir: self.internet_checker = internet_checker self.screen_width = screen_width self.screen_height = screen_height + self._suggestions_mapping = suggestions_mapping def is_system_x86_64(self): return self.arch_x86_64 @@ -60,3 +63,13 @@ def get_view_path(self): def is_internet_available(self) -> bool: return self.internet_checker.is_available() + + def get_suggestion_url(self, module: str, default: Optional[str] = None) -> Optional[str]: + if self._suggestions_mapping: + module_split = module.split(f'{self.app_name}.gems.') + + if len(module_split) > 1: + gem_name = module_split[1].split('.')[0] + return self._suggestions_mapping.get(gem_name, default) + + return default diff --git a/bauh/api/abstract/disk.py b/bauh/api/abstract/disk.py index 82b785a9..bcac5792 100644 --- a/bauh/api/abstract/disk.py +++ b/bauh/api/abstract/disk.py @@ -1,5 +1,5 @@ from abc import ABC, abstractmethod -from typing import Type +from typing import Type, Optional, Any, Dict from bauh.api.abstract.cache import MemoryCache from bauh.api.abstract.model import SoftwarePackage @@ -29,6 +29,12 @@ def fill(self, pkg: SoftwarePackage, sync: bool = False): """ pass + def read(self, pkg: SoftwarePackage) -> Optional[Dict[str, Any]]: + """ + returns the cached data from the given package + """ + pass + class DiskCacheLoaderFactory(ABC): diff --git a/bauh/api/abstract/model.py b/bauh/api/abstract/model.py index 8e63e257..0c9f1d7e 100644 --- a/bauh/api/abstract/model.py +++ b/bauh/api/abstract/model.py @@ -299,6 +299,14 @@ class SuggestionPriority(Enum): HIGH = 2 TOP = 3 + def __gt__(self, other): + if isinstance(other, SuggestionPriority): + return self.value > other.value + + def __lt__(self, other): + if isinstance(other, SuggestionPriority): + return self.value < other.value + class PackageSuggestion: diff --git a/bauh/api/abstract/view.py b/bauh/api/abstract/view.py index db4070ad..ce9fc0ec 100644 --- a/bauh/api/abstract/view.py +++ b/bauh/api/abstract/view.py @@ -9,6 +9,16 @@ class MessageType(Enum): ERROR = 2 +class ViewComponentAlignment(Enum): + CENTER = 0 + LEFT = 1 + RIGHT = 2 + BOTTOM = 3 + TOP = 4 + HORIZONTAL_CENTER = 5 + VERTICAL_CENTER = 6 + + class ViewObserver: def on_change(self, change): @@ -19,8 +29,10 @@ class ViewComponent(ABC): """ Represents a GUI component """ - def __init__(self, id_: Optional[str], observers: Optional[List[ViewObserver]] = None): + def __init__(self, id_: Optional[str], alignment: Optional[ViewComponentAlignment] = None, + observers: Optional[List[ViewObserver]] = None): self.id = id_ + self.alignment = alignment self.observers = observers if observers else [] def add_observer(self, obs): @@ -114,10 +126,11 @@ class SelectViewType(Enum): class SingleSelectComponent(InputViewComponent): - def __init__(self, type_: SelectViewType, label: str, options: List[InputOption], default_option: InputOption = None, - max_per_line: int = 1, tooltip: str = None, max_width: int = -1, id_: str = None, - capitalize_label: bool = True): - super(SingleSelectComponent, self).__init__(id_=id_) + def __init__(self, type_: SelectViewType, label: str, options: List[InputOption], + default_option: InputOption = None, max_per_line: int = 1, tooltip: str = None, + max_width: Optional[int] = -1, id_: str = None, capitalize_label: bool = True, + alignment: Optional[ViewComponentAlignment] = None): + super(SingleSelectComponent, self).__init__(id_=id_, alignment=alignment) self.type = type_ self.label = label self.options = options diff --git a/bauh/app.py b/bauh/app.py index 5ee5bcd2..68800f9c 100755 --- a/bauh/app.py +++ b/bauh/app.py @@ -53,6 +53,9 @@ def main(tray: bool = False): QCoreApplication.setAttribute(Qt.AA_UseHighDpiPixmaps) QCoreApplication.setAttribute(Qt.AA_EnableHighDpiScaling) + if bool(args.suggestions): + logger.info("Forcing loading software suggestions after the initialization process") + if tray or bool(args.tray): from bauh.tray import new_tray_icon app, widget = new_tray_icon(app_config, logger) diff --git a/bauh/app_args.py b/bauh/app_args.py index b6c1c386..a9a74792 100644 --- a/bauh/app_args.py +++ b/bauh/app_args.py @@ -9,6 +9,8 @@ def read() -> Namespace: parser.add_argument('-v', '--version', action='version', version='%(prog)s {}'.format(__version__)) parser.add_argument('--logs', action="store_true", help='It activates {} logs.'.format(__app_name__)) parser.add_argument('--offline', action="store_true", help='It assumes the internet connection is off') + parser.add_argument('--suggestions', action="store_true", + help='It forces loading software suggestions after the initialization process') exclusive_args = parser.add_mutually_exclusive_group() exclusive_args.add_argument('--tray', action="store_true", help='If {} should be attached to the system tray.'.format(__app_name__)) diff --git a/bauh/cli/app.py b/bauh/cli/app.py index e69ea83c..ad88fb53 100644 --- a/bauh/cli/app.py +++ b/bauh/cli/app.py @@ -35,6 +35,11 @@ def main(): cache_factory = DefaultMemoryCacheFactory(expiration_time=0) + downloader = AdaptableFileDownloader(logger=logger, multithread_enabled=app_config['download']['multithreaded'], + multithread_client=app_config['download']['multithreaded_client'], + i18n=i18n, http_client=http_client, + check_ssl=app_config['download']['check_ssl']) + context = ApplicationContext(i18n=i18n, http_client=http_client, download_icons=bool(app_config['download']['icons']), @@ -43,11 +48,11 @@ def main(): disk_loader_factory=DefaultDiskCacheLoaderFactory(logger), logger=logger, distro=util.get_distro(), - file_downloader=AdaptableFileDownloader(logger, bool(app_config['download']['multithreaded']), - i18n, http_client, app_config['download']['multithreaded_client']), + file_downloader=downloader, app_name=__app_name__, app_version=__version__, internet_checker=InternetChecker(offline=False), + suggestions_mapping=None, # TODO not needed at the moment root_user=user.is_root()) managers = gems.load_managers(context=context, locale=i18n.current_key, config=app_config, diff --git a/bauh/commons/custom_types.py b/bauh/commons/custom_types.py new file mode 100644 index 00000000..2ee4ef3c --- /dev/null +++ b/bauh/commons/custom_types.py @@ -0,0 +1,20 @@ +from typing import Optional, Any + + +class Value: + + def __init__(self, value: Optional[Any] = None): + self.value = value + + def __repr__(self): + return str(self.value) + + def __str__(self): + return self.__repr__() + + def __eq__(self, other): + if isinstance(other, Value): + return self.value == other.value + + def __hash__(self): + return hash(self.value) diff --git a/bauh/commons/suggestions.py b/bauh/commons/suggestions.py new file mode 100644 index 00000000..c7565e2c --- /dev/null +++ b/bauh/commons/suggestions.py @@ -0,0 +1,35 @@ +from logging import Logger +from typing import Dict, Optional, Tuple + +from bauh.api.abstract.model import SuggestionPriority + + +def parse(suggestions_str: str, logger: Optional[Logger] = None, type_: Optional[str] = None, + splitter: str = '=') \ + -> Dict[str, SuggestionPriority]: + output = dict() + + for line in suggestions_str.split('\n'): + clean_line = line.strip() + + if clean_line: + line_split = clean_line.split(splitter, 1) + + if len(line_split) == 2: + prio_str, name = line_split[0].strip(), line_split[1].strip() + + if prio_str and name: + try: + prio = int(line_split[0]) + except ValueError: + if logger: + logger.warning(f"Could not parse {type_ + ' ' if type_ else ''}suggestion: {line}") + continue + + output[line_split[1]] = SuggestionPriority(prio) + + return output + + +def sort_by_priority(names_prios: Dict[str, SuggestionPriority]) -> Tuple[str, ...]: + return tuple(pair[1] for pair in sorted(((names_prios[n], n) for n in names_prios), reverse=True)) diff --git a/bauh/commons/util.py b/bauh/commons/util.py index 76beacfe..9a8c5627 100644 --- a/bauh/commons/util.py +++ b/bauh/commons/util.py @@ -1,9 +1,14 @@ import logging +import re from abc import ABC from datetime import datetime from logging import Logger from typing import Optional, Union +re_command_forbidden_symbols = re.compile(r'[\'\"%$#*<>]') +re_several_spaces = re.compile(r'\s+') +re_command_parameter = re.compile(r'(^|\s)-+\w+') + class NullLoggerFactory(ABC): @@ -68,3 +73,16 @@ def datetime_as_milis(date: datetime = datetime.utcnow()) -> int: def map_timestamp_file(file_path: str) -> str: path_split = file_path.split('/') return '/'.join(path_split[0:-1]) + '/' + path_split[-1].split('.')[0] + '.ts' + + +def sanitize_command_input(input_: str) -> str: + final_input = input_ + + for op in ('|', '&'): + final_input = final_input.split(op)[0] + + for remove_re in (re_command_forbidden_symbols, re_command_parameter): + final_input = remove_re.sub('', final_input) + + final_input = re_several_spaces.sub(' ', final_input) + return final_input.strip() diff --git a/bauh/commons/view_utils.py b/bauh/commons/view_utils.py index 726e99b6..2138aaeb 100644 --- a/bauh/commons/view_utils.py +++ b/bauh/commons/view_utils.py @@ -4,7 +4,7 @@ from bauh.api.abstract.view import SelectViewType, InputOption, SingleSelectComponent -def new_select(label: str, tip: Optional[str], id_: str, opts: Iterable[Tuple[Optional[str], object, Optional[str]]], value: object, max_width: int, +def new_select(label: str, tip: Optional[str], id_: str, opts: Iterable[Tuple[Optional[str], object, Optional[str]]], value: object, max_width: Optional[int] = None, type_: SelectViewType = SelectViewType.RADIO, capitalize_label: bool = True): inp_opts = [InputOption(label=o[0].capitalize(), value=o[1], tooltip=o[2]) for o in opts] def_opt = [o for o in inp_opts if o.value == value] diff --git a/bauh/gems/appimage/__init__.py b/bauh/gems/appimage/__init__.py index 24133d96..4812d5cc 100644 --- a/bauh/gems/appimage/__init__.py +++ b/bauh/gems/appimage/__init__.py @@ -9,7 +9,6 @@ ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) APPIMAGE_SHARED_DIR = f'{SHARED_FILES_DIR}/appimage' INSTALLATION_DIR = f'{APPIMAGE_SHARED_DIR}/installed' -SUGGESTIONS_FILE = f'https://raw.githubusercontent.com/vinifmor/{__app_name__}-files/master/appimage/suggestions.txt' CONFIG_FILE = f'{CONFIG_DIR}/appimage.yml' APPIMAGE_CONFIG_DIR = f'{CONFIG_DIR}/appimage' UPDATES_IGNORED_FILE = f'{APPIMAGE_CONFIG_DIR}/updates_ignored.txt' @@ -19,8 +18,6 @@ DATABASE_APPS_FILE = f'{APPIMAGE_CACHE_DIR}/apps.db' DATABASE_RELEASES_FILE = f'{APPIMAGE_CACHE_DIR}/releases.db' DATABASES_TS_FILE = f'{APPIMAGE_CACHE_DIR}/dbs.ts' -SUGGESTIONS_CACHED_FILE = f'{APPIMAGE_CACHE_DIR}/suggestions.txt' -SUGGESTIONS_CACHED_TS_FILE = f'{APPIMAGE_CACHE_DIR}/suggestions.ts' DOWNLOAD_DIR = f'{TEMP_DIR}/appimage/download' diff --git a/bauh/gems/appimage/controller.py b/bauh/gems/appimage/controller.py index bf1775ea..88ba7f98 100644 --- a/bauh/gems/appimage/controller.py +++ b/bauh/gems/appimage/controller.py @@ -7,7 +7,6 @@ import subprocess import traceback from datetime import datetime -from math import floor from pathlib import Path from typing import Set, Type, List, Tuple, Optional, Iterable, Generator @@ -83,6 +82,7 @@ def __init__(self, context: ApplicationContext): self._action_self_install: Optional[CustomSoftwareAction] = None self._app_github: Optional[str] = None self._search_unfilled_attrs: Optional[Tuple[str, ...]] = None + self._suggestions_downloader: Optional[AppImageSuggestionsDownloader] = None def install_file(self, root_password: Optional[str], watcher: ProcessWatcher) -> bool: max_width = 350 @@ -746,9 +746,11 @@ def prepare(self, task_manager: TaskManager, root_password: Optional[str], inter create_config=create_config, http_client=self.context.http_client, logger=self.context.logger).start() - AppImageSuggestionsDownloader(taskman=task_manager, i18n=self.context.i18n, - http_client=self.context.http_client, logger=self.context.logger, - create_config=create_config).start() + if not self.suggestions_downloader.is_custom_local_file_mapped(): + self.suggestions_downloader.taskman = task_manager + self.suggestions_downloader.create_config = create_config + self.suggestions_downloader.register_task() + self.suggestions_downloader.start() def list_updates(self, internet_available: bool) -> List[PackageUpdate]: res = self.read_installed(disk_loader=None, internet_available=internet_available) @@ -768,23 +770,24 @@ def list_warnings(self, internet_available: bool) -> List[str]: return [self.i18n['appimage.warning.missing_db_files'].format(appimage=bold('AppImage'))] def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[PackageSuggestion]]: - res = [] + if limit == 0: + return connection = self._get_db_connection(DATABASE_APPS_FILE) if connection: - suggestions = AppImageSuggestionsDownloader(appimage_config=self.configman.get_config(), logger=self.logger, - i18n=self.i18n, http_client=self.http_client, - taskman=TaskManager()).read() + self.suggestions_downloader.taskman = TaskManager() + suggestions = tuple(self.suggestions_downloader.read()) if not suggestions: - self.logger.warning("Could not read suggestions") - return res + self.logger.warning("Could not read AppImage suggestions") + return else: - self.logger.info("Mapping suggestions") + self.logger.info("Mapping AppImage suggestions") try: if filter_installed: - installed = {i.name.lower() for i in self.read_installed(disk_loader=None, connection=connection).installed} + installed = {i.name.lower() for i in self.read_installed(disk_loader=None, + connection=connection).installed} else: installed = None @@ -795,7 +798,7 @@ def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[ name = lsplit[1].strip() - if limit <= 0 or len(sugs_map) < limit: + if limit < 0 or len(sugs_map) < limit: if not installed or not name.lower() in installed: sugs_map[name] = SuggestionPriority(int(lsplit[0])) else: @@ -804,17 +807,18 @@ def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[ cursor = connection.cursor() cursor.execute(query.FIND_APPS_BY_NAME_FULL.format(','.join([f"'{s}'" for s in sugs_map.keys()]))) + res = [] for t in cursor.fetchall(): app = AppImage(*t, i18n=self.i18n) res.append(PackageSuggestion(app, sugs_map[app.name.lower()])) - self.logger.info(f"Mapped {len(res)} suggestions") + + self.logger.info(f"Mapped {len(res)} AppImage suggestions") + return res except: traceback.print_exc() finally: connection.close() - return res - def is_default_enabled(self) -> bool: return True @@ -852,7 +856,6 @@ def clear_data(self, logs: bool = True): def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: config_ = self.configman.get_config() - max_width = 50 comps = [ TextInputComponent(label=self.i18n['appimage.config.database.expiration'], @@ -860,14 +863,12 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: config_['database']['expiration'], int) else '', tooltip=self.i18n['appimage.config.database.expiration.tip'], only_int=True, - max_width=max_width, id_='appim_db_exp'), TextInputComponent(label=self.i18n['appimage.config.suggestions.expiration'], value=int(config_['suggestions']['expiration']) if isinstance( config_['suggestions']['expiration'], int) else '', tooltip=self.i18n['appimage.config.suggestions.expiration.tip'], only_int=True, - max_width=max_width, id_='appim_sugs_exp') ] @@ -1083,3 +1084,18 @@ def app_github(self) -> str: self._app_github = f'vinifmor/{self.context.app_name}' return self._app_github + + @property + def suggestions_downloader(self) -> AppImageSuggestionsDownloader: + if not self._suggestions_downloader: + file_url = self.context.get_suggestion_url(self.__module__) + self._suggestions_downloader = AppImageSuggestionsDownloader(taskman=TaskManager(), + i18n=self.context.i18n, + http_client=self.context.http_client, + logger=self.context.logger, + file_url=file_url) + + if self._suggestions_downloader.is_custom_local_file_mapped(): + self.logger.info(f"Local AppImage suggestions file mapped: {file_url}") + + return self._suggestions_downloader diff --git a/bauh/gems/appimage/worker.py b/bauh/gems/appimage/worker.py index ea6ede95..95d8a3ae 100644 --- a/bauh/gems/appimage/worker.py +++ b/bauh/gems/appimage/worker.py @@ -8,7 +8,7 @@ from datetime import datetime, timedelta from pathlib import Path from threading import Thread -from typing import Optional, List +from typing import Optional, Generator import requests @@ -17,8 +17,7 @@ from bauh.commons.boot import CreateConfigFile from bauh.commons.html import bold from bauh.gems.appimage import get_icon_path, INSTALLATION_DIR, SYMLINKS_DIR, util, DATABASES_TS_FILE, \ - APPIMAGE_CACHE_DIR, DATABASE_APPS_FILE, DATABASE_RELEASES_FILE, URL_COMPRESSED_DATABASES, SUGGESTIONS_FILE, \ - SUGGESTIONS_CACHED_TS_FILE, SUGGESTIONS_CACHED_FILE + APPIMAGE_CACHE_DIR, DATABASE_APPS_FILE, DATABASE_RELEASES_FILE, URL_COMPRESSED_DATABASES from bauh.gems.appimage.model import AppImage from bauh.view.util.translation import I18n @@ -286,7 +285,9 @@ def run(self): class AppImageSuggestionsDownloader(Thread): - def __init__(self, logger: logging.Logger, http_client: HttpClient, i18n: I18n, taskman: TaskManager, create_config: Optional[CreateConfigFile] = None, appimage_config: Optional[dict] = None): + def __init__(self, logger: logging.Logger, http_client: HttpClient, i18n: I18n, file_url: Optional[str], + create_config: Optional[CreateConfigFile] = None, appimage_config: Optional[dict] = None, + taskman: Optional[TaskManager] = None): super(AppImageSuggestionsDownloader, self).__init__(daemon=True) self.create_config = create_config self.logger = logger @@ -295,133 +296,189 @@ def __init__(self, logger: logging.Logger, http_client: HttpClient, i18n: I18n, self.taskman = taskman self.config = appimage_config self.task_id = 'appim.suggestions' - self.taskman.register_task(id_=self.task_id, label=i18n['task.download_suggestions'], icon_path=get_icon_path()) + self._cached_file_path: Optional[str] = None + self._cached_ts_file_path: Optional[str] = None + + if file_url: + self._file_url = file_url + else: + self._file_url = f'https://raw.githubusercontent.com/vinifmor/bauh-files/master/appimage/suggestions.txt' + + @property + def cached_file_path(self) -> str: + if not self._cached_file_path: + self._cached_file_path = f'{APPIMAGE_CACHE_DIR}/suggestions.txt' + + return self._cached_file_path + + @property + def cached_ts_file_path(self) -> str: + if not self._cached_ts_file_path: + self._cached_ts_file_path = f'{APPIMAGE_CACHE_DIR}/suggestions.ts' + + return self._cached_ts_file_path + + def register_task(self): + self.taskman.register_task(id_=self.task_id, + label=self.i18n['task.download_suggestions'], icon_path=get_icon_path()) + + def is_custom_local_file_mapped(self) -> bool: + return self._file_url and self._file_url.startswith('/') def should_download(self, appimage_config: dict) -> bool: + if not self._file_url: + self.logger.error("No AppImage suggestions file URL defined") + return False + + if self.is_custom_local_file_mapped(): + return False + try: exp_hours = int(appimage_config['suggestions']['expiration']) except: - self.logger.error("An exception happened while trying to parse 'suggestions.expiration'") + self.logger.error("An exception happened while trying to parse the AppImage 'suggestions.expiration'") traceback.print_exc() return True if exp_hours <= 0: - self.logger.info("Suggestions cache is disabled") + self.logger.info("The AppImage suggestions cache is disabled") return True - if not os.path.exists(SUGGESTIONS_CACHED_FILE): - self.logger.info("'{}' not found. It must be downloaded".format(SUGGESTIONS_CACHED_FILE)) + if not os.path.exists(self.cached_file_path): + self.logger.info(f"File {self.cached_file_path} not found. It must be downloaded") return True - if not os.path.exists(SUGGESTIONS_CACHED_TS_FILE): - self.logger.info("'{}' not found. The suggestions file must be downloaded.") + if not os.path.exists(self.cached_ts_file_path): + self.logger.info(f"File {self.cached_ts_file_path} not found. The suggestions file must be downloaded.") return True - with open(SUGGESTIONS_CACHED_TS_FILE) as f: + with open(self.cached_ts_file_path) as f: timestamp_str = f.read() try: suggestions_timestamp = datetime.fromtimestamp(float(timestamp_str)) except: - self.logger.error('Could not parse the cached suggestions timestamp: {}'.format(timestamp_str)) + self.logger.error(f'Could not parse the cached AppImage suggestions timestamp: {timestamp_str}') traceback.print_exc() return True update = suggestions_timestamp + timedelta(hours=exp_hours) <= datetime.utcnow() return update - def read(self) -> Optional[List[str]]: - self.logger.info("Checking if suggestions should be downloaded") + def read(self) -> Generator[str, None, None]: + if not self._file_url: + self.logger.error("No AppImage suggestions file URL defined") + yield from () + + self.logger.info("Checking if AppImage suggestions should be downloaded") if self.should_download(self.config): suggestions_timestamp = datetime.utcnow().timestamp() suggestions_str = self.download() Thread(target=self.cache_suggestions, args=(suggestions_str, suggestions_timestamp), daemon=True).start() else: - self.logger.info("Reading cached suggestions from '{}'".format(SUGGESTIONS_CACHED_FILE)) - with open(SUGGESTIONS_CACHED_FILE) as f: + if self.is_custom_local_file_mapped(): + file_path, log_ref = self._file_url, 'local' + else: + file_path, log_ref = self.cached_file_path, 'cached' + + self.logger.info(f"Reading {log_ref} AppImage suggestions from {file_path}") + with open(file_path) as f: suggestions_str = f.read() - return self.map_suggestions(suggestions_str) if suggestions_str else None + yield from self.map_suggestions(suggestions_str) if suggestions_str else () def cache_suggestions(self, text: str, timestamp: float): - self.logger.info("Caching suggestions to '{}'".format(SUGGESTIONS_FILE)) + self.logger.info(f"Caching AppImage suggestions to {self.cached_file_path}") - cache_dir = os.path.dirname(SUGGESTIONS_CACHED_FILE) + cache_dir = os.path.dirname(self.cached_file_path) try: Path(cache_dir).mkdir(parents=True, exist_ok=True) cache_dir_ok = True except OSError: - self.logger.error("Could not create cache directory '{}'".format(cache_dir)) + self.logger.error(f"Could not create the caching directory {cache_dir}") traceback.print_exc() cache_dir_ok = False if cache_dir_ok: try: - with open(SUGGESTIONS_CACHED_FILE, 'w+') as f: + with open(self.cached_file_path, 'w+') as f: f.write(text) except: - self.logger.error("An exception happened while writing the file '{}'".format(SUGGESTIONS_FILE)) + self.logger.error(f"An exception happened while writing AppImage suggestions to {self.cached_file_path}") traceback.print_exc() try: - with open(SUGGESTIONS_CACHED_TS_FILE, 'w+') as f: + with open(self.cached_ts_file_path, 'w+') as f: f.write(str(timestamp)) except: - self.logger.error("An exception happened while writing the file '{}'".format(SUGGESTIONS_CACHED_TS_FILE)) + self.logger.error(f"An exception happened while writing the cached AppImage suggestions timestamp " + f"to {self.cached_ts_file_path}") traceback.print_exc() def download(self) -> Optional[str]: - self.logger.info("Downloading suggestions from {}".format(SUGGESTIONS_FILE)) + if not self._file_url: + self.logger.error("No AppImage suggestions file URL defined") + return + + if self.is_custom_local_file_mapped(): + self.logger.warning("Local AppImage suggestions file mapped. Nothing will be downloaded") + return + + self.logger.info(f"Downloading AppImage suggestions from {self._file_url}") try: - res = self.http_client.get(SUGGESTIONS_FILE) + res = self.http_client.get(self._file_url) except requests.exceptions.ConnectionError: - self.logger.warning("Could not download suggestion from '{}'".format(SUGGESTIONS_FILE)) + self.logger.warning(f"Could not download suggestion from {self._file_url}") return if not res: - self.logger.warning("Could not download suggestion from '{}'".format(SUGGESTIONS_FILE)) + self.logger.warning(f"Could not download suggestion from {self._file_url}") return if not res.text: - self.logger.warning("No suggestion found in {}".format(SUGGESTIONS_FILE)) + self.logger.warning(f"No AppImage suggestion found in {self._file_url}") return return res.text - def map_suggestions(self, text: str) -> List[str]: - return [line for line in text.split('\n') if line] + def map_suggestions(self, text: str) -> Generator[str, None, None]: + return (line for line in text.split('\n') if line) def run(self): - if self.create_config: - self.taskman.update_progress(self.task_id, 0, self.i18n['task.waiting_task'].format(bold(self.create_config.task_name))) - self.create_config.join() - self.config = self.create_config.config - ti = time.time() - self.taskman.update_progress(self.task_id, 1, None) - self.logger.info("Checking if suggestions should be downloaded") - should_download = self.should_download(self.config) - self.taskman.update_progress(self.task_id, 30, None) + if not self.is_custom_local_file_mapped(): + if self.create_config: + wait_msg = self.i18n['task.waiting_task'].format(bold(self.create_config.task_name)) + self.taskman.update_progress(self.task_id, 0, wait_msg) + self.create_config.join() + self.config = self.create_config.config - try: - if should_download: - suggestions_timestamp = datetime.utcnow().timestamp() - suggestions_str = self.download() - self.taskman.update_progress(self.task_id, 70, None) + ti = time.time() + self.taskman.update_progress(self.task_id, 1, None) - if suggestions_str: - self.cache_suggestions(suggestions_str, suggestions_timestamp) - else: - self.logger.info("Cached suggestions are up-to-date") - except: - self.logger.error("An unexpected exception happened") - traceback.print_exc() + self.logger.info("Checking if AppImage suggestions should be downloaded") + should_download = self.should_download(self.config) + self.taskman.update_progress(self.task_id, 30, None) + + try: + if should_download: + suggestions_timestamp = datetime.utcnow().timestamp() + suggestions_str = self.download() + self.taskman.update_progress(self.task_id, 70, None) + + if suggestions_str: + self.cache_suggestions(suggestions_str, suggestions_timestamp) + else: + self.logger.info("Cached AppImage suggestions are up-to-date") + except: + self.logger.error("An unexpected exception happened while downloading AppImage suggestions") + traceback.print_exc() self.taskman.update_progress(self.task_id, 100, None) self.taskman.finish_task(self.task_id) tf = time.time() - self.logger.info("Took {0:.9f} seconds to download suggestions".format(tf - ti)) + self.logger.info(f"Took {tf - ti:.9f} seconds to download suggestions") diff --git a/bauh/gems/arch/aur.py b/bauh/gems/arch/aur.py index 24cab409..fa1b02de 100644 --- a/bauh/gems/arch/aur.py +++ b/bauh/gems/arch/aur.py @@ -294,7 +294,7 @@ def map_update_data(self, pkgname: str, latest_version: Optional[str] = None, sr provided.add(pkgname) if info: - provided.add('{}={}'.format(pkgname, info['pkgver'])) + provided.add(f"{pkgname}={info['pkgver']}") if info.get('provides'): provided.update(info.get('provides')) @@ -303,7 +303,7 @@ def map_update_data(self, pkgname: str, latest_version: Optional[str] = None, sr 'b': info.get('pkgbase', pkgname)} else: if latest_version: - provided.add('{}={}'.format(pkgname, latest_version)) + provided.add(f'{pkgname}={latest_version}') return {'c': None, 's': None, 'p': provided, 'r': 'aur', 'v': latest_version, 'd': set(), 'b': pkgname} diff --git a/bauh/gems/arch/config.py b/bauh/gems/arch/config.py index b17800e3..615499b5 100644 --- a/bauh/gems/arch/config.py +++ b/bauh/gems/arch/config.py @@ -40,4 +40,5 @@ def get_default_config(self) -> dict: 'categories_exp': 24, 'aur_rebuild_detector': False, "aur_rebuild_detector_no_bin": True, - "prefer_repository_provider": True} + "prefer_repository_provider": True, + 'suggestions_exp': 24} diff --git a/bauh/gems/arch/controller.py b/bauh/gems/arch/controller.py index 7e6a266b..c27724b9 100644 --- a/bauh/gems/arch/controller.py +++ b/bauh/gems/arch/controller.py @@ -8,7 +8,6 @@ import time import traceback from datetime import datetime -from math import floor from pathlib import Path from pwd import getpwnam from threading import Thread @@ -21,8 +20,8 @@ TransactionResult, SoftwareAction, SettingsView, SettingsController from bauh.api.abstract.disk import DiskCacheLoader from bauh.api.abstract.handler import ProcessWatcher, TaskManager -from bauh.api.abstract.model import PackageUpdate, PackageHistory, SoftwarePackage, PackageSuggestion, PackageStatus, \ - SuggestionPriority, CustomSoftwareAction +from bauh.api.abstract.model import PackageUpdate, PackageHistory, SoftwarePackage, PackageStatus, \ + CustomSoftwareAction, PackageSuggestion from bauh.api.abstract.view import MessageType, FormComponent, InputOption, SingleSelectComponent, SelectViewType, \ ViewComponent, PanelComponent, MultipleSelectComponent, TextInputComponent, TextInputType, \ FileChooserComponent, TextComponent @@ -32,6 +31,7 @@ from bauh.commons.boot import CreateConfigFile from bauh.commons.category import CategoriesDownloader from bauh.commons.html import bold +from bauh.commons.suggestions import sort_by_priority from bauh.commons.system import SystemProcess, ProcessHandler, new_subprocess, run_cmd, SimpleProcess from bauh.commons.util import datetime_as_milis from bauh.commons.view_utils import new_select @@ -50,6 +50,7 @@ from bauh.gems.arch.output import TransactionStatusHandler from bauh.gems.arch.pacman import RE_DEP_OPERATORS from bauh.gems.arch.proc_util import write_as_user +from bauh.gems.arch.suggestions import RepositorySuggestionsDownloader from bauh.gems.arch.updates import UpdatesSummarizer from bauh.gems.arch.worker import AURIndexUpdater, ArchDiskCacheUpdater, ArchCompilationOptimizer, RefreshMirrors, \ SyncDatabases @@ -216,6 +217,7 @@ def __init__(self, context: ApplicationContext, disk_cache_updater: Optional[Arc self.re_file_conflict = re.compile(r'[\w\d\-_.]+:') self.disk_cache_updater = disk_cache_updater self.pkgbuilder_user: Optional[str] = f'{__app_name__}-aur' if context.root_user else None + self._suggestions_downloader: Optional[RepositorySuggestionsDownloader] = None def refresh_mirrors(self, root_password: Optional[str], watcher: ProcessWatcher) -> bool: handler = ProcessHandler(watcher) @@ -490,9 +492,9 @@ def _check_aur_package_update(self, pkg: ArchPackage, installed_data: dict, api_ try: pkg.install_date = datetime_as_milis(parse_date(install_date)) except ValueError: - self.logger.error("Could not parse 'install_date' ({}) from AUR package '{}'".format(install_date, pkg.name)) + self.logger.error(f"Could not parse 'install_date' ({install_date}) from AUR package '{pkg.name}'") else: - self.logger.error("AUR package '{}' install_date was not retrieved".format(pkg.name)) + self.logger.error(f"AUR package '{pkg.name}' install_date was not retrieved") return self.aur_mapper.check_update(pkg=pkg, last_modified=api_data['LastModified']) @@ -584,7 +586,7 @@ def read_installed(self, disk_loader: Optional[DiskCacheLoader], limit: int = -1 rebuild_ignored = Thread(target=self.__fill_ignored_by_rebuild_detector, args=(rebuild_output, ), daemon=True) rebuild_ignored.start() - installed = pacman.map_installed(names=names) + installed = pacman.map_packages(names=names) aur_pkgs, repo_pkgs, aur_index = None, None, None @@ -631,7 +633,7 @@ def read_installed(self, disk_loader: Optional[DiskCacheLoader], limit: int = -1 t.join() if pkgs: - ignored = self._list_ignored_updates() + ignored = self._fill_ignored_updates(set()) if ignored: for p in pkgs: @@ -672,7 +674,7 @@ def _downgrade_aur_pkg(self, context: TransactionContext) -> bool: context.project_dir = clone_dir srcinfo_path = f'{clone_dir}/.SRCINFO' - logs = git.log_shas_and_timestamps(clone_dir) + logs = git.list_commits(clone_dir) context.watcher.change_progress(40) if not logs or len(logs) == 1: @@ -1194,7 +1196,10 @@ def upgrade(self, requirements: UpgradeRequirements, root_password: Optional[str watcher.change_substatus('') return True - def _uninstall_pkgs(self, pkgs: Iterable[str], root_password: Optional[str], handler: ProcessHandler, ignore_dependencies: bool = False) -> bool: + def _uninstall_pkgs(self, pkgs: Collection[str], root_password: Optional[str], + handler: ProcessHandler, ignore_dependencies: bool = False, + replacers: Optional[Set[str]] = None) -> bool: + status_handler = TransactionStatusHandler(watcher=handler.watcher, i18n=self.i18n, names={*pkgs}, @@ -1206,6 +1211,9 @@ def _uninstall_pkgs(self, pkgs: Iterable[str], root_password: Optional[str], han if ignore_dependencies: cmd.append('-dd') + if replacers: + cmd.extend(f'--assume-installed={r}' for r in replacers) + status_handler.start() all_uninstalled, _ = handler.handle_simple(SimpleProcess(cmd=cmd, root_password=root_password, @@ -1299,9 +1307,53 @@ def _confirm_all_unneeded_removal(self, pkgs: Collection[str], context: Transact return True - def _uninstall(self, context: TransactionContext, names: Set[str], remove_unneeded: bool = False, disk_loader: Optional[DiskCacheLoader] = None, skip_requirements: bool = False): + def _fill_aur_providers(self, names: str, output: Set[str]): + for _, data in self.aur_client.gen_updates_data(names): + providers = data.get('p') + + if providers: + output.update(providers) + + def _map_actual_replacers(self, names: Set[str], context: TransactionContext) -> Optional[Set[str]]: + if not names: + return + + repo_replacers, aur_replacers = set(), set() + + for r in names: + repo = context.remote_repo_map.get(r) + + if repo and repo != 'aur': + repo_replacers.add(r) + elif repo == 'aur' or (context.aur_idx and r in context.aur_idx): + aur_replacers.add(r) + + actual_replacers = set() + + thread_fill_aur = None + if aur_replacers: + thread_fill_aur = Thread(target=self._fill_aur_providers, args=(aur_replacers, actual_replacers)) + thread_fill_aur.start() + + if repo_replacers: + repo_replace_providers = pacman.map_provided(remote=True, pkgs=repo_replacers) + + if repo_replace_providers: + actual_replacers.update(repo_replace_providers) + + if thread_fill_aur: + thread_fill_aur.join() + + return actual_replacers + + def _uninstall(self, context: TransactionContext, names: Set[str], remove_unneeded: bool = False, + disk_loader: Optional[DiskCacheLoader] = None, skip_requirements: bool = False, + replacers: Optional[Set[str]] = None): + self._update_progress(context, 10) + actual_replacers = self._map_actual_replacers(replacers, context) if replacers else None + net_available = self.context.internet_checker.is_available() if disk_loader else True hard_requirements = set() @@ -1309,13 +1361,14 @@ def _uninstall(self, context: TransactionContext, names: Set[str], remove_unneed if not skip_requirements: for n in names: try: - pkg_reqs = pacman.list_hard_requirements(n, self.logger) + pkg_reqs = pacman.list_hard_requirements(name=n, logger=self.logger, assume_installed=actual_replacers) if pkg_reqs: hard_requirements.update(pkg_reqs) + except PackageInHoldException: - context.watcher.show_message(title=self.i18n['error'].capitalize(), - body=self.i18n['arch.uninstall.error.hard_dep_in_hold'].format(bold(n)), + error_msg = self.i18n['arch.uninstall.error.hard_dep_in_hold'].format(bold(n)) + context.watcher.show_message(title=self.i18n['error'].capitalize(), body=error_msg, type_=MessageType.ERROR) return False @@ -1355,7 +1408,8 @@ def _uninstall(self, context: TransactionContext, names: Set[str], remove_unneed provided_by_uninstalled = pacman.map_provided(pkgs=to_uninstall) - uninstalled = self._uninstall_pkgs(to_uninstall, context.root_password, context.handler, ignore_dependencies=skip_requirements) + uninstalled = self._uninstall_pkgs(to_uninstall, context.root_password, context.handler, + ignore_dependencies=skip_requirements, replacers=actual_replacers) if uninstalled: self._remove_uninstalled_from_context(provided_by_uninstalled, context) @@ -1398,7 +1452,8 @@ def _uninstall(self, context: TransactionContext, names: Set[str], remove_unneed else: unnecessary_instances = None - unneded_uninstalled = self._uninstall_pkgs(all_unnecessary_to_uninstall, context.root_password, context.handler) + unneded_uninstalled = self._uninstall_pkgs(all_unnecessary_to_uninstall, context.root_password, + context.handler, replacers=actual_replacers) if unneded_uninstalled: to_uninstall.update(all_unnecessary_to_uninstall) @@ -1432,7 +1487,7 @@ def _uninstall(self, context: TransactionContext, names: Set[str], remove_unneed return uninstalled def _map_installed_data_for_removal(self, names: Iterable[str]) -> Optional[Dict[str, Dict[str, str]]]: - data = pacman.map_installed(names) + data = pacman.map_packages(names) if data: remapped_data = {} @@ -1644,7 +1699,7 @@ def _get_history_aur_pkg(self, pkg: ArchPackage) -> PackageHistory: if not os.path.exists(srcinfo_path): return PackageHistory.empyt(pkg) - logs = git.log_shas_and_timestamps(clone_dir) + logs = git.list_commits(clone_dir) if logs: srcfields = {'epoch', 'pkgver', 'pkgrel'} @@ -1769,10 +1824,13 @@ def get_history(self, pkg: ArchPackage) -> PackageHistory: else: return self._get_history_repo_pkg(pkg) - def _request_conflict_resolution(self, pkg: str, conflicting_pkg: str, context: TransactionContext, skip_requirements: bool = False) -> bool: - conflict_msg = '{} {} {}'.format(bold(pkg), self.i18n['and'], bold(conflicting_pkg)) - if not context.watcher.request_confirmation(title=self.i18n['arch.install.conflict.popup.title'], - body=self.i18n['arch.install.conflict.popup.body'].format(conflict_msg)): + def _request_conflict_resolution(self, pkg: str, conflicting_pkg: str, context: TransactionContext, + skip_requirements: bool = False) -> bool: + + conflict_msg = f"{bold(pkg)} {self.i18n['and']} {bold(conflicting_pkg)}" + msg_body = self.i18n['arch.install.conflict.popup.body'].format(conflict_msg) + + if not context.watcher.request_confirmation(title=self.i18n['arch.install.conflict.popup.title'], body=msg_body): context.watcher.print(self.i18n['action.cancelled']) return False else: @@ -1783,8 +1841,7 @@ def _request_conflict_resolution(self, pkg: str, conflicting_pkg: str, context: context.removed = {} res = self._uninstall(context=context, names={conflicting_pkg}, disk_loader=context.disk_loader, - remove_unneeded=False, skip_requirements=skip_requirements) - + remove_unneeded=False, skip_requirements=skip_requirements, replacers={pkg}) context.restabilish_progress() return res @@ -1802,7 +1859,7 @@ def _install_deps(self, context: TransactionContext, deps: List[Tuple[str, str]] if dep[1] == 'aur': dep_context = context.gen_dep_context(dep[0], dep[1]) dep_src = self.aur_client.get_src_info(dep[0]) - dep_context.base = dep_src['pkgbase'] + dep_context.base = dep_src['pkgbase'] if dep_src['pkgbase'] else dep[0] aur_deps_context.append(dep_context) else: repo_deps.append(dep) @@ -1856,14 +1913,33 @@ def _install_deps(self, context: TransactionContext, deps: List[Tuple[str, str]] else: return repo_dep_names - for aur_context in aur_deps_context: - installed = self._install_from_aur(aur_context) + if aur_deps_context: + aur_deps_info = self.aur_client.get_info((c.base for c in aur_deps_context)) + aur_deps_data = None - if not installed: - return {aur_context.name} - else: - progress += progress_increment - self._update_progress(context, progress) + if aur_deps_info: + aur_deps_data = {data['Name']: data for data in aur_deps_info} + + for aur_context in aur_deps_context: + if aur_deps_data: + dep_data = aur_deps_data.get(aur_context.base) + + if dep_data: + last_modified = dep_data.get('LastModified') + + if last_modified and isinstance(last_modified, int): + aur_context.last_modified = last_modified + else: + self.logger.warning(f"No valid 'LastModified' field returned for AUR package " + f"'{context.name}': {last_modified}") + + installed = self._install_from_aur(aur_context) + + if not installed: + return {aur_context.name} + else: + progress += progress_increment + self._update_progress(context, progress) self._update_progress(context, 100) @@ -2005,7 +2081,10 @@ def _build(self, context: TransactionContext) -> bool: cpus_changed, cpu_prev_governors = False, None if optimize: - cpus_changed, cpu_prev_governors = cpu_manager.set_all_cpus_to('performance', context.root_password, self.logger) + cpus_changed, cpu_prev_governors = cpu_manager.set_all_cpus_to('performance', context.root_password, + self.logger) + + pkgbuilt = False try: pkgbuilt, output = makepkg.build(pkgdir=context.project_dir, @@ -2016,18 +2095,21 @@ def _build(self, context: TransactionContext) -> bool: finally: if cpus_changed and cpu_prev_governors: self.logger.info("Restoring CPU governors") - cpu_manager.set_cpus(cpu_prev_governors, context.root_password, {'performance'}, self.logger) + cpu_manager.set_cpus(cpu_prev_governors, context.root_password, self.logger, {'performance'}) self._update_progress(context, 65) if pkgbuilt: self.__fill_aur_output_files(context) - self.logger.info("Reading '{}' cloned repository current commit".format(context.name)) - context.commit = git.get_current_commit(context.project_dir) + self.logger.info(f"Reading '{context.name}' cloned repository current commit") + commits = git.list_commits(context.project_dir, limit=1) + + if commits: + context.commit = commits[0][0] - if not context.commit: - self.logger.error("Could not read '{}' cloned repository current commit".format(context.name)) + else: + self.logger.error(f"Could not read '{context.name}' cloned repository current commit") if self._install(context=context): self._save_pkgbuild(context) @@ -2389,25 +2471,12 @@ def _install(self, context: TransactionContext) -> bool: if context.removed is None: context.removed = {} - to_install_replacements = pacman.map_replaces(names_to_install) - - skip_requirement_checking = False - if to_install_replacements: # checking if the packages to be installed replace the installed packages - all_replacements = set() - - for replacements in to_install_replacements: - all_replacements.update(replacements) - - if all_replacements: - for pkg in to_uninstall: - if pkg not in all_replacements: - break - - skip_requirement_checking = True - context.disable_progress_if_changing() + if not self._uninstall(names=to_uninstall, context=context, remove_unneeded=False, - disk_loader=context.disk_loader, skip_requirements=skip_requirement_checking): + disk_loader=context.disk_loader, + replacers=names_to_install): + context.watcher.show_message(title=self.i18n['error'], body=self.i18n['arch.uninstalling.conflict.fail'].format(', '.join((bold(p) for p in to_uninstall))), type_=MessageType.ERROR) @@ -2755,6 +2824,11 @@ def prepare(self, task_manager: TaskManager, root_password: Optional[str], inter self.index_aur = AURIndexUpdater(context=self.context, taskman=task_manager, create_config=create_config) # must always execute to properly determine the installed packages (even that AUR is disabled) self.index_aur.start() + if not self.suggestions_downloader.is_custom_local_file_mapped(): + self.suggestions_downloader.create_config = create_config + self.suggestions_downloader.register_task(task_manager) + self.suggestions_downloader.start() + refresh_mirrors = RefreshMirrors(taskman=task_manager, i18n=self.i18n, root_password=root_password, logger=self.logger, create_config=create_config) refresh_mirrors.start() @@ -2805,8 +2879,11 @@ def launch(self, pkg: ArchPackage): final_cmd = pkg.command.replace('%U', '') subprocess.Popen(final_cmd, shell=True) - def _gen_bool_selector(self, id_: str, label_key: str, tooltip_key: str, value: bool, max_width: int, - capitalize_label: bool = True, label_params: Optional[list] = None, tooltip_params: Optional[list] = None) -> SingleSelectComponent: + def _gen_bool_selector(self, id_: str, label_key: str, tooltip_key: str, value: bool, + max_width: Optional[int] = None, capitalize_label: bool = True, + label_params: Optional[list] = None, tooltip_params: Optional[list] = None) \ + -> SingleSelectComponent: + opts = [InputOption(label=self.i18n['yes'].capitalize(), value=True), InputOption(label=self.i18n['no'].capitalize(), value=False)] @@ -2830,12 +2907,11 @@ def _gen_bool_selector(self, id_: str, label_key: str, tooltip_key: str, value: id_=id_, capitalize_label=capitalize_label) - def _get_general_settings(self, arch_config: dict, max_width: int) -> SettingsView: + def _get_general_settings(self, arch_config: dict) -> SettingsView: db_sync_start = self._gen_bool_selector(id_='sync_dbs_start', label_key='arch.config.sync_dbs', tooltip_key='arch.config.sync_dbs_start.tip', - value=bool(arch_config['sync_databases_startup']), - max_width=max_width) + value=bool(arch_config['sync_databases_startup'])) db_sync_start.label += f" ({self.i18n['initialization'].capitalize()})" @@ -2843,95 +2919,86 @@ def _get_general_settings(self, arch_config: dict, max_width: int) -> SettingsVi self._gen_bool_selector(id_='repos', label_key='arch.config.repos', tooltip_key='arch.config.repos.tip', - value=bool(arch_config['repositories']), - max_width=max_width), + value=bool(arch_config['repositories'])), self._gen_bool_selector(id_='autoprovs', label_key='arch.config.automatch_providers', tooltip_key='arch.config.automatch_providers.tip', - value=bool(arch_config['automatch_providers']), - max_width=max_width), + value=bool(arch_config['automatch_providers'])), self._gen_bool_selector(id_='prefer_repo_provider', label_key='arch.config.prefer_repository_provider', tooltip_key='arch.config.prefer_repository_provider.tip', value=bool(arch_config['prefer_repository_provider']), - max_width=max_width, tooltip_params=['AUR']), self._gen_bool_selector(id_='check_dependency_breakage', label_key='arch.config.check_dependency_breakage', tooltip_key='arch.config.check_dependency_breakage.tip', - value=bool(arch_config['check_dependency_breakage']), - max_width=max_width), + value=bool(arch_config['check_dependency_breakage'])), self._gen_bool_selector(id_='mthread_download', label_key='arch.config.pacman_mthread_download', tooltip_key='arch.config.pacman_mthread_download.tip', value=arch_config['repositories_mthread_download'], - max_width=max_width, capitalize_label=True), self._gen_bool_selector(id_='sync_dbs', label_key='arch.config.sync_dbs', tooltip_key='arch.config.sync_dbs.tip', - value=bool(arch_config['sync_databases']), - max_width=max_width), + value=bool(arch_config['sync_databases'])), db_sync_start, self._gen_bool_selector(id_='clean_cached', label_key='arch.config.clean_cache', tooltip_key='arch.config.clean_cache.tip', - value=bool(arch_config['clean_cached']), - max_width=max_width), + value=bool(arch_config['clean_cached'])), self._gen_bool_selector(id_='suggest_unneeded_uninstall', label_key='arch.config.suggest_unneeded_uninstall', tooltip_params=['"{}"'.format(self.i18n['arch.config.suggest_optdep_uninstall'])], tooltip_key='arch.config.suggest_unneeded_uninstall.tip', - value=bool(arch_config['suggest_unneeded_uninstall']), - max_width=max_width), + value=bool(arch_config['suggest_unneeded_uninstall'])), self._gen_bool_selector(id_='suggest_optdep_uninstall', label_key='arch.config.suggest_optdep_uninstall', tooltip_key='arch.config.suggest_optdep_uninstall.tip', - value=bool(arch_config['suggest_optdep_uninstall']), - max_width=max_width), + value=bool(arch_config['suggest_optdep_uninstall'])), self._gen_bool_selector(id_='ref_mirs', label_key='arch.config.refresh_mirrors', tooltip_key='arch.config.refresh_mirrors.tip', - value=bool(arch_config['refresh_mirrors_startup']), - max_width=max_width), + value=bool(arch_config['refresh_mirrors_startup'])), TextInputComponent(id_='mirrors_sort_limit', label=self.i18n['arch.config.mirrors_sort_limit'], tooltip=self.i18n['arch.config.mirrors_sort_limit.tip'], only_int=True, - max_width=50, value=arch_config['mirrors_sort_limit'] if isinstance(arch_config['mirrors_sort_limit'], int) else ''), TextInputComponent(id_='arch_cats_exp', label=self.i18n['arch.config.categories_exp'], tooltip=self.i18n['arch.config.categories_exp.tip'], - max_width=50, only_int=True, capitalize_label=False, value=arch_config['categories_exp'] if isinstance(arch_config['categories_exp'], int) else ''), + TextInputComponent(id_='arch_sugs_exp', + label=self.i18n['arch.config.suggestions_exp'], + tooltip=self.i18n['arch.config.suggestions_exp.tip'], + only_int=True, + capitalize_label=False, + value=arch_config['suggestions_exp'] if isinstance(arch_config['suggestions_exp'], int) else '') ] return SettingsView(self, PanelComponent([FormComponent(fields, spaces=False)], id_="repo"), icon_path=get_repo_icon_path()) - def _get_aur_settings(self, arch_config: dict, max_width: int) -> SettingsView: + def _get_aur_settings(self, arch_config: dict) -> SettingsView: fields = [ self._gen_bool_selector(id_='aur', label_key='arch.config.aur', tooltip_key='arch.config.aur.tip', value=arch_config['aur'], - max_width=max_width, capitalize_label=False), self._gen_bool_selector(id_='opts', label_key='arch.config.optimize', tooltip_key='arch.config.optimize.tip', value=bool(arch_config['optimize']), - capitalize_label=False, - max_width=max_width), + capitalize_label=False), self._gen_bool_selector(id_='rebuild_detector', label_key='arch.config.aur_rebuild_detector', tooltip_key='arch.config.aur_rebuild_detector.tip', value=bool(arch_config['aur_rebuild_detector']), tooltip_params=["'rebuild-detector'"], - capitalize_label=False, - max_width=max_width), + capitalize_label=False), self._gen_bool_selector(id_='rebuild_detector_no_bin', label_key='arch.config.aur_rebuild_detector_no_bin', label_params=['rebuild-detector'], @@ -2939,8 +3006,7 @@ def _get_aur_settings(self, arch_config: dict, max_width: int) -> SettingsView: tooltip_params=['rebuild-detector', self.i18n['arch.config.aur_rebuild_detector'].format('')], value=bool(arch_config['aur_rebuild_detector_no_bin']), - capitalize_label=False, - max_width=max_width), + capitalize_label=False), new_select(id_='aur_build_only_chosen', label=self.i18n['arch.config.aur_build_only_chosen'], tip=self.i18n['arch.config.aur_build_only_chosen.tip'], @@ -2949,7 +3015,6 @@ def _get_aur_settings(self, arch_config: dict, max_width: int) -> SettingsView: (self.i18n['ask'].capitalize(), None, None), ], value=arch_config['aur_build_only_chosen'], - max_width=max_width, type_=SelectViewType.RADIO, capitalize_label=False), new_select(label=self.i18n['arch.config.edit_aur_pkgbuild'], @@ -2960,27 +3025,23 @@ def _get_aur_settings(self, arch_config: dict, max_width: int) -> SettingsView: (self.i18n['ask'].capitalize(), None, None), ], value=arch_config['edit_aur_pkgbuild'], - max_width=max_width, type_=SelectViewType.RADIO, capitalize_label=False), self._gen_bool_selector(id_='aur_remove_build_dir', label_key='arch.config.aur_remove_build_dir', tooltip_key='arch.config.aur_remove_build_dir.tip', value=bool(arch_config['aur_remove_build_dir']), - max_width=max_width, capitalize_label=False), FileChooserComponent(id_='aur_build_dir', label=self.i18n['arch.config.aur_build_dir'], tooltip=self.i18n['arch.config.aur_build_dir.tip'].format( get_build_dir(arch_config, self.pkgbuilder_user)), - max_width=round(max_width * 0.65), file_path=arch_config['aur_build_dir'], capitalize_label=False, directory=True), TextInputComponent(id_='aur_idx_exp', label=self.i18n['arch.config.aur_idx_exp'], tooltip=self.i18n['arch.config.aur_idx_exp.tip'], - max_width=50, only_int=True, capitalize_label=False, value=arch_config['aur_idx_exp'] if isinstance(arch_config['aur_idx_exp'], int) else '') @@ -2992,9 +3053,8 @@ def _get_aur_settings(self, arch_config: dict, max_width: int) -> SettingsView: def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: arch_config = self.configman.get_config() - max_width = floor(self.context.screen_width * 0.25) - yield self._get_general_settings(arch_config, max_width) - yield self._get_aur_settings(arch_config, max_width) + yield self._get_general_settings(arch_config) + yield self._get_aur_settings(arch_config) @staticmethod def fill_general_settings(arch_config: dict, form: FormComponent): @@ -3024,6 +3084,7 @@ def fill_general_settings(arch_config: dict, form: FormComponent): arch_config['suggest_unneeded_uninstall'] = sug_unneeded_uni arch_config['categories_exp'] = form.get_component('arch_cats_exp', TextInputComponent).get_int_value() + arch_config['suggestions_exp'] = form.get_component('arch_sugs_exp', TextInputComponent).get_int_value() def _fill_aur_settings(self, arch_config: dict, form: FormComponent): arch_config['optimize'] = form.get_component('opts', SingleSelectComponent).get_selected() @@ -3295,8 +3356,7 @@ def clean_cache(self, root_password: Optional[str], watcher: ProcessWatcher) -> return True - def _list_ignored_updates(self) -> Set[str]: - ignored = set() + def _fill_ignored_updates(self, output: Set[str]) -> Set[str]: if os.path.exists(UPDATES_IGNORED_FILE): with open(UPDATES_IGNORED_FILE) as f: ignored_lines = f.readlines() @@ -3306,9 +3366,9 @@ def _list_ignored_updates(self) -> Set[str]: line_clean = line.strip() if line_clean: - ignored.add(line_clean) + output.add(line_clean) - return ignored + return output def _write_ignored(self, names: Set[str]): Path(ARCH_CONFIG_DIR).mkdir(parents=True, exist_ok=True) @@ -3323,7 +3383,7 @@ def _write_ignored(self, names: Set[str]): f.write('') def ignore_update(self, pkg: ArchPackage): - ignored = self._list_ignored_updates() + ignored = self._fill_ignored_updates(set()) if pkg.name not in ignored: ignored.add(pkg.name) @@ -3332,7 +3392,7 @@ def ignore_update(self, pkg: ArchPackage): pkg.update_ignored = True def _revert_ignored_updates(self, pkgs: Iterable[str]): - ignored = self._list_ignored_updates() + ignored = self._fill_ignored_updates(set()) for p in pkgs: if p in ignored: @@ -3670,3 +3730,153 @@ def add_package_builder_user(self, handler: ProcessHandler) -> bool: return added return True + + def _fill_available_packages(self, output: Dict[str, Set[str]]): + output.update(pacman.map_available_packages()) + + def _fill_suggestions(self, output: Dict[str, int]): + self.suggestions_downloader.register_task(None) + suggestions = self.suggestions_downloader.read(self.configman.read_config()) + + if suggestions: + output.update(suggestions) + + def _fill_cached_if_unset(self, pkg: ArchPackage, loader: DiskCacheLoader): + data = loader.read(pkg) + + if data: + for attr, cached_val in data.items(): + if cached_val: + current_val = getattr(pkg, attr) + + if current_val is None: + setattr(pkg, attr, cached_val) + + def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[PackageSuggestion]]: + if limit == 0: + return + + arch_config = self.configman.get_config() + + if not arch_config['repositories']: + return + + name_priority = dict() + + fill_suggestions = Thread(target=self._fill_suggestions, args=(name_priority,)) + fill_suggestions.start() + + available_packages = dict() + fill_available = Thread(target=self._fill_available_packages, args=(available_packages,)) + fill_available.start() + + ignored_pkgs = set() + fill_ignored = Thread(target=pacman.fill_ignored_packages, args=(ignored_pkgs,)) + fill_ignored.start() + + fill_suggestions.join() + + if not name_priority: + self.logger.info("No Arch package suggestions found") + return + + self.logger.info(f"Found {len(name_priority)} named Arch package suggestions") + + if fill_available: + fill_available.join() + + if not available_packages: + self.logger.error("No available Arch package found. It will not be possible to return suggestions") + return + + fill_ignored.join() + + available_suggestions = dict() + + for n in name_priority: + if n not in ignored_pkgs: + data = available_packages.get(n) + + if data and (not filter_installed or not data['i']): + available_suggestions[n] = data + + if not available_suggestions: + self.logger.info("No Arch package suggestion to return") + return + + if filter_installed: + ignored_updates = set() + thread_fill_ignored_updates = Thread(target=self._fill_ignored_updates, args=(ignored_updates,)) + thread_fill_ignored_updates.start() + else: + ignored_updates, thread_fill_ignored_updates = None, None + + suggestion_by_priority = sort_by_priority({n: name_priority[n] for n in available_suggestions}) + + if available_suggestions and 0 < limit < len(available_suggestions): + suggestion_by_priority = suggestion_by_priority[0:limit] + + self.logger.info(f'Available Arch package suggestions: {len(suggestion_by_priority)}') + + if thread_fill_ignored_updates: + thread_fill_ignored_updates.join() + + full_data = pacman.map_packages(names=suggestion_by_priority, remote=filter_installed, not_signed=False, + skip_ignored=True) + + if full_data and full_data.get('signed'): + full_data = full_data['signed'] + + disk_loader, caching_threads = None, None + if not filter_installed: + disk_loader = self.context.disk_loader_factory.new() + caching_threads = list() + + suggestions = [] + for name in suggestion_by_priority: + pkg_data = available_suggestions[name] + pkg_full_data = full_data.get(name) + description = None + + if pkg_full_data: + description = pkg_full_data.get('description') + + pkg_updates_ignored = pkg_data['i'] and ignored_updates and name in ignored_updates + pkg = ArchPackage(name=name, + version=pkg_data['v'], + latest_version=pkg_data['v'], + repository=pkg_data['r'], + installed=pkg_data['i'], + description=description, + categories=self.categories.get(name), + i18n=self.i18n, + maintainer=pkg_data['r'], + update_ignored=pkg_updates_ignored) + + if disk_loader: + t = Thread(target=self._fill_cached_if_unset, args=(pkg, disk_loader)) + t.start() + caching_threads.append(t) + + suggestions.append(PackageSuggestion(package=pkg, priority=name_priority[name])) + + if caching_threads: + for t in caching_threads: + t.join() + + return suggestions + + @property + def suggestions_downloader(self) -> RepositorySuggestionsDownloader: + if not self._suggestions_downloader: + file_url = self.context.get_suggestion_url(self.__module__) + + self._suggestions_downloader = RepositorySuggestionsDownloader(logger=self.logger, + http_client=self.http_client, + i18n=self.i18n, + file_url=file_url) + + if self._suggestions_downloader.is_custom_local_file_mapped(): + self.logger.info(f"Local Arch suggestions file mapped: {file_url}") + + return self._suggestions_downloader diff --git a/bauh/gems/arch/cpu_manager.py b/bauh/gems/arch/cpu_manager.py index 71e7c87b..a3cc6221 100644 --- a/bauh/gems/arch/cpu_manager.py +++ b/bauh/gems/arch/cpu_manager.py @@ -3,6 +3,7 @@ import shutil import traceback from logging import Logger +from pathlib import Path from typing import Optional, Set, Tuple, Dict from bauh.api.paths import TEMP_DIR @@ -25,10 +26,21 @@ def current_governors() -> Dict[str, Set[int]]: return governors -def set_governor(governor: str, root_password: Optional[str], cpu_idxs: Optional[Set[int]] = None): +def set_governor(governor: str, root_password: Optional[str], logger: Logger, cpu_idxs: Optional[Set[int]] = None): new_gov_file = f'{TEMP_DIR}/bauh_scaling_governor' - with open(new_gov_file, 'w+') as f: - f.write(governor) + + try: + Path(TEMP_DIR).mkdir(exist_ok=True, parents=True) + except OSError: + logger.error(f"Could not create temporary directory for changing CPU governor: {TEMP_DIR}") + return + + try: + with open(new_gov_file, 'w+') as f: + f.write(governor) + except OSError: + logger.error(f"Could not write new governor ({governor}) to {new_gov_file}") + return for idx in (cpu_idxs if cpu_idxs else range(multiprocessing.cpu_count())): _change_governor(idx, new_gov_file, root_password) @@ -37,7 +49,7 @@ def set_governor(governor: str, root_password: Optional[str], cpu_idxs: Optional try: os.remove(new_gov_file) except OSError: - traceback.print_exc() + logger.error(f"Could not remove temporary governor file ({new_gov_file})") def _change_governor(cpu_idx: int, new_gov_file_path: str, root_password: Optional[str]): @@ -49,7 +61,7 @@ def _change_governor(cpu_idx: int, new_gov_file_path: str, root_password: Option traceback.print_exc() -def set_all_cpus_to(governor: str, root_password: Optional[str], logger: Optional[Logger] = None) \ +def set_all_cpus_to(governor: str, root_password: Optional[str], logger: Logger) \ -> Tuple[bool, Optional[Dict[str, Set[int]]]]: cpus_changed, cpu_governors = False, current_governors() @@ -63,18 +75,18 @@ def set_all_cpus_to(governor: str, root_password: Optional[str], logger: Optiona if logger: logger.info(f"Changing CPUs {not_in_performance} governors to '{governor}'") - set_governor(governor, root_password, not_in_performance) + set_governor(governor=governor, root_password=root_password, logger=logger, cpu_idxs=not_in_performance) cpus_changed = True return cpus_changed, cpu_governors -def set_cpus(governors: Dict[str, Set[int]], root_password: Optional[str], ignore_governors: Optional[Set[str]] = None, - logger: Optional[Logger] = None): +def set_cpus(governors: Dict[str, Set[int]], root_password: Optional[str], logger: Logger, + ignore_governors: Optional[Set[str]] = None): for gov, cpus in governors.items(): if not ignore_governors or gov not in ignore_governors: if logger: logger.info(f"Changing CPUs {cpus} governors to '{gov}'") - set_governor(gov, root_password, cpus) + set_governor(governor=gov, root_password=root_password, logger=logger, cpu_idxs=cpus) diff --git a/bauh/gems/arch/git.py b/bauh/gems/arch/git.py index 8320002d..632752d8 100644 --- a/bauh/gems/arch/git.py +++ b/bauh/gems/arch/git.py @@ -1,56 +1,49 @@ import shutil -from datetime import datetime +from io import StringIO +from logging import Logger from typing import List, Tuple, Optional from bauh.commons import system -from bauh.commons.system import new_subprocess, SimpleProcess +from bauh.commons.system import SimpleProcess def is_installed() -> bool: return bool(shutil.which('git')) -def list_commits(proj_dir: str) -> List[dict]: - logs = new_subprocess(['git', 'log', '--date=iso'], cwd=proj_dir).stdout +def list_commits(proj_dir: str, limit: int = -1, logger: Optional[Logger] = None) -> Optional[List[Tuple[str, int]]]: + if limit == 0: + return - commits, commit = [], {} - for out in new_subprocess(['grep', '-E', 'commit|Date:'], stdin=logs).stdout: - if out: - line = out.decode() - if line.startswith('commit'): - commit['commit'] = line.split(' ')[1].strip() - elif line.startswith('Date'): - commit['date'] = datetime.fromisoformat(line.split(':')[1].strip()) - commits.append(commit) - commit = {} + cmd = StringIO() + cmd.write('git log --format="%H %ct"') - return commits + if limit > 0: + cmd.write(f' -{limit}') + code, output = system.execute(cmd.getvalue(), cwd=proj_dir, shell=True) -def get_current_commit(repo_path: str) -> Optional[str]: - code, output = system.execute(cmd='git log -1 --format=%H', shell=True, cwd=repo_path) - - if code == 0: - for line in output.strip().split('\n'): + if code == 0 and output: + commits = [] + for line in output.split('\n'): line_strip = line.strip() if line_strip: - return line_strip - + line_split = line_strip.split(' ', 1) -def log_shas_and_timestamps(repo_path: str) -> Optional[List[Tuple[str, int]]]: - code, output = system.execute(cmd='git log --format="%H %at"', shell=True, cwd=repo_path) + if len(line_split) == 2: + commit_sha = line_split[0].strip() + try: + commit_date = int(line_split[1].strip()) + except ValueError: + commit_date = None - if code == 0: - logs = [] - for line in output.strip().split('\n'): - line_strip = line.strip() + if logger: + logger.error(f"Could not parse commit date {line_split[1]}") - if line_strip: - line_split = line_strip.split(' ') - logs.append((line_split[0].strip(), int(line_split[1].strip()))) + commits.append((commit_sha, commit_date)) - return logs + return commits def clone(url: str, target_dir: Optional[str], depth: int = -1, custom_user: Optional[str] = None) -> SimpleProcess: diff --git a/bauh/gems/arch/pacman.py b/bauh/gems/arch/pacman.py index 5f27b1bb..ffe83c18 100644 --- a/bauh/gems/arch/pacman.py +++ b/bauh/gems/arch/pacman.py @@ -2,8 +2,10 @@ import os import re import shutil +import traceback +from io import StringIO from threading import Thread -from typing import List, Set, Tuple, Dict, Iterable, Optional +from typing import List, Set, Tuple, Dict, Iterable, Optional, Any from colorama import Fore @@ -16,7 +18,7 @@ RE_OPTDEPS = re.compile(r'[\w._\-]+\s*:') RE_DEP_NOTFOUND = re.compile(r'error:.+\'(.+)\'') RE_DEP_OPERATORS = re.compile(r'[<>=]') -RE_INSTALLED_FIELDS = re.compile(r'(Name|Description|Version|Install Date|Validated By)\s*:\s*(.+)') +RE_REPOSITORY_FIELDS = re.compile(r'(Repository|Name|Description|Version|Install Date|Validated By)\s*:\s*(.+)') RE_INSTALLED_SIZE = re.compile(r'Installed Size\s*:\s*([0-9,.]+)\s(\w+)\n?', re.IGNORECASE) RE_DOWNLOAD_SIZE = re.compile(r'Download Size\s*:\s*([0-9,.]+)\s(\w+)\n?', re.IGNORECASE) RE_UPDATE_REQUIRED_FIELDS = re.compile(r'(\bProvides\b|\bInstalled Size\b|\bConflicts With\b)\s*:\s(.+)\n') @@ -91,23 +93,36 @@ def check_installed(pkg: str) -> bool: return bool(res) -def _fill_ignored(res: dict): - res['pkgs'] = list_ignored_packages() +def fill_ignored_packages(output: Set[str]): + output.update(list_ignored_packages()) -def map_installed(names: Optional[Iterable[str]] = None) -> Dict[str, Dict[str, str]]: - ignored = {} - thread_ignored = Thread(target=_fill_ignored, args=(ignored,), daemon=True) - thread_ignored.start() +def map_packages(names: Optional[Iterable[str]] = None, remote: bool = False, signed: bool = True, + not_signed: bool = True, skip_ignored: bool = False) -> Dict[str, Dict[str, Dict[str, str]]]: + if not signed and not not_signed: + return {} + + ignored, thread_ignored = None, None + if not skip_ignored: + ignored = set() + thread_ignored = Thread(target=fill_ignored_packages, args=(ignored,), daemon=True) + thread_ignored.start() + + env = system.gen_env() + env['LC_TIME'] = '' - allinfo = run_cmd('pacman -Qi{}'.format(' ' + ' '.join(names) if names else ''), print_error=False) + code, allinfo = system.execute(f"pacman -{'S' if remote else 'Q'}i {' '.join(names) if names else ''}", + shell=True, custom_env=env) pkgs = {'signed': {}, 'not_signed': {}} - current_pkg = {} - if allinfo: - for idx, field_tuple in enumerate(RE_INSTALLED_FIELDS.findall(allinfo)): - if field_tuple[0].startswith('N'): + if code == 0 and allinfo: + current_pkg = {} + + for idx, field_tuple in enumerate(RE_REPOSITORY_FIELDS.findall(allinfo)): + if field_tuple[0].startswith('R'): + current_pkg['repository'] = field_tuple[1].strip() + elif field_tuple[0].startswith('N'): current_pkg['name'] = field_tuple[1].strip() elif field_tuple[0].startswith('Ve'): current_pkg['version'] = field_tuple[1].strip() @@ -116,25 +131,25 @@ def map_installed(names: Optional[Iterable[str]] = None) -> Dict[str, Dict[str, elif field_tuple[0].startswith('I'): current_pkg['install_date'] = field_tuple[1].strip() elif field_tuple[0].startswith('Va'): - if field_tuple[1].strip().lower() == 'none': + if not_signed and field_tuple[1].strip().lower() == 'none': pkgs['not_signed'][current_pkg['name']] = current_pkg del current_pkg['name'] - else: + elif signed: pkgs['signed'][current_pkg['name']] = current_pkg del current_pkg['name'] current_pkg = {} - if pkgs['signed'] or pkgs['not_signed']: + if thread_ignored and (pkgs['signed'] or pkgs['not_signed']): thread_ignored.join() - if ignored['pkgs']: + if ignored: to_del = set() for key in ('signed', 'not_signed'): if pkgs.get(key): for pkg in pkgs[key].keys(): - if pkg in ignored['pkgs']: + if pkg in ignored: to_del.add(pkg) for pkg in to_del: @@ -218,20 +233,23 @@ def sign_key(key: str, root_password: Optional[str]) -> SystemProcess: def list_ignored_packages(config_path: str = '/etc/pacman.conf') -> Set[str]: - pacman_conf = new_subprocess(['cat', config_path]) - ignored = set() - grep = new_subprocess(['grep', '-Eo', r'\s*#*\s*ignorepkg\s*=\s*.+'], stdin=pacman_conf.stdout) - for o in grep.stdout: - if o: - line = o.decode().strip() + try: + pacman_conf = new_subprocess(['cat', config_path]) + grep = new_subprocess(['grep', '-Eo', r'\s*#*\s*ignorepkg\s*=\s*.+'], stdin=pacman_conf.stdout) + for o in grep.stdout: + if o: + line = o.decode().strip() - if not line.startswith('#'): - ignored.add(line.split('=')[1].strip()) + if not line.startswith('#'): + ignored.add(line.split('=')[1].strip()) - pacman_conf.terminate() - grep.terminate() - return ignored + pacman_conf.terminate() + grep.terminate() + return ignored + except: + traceback.print_exc() + return ignored def check_missing(names: Set[str]) -> Set[str]: @@ -514,7 +532,7 @@ def fill_provided_map(key: str, val: str, output: dict): current_val.add(val) -def map_provided(remote: bool = False, pkgs: Iterable[str] = None) -> Dict[str, Set[str]]: +def map_provided(remote: bool = False, pkgs: Iterable[str] = None) -> Optional[Dict[str, Set[str]]]: output = run_cmd('pacman -{}i {}'.format('S' if remote else 'Q', ' '.join(pkgs) if pkgs else '')) if output: @@ -982,8 +1000,15 @@ def is_snapd_installed() -> bool: return bool(run_cmd('pacman -Qq snapd', print_error=False)) -def list_hard_requirements(name: str, logger: Optional[logging.Logger] = None) -> Optional[Set[str]]: - code, output = system.execute('pacman -Rc {} --print-format=%n'.format(name), shell=True) +def list_hard_requirements(name: str, logger: Optional[logging.Logger] = None, + assume_installed: Optional[Set[str]] = None) -> Optional[Set[str]]: + cmd = StringIO() + cmd.write(f'pacman -Rc {name} --print-format=%n ') + + if assume_installed: + cmd.write(' '.join(f'--assume-installed={provider}' for provider in assume_installed)) + + code, output = system.execute(cmd.getvalue(), shell=True) if code != 0: if 'HoldPkg' in output: @@ -1029,3 +1054,24 @@ def find_one_match(name: str) -> Optional[str]: if matches and len(matches) == 1: return matches[0] + + +def map_available_packages() -> Optional[Dict[str, Any]]: + output = run_cmd('pacman -Sl') + + if output: + res = dict() + for line in output.split('\n'): + line_strip = line.strip() + + if line_strip: + package_data = line.split(' ') + + if len(package_data) >= 3: + pkgname = package_data[1].strip() + + if pkgname: + res[pkgname] = {'v': package_data[2].strip(), + 'r': package_data[0].strip(), + 'i': len(package_data) == 4 and 'installed' in package_data[3]} + return res diff --git a/bauh/gems/arch/resources/locale/ca b/bauh/gems/arch/resources/locale/ca index 2a5dda49..d1995e91 100644 --- a/bauh/gems/arch/resources/locale/ca +++ b/bauh/gems/arch/resources/locale/ca @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Uninstall optional dependencies arch.config.suggest_optdep_uninstall.tip=If the optional dependencies associated with uninstalled packages should be suggested for uninstallation. Only the optional dependencies that are not dependencies of other packages will be suggested. arch.config.suggest_unneeded_uninstall=Uninstall unneeded dependencies arch.config.suggest_unneeded_uninstall.tip=If the dependencies apparently no longer necessary associated with the uninstalled packages should be suggested for uninstallation. When this property is enabled it automatically disables the property {}. +arch.config.suggestions_exp=Suggestions expiration +arch.config.suggestions_exp.tip=It defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. arch.config.sync_dbs=Synchronize packages databases arch.config.sync_dbs.tip=Synchronizes the package databases once a day before the first package installation, upgrade or downgrade. This option helps to prevent errors during these operations. arch.config.sync_dbs_start.tip=Synchronizes the package databases during the initialization once a day diff --git a/bauh/gems/arch/resources/locale/de b/bauh/gems/arch/resources/locale/de index a286c61e..577f3181 100644 --- a/bauh/gems/arch/resources/locale/de +++ b/bauh/gems/arch/resources/locale/de @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Uninstall optional dependencies arch.config.suggest_optdep_uninstall.tip=If the optional dependencies associated with uninstalled packages should be suggested for uninstallation. Only the optional dependencies that are not dependencies of other packages will be suggested. arch.config.suggest_unneeded_uninstall=Uninstall unneeded dependencies arch.config.suggest_unneeded_uninstall.tip=If the dependencies apparently no longer necessary associated with the uninstalled packages should be suggested for uninstallation. When this property is enabled it automatically disables the property {}. +arch.config.suggestions_exp=Suggestions expiration +arch.config.suggestions_exp.tip=It defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. arch.config.sync_dbs=Synchronize packages databases arch.config.sync_dbs.tip=Synchronizes the package databases once a day before the first package installation, upgrade or downgrade. This option helps to prevent errors during these operations. arch.config.sync_dbs_start.tip=Synchronizes the package databases during the initialization once a day diff --git a/bauh/gems/arch/resources/locale/en b/bauh/gems/arch/resources/locale/en index a81625e8..d68de2e8 100644 --- a/bauh/gems/arch/resources/locale/en +++ b/bauh/gems/arch/resources/locale/en @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Uninstall optional dependencies arch.config.suggest_optdep_uninstall.tip=If the optional dependencies associated with uninstalled packages should be suggested for uninstallation. Only the optional dependencies that are not dependencies of other packages will be suggested. arch.config.suggest_unneeded_uninstall=Uninstall unneeded dependencies arch.config.suggest_unneeded_uninstall.tip=If the dependencies apparently no longer necessary associated with the uninstalled packages should be suggested for uninstallation. When this property is enabled it automatically disables the property {}. +arch.config.suggestions_exp=Suggestions expiration +arch.config.suggestions_exp.tip=It defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. arch.config.sync_dbs=Synchronize packages databases arch.config.sync_dbs.tip=Synchronizes the package databases once a day before the first package installation, upgrade or downgrade. This option helps to prevent errors during these operations. arch.config.sync_dbs_start.tip=Synchronizes the package databases during the initialization once a day diff --git a/bauh/gems/arch/resources/locale/es b/bauh/gems/arch/resources/locale/es index 751e2224..4a9faebd 100644 --- a/bauh/gems/arch/resources/locale/es +++ b/bauh/gems/arch/resources/locale/es @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Desinstalar dependencias opcionales arch.config.suggest_optdep_uninstall.tip=Si las dependencias opcionales asociadas con los paquetes desinstalados deben se sugeridas para desinstalación. Solo se sugerirán las dependencias opcionales que no sean dependencias de otros paquetes. arch.config.suggest_unneeded_uninstall=Desinstalar las dependencias innecesarias arch.config.suggest_unneeded_uninstall.tip=Si las dependencias aparentemente ya no necesarias asociadas con los paquetes desinstalados deben ser sugeridas para desinstalación. Cuando esta propiedad está habilitada, automáticamente deshabilita la propiedad {}. +arch.config.suggestions_exp=Expiración de sugerencias +arch.config.suggestions_exp.tip=Define el período (en horas) en el que la sugerencias almacenadas en disco seran consideradas actualizadas. Use 0 si desea siempre actualizarlas. arch.config.sync_dbs=Sincronizar las bases de paquetes arch.config.sync_dbs.tip=Sincroniza las bases de paquetes una vez al día antes de la primera instalación, actualización o reversión de paquete. Esta opción ayuda a prevenir errores durante estas operaciones. arch.config.sync_dbs_start.tip=Sincroniza las bases de paquetes durante la inicialización una vez al día diff --git a/bauh/gems/arch/resources/locale/fr b/bauh/gems/arch/resources/locale/fr index 6e22de0d..d9e51342 100644 --- a/bauh/gems/arch/resources/locale/fr +++ b/bauh/gems/arch/resources/locale/fr @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Désinstaller les dépendances optionnelles arch.config.suggest_optdep_uninstall.tip=Si les dépendances optionnelles liées au paquets désinstallés dévraient être suggérés pour désinstallation. Seules les dépendances n'étant pas liées à d'autres paquets seront suggérées. arch.config.suggest_unneeded_uninstall=Désinstaller les dépendances inutiles arch.config.suggest_unneeded_uninstall.tip=Si les dépendances apparemment inutiles liées au paquets désinstallés dévraient être suggérés pour désinstallation. Activer cette propriété désactive automatiquement {}. +arch.config.suggestions_exp=Suggestions expiration +arch.config.suggestions_exp.tip=It defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. arch.config.sync_dbs=Synchroniser les bases de données des paquets arch.config.sync_dbs.tip=Synchronise les bases de données du paquet une fois par jour avant sa première installation, mise à jour ou downgrade. Cette option aide à éviter les erreurs durant ces opérations. arch.config.sync_dbs_start.tip=Synchronise les bases de données du paquet durant l'initialisation journalière diff --git a/bauh/gems/arch/resources/locale/it b/bauh/gems/arch/resources/locale/it index 1700da7b..7dd9e9f7 100644 --- a/bauh/gems/arch/resources/locale/it +++ b/bauh/gems/arch/resources/locale/it @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Uninstall optional dependencies arch.config.suggest_optdep_uninstall.tip=If the optional dependencies associated with uninstalled packages should be suggested for uninstallation. Only the optional dependencies that are not dependencies of other packages will be suggested. arch.config.suggest_unneeded_uninstall=Uninstall unneeded dependencies arch.config.suggest_unneeded_uninstall.tip=If the dependencies apparently no longer necessary associated with the uninstalled packages should be suggested for uninstallation. When this property is enabled it automatically disables the property {}. +arch.config.suggestions_exp=Suggestions expiration +arch.config.suggestions_exp.tip=It defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. arch.config.sync_dbs=Synchronize packages databases arch.config.sync_dbs.tip=Synchronizes the package databases once a day before the first package installation, upgrade or downgrade. This option helps to prevent errors during these operations. arch.config.sync_dbs_start.tip=Synchronizes the package databases during the initialization once a day diff --git a/bauh/gems/arch/resources/locale/pt b/bauh/gems/arch/resources/locale/pt index f1c3d537..9823ae76 100644 --- a/bauh/gems/arch/resources/locale/pt +++ b/bauh/gems/arch/resources/locale/pt @@ -87,6 +87,8 @@ arch.config.suggest_optdep_uninstall=Desinstalar dependências opcionais arch.config.suggest_optdep_uninstall.tip=Se as dependências opcionais associadas a pacotes desinstalados devem ser sugeridas para desinstalação. Somente as dependências opcionais que não sejam dependências de outros pacotes serão sugeridas. arch.config.suggest_unneeded_uninstall=Desinstalar dependências desnecessárias arch.config.suggest_unneeded_uninstall.tip=Se as dependências aparentemente não mais necessárias associadas aos pacotes desinstalados devem ser sugeridas para desinstalação. Quando essa proprieda está habilitada automaticamente desabilita a propriedade {}. +arch.config.suggestions_exp=Validade de sugestões +arch.config.suggestions_exp.tip=Define o período (em horas) no qual as sugestões armazenadas em disco serão consideradas atualizadas. Use 0 se quiser que elas sempre sejam atualizadas. arch.config.sync_dbs=Sincronizar bases de pacotes arch.config.sync_dbs.tip=Sincroniza as bases de pacotes uma vez ao dia antes da primeira instalação, atualização ou reversão de pacote. Essa opção ajuda a evitar erros durante essa operações. arch.config.sync_dbs_start.tip=Sincroniza as bases de pacotes durante a inicialização uma vez ao dia diff --git a/bauh/gems/arch/resources/locale/ru b/bauh/gems/arch/resources/locale/ru index 6ae73ab6..5d32fd16 100644 --- a/bauh/gems/arch/resources/locale/ru +++ b/bauh/gems/arch/resources/locale/ru @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Uninstall optional dependencies arch.config.suggest_optdep_uninstall.tip=If the optional dependencies associated with uninstalled packages should be suggested for uninstallation. Only the optional dependencies that are not dependencies of other packages will be suggested. arch.config.suggest_unneeded_uninstall=Uninstall unneeded dependencies arch.config.suggest_unneeded_uninstall.tip=If the dependencies apparently no longer necessary associated with the uninstalled packages should be suggested for uninstallation. When this property is enabled it automatically disables the property {}. +arch.config.suggestions_exp=Suggestions expiration +arch.config.suggestions_exp.tip=It defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. arch.config.sync_dbs=Синхронизация баз данных пакетов arch.config.sync_dbs.tip=Синхронизируйте базы данных пакетов один раз в день перед первой установкой, обновлением или понижением версии пакета. Этот параметр помогает предотвратить ошибки во время этих операций. arch.config.sync_dbs_start.tip=Синхронизирует базы данных пакетов во время инициализации один раз в день diff --git a/bauh/gems/arch/resources/locale/tr b/bauh/gems/arch/resources/locale/tr index b193ff2e..9a943128 100644 --- a/bauh/gems/arch/resources/locale/tr +++ b/bauh/gems/arch/resources/locale/tr @@ -88,6 +88,8 @@ arch.config.suggest_optdep_uninstall=Uninstall optional dependencies arch.config.suggest_optdep_uninstall.tip=If the optional dependencies associated with uninstalled packages should be suggested for uninstallation. Only the optional dependencies that are not dependencies of other packages will be suggested. arch.config.suggest_unneeded_uninstall=Uninstall unneeded dependencies arch.config.suggest_unneeded_uninstall.tip=If the dependencies apparently no longer necessary associated with the uninstalled packages should be suggested for uninstallation. When this property is enabled it automatically disables the property {}. +arch.config.suggestions_exp=Suggestions expiration +arch.config.suggestions_exp.tip=It defines the period (in hours) in which the suggestions stored in disc will be considered up to date. Use 0 if you always want to update them. arch.config.sync_dbs=Paket veritabanlarını senkronize et arch.config.sync_dbs.tip=İlk veritabanını kurmadan, yükseltmeden veya indirmeden önce paket veritabanlarını günde bir kez senkronize eder. Bu seçenek, bu işlemler sırasında hataların önlenmesine yardımcı olur. arch.config.sync_dbs_start.tip=Başlatma sırasında paket veritabanlarını günde bir kez senkronize eder diff --git a/bauh/gems/arch/suggestions.py b/bauh/gems/arch/suggestions.py new file mode 100644 index 00000000..b87f9acf --- /dev/null +++ b/bauh/gems/arch/suggestions.py @@ -0,0 +1,209 @@ +import os +import traceback +from datetime import datetime, timedelta +from logging import Logger +from pathlib import Path +from threading import Thread +from typing import Optional, Dict + +from bauh.api.abstract.handler import TaskManager +from bauh.api.abstract.model import SuggestionPriority +from bauh.api.http import HttpClient +from bauh.commons.boot import CreateConfigFile +from bauh.gems.arch import ARCH_CACHE_DIR, get_icon_path +from bauh.view.util.translation import I18n +from bauh.commons.suggestions import parse + + +class RepositorySuggestionsDownloader(Thread): + + _file_suggestions: Optional[str] = None + _file_suggestions_ts: Optional[str] = None + + @classmethod + def file_suggestions(cls) -> str: + if cls._file_suggestions is None: + cls._file_suggestions = f'{ARCH_CACHE_DIR}/suggestions.txt' + + return cls._file_suggestions + + @classmethod + def file_suggestions_timestamp(cls) -> str: + if cls._file_suggestions_ts is None: + cls._file_suggestions_ts = f'{cls.file_suggestions()}.ts' + + return cls._file_suggestions_ts + + def __init__(self, logger: Logger, http_client: HttpClient, i18n: I18n, + create_config: Optional[CreateConfigFile] = None, file_url: Optional[str] = None): + super(RepositorySuggestionsDownloader, self).__init__() + self._log = logger + self.i18n = i18n + self.http_client = http_client + self._taskman: Optional[TaskManager] = None + self.create_config = create_config + self._file_url = file_url if file_url else 'https://raw.githubusercontent.com/vinifmor/bauh-files' \ + '/master/arch/suggestions.txt' + self.task_id = 'arch.suggs' + + def register_task(self, taskman: Optional[TaskManager]): + self._taskman = taskman + if taskman: + self._taskman.register_task(id_=self.task_id, label=self.i18n['task.download_suggestions'], + icon_path=get_icon_path()) + + @property + def taskman(self) -> TaskManager: + if self._taskman is None: + self._taskman = TaskManager() + + return self._taskman + + def should_download(self, arch_config: dict, only_positive_exp: bool = False) -> bool: + if not self._file_url: + self._log.error("No Arch suggestions file URL defined") + return False + + if self._file_url.startswith('/'): + return False + + try: + exp_hours = int(arch_config['suggestions_exp']) + except ValueError: + self._log.error(f"The Arch configuration property 'suggestions_exp' has a non int value set: " + f"{arch_config['suggestions']['expiration']}") + return not only_positive_exp + + if exp_hours <= 0: + self._log.info("Suggestions cache is disabled") + return not only_positive_exp + + if not os.path.exists(self.file_suggestions()): + self._log.info(f"'{self.file_suggestions()}' not found. It must be downloaded") + return True + + if not os.path.exists(self.file_suggestions_timestamp()): + self._log.info(f"'{self.file_suggestions()}' not found. The suggestions file must be downloaded.") + return True + + with open(self.file_suggestions_timestamp()) as f: + timestamp_str = f.read() + + try: + suggestions_timestamp = datetime.fromtimestamp(float(timestamp_str)) + except: + self._log.error(f'Could not parse the Arch cached suggestions timestamp: {timestamp_str}') + traceback.print_exc() + return True + + update = suggestions_timestamp + timedelta(hours=exp_hours) <= datetime.utcnow() + + if update: + self._log.info("The cached suggestions file is no longer valid") + else: + self._log.info("The cached suggestions file is up-to-date") + + return update + + def _save(self, text: str, timestamp: float): + self._log.info(f"Caching suggestions to '{self.file_suggestions()}'") + + cache_dir = os.path.dirname(self.file_suggestions()) + + try: + Path(cache_dir).mkdir(parents=True, exist_ok=True) + cache_dir_ok = True + except OSError: + self._log.error(f"Could not create cache directory '{cache_dir}'") + traceback.print_exc() + cache_dir_ok = False + + if cache_dir_ok: + try: + with open(self.file_suggestions(), 'w+') as f: + f.write(text) + except: + self._log.error(f"An exception happened while writing the file '{self.file_suggestions()}'") + traceback.print_exc() + + try: + with open(self.file_suggestions_timestamp(), 'w+') as f: + f.write(str(timestamp)) + except: + self._log.error(f"An exception happened while writing the file '{self.file_suggestions_timestamp()}'") + traceback.print_exc() + + def read_cached(self, custom_file: Optional[str] = None) -> Optional[Dict[str, SuggestionPriority]]: + if custom_file: + file_path, log_ref = custom_file, 'local' + else: + file_path, log_ref = self.file_suggestions(), 'cached' + + self._log.info(f"Reading {log_ref} Arch suggestions file '{file_path}'") + + try: + with open(file_path) as f: + sugs_str = f.read() + except FileNotFoundError: + self._log.warning(f"{log_ref.capitalize()} suggestions file does not exist ({file_path})") + return + + if not sugs_str: + self._log.warning(f"{log_ref.capitalize()} suggestions file '{file_path}' is empty") + return + + return parse(sugs_str, self._log, 'Arch') + + def download(self) -> Optional[Dict[str, SuggestionPriority]]: + self.taskman.update_progress(self.task_id, progress=1, substatus=None) + + self._log.info(f"Downloading suggestions from {self._file_url}") + res = self.http_client.get(self._file_url) + + suggestions = None + if res.status_code == 200 and res.text: + self.taskman.update_progress(self.task_id, progress=50, substatus=None) + suggestions = parse(res.text, self._log, 'Arch') + + if suggestions: + self._save(text=res.text, timestamp=datetime.utcnow().timestamp()) + else: + self._log.warning(f"Could not parse any Arch suggestion from {self._file_suggestions_ts}") + else: + self._log.warning(f"Could not retrieve Arch suggestions. " + f"Response (status={res.status_code}, text={res.text})") + + self.taskman.update_progress(self.task_id, progress=100, substatus=None) + self.taskman.finish_task(self.task_id) + return suggestions + + def read(self, arch_config: dict) -> Optional[Dict[str, int]]: + if self._file_url: + if self.is_custom_local_file_mapped(): + return self.read_cached(custom_file=self._file_url) + + if self.should_download(arch_config=arch_config): + return self.download() + + return self.read_cached() + + def is_custom_local_file_mapped(self) -> bool: + return self._file_url and self._file_url.startswith('/') + + def run(self): + if self.create_config: + if self.create_config.is_alive(): + self.taskman.update_progress(self.task_id, 0, + self.i18n['task.waiting_task'].format(self.create_config.task_name)) + self.create_config.join() + + if not self.should_download(arch_config=self.create_config.config, only_positive_exp=False): + self.taskman.update_progress(self.task_id, 100, self.i18n['task.canceled']) + self.taskman.finish_task(self.task_id) + return + + self.download() + else: + self._log.error(f"No {CreateConfigFile.__class__.__name__} instance set. Aborting..") + self.taskman.update_progress(self.task_id, 100, self.i18n['error']) + self.taskman.finish_task(self.task_id) diff --git a/bauh/gems/debian/controller.py b/bauh/gems/debian/controller.py index 53f2332a..3e8dbc49 100644 --- a/bauh/gems/debian/controller.py +++ b/bauh/gems/debian/controller.py @@ -16,7 +16,7 @@ from bauh.api.abstract.model import SoftwarePackage, PackageSuggestion, PackageUpdate, PackageHistory, \ CustomSoftwareAction from bauh.api.abstract.view import TextInputComponent, PanelComponent, FormComponent, MessageType, \ - SingleSelectComponent, InputOption, SelectViewType + SingleSelectComponent, InputOption, SelectViewType, ViewComponentAlignment from bauh.api.paths import CONFIG_DIR from bauh.commons.html import bold from bauh.commons.system import ProcessHandler @@ -508,7 +508,7 @@ def prepare(self, task_manager: Optional[TaskManager], root_password: Optional[s root_password=root_password, aptitude=self.aptitude) sync_pkgs.start() - if DebianSuggestionsDownloader.should_download(deb_config, self._log, only_positive_exp=True): + if self.suggestions_downloader.should_download(deb_config, only_positive_exp=True): self.suggestions_downloader.register_task(task_manager) self.suggestions_downloader.start() @@ -603,7 +603,6 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: options=purge_opts, default_option=purge_current, type_=SelectViewType.RADIO, - max_width=200, max_per_line=2) sources_app = config_.get('pkg_sources.app') @@ -623,8 +622,8 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: tooltip=source_auto_tip, options=source_opts, default_option=next(o for o in source_opts if o.value == sources_app), - type_=SelectViewType.COMBO, - max_width=200) + alignment=ViewComponentAlignment.CENTER, + type_=SelectViewType.COMBO) try: app_cache_exp = int(config_.get('index_apps.exp', 0)) @@ -636,8 +635,7 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: ti_index_apps_exp = TextInputComponent(id_='index_apps.exp', label=self._i18n['debian.config.index_apps.exp'], tooltip=self._i18n['debian.config.index_apps.exp.tip'], - value=str(app_cache_exp), only_int=True, - max_width=60) + value=str(app_cache_exp), only_int=True) try: sync_pkgs_time = int(config_.get('sync_pkgs.time', 0)) @@ -649,8 +647,7 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: ti_sync_pkgs = TextInputComponent(id_='sync_pkgs.time', label=self._i18n['debian.config.sync_pkgs.time'], tooltip=self._i18n['debian.config.sync_pkgs.time.tip'], - value=str(sync_pkgs_time), only_int=True, - max_width=60) + value=str(sync_pkgs_time), only_int=True) try: suggestions_exp = int(config_.get('suggestions.exp', 0)) @@ -662,8 +659,7 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: ti_suggestions_exp = TextInputComponent(id_='suggestions.exp', label=self._i18n['debian.config.suggestions.exp'], tooltip=self._i18n['debian.config.suggestions.exp.tip'], - value=str(suggestions_exp), only_int=True, - max_width=60) + value=str(suggestions_exp), only_int=True) panel = PanelComponent([FormComponent([input_sources, sel_purge, ti_sync_pkgs, ti_index_apps_exp, ti_suggestions_exp])]) @@ -891,8 +887,13 @@ def file_ignored_updates(self) -> str: @property def suggestions_downloader(self) -> DebianSuggestionsDownloader: if not self._suggestions_downloader: + file_url = self.context.get_suggestion_url(self.__module__) self._suggestions_downloader = DebianSuggestionsDownloader(i18n=self._i18n, logger=self._log, - http_client=self.context.http_client) + http_client=self.context.http_client, + file_url=file_url) + + if self._suggestions_downloader.is_local_suggestions_file(): + self._log.info(f"Local Debian suggestions file mapped: {file_url}") return self._suggestions_downloader diff --git a/bauh/gems/debian/suggestions.py b/bauh/gems/debian/suggestions.py index da361ebf..878a6a54 100644 --- a/bauh/gems/debian/suggestions.py +++ b/bauh/gems/debian/suggestions.py @@ -9,6 +9,7 @@ from bauh.api.abstract.handler import TaskManager from bauh.api.abstract.model import SuggestionPriority from bauh.api.http import HttpClient +from bauh.commons.suggestions import parse from bauh.gems.debian import DEBIAN_ICON_PATH, DEBIAN_CACHE_DIR from bauh.view.util.translation import I18n @@ -33,20 +34,18 @@ def file_suggestions_timestamp(cls) -> str: return cls._file_suggestions_ts - @classmethod - def url_suggestions(cls) -> str: - if cls._url_suggestions is None: - cls._url_suggestions = 'https://raw.githubusercontent.com/vinifmor/bauh-files' \ - '/master/debian/suggestions.txt' - - return cls._url_suggestions - - def __init__(self, logger: Logger, http_client: HttpClient, i18n: I18n): + def __init__(self, logger: Logger, http_client: HttpClient, i18n: I18n, file_url: Optional[str]): super(DebianSuggestionsDownloader, self).__init__() self._log = logger self.i18n = i18n self.http_client = http_client self._taskman: Optional[TaskManager] = None + + if file_url: + self._file_url = file_url + else: + self._file_url = 'https://raw.githubusercontent.com/vinifmor/bauh-files/master/debian/suggestions_v1.txt' + self.task_id = 'debian.suggs' def register_task(self, taskman: Optional[TaskManager]): @@ -55,6 +54,9 @@ def register_task(self, taskman: Optional[TaskManager]): self._taskman.register_task(id_=self.task_id, label=self.i18n['task.download_suggestions'], icon_path=DEBIAN_ICON_PATH) + def is_local_suggestions_file(self) -> bool: + return self._file_url and self._file_url.startswith('/') + @property def taskman(self) -> TaskManager: if self._taskman is None: @@ -62,34 +64,40 @@ def taskman(self) -> TaskManager: return self._taskman - @classmethod - def should_download(cls, debian_config: dict, logger: Logger, only_positive_exp: bool = False) -> bool: + def should_download(self, debian_config: dict, only_positive_exp: bool = False) -> bool: + if not self._file_url: + self._log.error("No Debian suggestions file URL defined") + return False + + if self.is_local_suggestions_file(): + return False + try: exp_hours = int(debian_config['suggestions.exp']) except ValueError: - logger.error(f"The Debian configuration property 'suggestions.expiration' has a non int value set: " - f"{debian_config['suggestions']['expiration']}") + self._log.error(f"The Debian configuration property 'suggestions.expiration' has a non int value set: " + f"{debian_config['suggestions']['expiration']}") return not only_positive_exp if exp_hours <= 0: - logger.info("Suggestions cache is disabled") + self._log.info("Suggestions cache is disabled") return not only_positive_exp - if not os.path.exists(cls.file_suggestions()): - logger.info(f"'{cls.file_suggestions()}' not found. It must be downloaded") + if not os.path.exists(self.file_suggestions()): + self._log.info(f"'{self.file_suggestions()}' not found. It must be downloaded") return True - if not os.path.exists(cls.file_suggestions()): - logger.info(f"'{cls.file_suggestions()}' not found. The suggestions file must be downloaded.") + if not os.path.exists(self.file_suggestions()): + self._log.info(f"'{self.file_suggestions()}' not found. The suggestions file must be downloaded.") return True - with open(cls.file_suggestions_timestamp()) as f: + with open(self.file_suggestions_timestamp()) as f: timestamp_str = f.read() try: suggestions_timestamp = datetime.fromtimestamp(float(timestamp_str)) except: - logger.error(f'Could not parse the Debian cached suggestions timestamp: {timestamp_str}') + self._log.error(f'Could not parse the Debian cached suggestions timestamp: {timestamp_str}') traceback.print_exc() return True @@ -97,6 +105,13 @@ def should_download(cls, debian_config: dict, logger: Logger, only_positive_exp: return update def _save(self, text: str, timestamp: float): + if not self._file_url: + self._log.error("No Debian suggestions file URL defined") + return + + if self.is_local_suggestions_file(): + return False + self._log.info(f"Caching suggestions to '{self.file_suggestions()}'") cache_dir = os.path.dirname(self.file_suggestions()) @@ -124,51 +139,49 @@ def _save(self, text: str, timestamp: float): self._log.error(f"An exception happened while writing the file '{self.file_suggestions_timestamp()}'") traceback.print_exc() - def parse_suggestions(self, suggestions_str: str) -> Dict[str, SuggestionPriority]: - output = dict() - for line in suggestions_str.split('\n'): - clean_line = line.strip() - - if clean_line: - line_split = clean_line.split(':', 1) - - if len(line_split) == 2: - try: - prio = int(line_split[0]) - except ValueError: - self._log.warning(f"Could not parse Debian package suggestion: {line}") - continue - - output[line_split[1]] = SuggestionPriority(prio) + def read_cached(self) -> Optional[Dict[str, SuggestionPriority]]: + if not self._file_url: + self._log.error("No Debian suggestions file URL defined") + return - return output + if self.is_local_suggestions_file(): + file_path, log_ref = self._file_url, 'local' + else: + file_path, log_ref = self.file_suggestions(), 'cached' - def read_cached(self) -> Optional[Dict[str, SuggestionPriority]]: - self._log.info(f"Reading cached suggestions file '{self.file_suggestions()}'") + self._log.info(f"Reading {log_ref} suggestions file {file_path}") try: - with open(self.file_suggestions()) as f: + with open(file_path) as f: sugs_str = f.read() except FileNotFoundError: - self._log.warning(f"Cached suggestions file does not exist ({self.file_suggestions()})") + self._log.warning(f"The {log_ref} suggestions file does not exist ({file_path})") + return + except OSError: + self._log.warning(f"Could not read from the {log_ref} suggestions file ({file_path})") + traceback.print_exc() return if not sugs_str: - self._log.warning(f"Cached suggestions file '{self.file_suggestions()}' is empty") + self._log.warning(f"The {log_ref} suggestions file '{file_path}' is empty") return - return self.parse_suggestions(sugs_str) + return parse(sugs_str, self._log, 'Debian') def download(self) -> Optional[Dict[str, SuggestionPriority]]: + if not self._file_url: + self._log.error("No Debian suggestions file URL defined") + return + self.taskman.update_progress(self.task_id, progress=1, substatus=None) - self._log.info(f"Downloading suggestions from {self.url_suggestions()}") - res = self.http_client.get(self.url_suggestions()) + self._log.info(f"Downloading Debian suggestions from {self._file_url}") + res = self.http_client.get(self._file_url) suggestions = None if res.status_code == 200 and res.text: self.taskman.update_progress(self.task_id, progress=50, substatus=None) - suggestions = self.parse_suggestions(res.text) + suggestions = parse(res.text, self._log, 'Debian') if suggestions: self._save(text=res.text, timestamp=datetime.utcnow().timestamp()) @@ -183,10 +196,14 @@ def download(self) -> Optional[Dict[str, SuggestionPriority]]: return suggestions def read(self, debian_config: dict) -> Optional[Dict[str, int]]: - if self.should_download(debian_config=debian_config, logger=self._log): - return self.download() + if not self._file_url: + self._log.error("No Debian suggestions file URL defined") + return + + if self.is_local_suggestions_file() or not self.should_download(debian_config=debian_config): + return self.read_cached() - return self.read_cached() + return self.download() def run(self): self.download() diff --git a/bauh/gems/flatpak/__init__.py b/bauh/gems/flatpak/__init__.py index a775678b..a577ba88 100644 --- a/bauh/gems/flatpak/__init__.py +++ b/bauh/gems/flatpak/__init__.py @@ -9,7 +9,6 @@ from bauh.commons import resource ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) -SUGGESTIONS_FILE = f'https://raw.githubusercontent.com/vinifmor/{__app_name__}-files/master/flatpak/suggestions.txt' CONFIG_FILE = f'{CONFIG_DIR}/flatpak.yml' FLATPAK_CONFIG_DIR = f'{CONFIG_DIR}/flatpak' UPDATES_IGNORED_FILE = f'{FLATPAK_CONFIG_DIR}/updates_ignored.txt' diff --git a/bauh/gems/flatpak/controller.py b/bauh/gems/flatpak/controller.py index 53dbaac8..a2cc9112 100644 --- a/bauh/gems/flatpak/controller.py +++ b/bauh/gems/flatpak/controller.py @@ -2,7 +2,7 @@ import re import traceback from datetime import datetime -from math import floor +from operator import attrgetter from pathlib import Path from threading import Thread from typing import List, Set, Type, Tuple, Optional, Generator, Dict @@ -15,13 +15,14 @@ from bauh.api.abstract.disk import DiskCacheLoader from bauh.api.abstract.handler import ProcessWatcher, TaskManager from bauh.api.abstract.model import PackageHistory, PackageUpdate, SoftwarePackage, PackageSuggestion, \ - SuggestionPriority, PackageStatus, CustomSoftwareAction + PackageStatus, CustomSoftwareAction, SuggestionPriority from bauh.api.abstract.view import MessageType, FormComponent, SingleSelectComponent, InputOption, SelectViewType, \ - PanelComponent + PanelComponent, ViewComponentAlignment +from bauh.commons import suggestions from bauh.commons.boot import CreateConfigFile from bauh.commons.html import strip_html, bold from bauh.commons.system import ProcessHandler -from bauh.gems.flatpak import flatpak, SUGGESTIONS_FILE, CONFIG_FILE, UPDATES_IGNORED_FILE, FLATPAK_CONFIG_DIR, \ +from bauh.gems.flatpak import flatpak, CONFIG_FILE, UPDATES_IGNORED_FILE, FLATPAK_CONFIG_DIR, \ EXPORTS_PATH, \ get_icon_path, VERSION_1_5, VERSION_1_2, VERSION_1_12 from bauh.gems.flatpak.config import FlatpakConfigManager @@ -47,6 +48,7 @@ def __init__(self, context: ApplicationContext): self.logger = context.logger self.configman = FlatpakConfigManager() self._action_full_update: Optional[CustomSoftwareAction] = None + self._suggestions_file_url: Optional[str] = None def get_managed_types(self) -> Set["type"]: return {FlatpakApplication} @@ -231,7 +233,7 @@ def read_installed(self, disk_loader: Optional[DiskCacheLoader], limit: int = -1 latest_version=ref_split[-1], runtime=True, installation=installation, - installed=True, + installed=False, update_component=True, update=True) @@ -297,12 +299,14 @@ def upgrade(self, requirements: UpgradeRequirements, root_password: Optional[str try: if req.pkg.update_component: + self.logger.info(f"Installing {req.pkg}") res, _ = ProcessHandler(watcher).handle_simple(flatpak.install(app_id=ref, installation=req.pkg.installation, origin=req.pkg.origin, version=flatpak_version)) else: + self.logger.info(f"Updating {req.pkg}") res, _ = ProcessHandler(watcher).handle_simple(flatpak.update(app_ref=ref, installation=req.pkg.installation, related=related, @@ -580,47 +584,99 @@ def list_updates(self, internet_available: bool) -> List[PackageUpdate]: def list_warnings(self, internet_available: bool) -> Optional[List[str]]: pass - def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[PackageSuggestion]]: - cli_version = flatpak.get_version() - res = [] + def _read_local_suggestions_file(self) -> Optional[str]: + try: + with open(self.suggestions_file_url) as f: + suggestions_str = f.read() + + return suggestions_str + except FileNotFoundError: + self.logger.error(f"Local Flatpak suggestions file not found: {self.suggestions_file_url}") + except OSError: + self.logger.error(f"Could not read local Flatpak suggestions file: {self.suggestions_file_url}") + traceback.print_exc() + + def _download_remote_suggestions_file(self) -> Optional[str]: + self.logger.info(f"Downloading the Flatpak suggestions from {self.suggestions_file_url}") + file = self.http_client.get(self.suggestions_file_url) + + if file: + return file.text + + def _fill_suggestion(self, appid: str, priority: SuggestionPriority, flatpak_version: Version, remote: str, + output: List[PackageSuggestion]): + app_json = flatpak.search(flatpak_version, appid, remote, app_id=True) + + if app_json: + model = PackageSuggestion(self._map_to_model(app_json[0], False, None)[0], priority) + self.suggestions_cache.add(appid, model) + output.append(model) + else: + self.logger.warning(f"Could not find Flatpak suggestions '{appid}'") - self.logger.info("Downloading the suggestions file {}".format(SUGGESTIONS_FILE)) - file = self.http_client.get(SUGGESTIONS_FILE) + def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[PackageSuggestion]]: + if limit == 0: + return - if not file or not file.text: - self.logger.warning("No suggestion found in {}".format(SUGGESTIONS_FILE)) - return res + if self.is_local_suggestions_file_mapped(): + suggestions_str = self._read_local_suggestions_file() else: - self.logger.info("Mapping suggestions") - remote_level = self._get_search_remote() - installed = {i.id for i in self.read_installed(disk_loader=None).installed} if filter_installed else None + suggestions_str = self._download_remote_suggestions_file() + + if suggestions_str is None: + return + + if not suggestions_str: + self.logger.warning(f"No Flatpak suggestion found in {self.suggestions_file_url}") + return + + ids_prios = suggestions.parse(suggestions_str, self.logger, 'Flatpak') - for line in file.text.split('\n'): - if line: - if limit <= 0 or len(res) < limit: - sug = line.split('=') - appid = sug[1].strip() + if not ids_prios: + self.logger.warning(f"No Flatpak suggestion could be parsed from {self.suggestions_file_url}") + return - if installed and appid in installed: - continue + suggestion_by_priority = suggestions.sort_by_priority(ids_prios) - priority = SuggestionPriority(int(sug[0])) + if filter_installed: + installed = {i.id for i in self.read_installed(disk_loader=None).installed} - cached_sug = self.suggestions_cache.get(appid) + if installed: + suggestion_by_priority = tuple(id_ for id_ in suggestion_by_priority if id_ not in installed) - if cached_sug: - res.append(cached_sug) - else: - app_json = flatpak.search(cli_version, appid, remote_level, app_id=True) + if suggestion_by_priority and 0 < limit < len(suggestion_by_priority): + suggestion_by_priority = suggestion_by_priority[0:limit] - if app_json: - model = PackageSuggestion(self._map_to_model(app_json[0], False, None)[0], priority) - self.suggestions_cache.add(appid, model) - res.append(model) - else: - break + self.logger.info(f'Available Flatpak suggestions: {len(suggestion_by_priority)}') + + if not suggestion_by_priority: + return + + flatpak_version = flatpak.get_version() + remote = self._get_search_remote() + + self.logger.info("Mapping Flatpak suggestions") + res, fill_suggestions = [], [] + cached_count = 0 + + for appid in suggestion_by_priority: + cached_instance = self.suggestions_cache.get(appid) + + if cached_instance: + res.append(cached_instance) + cached_count += 1 + else: + fill = Thread(target=self._fill_suggestion, args=(appid, ids_prios[appid], flatpak_version, + remote, res)) + fill.start() + fill_suggestions.append(fill) + + for fill in fill_suggestions: + fill.join() + + if cached_count > 0: + self.logger.info(f"Returning {cached_count} cached Flatpak suggestions") - res.sort(key=lambda s: s.priority.value, reverse=True) return res def is_default_enabled(self) -> bool: @@ -665,8 +721,8 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: options=install_opts, default_option=[o for o in install_opts if o.value == flatpak_config['installation_level']][0], max_per_line=len(install_opts), - max_width=160, type_=SelectViewType.COMBO, + alignment=ViewComponentAlignment.CENTER, id_='install')) yield SettingsView(self, PanelComponent([FormComponent(fields, self.i18n['installation'].capitalize())])) @@ -705,31 +761,22 @@ def get_upgrade_requirements(self, pkgs: List[FlatpakApplication], root_password return UpgradeRequirements(None, None, to_update, []) def sort_update_order(self, pkgs: List[FlatpakApplication]) -> List[FlatpakApplication]: - partials, runtimes, apps = [], [], [] + runtimes, apps = set(), set() for p in pkgs: if p.runtime: - if p.partial: - partials.append(p) - else: - runtimes.append(p) + runtimes.add(p) else: - apps.append(p) - - if not runtimes: - return [*partials, *apps] - elif partials: - all_runtimes = [] - for runtime in runtimes: - for partial in partials: - if partial.installation == runtime.installation and partial.base_id == runtime.id: - all_runtimes.append(partial) - break - - all_runtimes.append(runtime) - return [*all_runtimes, *apps] - else: - return [*runtimes, *apps] + apps.add(p) + + sorted_list = [] + for comps in (runtimes, apps): + if comps: + comp_list = list(comps) + comp_list.sort(key=attrgetter('installation', 'name', 'id')) + sorted_list.extend(comp_list) + + return sorted_list def _read_ignored_updates(self) -> Set[str]: ignored = set() @@ -802,3 +849,21 @@ def action_full_update(self) -> CustomSoftwareAction: requires_root=False) return self._action_full_update + + @property + def suggestions_file_url(self) -> str: + if self._suggestions_file_url is None: + file_url = self.context.get_suggestion_url(self.__module__) + + if not file_url: + file_url = 'https://raw.githubusercontent.com/vinifmor/bauh-files/master/flatpak/suggestions.txt' + + self._suggestions_file_url = file_url + + if file_url.startswith('/'): + self.logger.info(f"Local Flatpak suggestions file mapped: {file_url}") + + return self._suggestions_file_url + + def is_local_suggestions_file_mapped(self) -> bool: + return self.suggestions_file_url.startswith('/') diff --git a/bauh/gems/flatpak/flatpak.py b/bauh/gems/flatpak/flatpak.py index 87ac1424..a8336bd7 100755 --- a/bauh/gems/flatpak/flatpak.py +++ b/bauh/gems/flatpak/flatpak.py @@ -326,15 +326,18 @@ def get_app_commits_data(app_ref: str, origin: str, installation: str, full_str: return commits -def search(version: Version, word: str, installation: str, app_id: bool = False) -> List[dict]: +def search(version: Version, word: str, installation: str, app_id: bool = False) -> Optional[List[dict]]: res = run_cmd(f'flatpak search {word} --{installation}', lang=None) - found = [] + if not res: + return + found = None split_res = res.strip().split('\n') if split_res and '\t' in split_res[0]: + found = [] for info in split_res: if info: info_list = info.split('\t') diff --git a/bauh/gems/flatpak/model.py b/bauh/gems/flatpak/model.py index 0e10a282..e2bac51e 100644 --- a/bauh/gems/flatpak/model.py +++ b/bauh/gems/flatpak/model.py @@ -2,7 +2,7 @@ from bauh.api.abstract.model import SoftwarePackage, PackageStatus from bauh.commons import resource -from bauh.gems.flatpak import ROOT_DIR, VERSION_1_2, VERSION_1_5 +from bauh.gems.flatpak import ROOT_DIR, VERSION_1_2 from bauh.view.util.translation import I18n @@ -125,7 +125,16 @@ def get_update_ignore_key(self) -> str: def __eq__(self, other): if isinstance(other, FlatpakApplication): - return self.id == other.id and self.installation == other.installation and self.branch == other.branch + return self.id == other.id and self.installation == other.installation and self.branch == other.branch \ + and self.runtime == other.runtime and self.partial == other.partial and \ + self.update_component == other.update_component + + def __hash__(self) -> int: + hash_sum = 0 + for attr in ('id', 'installation', 'branch', 'runtime', 'partial', 'update_component'): + hash_sum += hash(getattr(self, attr)) + + return hash_sum def get_disk_icon_path(self) -> str: if not self.runtime: @@ -137,9 +146,22 @@ def get_update_id(self, flatpak_version: Version) -> str: else: return f'{self.installation}/{self.ref}' + def can_be_installed(self) -> bool: + return not self.update_component and not self.installed + + def can_be_updated(self) -> bool: + return self.update_component or super(FlatpakApplication, self).can_be_updated() + def can_be_uninstalled(self) -> bool: return not self.update_component and super(FlatpakApplication, self).can_be_uninstalled() def update_ref(self): if self.id and self.arch and self.branch: self.ref = f'{self.id}/{self.arch}/{self.branch}' + + def __repr__(self) -> str: + return f'Flatpak (id={self.id}, branch={self.branch}, origin={self.origin}, installation={self.installation},' \ + f' partial={self.partial}, update_component={self.update_component})' + + def __str__(self): + return self.__repr__() diff --git a/bauh/gems/snap/__init__.py b/bauh/gems/snap/__init__.py index 582ce071..978c1d33 100644 --- a/bauh/gems/snap/__init__.py +++ b/bauh/gems/snap/__init__.py @@ -8,7 +8,6 @@ CONFIG_FILE = f'{CONFIG_DIR}/snap.yml' CATEGORIES_FILE_PATH = f'{SNAP_CACHE_DIR}/categories.txt' URL_CATEGORIES_FILE = 'https://raw.githubusercontent.com/vinifmor/bauh-files/master/snap/categories.txt' -SUGGESTIONS_FILE = 'https://raw.githubusercontent.com/vinifmor/bauh-files/master/snap/suggestions.txt' def get_icon_path() -> str: diff --git a/bauh/gems/snap/controller.py b/bauh/gems/snap/controller.py index df7d9236..7cabfd17 100644 --- a/bauh/gems/snap/controller.py +++ b/bauh/gems/snap/controller.py @@ -13,12 +13,13 @@ from bauh.api.abstract.view import SingleSelectComponent, SelectViewType, InputOption, PanelComponent, \ FormComponent, TextInputComponent from bauh.api.exception import NoInternetException +from bauh.commons import suggestions from bauh.commons.boot import CreateConfigFile from bauh.commons.category import CategoriesDownloader from bauh.commons.html import bold from bauh.commons.system import SystemProcess, ProcessHandler, new_root_subprocess from bauh.commons.view_utils import new_select, get_human_size_str -from bauh.gems.snap import snap, URL_CATEGORIES_FILE, CATEGORIES_FILE_PATH, SUGGESTIONS_FILE, \ +from bauh.gems.snap import snap, URL_CATEGORIES_FILE, CATEGORIES_FILE_PATH, \ get_icon_path, snapd from bauh.gems.snap.config import SnapConfigManager from bauh.gems.snap.model import SnapApplication @@ -42,6 +43,7 @@ def __init__(self, context: ApplicationContext): self.suggestions_cache = context.cache_factory.new() self.info_path = None self.configman = SnapConfigManager() + self._suggestions_url: Optional[str] = None def _fill_categories(self, app: SnapApplication): categories = self.categories.get(app.name.lower()) @@ -315,7 +317,8 @@ def list_warnings(self, internet_available: bool) -> Optional[List[str]]: self.logger.warning(f'It seems Snap API is not available. Search output: {output}') return [self.i18n['snap.notifications.api.unavailable'].format(bold('Snaps'), bold('Snap'))] - def _fill_suggestion(self, name: str, priority: SuggestionPriority, snapd_client: SnapdClient, out: List[PackageSuggestion]): + def _fill_suggestion(self, name: str, priority: SuggestionPriority, snapd_client: SnapdClient, + out: List[PackageSuggestion]): res = snapd_client.find_by_name(name) if res: @@ -361,46 +364,87 @@ def _map_to_app(self, app_json: dict, installed: bool, disk_loader: Optional[Dis app.status = PackageStatus.READY return app + def _read_local_suggestions_file(self) -> Optional[str]: + try: + with open(self.suggestions_url) as f: + suggestions_str = f.read() + + return suggestions_str + except FileNotFoundError: + self.logger.error(f"Local Snap suggestions file not found: {self.suggestions_url}") + except OSError: + self.logger.error(f"Could not read local Snap suggestions file: {self.suggestions_url}") + traceback.print_exc() + + def _download_remote_suggestions_file(self) -> Optional[str]: + self.logger.info(f"Downloading the Snap suggestions from {self.suggestions_url}") + file = self.http_client.get(self.suggestions_url) + + if file: + return file.text + def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[PackageSuggestion]]: - res = [] + if limit == 0 or not snapd.is_running(): + return + + if self.is_local_suggestions_file_mapped(): + suggestions_str = self._read_local_suggestions_file() + else: + suggestions_str = self._download_remote_suggestions_file() + + if suggestions_str is None: + return + + if not suggestions_str: + self.logger.warning(f"No Snap suggestion found in {self.suggestions_url}") + return + + ids_prios = suggestions.parse(suggestions_str, self.logger, 'Snap') + + if not ids_prios: + self.logger.warning(f"No Snap suggestion could be parsed from {self.suggestions_url}") + return + + suggestion_by_priority = suggestions.sort_by_priority(ids_prios) + snapd_client = SnapdClient(self.logger) + + if filter_installed: + installed = {s['name'].lower() for s in snapd_client.list_all_snaps()} - if snapd.is_running(): - self.logger.info(f'Downloading suggestions file {SUGGESTIONS_FILE}') - file = self.http_client.get(SUGGESTIONS_FILE) + if installed: + suggestion_by_priority = tuple(n for n in suggestion_by_priority if n not in installed) - if not file or not file.text: - self.logger.warning(f"No suggestion found in {SUGGESTIONS_FILE}") - return res + if suggestion_by_priority and 0 < limit < len(suggestion_by_priority): + suggestion_by_priority = suggestion_by_priority[0:limit] + + self.logger.info(f'Available Snap suggestions: {len(suggestion_by_priority)}') + + if not suggestion_by_priority: + return + + self.logger.info("Mapping Snap suggestions") + + instances, threads = [], [] + + res, cached_count = [], 0 + for name in suggestion_by_priority: + cached_sug = self.suggestions_cache.get(name) + + if cached_sug: + res.append(cached_sug) + cached_count += 1 else: - self.logger.info('Mapping suggestions') - - suggestions, threads = [], [] - snapd_client = SnapdClient(self.logger) - installed = {s['name'].lower() for s in snapd_client.list_all_snaps()} - - for l in file.text.split('\n'): - if l: - if limit <= 0 or len(suggestions) < limit: - sug = l.strip().split('=') - name = sug[1] - - if not installed or name not in installed: - cached_sug = self.suggestions_cache.get(name) - - if cached_sug: - res.append(cached_sug) - else: - t = Thread(target=self._fill_suggestion, args=(name, SuggestionPriority(int(sug[0])), snapd_client, res)) - t.start() - threads.append(t) - time.sleep(0.001) # to avoid being blocked - else: - break + t = Thread(target=self._fill_suggestion, args=(name, ids_prios[name], snapd_client, res)) + t.start() + threads.append(t) + time.sleep(0.001) # to avoid being blocked - for t in threads: - t.join() + for t in threads: + t.join() + + if cached_count > 0: + self.logger.info(f"Returning {cached_count} cached Snap suggestions") - res.sort(key=lambda s: s.priority.value, reverse=True) return res def is_default_enabled(self) -> bool: @@ -435,13 +479,11 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: (self.i18n['no'].capitalize(), False, None)], value=bool(snap_config['install_channel']), id_='snap_install_channel', - max_width=200, tip=self.i18n['snap.config.install_channel.tip']) cat_exp_val = snap_config['categories_exp'] if isinstance(snap_config['categories_exp'], int) else '' categories_exp = TextInputComponent(id_='snap_cat_exp', value=cat_exp_val, - max_width=60, only_int=True, label=self.i18n['snap.config.categories_exp'], tooltip=self.i18n['snap.config.categories_exp.tip']) @@ -511,3 +553,21 @@ def _request_channel_installation(self, pkg: SnapApplication, snap_config: Optio raise Exception('aborted') else: return select.get_selected() + + @property + def suggestions_url(self) -> str: + if not self._suggestions_url: + file_url = self.context.get_suggestion_url(self.__module__) + + if not file_url: + file_url = 'https://raw.githubusercontent.com/vinifmor/bauh-files/master/snap/suggestions.txt' + + self._suggestions_url = file_url + + if file_url.startswith('/'): + self.logger.info(f"Local Snap suggestions file mapped: {file_url}") + + return self._suggestions_url + + def is_local_suggestions_file_mapped(self) -> bool: + return self.suggestions_url.startswith('/') diff --git a/bauh/gems/web/__init__.py b/bauh/gems/web/__init__.py index dbb27a29..28802793 100644 --- a/bauh/gems/web/__init__.py +++ b/bauh/gems/web/__init__.py @@ -22,12 +22,9 @@ DESKTOP_ENTRY_PATH_PATTERN = f'{DESKTOP_ENTRIES_DIR}/{__app_name__}.web.' + '{name}.desktop' URL_FIX_PATTERN = "https://raw.githubusercontent.com/vinifmor/bauh-files/master/web/env/v2/fix/{domain}/{electron_branch}/fix.js" URL_PROPS_PATTERN = "https://raw.githubusercontent.com/vinifmor/bauh-files/master/web/env/v2/fix/{domain}/{electron_branch}/properties" -URL_SUGGESTIONS = "https://raw.githubusercontent.com/vinifmor/bauh-files/master/web/env/v2/suggestions.yml" UA_CHROME = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36' TEMP_PATH = f'{TEMP_DIR}/web' SEARCH_INDEX_FILE = f'{WEB_CACHE_DIR}/index.yml' -SUGGESTIONS_CACHE_FILE = f'{WEB_CACHE_DIR}/suggestions.yml' -SUGGESTIONS_CACHE_TS_FILE = map_timestamp_file(SUGGESTIONS_CACHE_FILE) CONFIG_FILE = f'{CONFIG_DIR}/web.yml' ENVIRONMENT_SETTINGS_CACHED_FILE = f'{WEB_CACHE_DIR}/environment.yml' ENVIRONMENT_SETTINGS_TS_FILE = f'{WEB_CACHE_DIR}/environment.ts' diff --git a/bauh/gems/web/controller.py b/bauh/gems/web/controller.py index c2cf3aed..7de6698e 100644 --- a/bauh/gems/web/controller.py +++ b/bauh/gems/web/controller.py @@ -6,10 +6,9 @@ import shutil import subprocess import traceback -from math import floor from pathlib import Path from threading import Thread -from typing import List, Type, Set, Tuple, Optional, Dict, Generator, Iterable +from typing import List, Type, Set, Tuple, Optional, Dict, Generator, Iterable, Pattern import requests import yaml @@ -25,16 +24,15 @@ PackageHistory, \ SuggestionPriority, PackageStatus from bauh.api.abstract.view import MessageType, MultipleSelectComponent, InputOption, SingleSelectComponent, \ - SelectViewType, TextInputComponent, FormComponent, FileChooserComponent, PanelComponent + SelectViewType, TextInputComponent, FormComponent, FileChooserComponent, PanelComponent, ViewComponentAlignment from bauh.api.paths import DESKTOP_ENTRIES_DIR from bauh.commons import resource from bauh.commons.boot import CreateConfigFile from bauh.commons.html import bold from bauh.commons.system import ProcessHandler, get_dir_size, SimpleProcess from bauh.commons.view_utils import get_human_size_str -from bauh.gems.web import INSTALLED_PATH, nativefier, DESKTOP_ENTRY_PATH_PATTERN, URL_FIX_PATTERN, ENV_PATH, UA_CHROME, \ - SUGGESTIONS_CACHE_FILE, ROOT_DIR, TEMP_PATH, FIX_FILE_PATH, ELECTRON_CACHE_DIR, \ - get_icon_path, URL_PROPS_PATTERN +from bauh.gems.web import INSTALLED_PATH, nativefier, DESKTOP_ENTRY_PATH_PATTERN, URL_FIX_PATTERN, ENV_PATH, \ + ROOT_DIR, TEMP_PATH, FIX_FILE_PATH, ELECTRON_CACHE_DIR, UA_CHROME, get_icon_path, URL_PROPS_PATTERN from bauh.gems.web.config import WebConfigManager from bauh.gems.web.environment import EnvironmentUpdater, EnvironmentComponent from bauh.gems.web.model import WebApplication @@ -44,18 +42,18 @@ try: from bs4 import BeautifulSoup, SoupStrainer + BS4_AVAILABLE = True except: BS4_AVAILABLE = False - try: import lxml + LXML_AVAILABLE = True except: LXML_AVAILABLE = False -RE_PROTOCOL_STRIP = re.compile(r'[a-zA-Z]+://') RE_SEVERAL_SPACES = re.compile(r'\s+') RE_SYMBOLS_SPLIT = re.compile(r'[\-|_\s:.]') @@ -64,7 +62,7 @@ class WebApplicationManager(SoftwareManager, SettingsController): - def __init__(self, context: ApplicationContext, suggestions_loader: Optional[SuggestionsLoader] = None): + def __init__(self, context: ApplicationContext): super(WebApplicationManager, self).__init__(context=context) self.http_client = context.http_client self.env_updater = EnvironmentUpdater(logger=context.logger, http_client=context.http_client, @@ -73,11 +71,13 @@ def __init__(self, context: ApplicationContext, suggestions_loader: Optional[Sug self.i18n = context.i18n self.logger = context.logger self.env_thread = None - self.suggestions_loader = suggestions_loader + self.suggestions_loader: Optional[SuggestionsLoader] = None + self._suggestions_manager: Optional[SuggestionsManager] = None self.suggestions = {} self.configman = WebConfigManager() self.idxman = SearchIndexManager(logger=context.logger) self._custom_actions: Optional[Iterable[CustomSoftwareAction]] = None + self._re_protocol_strip: Optional[Pattern] = None def get_accept_language_header(self) -> str: try: @@ -174,7 +174,7 @@ def _get_app_icon_url(self, url: str, soup: "BeautifulSoup") -> str: if icon_url: return icon_url - def _get_app_description(self, url: str, soup: "BeautifulSoup") -> str: + def _get_app_description(self, url: str, soup: "BeautifulSoup") -> str: description = None desc_tag = soup.head.find('meta', attrs={'name': 'description'}) @@ -227,13 +227,17 @@ def _get_custom_properties(self, url_domain: str, electron_branch: str) -> Optio return props except Exception as e: - self.logger.warning(f"Error when trying to retrieve custom installation properties for {props_url}: {e.__class__.__name__}") + self.logger.warning( + f"Error when trying to retrieve custom installation properties for {props_url}: {e.__class__.__name__}") def _map_electron_branch(self, version: str) -> str: return f"electron_{'_'.join(version.split('.')[0:-1])}_X" - def _strip_url_protocol(self, url: str) -> str: - return RE_PROTOCOL_STRIP.split(url)[1].strip().lower() + def strip_url_protocol(self, url: str) -> str: + if not self._re_protocol_strip: + self._re_protocol_strip = re.compile(r'^[a-zA-Z]+://(www\.)?') + + return self._re_protocol_strip.split(url)[-1].strip().lower() def serialize_to_disk(self, pkg: SoftwarePackage, icon_bytes: Optional[bytes], only_icon: bool): super(WebApplicationManager, self).serialize_to_disk(pkg=pkg, icon_bytes=None, only_icon=False) @@ -242,7 +246,8 @@ def _request_url(self, url: str) -> Optional[Response]: headers = {'Accept-language': self.get_accept_language_header(), 'User-Agent': UA_CHROME} try: - return self.http_client.get(url, headers=headers, ignore_ssl=True, single_call=True, session=False, allow_redirects=True) + return self.http_client.get(url, headers=headers, ignore_ssl=True, single_call=True, session=False, + allow_redirects=True) except Exception as e: self.logger.warning(f"Could not GET '{url}'. Exception: {e.__class__.__name__}") @@ -263,9 +268,9 @@ def search(self, words: str, disk_loader: DiskCacheLoader, limit: int = -1, is_u if is_url: url = words[0:-1] if words.endswith('/') else words - url_no_protocol = self._strip_url_protocol(url) + url_no_protocol = self.strip_url_protocol(url) - installed_matches = [app for app in installed if self._strip_url_protocol(app.url) == url_no_protocol] + installed_matches = [app for app in installed if self.strip_url_protocol(app.url) == url_no_protocol] if installed_matches: res.installed.extend(installed_matches) @@ -320,25 +325,28 @@ def search(self, words: str, disk_loader: DiskCacheLoader, limit: int = -1, is_u self.logger.info("Query '{}' was not found in the suggestion's index".format(words)) res.installed.extend(installed_matches) else: - if not os.path.exists(SUGGESTIONS_CACHE_FILE): + cached_file_path = self.suggestions_manager.get_cached_file_path() + + if not os.path.exists(cached_file_path): # if the suggestions cache was not found, it will not be possible to retrieve the matched apps # so only the installed matches will be returned - self.logger.warning("Suggestion cached file {} was not found".format(SUGGESTIONS_CACHE_FILE)) + self.logger.warning(f"Suggestion file {cached_file_path} was not found") res.installed.extend(installed_matches) else: - with open(SUGGESTIONS_CACHE_FILE) as f: + with open(cached_file_path) as f: cached_suggestions = yaml.safe_load(f.read()) if not cached_suggestions: # if no suggestion is found, it will not be possible to retrieve the matched apps # so only the installed matches will be returned - self.logger.warning("No suggestion found in {}".format(SUGGESTIONS_CACHE_FILE)) + self.logger.warning(f"No suggestion found in {cached_file_path}") res.installed.extend(installed_matches) else: - matched_suggestions = [cached_suggestions[key] for key in index_match_keys if cached_suggestions.get(key)] + matched_suggestions = [cached_suggestions[key] for key in index_match_keys if + cached_suggestions.get(key)] if not matched_suggestions: - self.logger.warning("No suggestion found for the search index keys: {}".format(index_match_keys)) + self.logger.warning(f"No suggestion found for the query index keys: {index_match_keys}") res.installed.extend(installed_matches) else: matched_suggestions.sort(key=lambda s: s.get('priority', 0), reverse=True) @@ -376,7 +384,8 @@ def search(self, words: str, disk_loader: DiskCacheLoader, limit: int = -1, is_u return res - def read_installed(self, disk_loader: Optional[DiskCacheLoader], limit: int = -1, only_apps: bool = False, pkg_types: Set[Type[SoftwarePackage]] = None, internet_available: bool = True) -> SearchResult: + def read_installed(self, disk_loader: Optional[DiskCacheLoader], limit: int = -1, only_apps: bool = False, + pkg_types: Set[Type[SoftwarePackage]] = None, internet_available: bool = True) -> SearchResult: res = SearchResult([], [], 0) if os.path.exists(INSTALLED_PATH): @@ -393,12 +402,14 @@ def downgrade(self, pkg: SoftwarePackage, root_password: Optional[str], handler: def upgrade(self, requirements: UpgradeRequirements, root_password: Optional[str], watcher: ProcessWatcher) -> bool: pass - def uninstall(self, pkg: WebApplication, root_password: Optional[str], watcher: ProcessWatcher, disk_loader: DiskCacheLoader) -> TransactionResult: + def uninstall(self, pkg: WebApplication, root_password: Optional[str], watcher: ProcessWatcher, + disk_loader: DiskCacheLoader) -> TransactionResult: self.logger.info("Checking if {} installation directory {} exists".format(pkg.name, pkg.installation_dir)) if not os.path.exists(pkg.installation_dir): watcher.show_message(title=self.i18n['error'], - body=self.i18n['web.uninstall.error.install_dir.not_found'].format(bold(pkg.installation_dir)), + body=self.i18n['web.uninstall.error.install_dir.not_found'].format( + bold(pkg.installation_dir)), type_=MessageType.ERROR) return TransactionResult.fail() @@ -464,7 +475,8 @@ def get_managed_types(self) -> Set[Type[SoftwarePackage]]: def get_info(self, pkg: WebApplication) -> dict: if pkg.installed: - info = {'0{}_{}'.format(idx + 1, att): getattr(pkg, att) for idx, att in enumerate(('url', 'description', 'version', 'categories', 'installation_dir', 'desktop_entry'))} + info = {'0{}_{}'.format(idx + 1, att): getattr(pkg, att) for idx, att in + enumerate(('url', 'description', 'version', 'categories', 'installation_dir', 'desktop_entry'))} info['07_exec_file'] = pkg.get_exec_path() info['08_icon_path'] = pkg.get_disk_icon_path() @@ -481,12 +493,14 @@ def get_info(self, pkg: WebApplication) -> dict: return info else: - return {'0{}_{}'.format(idx + 1, att): getattr(pkg, att) for idx, att in enumerate(('url', 'description', 'version', 'categories'))} + return {'0{}_{}'.format(idx + 1, att): getattr(pkg, att) for idx, att in + enumerate(('url', 'description', 'version', 'categories'))} def get_history(self, pkg: SoftwarePackage) -> PackageHistory: pass - def _ask_install_options(self, app: WebApplication, watcher: ProcessWatcher, pre_validated: bool) -> Tuple[bool, List[str]]: + def _ask_install_options(self, app: WebApplication, watcher: ProcessWatcher, pre_validated: bool) -> Tuple[ + bool, List[str]]: watcher.change_substatus(self.i18n['web.install.substatus.options']) max_width = 350 @@ -703,9 +717,11 @@ def _download_suggestion_icon(self, pkg: WebApplication, app_dir: str) -> Tuple[ self.logger.error("An exception has happened when downloading {}".format(pkg.icon_url)) traceback.print_exc() else: - self.logger.warning('Could no retrieve the icon {} defined for the suggestion {}'.format(pkg.icon_url, pkg.name)) + self.logger.warning( + 'Could no retrieve the icon {} defined for the suggestion {}'.format(pkg.icon_url, pkg.name)) except: - self.logger.warning('An exception happened when trying to retrieve the icon {} for the suggestion {}'.format(pkg.icon_url, + self.logger.warning( + 'An exception happened when trying to retrieve the icon {} for the suggestion {}'.format(pkg.icon_url, pkg.name)) traceback.print_exc() @@ -749,7 +765,7 @@ def _install(self, pkg: WebApplication, install_options: List[str], watcher: Pro electron_version = str(next((c for c in env_components if c.id == 'electron')).version) - url_domain, electron_branch = self._strip_url_protocol(pkg.url), self._map_electron_branch(electron_version) + url_domain, electron_branch = self.strip_url_protocol(pkg.url), self._map_electron_branch(electron_version) fix = self._get_fix_for(url_domain=url_domain, electron_branch=electron_branch) if fix: @@ -886,7 +902,8 @@ def _install(self, pkg: WebApplication, install_options: List[str], watcher: Pro return TransactionResult(success=True, installed=[pkg], removed=[]) - def install(self, pkg: WebApplication, root_password: Optional[str], disk_loader: DiskCacheLoader, watcher: ProcessWatcher) -> TransactionResult: + def install(self, pkg: WebApplication, root_password: Optional[str], disk_loader: DiskCacheLoader, + watcher: ProcessWatcher) -> TransactionResult: continue_install, install_options = self._ask_install_options(pkg, watcher, pre_validated=True) if not continue_install: @@ -967,9 +984,7 @@ def prepare(self, task_manager: TaskManager, root_password: Optional[str], inter create_config=create_config, i18n=self.i18n).start() - self.suggestions_loader = SuggestionsLoader(manager=SuggestionsManager(logger=self.logger, - http_client=self.http_client, - i18n=self.i18n), + self.suggestions_loader = SuggestionsLoader(manager=self.suggestions_manager, logger=self.logger, i18n=self.i18n, taskman=task_manager, @@ -1008,7 +1023,8 @@ def _fill_suggestion(self, app: WebApplication): app.description = self._get_app_description(app.url, soup) try: - find_url = not app.icon_url or (app.icon_url and not self.http_client.exists(app.icon_url, session=False)) + find_url = not app.icon_url or ( + app.icon_url and not self.http_client.exists(app.icon_url, session=False)) except (requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout): find_url = None @@ -1049,6 +1065,9 @@ def _fill_config_async(self, output: dict): output.update(self.configman.get_config()) def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[PackageSuggestion]]: + if limit == 0: + return + web_config = {} thread_config = Thread(target=self._fill_config_async, args=(web_config,)) @@ -1057,10 +1076,19 @@ def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[ if self.suggestions: suggestions = self.suggestions elif self.suggestions_loader: - self.suggestions_loader.join(5) + if self.suggestions_loader.is_alive(): + self.suggestions_loader.join(5) + suggestions = self.suggestions + elif self.suggestions_manager.is_custom_local_file_mapped(): + suggestions = self.suggestions_manager.read_cached() else: - suggestions = SuggestionsManager(logger=self.logger, http_client=self.http_client, i18n=self.i18n).download() + thread_config.join() + + if self.suggestions_manager.should_download(web_config): + suggestions = self.suggestions_manager.download() + else: + suggestions = self.suggestions_manager.read_cached() # cleaning memory self.suggestions_loader = None @@ -1071,22 +1099,24 @@ def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[ suggestion_list.sort(key=lambda s: s.get('priority', 0), reverse=True) if filter_installed: - installed = {self._strip_url_protocol(i.url) for i in self.read_installed(disk_loader=None).installed} + installed = {self.strip_url_protocol(i.url) for i in self.read_installed(disk_loader=None).installed} else: installed = None env_settings, res = None, [] for s in suggestion_list: - if limit <= 0 or len(res) < limit: + if limit < 0 or len(res) < limit: if installed: - surl = self._strip_url_protocol(s['url']) + surl = self.strip_url_protocol(s['url']) if surl in installed: continue if env_settings is None: # reading settings if not already loaded - thread_config.join() + if thread_config.is_alive(): + thread_config.join() + env_settings = self.env_updater.read_settings(web_config=web_config) res.append(self._map_suggestion(s, env_settings)) @@ -1107,7 +1137,8 @@ def list_suggestions(self, limit: int, filter_installed: bool) -> Optional[List[ return res - def execute_custom_action(self, action: CustomSoftwareAction, pkg: SoftwarePackage, root_password: Optional[str], watcher: ProcessWatcher) -> bool: + def execute_custom_action(self, action: CustomSoftwareAction, pkg: SoftwarePackage, root_password: Optional[str], + watcher: ProcessWatcher) -> bool: pass def is_default_enabled(self) -> bool: @@ -1127,7 +1158,8 @@ def clear_data(self, logs: bool = True): print('{}[bauh][web] Directory {} deleted{}'.format(Fore.YELLOW, ENV_PATH, Fore.RESET)) except: if logs: - print('{}[bauh][web] An exception has happened when deleting {}{}'.format(Fore.RED, ENV_PATH, Fore.RESET)) + print('{}[bauh][web] An exception has happened when deleting {}{}'.format(Fore.RED, ENV_PATH, + Fore.RESET)) traceback.print_exc() def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: @@ -1137,28 +1169,30 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: value=web_config['environment']['electron']['version'], tooltip=self.i18n['web.settings.electron.version.tooltip'], placeholder='{}: 7.1.0'.format(self.i18n['example.short']), - max_width=150, id_='electron_branch') native_opts = [ - InputOption(label=self.i18n['web.settings.nativefier.env'].capitalize(), value=False, tooltip=self.i18n['web.settings.nativefier.env.tooltip'].format(app=self.context.app_name)), - InputOption(label=self.i18n['web.settings.nativefier.system'].capitalize(), value=True, tooltip=self.i18n['web.settings.nativefier.system.tooltip']) + InputOption(label=self.i18n['web.settings.nativefier.env'].capitalize(), value=False, + tooltip=self.i18n['web.settings.nativefier.env.tooltip'].format(app=self.context.app_name)), + InputOption(label=self.i18n['web.settings.nativefier.system'].capitalize(), value=True, + tooltip=self.i18n['web.settings.nativefier.system.tooltip']) ] select_nativefier = SingleSelectComponent(label="Nativefier", options=native_opts, - default_option=[o for o in native_opts if o.value == web_config['environment']['system']][0], + default_option=[o for o in native_opts if + o.value == web_config['environment']['system']][0], type_=SelectViewType.COMBO, tooltip=self.i18n['web.settings.nativefier.tip'], - max_width=150, + alignment=ViewComponentAlignment.CENTER, id_='nativefier') env_settings_exp = TextInputComponent(label=self.i18n['web.settings.cache_exp'], tooltip=self.i18n['web.settings.cache_exp.tip'], capitalize_label=False, - value=int(web_config['environment']['cache_exp']) if isinstance(web_config['environment']['cache_exp'], int) else '', + value=int(web_config['environment']['cache_exp']) if isinstance( + web_config['environment']['cache_exp'], int) else '', only_int=True, - max_width=60, id_='web_cache_exp') sugs_exp = TextInputComponent(label=self.i18n['web.settings.suggestions.cache_exp'], @@ -1167,7 +1201,6 @@ def get_settings(self) -> Optional[Generator[SettingsView, None, None]]: value=int(web_config['suggestions']['cache_exp']) if isinstance( web_config['suggestions']['cache_exp'], int) else '', only_int=True, - max_width=60, id_='web_sugs_exp') form_env = FormComponent(label=self.i18n['web.settings.nativefier.env'].capitalize(), @@ -1224,3 +1257,13 @@ def gen_custom_actions(self) -> Generator[CustomSoftwareAction, None, None]: ) yield from self._custom_actions + + @property + def suggestions_manager(self): + if self._suggestions_manager is None: + self._suggestions_manager = SuggestionsManager(logger=self.logger, + http_client=self.http_client, + i18n=self.i18n, + file_url=self.context.get_suggestion_url(self.__module__)) + + return self._suggestions_manager diff --git a/bauh/gems/web/suggestions.py b/bauh/gems/web/suggestions.py index 5797958c..90256ee1 100644 --- a/bauh/gems/web/suggestions.py +++ b/bauh/gems/web/suggestions.py @@ -3,23 +3,44 @@ from datetime import datetime, timedelta from logging import Logger from pathlib import Path +from typing import Optional import requests import yaml from bauh.api.http import HttpClient -from bauh.gems.web import URL_SUGGESTIONS, SUGGESTIONS_CACHE_FILE, SUGGESTIONS_CACHE_TS_FILE +from bauh.commons.util import map_timestamp_file +from bauh.gems.web import WEB_CACHE_DIR from bauh.view.util.translation import I18n class SuggestionsManager: - def __init__(self, http_client: HttpClient, logger: Logger, i18n: I18n): + def __init__(self, http_client: HttpClient, logger: Logger, i18n: I18n, file_url: Optional[str]): self.http_client = http_client self.logger = logger self.i18n = i18n + if file_url: + self._file_url = file_url + else: + self._file_url = "https://raw.githubusercontent.com/vinifmor/bauh-files/master/web/env/v2/suggestions.yml" + self._cached_file_path = f'{WEB_CACHE_DIR}/suggestions.yml' + self._cached_file_ts_path = map_timestamp_file(self._cached_file_path) + + @property + def file_url(self) -> Optional[str]: + return self._file_url + + def is_custom_local_file_mapped(self) -> bool: + return self._file_url and self._file_url.startswith('/') def should_download(self, web_config: dict) -> bool: + if not self._file_url: + return False + + if self.is_custom_local_file_mapped(): + return False + exp = web_config['suggestions']['cache_exp'] if web_config['suggestions']['cache_exp'] is None: self.logger.info("No cache expiration defined for suggestions") @@ -28,28 +49,28 @@ def should_download(self, web_config: dict) -> bool: try: exp = int(exp) except ValueError: - self.logger.error("Error while parsing the 'suggestions.cache_exp' ({}) settings property".format(exp)) + self.logger.error(f"Error while parsing the 'suggestions.cache_exp' ({exp}) settings property") return True if exp <= 0: - self.logger.info("No cache expiration defined for suggestions ({})".format(exp)) + self.logger.info(f"No cache expiration defined for suggestions ({exp})") return True - if not os.path.exists(SUGGESTIONS_CACHE_FILE): - self.logger.info("No suggestions cached file found '{}'".format(SUGGESTIONS_CACHE_FILE)) + if not os.path.exists(self._cached_file_path): + self.logger.info(f"No suggestions cached file found '{self._cached_file_path}'") return True - if not os.path.exists(SUGGESTIONS_CACHE_TS_FILE): - self.logger.info("No suggestions cache file timestamp found '{}'".format(SUGGESTIONS_CACHE_TS_FILE)) + if not os.path.exists(self._cached_file_ts_path): + self.logger.info(f"No suggestions cache file timestamp found '{self._cached_file_ts_path}'") return True - with open(SUGGESTIONS_CACHE_TS_FILE) as f: + with open(self._cached_file_ts_path) as f: timestamp_str = f.read() try: sugs_timestamp = datetime.fromtimestamp(float(timestamp_str)) except: - self.logger.error("Could not parse the cached suggestions file timestamp: {}".format(timestamp_str)) + self.logger.error(f"Could not parse the cached suggestions file timestamp: {timestamp_str}") return True expired = sugs_timestamp + timedelta(days=exp) <= datetime.utcnow() @@ -61,17 +82,25 @@ def should_download(self, web_config: dict) -> bool: self.logger.info("Cached suggestions file is up to date") return False + def get_cached_file_path(self) -> str: + return self._file_url if self.is_custom_local_file_mapped() else self._cached_file_path + def read_cached(self, check_file: bool = True) -> dict: - if check_file and not os.path.exists(SUGGESTIONS_CACHE_FILE): - self.logger.warning("Cached suggestions file does not exist ({})".format(SUGGESTIONS_CACHE_FILE)) + if self.is_custom_local_file_mapped(): + file_path, log_ref = self._file_url, 'local' + else: + file_path, log_ref = self._cached_file_path, 'cached' + + if check_file and not os.path.exists(file_path): + self.logger.warning(f"{log_ref.capitalize()} suggestions file does not exist ({file_path})") return {} - self.logger.info("Reading cached suggestions file '{}'".format(SUGGESTIONS_CACHE_FILE)) - with open(SUGGESTIONS_CACHE_FILE) as f: + self.logger.info(f"Reading {log_ref} suggestions file '{file_path}'") + with open(file_path) as f: sugs_str = f.read() if not sugs_str: - self.logger.warning("Cached suggestions file '{}' is empty".format(SUGGESTIONS_CACHE_FILE)) + self.logger.warning(f"{log_ref.capitalize()} suggestions file '{file_path}' is empty") return {} try: @@ -82,14 +111,14 @@ def read_cached(self, check_file: bool = True) -> dict: return {} def download(self) -> dict: - self.logger.info("Reading suggestions from {}".format(URL_SUGGESTIONS)) + self.logger.info(f"Reading suggestions from {self._file_url}") try: - suggestions = self.http_client.get_yaml(URL_SUGGESTIONS, session=False) + suggestions = self.http_client.get_yaml(self._file_url, session=False) if suggestions: - self.logger.info("{} suggestions successfully read".format(len(suggestions))) + self.logger.info(f"{len(suggestions)} suggestions successfully read") else: - self.logger.warning("Could not read suggestions from {}".format(URL_SUGGESTIONS)) + self.logger.warning(f"Could not read suggestions from {self._file_url}") except (requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout): self.logger.warning("Internet seems to be off: it was not possible to retrieve the suggestions") @@ -102,31 +131,36 @@ def save_to_disk(self, suggestions: dict, timestamp: float): if not suggestions: return - self.logger.info('Caching {} suggestions to the disk'.format(len(suggestions))) - suggestions_file_dir = os.path.dirname(SUGGESTIONS_CACHE_FILE) + if self.is_custom_local_file_mapped(): + return + + self.logger.info(f'Caching {len(suggestions)} suggestions to the disk') + suggestions_file_dir = os.path.dirname(self._cached_file_path) + try: Path(suggestions_file_dir).mkdir(parents=True, exist_ok=True) except OSError: - self.logger.error("Could not generate the directory {}".format(suggestions_file_dir)) + self.logger.error(f"Could not generate the directory {suggestions_file_dir}") traceback.print_exc() return try: - with open(SUGGESTIONS_CACHE_FILE, 'w+') as f: + with open(self._cached_file_path, 'w+') as f: f.write(yaml.safe_dump(suggestions)) except: - self.logger.error("Could write to {}".format(SUGGESTIONS_CACHE_FILE)) + self.logger.error(f"Could write to {self._cached_file_path}") traceback.print_exc() return - self.logger.info("{} suggestions successfully cached to file '{}'".format(len(suggestions), SUGGESTIONS_CACHE_FILE)) + self.logger.info(f"{len(suggestions)} suggestions successfully cached to file '{self._cached_file_path}'") try: - with open(SUGGESTIONS_CACHE_TS_FILE, 'w+') as f: + with open(self._cached_file_ts_path, 'w+') as f: f.write(str(timestamp)) except: - self.logger.error("Could not write to {}".format(SUGGESTIONS_CACHE_TS_FILE)) + self.logger.error(f"Could not write to {self._cached_file_ts_path}") traceback.print_exc() return - self.logger.info("Suggestions cached file timestamp ({}) successfully saved at '{}'".format(timestamp, SUGGESTIONS_CACHE_TS_FILE)) + self.logger.info(f"Suggestions cached file timestamp ({timestamp}) " + f"successfully saved at '{self._cached_file_ts_path}'") diff --git a/bauh/gems/web/worker.py b/bauh/gems/web/worker.py index bcff6488..df0f3916 100644 --- a/bauh/gems/web/worker.py +++ b/bauh/gems/web/worker.py @@ -35,27 +35,33 @@ def __init__(self, taskman: TaskManager, manager: SuggestionsManager, def run(self): ti = time.time() - self.taskman.update_progress(self.task_id, 0, self.i18n['task.waiting_task'].format(bold(self.create_config.task_name))) - self.create_config.join() - - self.taskman.update_progress(self.task_id, 10, None) - - if not self.internet_connection: - self.logger.warning("No internet connection. Only cached suggestions can be loaded") - self.suggestions = self.manager.read_cached(check_file=True) - elif not self.manager.should_download(self.create_config.config): - self.suggestions = self.manager.read_cached(check_file=False) + if self.manager.is_custom_local_file_mapped(): + self.taskman.update_progress(self.task_id, 50, None) + self.logger.info(f"Local Web suggestions file mapped: {self.manager.file_url}") + self.suggestions = self.manager.read_cached() else: - try: - timestamp = datetime.utcnow().timestamp() - self.suggestions = self.manager.download() - - if self.suggestions: - self.taskman.update_progress(self.task_id, 50, self.i18n['web.task.suggestions.saving']) - self.manager.save_to_disk(self.suggestions, timestamp) - except: - self.logger.error("Unexpected exception") - traceback.print_exc() + wait_msg = self.i18n['task.waiting_task'].format(bold(self.create_config.task_name)) + self.taskman.update_progress(self.task_id, 0, wait_msg) + self.create_config.join() + + self.taskman.update_progress(self.task_id, 10, None) + + if not self.internet_connection: + self.logger.warning("No internet connection. Only cached suggestions can be loaded") + self.suggestions = self.manager.read_cached() + elif not self.manager.should_download(self.create_config.config): + self.suggestions = self.manager.read_cached(check_file=False) + else: + try: + timestamp = datetime.utcnow().timestamp() + self.suggestions = self.manager.download() + + if self.suggestions: + self.taskman.update_progress(self.task_id, 50, self.i18n['web.task.suggestions.saving']) + self.manager.save_to_disk(self.suggestions, timestamp) + except: + self.logger.error("Unexpected exception") + traceback.print_exc() if self.suggestions_callback: self.taskman.update_progress(self.task_id, 75, None) @@ -73,7 +79,8 @@ def run(self): class SearchIndexGenerator(Thread): - def __init__(self, taskman: TaskManager, idxman: SearchIndexManager, suggestions_loader: SuggestionsLoader, i18n: I18n, logger: logging.Logger): + def __init__(self, taskman: TaskManager, idxman: SearchIndexManager, suggestions_loader: SuggestionsLoader, + i18n: I18n, logger: logging.Logger): super(SearchIndexGenerator, self).__init__(daemon=True) self.taskman = taskman self.idxman = idxman @@ -85,8 +92,11 @@ def __init__(self, taskman: TaskManager, idxman: SearchIndexManager, suggestions def run(self): ti = time.time() - self.taskman.update_progress(self.task_id, 0, self.i18n['task.waiting_task'].format(bold(self.suggestions_loader.task_name))) - self.suggestions_loader.join() + + if self.suggestions_loader.is_alive(): + wait_msg = self.i18n['task.waiting_task'].format(bold(self.suggestions_loader.task_name)) + self.taskman.update_progress(self.task_id, 0, wait_msg) + self.suggestions_loader.join() if self.suggestions_loader.suggestions: self.taskman.update_progress(self.task_id, 1, None) diff --git a/bauh/manage.py b/bauh/manage.py index 0f8bf736..789ea3d3 100644 --- a/bauh/manage.py +++ b/bauh/manage.py @@ -13,6 +13,7 @@ from bauh.view.core import gems from bauh.view.core.controller import GenericSoftwareManager from bauh.view.core.downloader import AdaptableFileDownloader +from bauh.view.core.suggestions import read_suggestions_mapping from bauh.view.qt.prepare import PreparePanel from bauh.view.qt.settings import SettingsWindow from bauh.view.qt.window import ManageWindow @@ -31,6 +32,11 @@ def new_manage_panel(app_args: Namespace, app_config: dict, logger: logging.Logg http_client = HttpClient(logger) + downloader = AdaptableFileDownloader(logger=logger, multithread_enabled=app_config['download']['multithreaded'], + multithread_client=app_config['download']['multithreaded_client'], + i18n=i18n, http_client=http_client, + check_ssl=app_config['download']['check_ssl']) + context = ApplicationContext(i18n=i18n, http_client=http_client, download_icons=bool(app_config['download']['icons']), @@ -39,11 +45,11 @@ def new_manage_panel(app_args: Namespace, app_config: dict, logger: logging.Logg disk_loader_factory=DefaultDiskCacheLoaderFactory(logger), logger=logger, distro=util.get_distro(), - file_downloader=AdaptableFileDownloader(logger, bool(app_config['download']['multithreaded']), - i18n, http_client, app_config['download']['multithreaded_client']), + file_downloader=downloader, app_name=__app_name__, app_version=__version__, internet_checker=InternetChecker(offline=app_args.offline), + suggestions_mapping=read_suggestions_mapping(), root_user=user.is_root()) managers = gems.load_managers(context=context, locale=i18n.current_key, config=app_config, @@ -53,12 +59,15 @@ def new_manage_panel(app_args: Namespace, app_config: dict, logger: logging.Logg util.clean_app_files(managers) exit(0) - manager = GenericSoftwareManager(managers, context=context, config=app_config) + force_suggestions = bool(app_args.suggestions) + manager = GenericSoftwareManager(managers, context=context, config=app_config, force_suggestions=force_suggestions) app = new_qt_application(app_config=app_config, logger=logger, quit_on_last_closed=True) screen_size = app.primaryScreen().size() context.screen_width, context.screen_height = screen_size.width(), screen_size.height() + logger.info(f"Screen: {screen_size.width()} x {screen_size.height()} " + f"(DPI: {int(app.primaryScreen().logicalDotsPerInch())})") if app_args.settings: # only settings window manager.cache_available_managers() @@ -72,6 +81,7 @@ def new_manage_panel(app_args: Namespace, app_config: dict, logger: logging.Logg context=context, http_client=http_client, icon=util.get_default_icon()[1], + force_suggestions=force_suggestions, logger=logger) prepare = PreparePanel(screen_size=screen_size, @@ -79,7 +89,8 @@ def new_manage_panel(app_args: Namespace, app_config: dict, logger: logging.Logg manager=manager, i18n=i18n, manage_window=manage_window, - app_config=app_config) + app_config=app_config, + force_suggestions=force_suggestions) cache_cleaner.start() return app, prepare diff --git a/bauh/view/core/config.py b/bauh/view/core/config.py index e9c7c5d8..2b2c5ee4 100644 --- a/bauh/view/core/config.py +++ b/bauh/view/core/config.py @@ -30,7 +30,7 @@ def get_default_config(self) -> dict: }, 'suggestions': { 'enabled': True, - 'by_type': 10 + 'by_type': 15 }, 'ui': { 'table': { @@ -51,7 +51,8 @@ def get_default_config(self) -> dict: 'download': { 'multithreaded': False, 'multithreaded_client': None, - 'icons': True + 'icons': True, + 'check_ssl': True }, 'store_root_password': True, 'disk': { diff --git a/bauh/view/core/controller.py b/bauh/view/core/controller.py index 44a93cc2..66f9acaa 100755 --- a/bauh/view/core/controller.py +++ b/bauh/view/core/controller.py @@ -1,10 +1,11 @@ +import os import re import shutil import time import traceback from subprocess import Popen, STDOUT from threading import Thread -from typing import List, Set, Type, Tuple, Dict, Optional, Generator, Callable +from typing import List, Set, Type, Tuple, Dict, Optional, Generator, Callable, Pattern from bauh.api.abstract.controller import SoftwareManager, SearchResult, ApplicationContext, UpgradeRequirements, \ UpgradeRequirement, TransactionResult, SoftwareAction, SettingsView, SettingsController @@ -16,6 +17,7 @@ from bauh.api.exception import NoInternetException from bauh.commons.boot import CreateConfigFile from bauh.commons.html import bold +from bauh.commons.util import sanitize_command_input from bauh.view.core.config import CoreConfigManager from bauh.view.core.settings import GenericSettingsManager from bauh.view.core.update import check_for_update @@ -38,7 +40,9 @@ def __init__(self, to_install: List[UpgradeRequirement], to_remove: List[Upgrade class GenericSoftwareManager(SoftwareManager, SettingsController): - def __init__(self, managers: List[SoftwareManager], context: ApplicationContext, config: dict): + def __init__(self, managers: List[SoftwareManager], context: ApplicationContext, config: dict, + force_suggestions: bool = False): + super(GenericSoftwareManager, self).__init__(context=context) self.managers = managers self.map = {t: m for m in self.managers for t in m.get_managed_types()} @@ -55,6 +59,7 @@ def __init__(self, managers: List[SoftwareManager], context: ApplicationContext, self.configman = CoreConfigManager() self._action_reset: Optional[CustomSoftwareAction] = None self._dynamic_extra_actions: Optional[Dict[CustomSoftwareAction, Callable[[dict], bool]]] = None + self.force_suggestions = force_suggestions @property def dynamic_extra_actions(self) -> Dict[CustomSoftwareAction, Callable[[dict], bool]]: @@ -155,25 +160,27 @@ def search(self, words: str, disk_loader: DiskCacheLoader = None, limit: int = - res = SearchResult.empty() if self.context.is_internet_available(): - norm_word = words.strip().lower() + norm_query = sanitize_command_input(words).lower() + self.logger.info(f"Search query: {norm_query}") - is_url = bool(RE_IS_URL.match(norm_word)) - disk_loader = self.disk_loader_factory.new() - disk_loader.start() + if norm_query: + is_url = bool(RE_IS_URL.match(norm_query)) + disk_loader = self.disk_loader_factory.new() + disk_loader.start() - threads = [] + threads = [] - for man in self.managers: - t = Thread(target=self._search, args=(norm_word, is_url, man, disk_loader, res)) - t.start() - threads.append(t) + for man in self.managers: + t = Thread(target=self._search, args=(norm_query, is_url, man, disk_loader, res)) + t.start() + threads.append(t) - for t in threads: - t.join() + for t in threads: + t.join() - if disk_loader: - disk_loader.stop_working() - disk_loader.join() + if disk_loader: + disk_loader.stop_working() + disk_loader.join() # res.installed = self._sort(res.installed, norm_word) # res.new = self._sort(res.new, norm_word) @@ -426,6 +433,7 @@ def prepare(self, task_manager: TaskManager, root_password: Optional[str], inter for t in prepare_tasks: t.join() + create_config.join() tf = time.time() self.logger.info(f'Finished ({tf - ti:.2f} seconds)') @@ -485,11 +493,12 @@ def _fill_suggestions(self, suggestions: list, man: SoftwareManager, limit: int, suggestions.extend(man_sugs) def list_suggestions(self, limit: int, filter_installed: bool) -> List[PackageSuggestion]: - if bool(self.config['suggestions']['enabled']): + if self.force_suggestions or bool(self.config['suggestions']['enabled']): if self.managers and self.context.is_internet_available(): suggestions, threads = [], [] for man in self.managers: - t = Thread(target=self._fill_suggestions, args=(suggestions, man, int(self.config['suggestions']['by_type']), filter_installed)) + t = Thread(target=self._fill_suggestions, + args=(suggestions, man, int(self.config['suggestions']['by_type']), filter_installed)) t.start() threads.append(t) diff --git a/bauh/view/core/downloader.py b/bauh/view/core/downloader.py index c1d17243..1800191c 100644 --- a/bauh/view/core/downloader.py +++ b/bauh/view/core/downloader.py @@ -23,13 +23,15 @@ class AdaptableFileDownloader(FileDownloader): - def __init__(self, logger: logging.Logger, multithread_enabled: bool, i18n: I18n, http_client: HttpClient, multithread_client: str): + def __init__(self, logger: logging.Logger, multithread_enabled: bool, i18n: I18n, http_client: HttpClient, + multithread_client: str, check_ssl: bool): self.logger = logger self.multithread_enabled = multithread_enabled self.i18n = i18n self.http_client = http_client self.supported_multithread_clients = ['aria2', 'axel'] self.multithread_client = multithread_client + self.check_ssl = check_ssl @staticmethod def is_aria2c_available() -> bool: @@ -75,6 +77,9 @@ def _get_aria2c_process(self, url: str, output_path: str, cwd: str, root_passwor def _get_axel_process(self, url: str, output_path: str, cwd: str, root_password: Optional[str], threads: int) -> SimpleProcess: cmd = ['axel', url, '-n', str(threads), '-4', '-c', '-T', '5'] + if not self.check_ssl: + cmd.append('-k') + if output_path: cmd.append(f'--output={output_path}') @@ -83,6 +88,9 @@ def _get_axel_process(self, url: str, output_path: str, cwd: str, root_password: def _get_wget_process(self, url: str, output_path: str, cwd: str, root_password: Optional[str]) -> SimpleProcess: cmd = ['wget', url, '-c', '--retry-connrefused', '-t', '10', '--no-config', '-nc'] + if not self.check_ssl: + cmd.append('--no-check-certificate') + if output_path: cmd.append('-O') cmd.append(output_path) diff --git a/bauh/view/core/settings.py b/bauh/view/core/settings.py index 2abf9d6d..4b9d99bf 100644 --- a/bauh/view/core/settings.py +++ b/bauh/view/core/settings.py @@ -12,7 +12,7 @@ from bauh.api.abstract.controller import SoftwareManager, SettingsController, SettingsView from bauh.api.abstract.view import TabComponent, InputOption, TextComponent, MultipleSelectComponent, \ PanelComponent, FormComponent, TabGroupComponent, SingleSelectComponent, SelectViewType, TextInputComponent, \ - FileChooserComponent, RangeInputComponent + FileChooserComponent, RangeInputComponent, ViewComponentAlignment from bauh.commons.view_utils import new_select from bauh.view.core import timeshift from bauh.view.core.config import CoreConfigManager, BACKUP_REMOVE_METHODS, BACKUP_DEFAULT_REMOVE_METHOD @@ -87,7 +87,7 @@ def get_settings(self) -> TabGroupComponent: id_='core.types')) tabs.append(self._gen_general_settings(core_config)) - tabs.append(self._gen_ui_settings(core_config)) + tabs.append(self._gen_interface_settings(core_config)) tabs.append(self._gen_tray_settings(core_config)) tabs.append(self._gen_adv_settings(core_config)) @@ -102,26 +102,22 @@ def get_settings(self) -> TabGroupComponent: return TabGroupComponent(tabs) def _gen_adv_settings(self, core_config: dict) -> TabComponent: - default_width = 300 input_data_exp = TextInputComponent(label=self.i18n['core.config.mem_cache.data_exp'], tooltip=self.i18n['core.config.mem_cache.data_exp.tip'], value=str(core_config['memory_cache']['data_expiration']), only_int=True, - max_width=60, id_="data_exp") input_icon_exp = TextInputComponent(label=self.i18n['core.config.mem_cache.icon_exp'], tooltip=self.i18n['core.config.mem_cache.icon_exp.tip'], value=str(core_config['memory_cache']['icon_expiration']), only_int=True, - max_width=60, id_="icon_exp") select_trim = new_select(label=self.i18n['core.config.trim.after_upgrade'], tip=self.i18n['core.config.trim.after_upgrade.tip'], value=core_config['disk']['trim']['after_upgrade'], - max_width=default_width, opts=[(self.i18n['yes'].capitalize(), True, None), (self.i18n['no'].capitalize(), False, None), (self.i18n['ask'].capitalize(), None, None)], @@ -130,22 +126,26 @@ def _gen_adv_settings(self, core_config: dict) -> TabComponent: select_dep_check = self._gen_bool_component(label=self.i18n['core.config.system.dep_checking'], tooltip=self.i18n['core.config.system.dep_checking.tip'], value=core_config['system']['single_dependency_checking'], - max_width=default_width, id_='dep_check') + select_check_ssl = self._gen_bool_component(label=self.i18n['core.config.download.check_ssl'], + tooltip=self.i18n['core.config.download.check_ssl.tip'], + value=core_config['download']['check_ssl'], + id_='download.check_ssl') + select_dmthread = self._gen_bool_component(label=self.i18n['core.config.download.multithreaded'], tooltip=self.i18n['core.config.download.multithreaded.tip'], id_="down_mthread", - max_width=default_width, value=core_config['download']['multithreaded']) - select_mthread_client = self._gen_multithread_client_select(core_config, default_width) + select_mthread_client = self._gen_multithread_client_select(core_config) - inputs = [select_dmthread, select_mthread_client, select_trim, select_dep_check, input_data_exp, input_icon_exp] - panel = PanelComponent([FormComponent(inputs, spaces=False)]) + inputs = [select_dmthread, select_mthread_client, select_check_ssl, select_trim, select_dep_check, + input_data_exp, input_icon_exp] + panel = PanelComponent([FormComponent(inputs, spaces=False)], id_='advanced') return TabComponent(self.i18n['core.config.tab.advanced'].capitalize(), panel, None, 'core.adv') - def _gen_multithread_client_select(self, core_config: dict, default_width: int) -> SingleSelectComponent: + def _gen_multithread_client_select(self, core_config: dict) -> SingleSelectComponent: available_mthread_clients = self.file_downloader.list_available_multithreaded_clients() available_mthread_clients.sort() @@ -163,18 +163,14 @@ def _gen_multithread_client_select(self, core_config: dict, default_width: int) return new_select(label=self.i18n['core.config.download.multithreaded_client'], tip=self.i18n['core.config.download.multithreaded_client.tip'], id_="mthread_client", - max_width=default_width, opts=mthread_client_opts, value=current_mthread_client) def _gen_tray_settings(self, core_config: dict) -> TabComponent: - default_width = 350 - input_update_interval = TextInputComponent(label=self.i18n['core.config.updates.interval'].capitalize(), tooltip=self.i18n['core.config.updates.interval.tip'], only_int=True, value=str(core_config['updates']['check_interval']), - max_width=60, id_="updates_interval") allowed_exts = {'png', 'svg', 'jpg', 'jpeg', 'ico', 'xpm'} @@ -183,7 +179,6 @@ def _gen_tray_settings(self, core_config: dict) -> TabComponent: label=self.i18n["core.config.ui.tray.default_icon"], tooltip=self.i18n["core.config.ui.tray.default_icon.tip"], file_path=de_path, - max_width=default_width, allowed_extensions=allowed_exts) up_path = str(core_config['ui']['tray']['updates_icon']) if core_config['ui']['tray']['updates_icon'] else None @@ -191,27 +186,22 @@ def _gen_tray_settings(self, core_config: dict) -> TabComponent: label=self.i18n["core.config.ui.tray.updates_icon"].capitalize(), tooltip=self.i18n["core.config.ui.tray.updates_icon.tip"].capitalize(), file_path=up_path, - max_width=default_width, allowed_extensions=allowed_exts) sub_comps = [FormComponent([select_def_icon, select_up_icon, input_update_interval], spaces=False)] return TabComponent(self.i18n['core.config.tab.tray'].capitalize(), - PanelComponent(sub_comps), None, 'core.tray') - - def _gen_ui_settings(self, core_config: dict) -> TabComponent: - default_width = 200 + PanelComponent(sub_comps, id_='tray'), None, 'core.tray') + def _gen_interface_settings(self, core_config: dict) -> TabComponent: select_hdpi = self._gen_bool_component(label=self.i18n['core.config.ui.hdpi'], tooltip=self.i18n['core.config.ui.hdpi.tip'], value=bool(core_config['ui']['hdpi']), - max_width=default_width, id_='hdpi') scale_tip = self.i18n['core.config.ui.auto_scale.tip'].format('QT_AUTO_SCREEN_SCALE_FACTOR') select_ascale = self._gen_bool_component(label=self.i18n['core.config.ui.auto_scale'], tooltip=scale_tip, value=bool(core_config['ui']['auto_scale']), - max_width=default_width, id_='auto_scale') try: @@ -224,8 +214,7 @@ def _gen_ui_settings(self, core_config: dict) -> TabComponent: select_scale = RangeInputComponent(id_="scalef", label=self.i18n['core.config.ui.scale_factor'] + ' (%)', tooltip=self.i18n['core.config.ui.scale_factor.tip'], - min_value=100, max_value=400, step_value=5, value=int(scale * 100), - max_width=60) + min_value=100, max_value=400, step_value=5, value=int(scale * 100)) if not core_config['ui']['qt_style']: cur_style = QApplication.instance().property('qt_style') @@ -250,38 +239,34 @@ def _gen_ui_settings(self, core_config: dict) -> TabComponent: options=style_opts, default_option=default_style, type_=SelectViewType.COMBO, - max_width=default_width, + alignment=ViewComponentAlignment.CENTER, id_="style") systheme_tip = self.i18n['core.config.ui.system_theme.tip'].format(app=__app_name__) select_system_theme = self._gen_bool_component(label=self.i18n['core.config.ui.system_theme'], tooltip=systheme_tip, value=bool(core_config['ui']['system_theme']), - max_width=default_width, id_='system_theme') input_maxd = TextInputComponent(label=self.i18n['core.config.ui.max_displayed'], tooltip=self.i18n['core.config.ui.max_displayed.tip'], only_int=True, id_="table_max", - max_width=50, value=str(core_config['ui']['table']['max_displayed'])) select_dicons = self._gen_bool_component(label=self.i18n['core.config.download.icons'], tooltip=self.i18n['core.config.download.icons.tip'], id_="down_icons", - max_width=default_width, value=core_config['download']['icons']) sub_comps = [FormComponent([select_hdpi, select_ascale, select_scale, select_dicons, select_system_theme, select_style, input_maxd], spaces=False)] - return TabComponent(self.i18n['core.config.tab.ui'].capitalize(), PanelComponent(sub_comps), None, 'core.ui') + return TabComponent(self.i18n['core.config.tab.ui'].capitalize(), + PanelComponent(sub_comps, id_='interface'), None, 'core.ui') def _gen_general_settings(self, core_config: dict) -> TabComponent: - default_width = floor(0.15 * self.context.screen_width) - locale_keys = translation.get_available_keys() locale_opts = [InputOption(label=self.i18n[f'locale.{k}'].capitalize(), value=k) for k in locale_keys] @@ -303,55 +288,49 @@ def _gen_general_settings(self, core_config: dict) -> TabComponent: options=locale_opts, default_option=current_locale, type_=SelectViewType.COMBO, - max_width=default_width, + alignment=ViewComponentAlignment.CENTER, id_='locale') sel_store_pwd = self._gen_bool_component(label=self.i18n['core.config.store_password'].capitalize(), tooltip=self.i18n['core.config.store_password.tip'].capitalize(), id_="store_pwd", - max_width=default_width, value=bool(core_config['store_root_password'])) notify_tip = self.i18n['core.config.system.notifications.tip'].capitalize() sel_sys_notify = self._gen_bool_component(label=self.i18n['core.config.system.notifications'].capitalize(), tooltip=notify_tip, value=bool(core_config['system']['notifications']), - max_width=default_width, id_="sys_notify") sel_load_apps = self._gen_bool_component(label=self.i18n['core.config.boot.load_apps'], tooltip=self.i18n['core.config.boot.load_apps.tip'], value=bool(core_config['boot']['load_apps']), - id_='boot.load_apps', - max_width=default_width) + id_='boot.load_apps') sel_sugs = self._gen_bool_component(label=self.i18n['core.config.suggestions.activated'].capitalize(), tooltip=self.i18n['core.config.suggestions.activated.tip'].capitalize(), id_="sugs_enabled", - max_width=default_width, value=bool(core_config['suggestions']['enabled'])) inp_sugs = TextInputComponent(label=self.i18n['core.config.suggestions.by_type'], tooltip=self.i18n['core.config.suggestions.by_type.tip'], value=str(core_config['suggestions']['by_type']), only_int=True, - max_width=50, id_="sugs_by_type") inp_reboot = new_select(label=self.i18n['core.config.updates.reboot'], tip=self.i18n['core.config.updates.reboot.tip'], id_='ask_for_reboot', - max_width=default_width, + max_width=None, value=bool(core_config['updates']['ask_for_reboot']), opts=[(self.i18n['ask'].capitalize(), True, None), (self.i18n['no'].capitalize(), False, None)]) inputs = [sel_locale, sel_store_pwd, sel_sys_notify, sel_load_apps, inp_reboot, sel_sugs, inp_sugs] - panel = PanelComponent([FormComponent(inputs, spaces=False)]) + panel = PanelComponent([FormComponent(inputs, spaces=False)], id_='general') return TabComponent(self.i18n['core.config.tab.general'].capitalize(), panel, None, 'core.gen') - def _gen_bool_component(self, label: str, tooltip: Optional[str], value: bool, id_: str, max_width: int = 200) \ - -> SingleSelectComponent: + def _gen_bool_component(self, label: str, tooltip: Optional[str], value: bool, id_: str) -> SingleSelectComponent: opts = [InputOption(label=self.i18n['yes'].capitalize(), value=True), InputOption(label=self.i18n['no'].capitalize(), value=False)] @@ -362,7 +341,6 @@ def _gen_bool_component(self, label: str, tooltip: Optional[str], value: bool, i type_=SelectViewType.RADIO, tooltip=tooltip, max_per_line=len(opts), - max_width=max_width, id_=id_) def _save_settings(self, general: PanelComponent, @@ -408,9 +386,13 @@ def _save_settings(self, general: PanelComponent, mthread_client = adv_form.get_component('mthread_client', SingleSelectComponent).get_selected() core_config['download']['multithreaded_client'] = mthread_client + check_ssl = adv_form.get_component('download.check_ssl', SingleSelectComponent).get_selected() + core_config['download']['check_ssl'] = check_ssl + if isinstance(self.file_downloader, AdaptableFileDownloader): self.file_downloader.multithread_client = mthread_client self.file_downloader.multithread_enabled = download_mthreaded + self.file_downloader.check_ssl = check_ssl single_dep_check = adv_form.get_component('dep_check', SingleSelectComponent).get_selected() core_config['system']['single_dependency_checking'] = single_dep_check @@ -570,15 +552,12 @@ def save_settings(self, component: TabGroupComponent) -> Tuple[bool, Optional[Li self.logger.info(f"Saving all settings took {tf - ti:.8f} seconds") return success, warnings - def _gen_backup_settings(self, core_config: dict) -> TabComponent: + def _gen_backup_settings(self, core_config: dict) -> Optional[TabComponent]: if timeshift.is_available(): - default_width = 350 - enabled_opt = self._gen_bool_component(label=self.i18n['core.config.backup'], tooltip=None, value=bool(core_config['backup']['enabled']), - id_='enabled', - max_width=default_width) + id_='enabled') ops_opts = [(self.i18n['yes'].capitalize(), True, None), (self.i18n['no'].capitalize(), False, None), @@ -588,28 +567,24 @@ def _gen_backup_settings(self, core_config: dict) -> TabComponent: tip=None, value=core_config['backup']['install'], opts=ops_opts, - max_width=default_width, id_='install') uninstall_mode = new_select(label=self.i18n['core.config.backup.uninstall'], tip=None, value=core_config['backup']['uninstall'], opts=ops_opts, - max_width=default_width, id_='uninstall') upgrade_mode = new_select(label=self.i18n['core.config.backup.upgrade'], tip=None, value=core_config['backup']['upgrade'], opts=ops_opts, - max_width=default_width, id_='upgrade') downgrade_mode = new_select(label=self.i18n['core.config.backup.downgrade'], tip=None, value=core_config['backup']['downgrade'], opts=ops_opts, - max_width=default_width, id_='downgrade') mode = new_select(label=self.i18n['core.config.backup.mode'], @@ -621,13 +596,11 @@ def _gen_backup_settings(self, core_config: dict) -> TabComponent: (self.i18n['core.config.backup.mode.only_one'], 'only_one', self.i18n['core.config.backup.mode.only_one.tip']) ], - max_width=default_width, id_='mode') type_ = new_select(label=self.i18n['type'].capitalize(), tip=None, value=core_config['backup']['type'], opts=[('rsync', 'rsync', None), ('btrfs', 'btrfs', None)], - max_width=default_width, id_='type') remove_method = core_config['backup']['remove_method'] @@ -646,10 +619,9 @@ def _gen_backup_settings(self, core_config: dict) -> TabComponent: tip=None, value=remove_method, opts=remove_opts, - max_width=default_width, capitalize_label=False, id_='remove_method') inputs = [enabled_opt, type_, mode, sel_remove, install_mode, uninstall_mode, upgrade_mode, downgrade_mode] - panel = PanelComponent([FormComponent(inputs, spaces=False)]) + panel = PanelComponent([FormComponent(inputs, spaces=False)], id_='backup') return TabComponent(self.i18n['core.config.tab.backup'].capitalize(), panel, None, 'core.bkp') diff --git a/bauh/view/core/suggestions.py b/bauh/view/core/suggestions.py new file mode 100644 index 00000000..280f3a88 --- /dev/null +++ b/bauh/view/core/suggestions.py @@ -0,0 +1,32 @@ +import os +from typing import Optional, Dict +from bauh import __app_name__ + + +def read_suggestions_mapping() -> Optional[Dict[str, str]]: + file_path = f'/etc/{__app_name__}/suggestions.conf' + + if os.path.isfile(file_path): + try: + with open(file_path) as f: + file_content = f.read() + except FileNotFoundError: + return + + if not file_content: + return + + mapping = {} + for line in file_content.split('\n'): + line_strip = line.strip() + + if not line_strip.startswith('#'): + gem_file = line_strip.split('=') + + if len(gem_file) == 2: + gem_name, file_url = gem_file[0].strip(), gem_file[1].strip() + + if gem_name and file_url: + mapping[gem_name] = file_url + + return mapping if mapping else None diff --git a/bauh/view/qt/apps_table.py b/bauh/view/qt/apps_table.py index 5a75367f..87dfe125 100644 --- a/bauh/view/qt/apps_table.py +++ b/bauh/view/qt/apps_table.py @@ -18,10 +18,6 @@ from bauh.view.qt.view_model import PackageView from bauh.view.util.translation import I18n -NAME_MAX_SIZE = 30 -DESC_MAX_SIZE = 40 -PUBLISHER_MAX_SIZE = 25 - class UpgradeToggleButton(QToolButton): @@ -72,8 +68,9 @@ class PackagesTable(QTableWidget): COL_NUMBER = 9 DEFAULT_ICON_SIZE = QSize(16, 16) - def __init__(self, parent: QWidget, icon_cache: MemoryCache, download_icons: bool): + def __init__(self, parent: QWidget, icon_cache: MemoryCache, download_icons: bool, screen_width: int): super(PackagesTable, self).__init__() + self.screen_width = screen_width self.setObjectName('table_packages') self.setParent(parent) self.window = parent @@ -85,7 +82,7 @@ def __init__(self, parent: QWidget, icon_cache: MemoryCache, download_icons: boo self.horizontalHeader().setVisible(False) self.horizontalHeader().setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Preferred) self.setSelectionBehavior(QTableView.SelectRows) - self.setHorizontalHeaderLabels(['' for _ in range(self.columnCount())]) + self.setHorizontalHeaderLabels(('' for _ in range(self.columnCount()))) self.setHorizontalScrollBarPolicy(Qt.ScrollBarAlwaysOff) self.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Preferred) self.horizontalScrollBar().setCursor(QCursor(Qt.PointingHandCursor)) @@ -156,8 +153,7 @@ def ignore_updates(): custom_actions = pkg.model.get_custom_actions() if custom_actions: - actions = [self._map_custom_action(pkg, a, menu_row) for a in custom_actions] - menu_row.addActions(actions) + menu_row.addActions((self._map_custom_action(pkg, a, menu_row) for a in custom_actions)) menu_row.adjustSize() menu_row.popup(QCursor.pos()) @@ -383,8 +379,12 @@ def _set_col_version(self, col: int, pkg: PackageView): tooltip = self.i18n['version.updates_ignored'] if pkg.model.installed and pkg.model.update and not pkg.model.is_update_ignored() and pkg.model.version and pkg.model.latest_version and pkg.model.version != pkg.model.latest_version: - tooltip = '{}. {}: {}'.format(tooltip, self.i18n['version.latest'], pkg.model.latest_version) - label_version.setText(label_version.text() + ' > {}'.format(pkg.model.latest_version)) + tooltip = f"{tooltip} ({self.i18n['version.installed']}: {pkg.model.version} | " \ + f"{self.i18n['version.latest']}: {pkg.model.latest_version})" + label_version.setText(f"{label_version.text()} > {pkg.model.latest_version}") + + if label_version.sizeHint().width() / self.screen_width > 0.22: + label_version.setText(pkg.model.latest_version) item.setToolTip(tooltip) self.setCellWidget(pkg.table_index, col, item) @@ -431,20 +431,20 @@ def _set_col_name(self, col: int, pkg: PackageView): col_name.setObjectName('app_name') col_name.setSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.Preferred) - name = pkg.model.get_display_name() + name = pkg.model.get_display_name().strip() if name: col_name.setToolTip('{}: {}'.format(self.i18n['app.name'].lower(), pkg.model.get_name_tooltip())) else: name = '...' col_name.setToolTip(self.i18n['app.name'].lower()) - if len(name) > NAME_MAX_SIZE: - name = name[0:NAME_MAX_SIZE - 3] + '...' + col_name.setText(name) + screen_perc = col_name.sizeHint().width() / self.screen_width - if len(name) < NAME_MAX_SIZE: - name = name + ' ' * (NAME_MAX_SIZE - len(name)) + if screen_perc > 0.15: + max_chars = int(len(name) * 0.15 / screen_perc) - 3 + col_name.setText(name[0:max_chars] + '...') - col_name.setText(name) self.setCellWidget(pkg.table_index, col, col_name) def _update_icon(self, label: QLabel, icon: QIcon): @@ -463,11 +463,17 @@ def _set_col_description(self, col: int, pkg: PackageView): else: desc = '...' - if desc and desc != '...' and len(desc) > DESC_MAX_SIZE: - desc = strip_html(desc[0: DESC_MAX_SIZE - 1]) + '...' + if desc and desc != '...': + desc = strip_html(desc) item.setText(desc) + current_width_perc = item.sizeHint().width() / self.screen_width + if current_width_perc > 0.18: + max_width = int(len(desc) * 0.18 / current_width_perc) - 3 + desc = desc[0:max_width] + '...' + item.setText(desc) + if pkg.model.description: item.setToolTip(pkg.model.description) @@ -479,15 +485,21 @@ def _set_col_publisher(self, col: int, pkg: PackageView): publisher = pkg.model.get_publisher() full_publisher = None + lb_name = QLabel() + lb_name.setObjectName('app_publisher') + if publisher: publisher = publisher.strip() full_publisher = publisher - if len(publisher) > PUBLISHER_MAX_SIZE: - publisher = full_publisher[0: PUBLISHER_MAX_SIZE - 3] + '...' + if publisher: + lb_name.setText(publisher) + screen_perc = lb_name.sizeHint().width() / self.screen_width - lb_name = QLabel() - lb_name.setObjectName('app_publisher') + if screen_perc > 0.12: + max_chars = int(len(publisher) * 0.12 / screen_perc) - 3 + publisher = publisher[0: max_chars] + '...' + lb_name.setText(publisher) if not publisher: if not pkg.model.installed: @@ -495,7 +507,7 @@ def _set_col_publisher(self, col: int, pkg: PackageView): publisher = self.i18n['unknown'] - lb_name.setText(' {}'.format(publisher)) + lb_name.setText(f' {publisher}') item.addWidget(lb_name) if publisher and full_publisher: @@ -570,10 +582,10 @@ def change_headers_policy(self, policy: QHeaderView = QHeaderView.ResizeToConten header_horizontal = self.horizontalHeader() for i in range(self.columnCount()): if maximized: - if i not in (4, 5, 8): - header_horizontal.setSectionResizeMode(i, QHeaderView.ResizeToContents) - else: + if i in (2, 3): header_horizontal.setSectionResizeMode(i, QHeaderView.Stretch) + else: + header_horizontal.setSectionResizeMode(i, QHeaderView.ResizeToContents) else: header_horizontal.setSectionResizeMode(i, policy) diff --git a/bauh/view/qt/components.py b/bauh/view/qt/components.py index 7b9da50f..7c04a9cc 100644 --- a/bauh/view/qt/components.py +++ b/bauh/view/qt/components.py @@ -11,7 +11,8 @@ from bauh.api.abstract.view import SingleSelectComponent, InputOption, MultipleSelectComponent, SelectViewType, \ TextInputComponent, FormComponent, FileChooserComponent, ViewComponent, TabGroupComponent, PanelComponent, \ - TwoStateButtonComponent, TextComponent, SpacerComponent, RangeInputComponent, ViewObserver, TextInputType + TwoStateButtonComponent, TextComponent, SpacerComponent, RangeInputComponent, ViewObserver, TextInputType, \ + ViewComponentAlignment from bauh.view.util.translation import I18n @@ -256,6 +257,25 @@ def remove_saved_state(self, state_id: int): del self._saved_states[state_id] +def map_alignment(alignment: ViewComponentAlignment) -> Optional[int]: + if alignment == ViewComponentAlignment.CENTER: + return Qt.AlignCenter + elif alignment == ViewComponentAlignment.LEFT: + return Qt.AlignLeft + elif alignment == ViewComponentAlignment.RIGHT: + return Qt.AlignRight + elif alignment == ViewComponentAlignment.HORIZONTAL_CENTER: + return Qt.AlignHCenter + elif alignment == ViewComponentAlignment.VERTICAL_CENTER: + return Qt.AlignVCenter + elif alignment == ViewComponentAlignment.BOTTOM: + return Qt.AlignBottom + elif alignment == ViewComponentAlignment.TOP: + return Qt.AlignTop + else: + return + + class RadioButtonQt(QRadioButton): def __init__(self, model: InputOption, model_parent: SingleSelectComponent): @@ -265,6 +285,9 @@ def __init__(self, model: InputOption, model_parent: SingleSelectComponent): self.toggled.connect(self._set_checked) self.setCursor(QCursor(Qt.PointingHandCursor)) + if model_parent.id: + self.setProperty('parent', model_parent.id) + if model.icon_path: if model.icon_path.startswith('/'): self.setIcon(QIcon(model.icon_path)) @@ -344,8 +367,17 @@ class FormComboBoxQt(QComboBox): def __init__(self, model: SingleSelectComponent): super(FormComboBoxQt, self).__init__() self.model = model + self.setSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.Fixed) self.setCursor(QCursor(Qt.PointingHandCursor)) self.view().setCursor(QCursor(Qt.PointingHandCursor)) + self.setEditable(True) + self.lineEdit().setReadOnly(True) + + if model.alignment: + comp_alignment = map_alignment(model.alignment) + + if comp_alignment is not None: + self.lineEdit().setAlignment(comp_alignment) if model.max_width > 0: self.setMaximumWidth(int(model.max_width)) @@ -363,6 +395,9 @@ def __init__(self, model: SingleSelectComponent): self.currentIndexChanged.connect(self._set_selected) + if model.id: + self.setObjectName(model.id) + def _set_selected(self, idx: int): self.model.value = self.model.options[idx] self.setToolTip(self.model.value.tooltip) @@ -373,9 +408,13 @@ class FormRadioSelectQt(QWidget): def __init__(self, model: SingleSelectComponent, parent: QWidget = None): super(FormRadioSelectQt, self).__init__(parent=parent) self.model = model - self.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Minimum) + self.setSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.Minimum) + self.setProperty('opts', str(len(self.model.options) if self.model.options else 0)) - if model.max_width > 0: + if model.id: + self.setObjectName(model.id) + + if model.max_width and model.max_width > 0: self.setMaximumWidth(int(model.max_width)) grid = QGridLayout() @@ -400,7 +439,7 @@ def __init__(self, model: SingleSelectComponent, parent: QWidget = None): else: col += 1 - if model.max_width <= 0: + if model.max_width is not None and model.max_width <= 0: self.setMaximumWidth(int(self.sizeHint().width())) @@ -408,14 +447,20 @@ class RadioSelectQt(QGroupBox): def __init__(self, model: SingleSelectComponent): super(RadioSelectQt, self).__init__(model.label + ' :' if model.label else None) + + if model.id: + self.setObjectName(model.id) + if not model.label: - self.setObjectName('radio_select_notitle') + self.setProperty('no_label', 'true') self.model = model grid = QGridLayout() self.setLayout(grid) + self.setProperty('opts', str(len(model.options)) if model.options else '0') + line, col = 0, 0 for op in model.options: comp = RadioButtonQt(op, model) @@ -445,6 +490,9 @@ def __init__(self, model: SingleSelectComponent): self._layout.addWidget(QLabel(model.label + ' :' if model.label else ''), 0, 0) self._layout.addWidget(FormComboBoxQt(model), 0, 1) + if model.id: + self.setObjectName(model.id) + class QLineEditObserver(QLineEdit, ViewObserver): @@ -479,7 +527,10 @@ def __init__(self, model: TextInputComponent): self.model = model self.setLayout(QGridLayout()) - if self.model.max_width > 0: + if model.id: + self.setObjectName(model.id) + + if self.model.max_width and self.model.max_width > 0: self.setMaximumWidth(int(self.model.max_width)) self.text_input = QLineEditObserver() if model.type == TextInputType.SINGLE_LINE else QPlainTextEditObserver() @@ -578,6 +629,9 @@ def __init__(self, model: MultipleSelectComponent, callback): pos_label = QLabel() self.layout().addWidget(pos_label, line + 1, 1) + if model.id: + self.setObjectName(model.id) + class FormMultipleSelectQt(QWidget): @@ -644,6 +698,9 @@ def __init__(self, model: MultipleSelectComponent, parent: QWidget = None): if model.label: self.layout().addWidget(QLabel(), line + 1, 1) + if model.id: + self.setObjectName(model.id) + class InputFilter(QLineEdit): @@ -705,6 +762,10 @@ def __init__(self, model: PanelComponent, i18n: I18n, parent: QWidget = None): super(PanelQt, self).__init__(parent=parent) self.model = model self.i18n = i18n + + if model.id: + self.setObjectName(model.id) + self.setLayout(QVBoxLayout()) self.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Minimum) @@ -721,6 +782,9 @@ def __init__(self, model: FormComponent, i18n: I18n): self.i18n = i18n self.setLayout(QFormLayout()) + if model.id: + self.setObjectName(model.id) + if model.min_width and model.min_width > 0: self.setMinimumWidth(model.min_width) @@ -809,6 +873,10 @@ def gen_tip_icon(self, tip: str) -> QLabel: def _new_text_input(self, c: TextInputComponent) -> Tuple[QLabel, QLineEdit]: view = QLineEditObserver() if c.type == TextInputType.SINGLE_LINE else QPlainTextEditObserver() + view.setSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.Fixed) + + if c.id: + view.setObjectName(c.id) if c.min_width >= 0: view.setMinimumWidth(int(c.min_width)) @@ -855,12 +923,16 @@ def update_model(text: str): def _new_range_input(self, model: RangeInputComponent) -> QSpinBox: spinner = QSpinBox() + spinner.setSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.Fixed) spinner.setCursor(QCursor(Qt.PointingHandCursor)) spinner.setMinimum(model.min) spinner.setMaximum(model.max) spinner.setSingleStep(model.step) spinner.setValue(model.value if model.value is not None else model.min) + if model.id: + spinner.setObjectName(model.id) + if model.tooltip: spinner.setToolTip(model.tooltip) @@ -872,10 +944,18 @@ def _update_value(): def _wrap(self, comp: QWidget, model: ViewComponent) -> QWidget: field_container = QWidget() + field_container.setSizePolicy(QSizePolicy.Minimum, QSizePolicy.Minimum) field_container.setLayout(QHBoxLayout()) field_container.layout().setContentsMargins(0, 0, 0, 0) + field_container.layout().setSpacing(0) + field_container.layout().setAlignment(Qt.AlignLeft) + field_container.setProperty('wrapper', 'true') + field_container.setProperty('wrapped_type', comp.__class__.__name__) - if model.max_width > 0: + if model.id: + field_container.setProperty('wrapped', model.id) + + if model.max_width and model.max_width > 0: field_container.setMaximumWidth(int(model.max_width)) field_container.layout().addWidget(comp) @@ -883,9 +963,14 @@ def _wrap(self, comp: QWidget, model: ViewComponent) -> QWidget: def _new_file_chooser(self, c: FileChooserComponent) -> Tuple[QLabel, QLineEdit]: chooser = QLineEditObserver() + chooser.setProperty('file_chooser', 'true') + chooser.setSizePolicy(QSizePolicy.MinimumExpanding, QSizePolicy.Fixed) chooser.setReadOnly(True) - if c.max_width > 0: + if c.id: + chooser.setObjectName(c.id) + + if c.max_width and c.max_width > 0: chooser.setMaximumWidth(int(c.max_width)) if c.file_path: @@ -957,6 +1042,7 @@ def __init__(self, model: TabGroupComponent, i18n: I18n, parent: QWidget = None) icon = QIcon() scroll = QScrollArea() + scroll.setSizePolicy(QSizePolicy.Expanding, QSizePolicy.Expanding) scroll.setFrameShape(QFrame.NoFrame) scroll.setWidgetResizable(True) scroll.setWidget(to_widget(c.get_content(), i18n)) @@ -974,13 +1060,19 @@ def new_single_select(model: SingleSelectComponent) -> QWidget: raise Exception("Unsupported type {}".format(model.type)) -def new_spacer(min_width: int = None) -> QWidget: +def new_spacer(min_width: Optional[int] = None, min_height: Optional[int] = None, max_width: Optional[int] = None) -> QWidget: spacer = QWidget() spacer.setProperty('spacer', 'true') - if min_width: + if min_width is not None and min_width >= 0: spacer.setMinimumWidth(int(min_width)) + if max_width is not None and max_width >= 0: + spacer.setMaximumWidth(max_width) + + if min_height is not None and min_height >= 0: + spacer.setMaximumHeight(int(min_height)) + spacer.setSizePolicy(QSizePolicy.Expanding, QSizePolicy.Expanding) return spacer diff --git a/bauh/view/qt/prepare.py b/bauh/view/qt/prepare.py index 71ee58b7..b93ee42e 100644 --- a/bauh/view/qt/prepare.py +++ b/bauh/view/qt/prepare.py @@ -146,7 +146,7 @@ class PreparePanel(QWidget, TaskManager): signal_password_response = pyqtSignal(bool, str) def __init__(self, context: ApplicationContext, manager: SoftwareManager, screen_size: QSize, - i18n: I18n, manage_window: QWidget, app_config: dict): + i18n: I18n, manage_window: QWidget, app_config: dict, force_suggestions: bool = False): super(PreparePanel, self).__init__(flags=Qt.CustomizeWindowHint | Qt.WindowTitleHint) self.i18n = i18n self.context = context @@ -165,6 +165,7 @@ def __init__(self, context: ApplicationContext, manager: SoftwareManager, screen self.ftasks = 0 self.started_at = None self.self_close = False + self.force_suggestions = force_suggestions self.prepare_thread = Prepare(self.context, manager, self.i18n) self.prepare_thread.signal_register.connect(self.register_task) @@ -438,7 +439,9 @@ def finish(self): if self.isVisible(): self.manage_window.show() - if self.app_config['boot']['load_apps']: + if self.force_suggestions: + self.manage_window.begin_load_suggestions(filter_installed=True) + elif self.app_config['boot']['load_apps']: self.manage_window.begin_refresh_packages() else: self.manage_window.load_without_packages() diff --git a/bauh/view/qt/settings.py b/bauh/view/qt/settings.py index bb7bf0df..ae732c7c 100644 --- a/bauh/view/qt/settings.py +++ b/bauh/view/qt/settings.py @@ -83,6 +83,7 @@ def __init__(self, manager: SoftwareManager, i18n: I18n, window: QWidget, parent def show(self): super(SettingsWindow, self).show() centralize(self) + self.setMinimumWidth(int(self.sizeHint().width())) def closeEvent(self, event): if self.window and self.window.settings_window == self: diff --git a/bauh/view/qt/window.py b/bauh/view/qt/window.py index 2cb6b538..dfb90663 100755 --- a/bauh/view/qt/window.py +++ b/bauh/view/qt/window.py @@ -97,7 +97,8 @@ class ManageWindow(QWidget): signal_stop_notifying = pyqtSignal() def __init__(self, i18n: I18n, icon_cache: MemoryCache, manager: SoftwareManager, screen_size: QSize, config: dict, - context: ApplicationContext, http_client: HttpClient, logger: logging.Logger, icon: QIcon): + context: ApplicationContext, http_client: HttpClient, logger: logging.Logger, icon: QIcon, + force_suggestions: bool = False): super(ManageWindow, self).__init__() self.setObjectName('manage_window') self.comp_manager = QtComponentsManager() @@ -277,7 +278,9 @@ def __init__(self, i18n: I18n, icon_cache: MemoryCache, manager: SoftwareManager self.table_container.setLayout(QVBoxLayout()) self.table_container.layout().setContentsMargins(0, 0, 0, 0) - self.table_apps = PackagesTable(self, self.icon_cache, download_icons=bool(self.config['download']['icons'])) + self.table_apps = PackagesTable(self, self.icon_cache, + download_icons=bool(self.config['download']['icons']), + screen_width=int(screen_size.width())) self.table_apps.change_headers_policy() self.table_container.layout().addWidget(self.table_apps) @@ -373,8 +376,11 @@ def __init__(self, i18n: I18n, icon_cache: MemoryCache, manager: SoftwareManager self.container_bottom.layout().addWidget(new_spacer()) - if config['suggestions']['enabled']: - bt_sugs = IconButton(action=lambda: self._begin_load_suggestions(filter_installed=True), + self.load_suggestions = force_suggestions or bool(config['suggestions']['enabled']) + self.suggestions_requested = False + + if self.load_suggestions: + bt_sugs = IconButton(action=lambda: self.begin_load_suggestions(filter_installed=True), i18n=i18n, tooltip=self.i18n['manage_window.bt.suggestions.tooltip']) bt_sugs.setObjectName('suggestions') @@ -440,8 +446,6 @@ def __init__(self, i18n: I18n, icon_cache: MemoryCache, manager: SoftwareManager self.types_changed = False self.dialog_about = None - self.load_suggestions = bool(config['suggestions']['enabled']) - self.suggestions_requested = False self.first_refresh = True self.thread_warnings = ListWarnings(man=manager, i18n=i18n) @@ -454,7 +458,9 @@ def __init__(self, i18n: I18n, icon_cache: MemoryCache, manager: SoftwareManager self.thread_load_installed = NotifyInstalledLoaded() self.thread_load_installed.signal_loaded.connect(self._finish_loading_installed) self.setMinimumHeight(int(screen_size.height() * 0.5)) - self.setMinimumWidth(int(screen_size.width() * 0.6)) + self.setMaximumHeight(int(screen_size.height())) + self.setMinimumWidth(int(screen_size.width() * 0.5)) + self.setMaximumWidth(int(screen_size.width() - screen_size.width() * 0.015)) self._register_groups() def _register_groups(self): @@ -512,6 +518,7 @@ def begin_apply_filters(self): def _finish_apply_filters(self): self._finish_action(ACTION_APPLY_FILTERS) self.update_bt_upgrade() + self._resize() def stop_notifying_package_states(self): if self.thread_notify_pkgs_ready.isRunning(): @@ -735,7 +742,7 @@ def load_without_packages(self): self._handle_console_option(False) self._finish_refresh_packages({'installed': None, 'types': None}, as_installed=False) - def _begin_load_suggestions(self, filter_installed: bool): + def begin_load_suggestions(self, filter_installed: bool): self.search_bar.clear() self._begin_action(self.i18n['manage_window.status.suggestions']) self._handle_console_option(False) @@ -811,6 +818,9 @@ def _finish_uninstall(self, res: dict): self._show_console_checkbox_if_output() self._update_installed_filter() self.begin_apply_filters() + self.table_apps.change_headers_policy(policy=QHeaderView.Stretch, maximized=self._maximized) + self.table_apps.change_headers_policy(policy=QHeaderView.ResizeToContents, maximized=self._maximized) + self._resize(accept_lower_width=True) notify_tray() else: self._show_console_errors() @@ -960,12 +970,12 @@ def update_pkgs(self, new_pkgs: Optional[List[SoftwarePackage]], as_installed: b commons.update_info(pkgv, pkgs_info) commons.apply_filters(pkgv, filters, pkgs_info) - if pkgs_info['apps_count'] == 0: + if pkgs_info['apps_count'] == 0 and not self.suggestions_requested: if self.load_suggestions or self.types_changed: if as_installed: self.pkgs_installed = pkgs_info['pkgs'] - self._begin_load_suggestions(filter_installed=False) + self.begin_load_suggestions(filter_installed=True) self.load_suggestions = False return False else: @@ -1127,9 +1137,14 @@ def _resize(self, accept_lower_width: bool = True): new_width = max(table_width, toolbar_width, topbar_width) new_width *= 1.05 # this extra size is not because of the toolbar button, but the table upgrade buttons + new_width = int(new_width) + + if new_width >= self.maximumWidth(): + new_width = self.maximumWidth() if (self.pkgs and accept_lower_width) or new_width > self.width(): - self.resize(int(new_width), self.height()) + self.resize(new_width, self.height()) + self.setMinimumWidth(new_width) def set_progress_controll(self, enabled: bool): self.progress_controll_enabled = enabled diff --git a/bauh/view/resources/locale/ca b/bauh/view/resources/locale/ca index 161241d7..50abfaf6 100644 --- a/bauh/view/resources/locale/ca +++ b/bauh/view/resources/locale/ca @@ -221,6 +221,8 @@ core.config.backup.uninstall=Before uninstalling core.config.backup.upgrade=Before upgrading core.config.boot.load_apps=Load apps after startup core.config.boot.load_apps.tip=If the installed applications or suggestions should be loaded on the management panel after the initialization process +core.config.download.check_ssl=Check security certificate +core.config.download.check_ssl.tip=If the security certificate (SSL) should be checked before downloading files core.config.download.icons=Download icons core.config.download.icons.tip=If the application icons should be downloaded and displayed on the table core.config.download.multithreaded=Multithreaded download diff --git a/bauh/view/resources/locale/de b/bauh/view/resources/locale/de index 034d38d8..76926afc 100644 --- a/bauh/view/resources/locale/de +++ b/bauh/view/resources/locale/de @@ -220,6 +220,8 @@ core.config.backup.uninstall=Before uninstalling core.config.backup.upgrade=Before upgrading core.config.boot.load_apps=Load apps after startup core.config.boot.load_apps.tip=If the installed applications or suggestions should be loaded on the management panel after the initialization process +core.config.download.check_ssl=Check security certificate +core.config.download.check_ssl.tip=If the security certificate (SSL) should be checked before downloading files core.config.download.icons=Download Icons core.config.download.icons.tip=Falls aktiviert werden die Anwendungs-Icons in der Tabelle angezeigt core.config.download.multithreaded=Paralleler Download diff --git a/bauh/view/resources/locale/en b/bauh/view/resources/locale/en index eb30faa6..caff3dc1 100644 --- a/bauh/view/resources/locale/en +++ b/bauh/view/resources/locale/en @@ -222,6 +222,8 @@ core.config.backup.upgrade=Before upgrading core.config.backup=Enabled core.config.boot.load_apps=Load apps after startup core.config.boot.load_apps.tip=If the installed applications or suggestions should be loaded on the management panel after the initialization process +core.config.download.check_ssl=Check security certificate +core.config.download.check_ssl.tip=If the security certificate (SSL) should be checked before downloading files core.config.download.icons.tip=If the application icons should be downloaded and displayed on the table core.config.download.icons=Download icons core.config.download.multithreaded=Multi-threaded download diff --git a/bauh/view/resources/locale/es b/bauh/view/resources/locale/es index dd06cac2..9ac06a98 100644 --- a/bauh/view/resources/locale/es +++ b/bauh/view/resources/locale/es @@ -221,6 +221,8 @@ core.config.backup.uninstall=Antes de desinstalar core.config.backup.upgrade=Antes de actualizar core.config.boot.load_apps=Cargar aplicaciones al inicio core.config.boot.load_apps.tip= Si las aplicaciones instaladas o sugerencias deben ser cargadas en el panel de administración después del proceso de inicialización +core.config.download.check_ssl=Comprobar certificado de seguridad +core.config.download.check_ssl.tip=Si se debe verificar el certificado de seguridad (SSL) antes de descargar archivos core.config.download.icons=Descargar iconos core.config.download.icons.tip=Si los íconos de las aplicaciones se deben descargar y mostrar en la tabla core.config.download.multithreaded=Descarga segmentada diff --git a/bauh/view/resources/locale/fr b/bauh/view/resources/locale/fr index 710e3b42..5b0a2a72 100644 --- a/bauh/view/resources/locale/fr +++ b/bauh/view/resources/locale/fr @@ -220,6 +220,8 @@ core.config.backup.upgrade=Avant de mettre à jour core.config.backup=Activé core.config.boot.load_apps=Load apps after startup core.config.boot.load_apps.tip=If the installed applications or suggestions should be loaded on the management panel after the initialization process +core.config.download.check_ssl=Check security certificate +core.config.download.check_ssl.tip=If the security certificate (SSL) should be checked before downloading files core.config.download.icons.tip=Si les icônes de l'application devraient être téléchargées et affichées sur le tableau core.config.download.icons=Télécharger les icônes core.config.download.multithreaded=Téléchargement parallèles diff --git a/bauh/view/resources/locale/it b/bauh/view/resources/locale/it index d8ba47c0..ecf36639 100644 --- a/bauh/view/resources/locale/it +++ b/bauh/view/resources/locale/it @@ -220,6 +220,8 @@ core.config.backup.uninstall=Before uninstalling core.config.backup.upgrade=Before upgrading core.config.boot.load_apps=Load apps after startup core.config.boot.load_apps.tip=If the installed applications or suggestions should be loaded on the management panel after the initialization process +core.config.download.check_ssl=Check security certificate +core.config.download.check_ssl.tip=If the security certificate (SSL) should be checked before downloading files core.config.download.icons=Download icons core.config.download.icons.tip=If the application icons should be downloaded and displayed on the table core.config.download.multithreaded=Multithreaded download diff --git a/bauh/view/resources/locale/pt b/bauh/view/resources/locale/pt index 1cf4b38e..17e898f7 100644 --- a/bauh/view/resources/locale/pt +++ b/bauh/view/resources/locale/pt @@ -220,6 +220,8 @@ core.config.backup.upgrade=Antes de atualizar core.config.backup=Habilitada core.config.boot.load_apps=Carregar aplicativos após iniciar core.config.boot.load_apps.tip=Se os aplicativos instalados ou sugestões devem ser carregados no painel de gerenciamento após a inicialização. +core.config.download.check_ssl=Verificar certificado de segurança +core.config.download.check_ssl.tip=Se o certificado de segurança (SSL) deve ser verificado antes de baixar arquivos core.config.download.icons.tip=Se os ícones dos aplicativos devem ser baixados e exibidos na tabela core.config.download.icons=Baixar ícones core.config.download.multithreaded.tip=Se os aplicativos, pacotes e arquivos devem ser baixados através de uma ferramenta que trabalha com segmentação / threads (pode ser mais rápido). diff --git a/bauh/view/resources/locale/ru b/bauh/view/resources/locale/ru index d543de5a..c68bf42b 100644 --- a/bauh/view/resources/locale/ru +++ b/bauh/view/resources/locale/ru @@ -220,6 +220,8 @@ core.config.backup.uninstall=Before uninstalling core.config.backup.upgrade=Before upgrading core.config.boot.load_apps=Load apps after startup core.config.boot.load_apps.tip=If the installed applications or suggestions should be loaded on the management panel after the initialization process +core.config.download.check_ssl=Check security certificate +core.config.download.check_ssl.tip=If the security certificate (SSL) should be checked before downloading files core.config.download.icons=Скачать иконки core.config.download.icons.tip=Загружать иконки приложения для отображаения на рабочем столе core.config.download.multithreaded=Многопоточная загрузка diff --git a/bauh/view/resources/locale/tr b/bauh/view/resources/locale/tr index 22d26e06..0f27f2c7 100644 --- a/bauh/view/resources/locale/tr +++ b/bauh/view/resources/locale/tr @@ -220,6 +220,8 @@ core.config.backup.upgrade=Önce yükseltiliyor core.config.backup=Etkin core.config.boot.load_apps=Load apps after startup core.config.boot.load_apps.tip=If the installed applications or suggestions should be loaded on the management panel after the initialization process +core.config.download.check_ssl=Check security certificate +core.config.download.check_ssl.tip=If the security certificate (SSL) should be checked before downloading files core.config.download.icons.tip=Tabloda görüntüleniyorsa uygulama simgeleri indirilmeli core.config.download.icons=Simgeleri indir core.config.download.multithreaded.tip=Uygulamaların, paketlerin ve dosyaların iş parçacıklarıyla (daha hızlı) çalışan bir araçla indirilip indirilmeyeceği. diff --git a/bauh/view/resources/style/default/default.qss b/bauh/view/resources/style/default/default.qss index 48dd1fc9..ea2219dc 100644 --- a/bauh/view/resources/style/default/default.qss +++ b/bauh/view/resources/style/default/default.qss @@ -161,7 +161,7 @@ RadioSelectQt { font-weight: bold; } -RadioSelectQt#radio_select_notitle { +RadioSelectQt[no_label = "true"] { padding-top: 5px; } @@ -396,10 +396,59 @@ ScreenshotsDialog QPushButton#close { min-width: 25px; } +SettingsWindow * { + font-size: 14px; +} + +SettingsWindow QLabel[tip_icon = "true"] { + qproperty-scaledContents: True; + min-height: 14px; + max-height: 14px; + min-width: 14px; + max-width: 14px; +} + SettingsWindow TabGroupQt#settings { min-width: 400px; } +SettingsWindow PanelQt FormComboBoxQt { + max-width: 190px; +} + +SettingsWindow PanelQt QLineEdit, SettingsWindow PanelQt QSpinBox { + max-width: 60px; +} + +SettingsWindow PanelQt QLineEdit[file_chooser = 'true'] { + max-width: 300px; +} + +SettingsWindow PanelQt FormRadioSelectQt[opts = '2'] { + min-width: 280px; + max-width: 280px; +} + +SettingsWindow PanelQt FormRadioSelectQt[opts = '3'] { + min-width: 280px; + max-width: 400px; +} + +SettingsWindow FormMultipleSelectQt#gems { + max-width: 220px; +} + +SettingsWindow FormMultipleSelectQt#gems QCheckBox { + qproperty-iconSize: 18px 18px; +} + +SettingsWindow FormMultipleSelectQt#gems QLabel[help_icon = "true"], SettingsWindow FormMultipleSelectQt#gems QLabel[warning_icon = "true"] { + min-height: 18px; + max-height: 18px; + min-width: 18px; + max-width: 18px; +} + QMenu QPushButton[current = "true"] { font-weight: bold; } diff --git a/bauh/view/util/disk.py b/bauh/view/util/disk.py index 786ed4e0..f460f45f 100644 --- a/bauh/view/util/disk.py +++ b/bauh/view/util/disk.py @@ -3,7 +3,7 @@ import os import time from threading import Thread, Lock -from typing import Type, Dict +from typing import Type, Dict, Any, Optional import yaml @@ -37,6 +37,33 @@ def fill(self, pkg: SoftwarePackage, sync: bool = False): else: self.pkgs.append(pkg) + def read(self, pkg: SoftwarePackage) -> Optional[Dict[str, Any]]: + if pkg and pkg.supports_disk_cache(): + data_path = pkg.get_disk_data_path() + + if data_path and os.path.isfile(data_path): + ext = data_path.split('.')[-1] + + try: + with open(data_path) as f: + file_content = f.read() + except FileNotFoundError: + return + + if file_content: + if ext == 'json': + cached_data = json.loads(file_content) + elif ext in {'yml', 'yaml'}: + cached_data = yaml.load(file_content) + else: + raise Exception(f'The cached data file {data_path} has an unsupported format') + + if cached_data: + return cached_data + + else: + self.logger.warning(f"No cached content in file {data_path}") + def stop_working(self): self._work = False @@ -58,26 +85,16 @@ def run(self): self._working = False def _fill_cached_data(self, pkg: SoftwarePackage) -> bool: - if os.path.exists(pkg.get_disk_data_path()): - disk_path = pkg.get_disk_data_path() - ext = disk_path.split('.')[-1] - - with open(disk_path) as f: - if ext == 'json': - cached_data = json.loads(f.read()) - elif ext in {'yml', 'yaml'}: - cached_data = yaml.load(f.read()) - else: - raise Exception('The cached data file {} has an unsupported format'.format(disk_path)) + cached_data = self.read(pkg) - if cached_data: - pkg.fill_cached_data(cached_data) - cache = self.cache_map.get(pkg.__class__) + if cached_data: + pkg.fill_cached_data(cached_data) + cache = self.cache_map.get(pkg.__class__) - if cache: - cache.add_non_existing(str(pkg.id), cached_data) + if cache: + cache.add_non_existing(str(pkg.id), cached_data) - return True + return True return False diff --git a/tests/common/test_util.py b/tests/common/test_util.py index 3e9ecd0f..22086bd0 100644 --- a/tests/common/test_util.py +++ b/tests/common/test_util.py @@ -1,6 +1,7 @@ +import time from unittest import TestCase -from bauh.commons.util import size_to_byte +from bauh.commons.util import size_to_byte, sanitize_command_input class SizeToByteTest(TestCase): @@ -46,3 +47,46 @@ def test_must_return_converted_string_sizes(self): def test_must_treat_string_sizes_before_converting(self): self.assertEqual(57300, size_to_byte(' 57 , 3 ', ' K ')) + + +class SanitizeCommandInputTest(TestCase): + + def test__must_remove_any_forbidden_symbols(self): + input_ = ' #$%* abc-def@ #%<<>> ' + res = sanitize_command_input(input_) + self.assertEqual('abc-def@', res) + + def test__must_remove_single_quotes(self): + input_ = " 'abc'-'xpto' " + res = sanitize_command_input(input_) + self.assertEqual('abc-xpto', res) + + def test__must_remove_double_quotes(self): + input_ = ' "abc"-"xpto" ' + res = sanitize_command_input(input_) + self.assertEqual('abc-xpto', res) + + def test__must_remove_the_pipe_operator(self): + input_ = ' abc | ls /home/xpto | cat /home/xpto/secret.txt' + res = sanitize_command_input(input_) + self.assertEqual('abc', res) + + def test__must_remove_the_and_operator(self): + input_ = ' abc && ls /home/xpto & cat /home/xpto/secret.txt' + res = sanitize_command_input(input_) + self.assertEqual('abc', res) + + def test__must_remove_several_operator(self): + input_ = ' abc | ls /home/xpto && cat /home/xpto/secret.txt' + res = sanitize_command_input(input_) + self.assertEqual('abc', res) + + def test__must_remove_single_dash_parameters(self): + input_ = '-cat abc-def -user -system ghi--jkl -xpto ' + res = sanitize_command_input(input_) + self.assertEqual('abc-def ghi--jkl', res) + + def test__must_remove_double_dash_parameters(self): + input_ = '--cat abc-def --user -system ghi--jkl --xpto ' + res = sanitize_command_input(input_) + self.assertEqual('abc-def ghi--jkl', res) diff --git a/tests/gems/flatpak/test_controller.py b/tests/gems/flatpak/test_controller.py index cd7749ca..11f7afa3 100644 --- a/tests/gems/flatpak/test_controller.py +++ b/tests/gems/flatpak/test_controller.py @@ -96,7 +96,7 @@ def test_sort_deps_two_apps_two_runtimes_two_partials(self): gnome_locale = FlatpakApplication(id="org.gnome.Platform.Locale", name="Locale", runtime=True, partial=True) gnome_locale.base_id = gnome_platform.id - platform_default = FlatpakApplication(id="org.freedesktop.Platform.GL.default", name='default', runtime=True) + platform_default = FlatpakApplication(id="org.freedesktop.Platform.GL.default", name='Default', runtime=True) platform_locale = FlatpakApplication(id="org.freedesktop.Platform.GL.Locale", name='Locale', runtime=True, partial=True) platform_locale.base_id = platform_default.id @@ -113,9 +113,9 @@ def test_sort_deps_two_apps_two_runtimes_two_partials(self): self.assertIsInstance(sorted_list, list) self.assertEqual(len(pkgs), len(sorted_list)) - self.assertEqual(gnome_locale.id, sorted_list[0].id) - self.assertEqual(gnome_platform.id, sorted_list[1].id) - self.assertEqual(platform_locale.id, sorted_list[2].id) - self.assertEqual(platform_default.id, sorted_list[3].id) + self.assertEqual(platform_default.id, sorted_list[0].id) + self.assertEqual(platform_locale.id, sorted_list[1].id) + self.assertEqual(gnome_locale.id, sorted_list[2].id) + self.assertEqual(gnome_platform.id, sorted_list[3].id) self.assertEqual('org.gnome.gedit', sorted_list[4].id) self.assertEqual('com.spotify.Client', sorted_list[5].id) diff --git a/tests/gems/web/test_controller.py b/tests/gems/web/test_controller.py index 442ffe24..12dc76eb 100644 --- a/tests/gems/web/test_controller.py +++ b/tests/gems/web/test_controller.py @@ -45,3 +45,20 @@ def test_get_accept_language_header__must_not_concatenate_default_locale_if_syst returned = self.manager.get_accept_language_header() self.assertEqual(f'en-IN, en', returned) getdefaultlocale.assert_called_once() + + def test_strip_url_protocol__http_no_www(self): + res = self.manager.strip_url_protocol('http://test.com') + self.assertEqual('test.com', res) + + def test_strip_url_protocol__http_with_www(self): + res = self.manager.strip_url_protocol('http://www.test.com') + self.assertEqual('test.com', res) + + def test_strip_url_protocol__https_no_www(self): + res = self.manager.strip_url_protocol('https://test.com') + self.assertEqual('test.com', res) + + def test_strip_url_protocol__https_with_www(self): + res = self.manager.strip_url_protocol('https://www.test.com') + self.assertEqual('test.com', res) +