Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AlmaLinux: Issue with H.264 when client request color depth 32 bit (xorgxrdp) #3405

Open
hauihau opened this issue Jan 22, 2025 · 10 comments
Open

Comments

@hauihau
Copy link

hauihau commented Jan 22, 2025

xrdp version

0.10.2

Detailed xrdp version, build options

xrdp 0.10.2
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --build=x86_64-redhat-linux-gnu
      --host=x86_64-redhat-linux-gnu
      --program-prefix=
      --disable-dependency-tracking
      --prefix=/usr
      --exec-prefix=/usr
      --bindir=/usr/bin
      --sbindir=/usr/sbin
      --sysconfdir=/etc
      --datadir=/usr/share
      --includedir=/usr/include
      --libdir=/usr/lib64
      --libexecdir=/usr/libexec
      --localstatedir=/var
      --sharedstatedir=/var/lib
      --mandir=/usr/share/man
      --infodir=/usr/share/info
      --enable-fuse
      --enable-pixman
      --enable-painter
      --enable-vsock
      --enable-ipv6
      --enable-openh264
      --with-socketdir=/run/xrdp
      --with-imlib2
      build_alias=x86_64-redhat-linux-gnu
      host_alias=x86_64-redhat-linux-gnu
      CC=gcc
      CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
      LDFLAGS=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
      CXX=g++
      CXXFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
      LT_SYS_LIBRARY_PATH=/usr/lib64:
      PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig

  Compiled with OpenSSL 3.0.7 1 Nov 2022

Operating system & version

RedHat Enterprise Linix 9.4 / Almalinux 9.5

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

xorgxrdp 0.10.3

What desktop environment do you use?

Xfce

Environment xrdp running on

Physical and virtual systems

What's your client?

Windows 11 Microsoft Client

Area(s) with issue?

Graphic glitches, Other

Steps to reproduce

We have just updates to the lastest version of xrdp and xorgxrdp on our RedHat and Alma Linux systems.

When we connect to the systems with 32 bit color depth, we only see a blue screen and the systemd journal contains lot's of lines with

[ERROR] Failed to create H.264 encoder

If we reduce the color depth to 24 or less, everything is working as expected.

There is also in the logs when conencting:

Color depth 32 bit:

[INFO ] client supports gfx protocol

Color depth <32 bit:

[WARN ] client requested gfx protocol with insufficient color depth

✔️ Expected Behavior

Seeing the Xfce desktop

❌ Actual Behavior

The window has just blue background, no Xfce desktop items visible

Anything else?

Please let me know if you need further details or logs.

@hauihau hauihau added the bug label Jan 22, 2025
@matt335672
Copy link
Member

@hauihau - did you install xrdp using dnf? It isn't clear from your description above.

@matt335672
Copy link
Member

I've had a poke around in the source, and this looks currently like a memory allocation error of some sort.

Are you able to provide the following?

  • An mstsc.exe ini file which exhibits this?
  • The layout of the monitor or monitors you are using with xrdp?

Thanks.

@hauihau
Copy link
Author

hauihau commented Jan 23, 2025

@hauihau - did you install xrdp using dnf? It isn't clear from your description above.

Both xrdp and xorgxrdp were installed with dnf from EPEL 9 repository,

@hauihau
Copy link
Author

hauihau commented Jan 23, 2025

I've had a poke around in the source, and this looks currently like a memory allocation error of some sort.

Are you able to provide the following?

  • An mstsc.exe ini file which exhibits this?
  • The layout of the monitor or monitors you are using with xrdp?

Thanks.

I'm not aware of any ini file, bit this is the content of the Default.rdp:

screen mode id:i:2
use multimon:i:0
desktopwidth:i:2560
desktopheight:i:1440
session bpp:i:24
winposstr:s:0,1,449,203,1249,803
compression:i:1
keyboardhook:i:2
audiocapturemode:i:0
videoplaybackmode:i:1
connection type:i:7
networkautodetect:i:1
bandwidthautodetect:i:1
displayconnectionbar:i:1
enableworkspacereconnect:i:0
disable wallpaper:i:0
allow font smoothing:i:0
allow desktop composition:i:0
disable full window drag:i:1
disable menu anims:i:1
disable themes:i:0
disable cursor setting:i:0
bitmapcachepersistenable:i:1
full address:s:
audiomode:i:0
redirectprinters:i:1
redirectlocation:i:0
redirectcomports:i:0
redirectsmartcards:i:1
redirectwebauthn:i:1
redirectclipboard:i:1
redirectposdevices:i:0
autoreconnection enabled:i:1
authentication level:i:2
prompt for credentials:i:0
negotiate security layer:i:1
remoteapplicationmode:i:0
alternate shell:s:
shell working directory:s:
gatewayhostname:s:
gatewayusagemethod:i:4
gatewaycredentialssource:i:4
gatewayprofileusagemethod:i:0
promptcredentialonce:i:0
gatewaybrokeringtype:i:0
use redirection server name:i:0
rdgiskdcproxy:i:0
kdcproxyname:s:
enablerdsaadauth:i:0
drivestoredirect:s:
username:s:

I have two monitors connected to a USB-C/Thunderbolt docking station, both displays have 2560x1440.

@matt335672
Copy link
Member

Thanks @hauihau - I'll try to reproduce this here using similar settings.

@matt335672
Copy link
Member

I'm having a few problems with the debuffer on Alma 9, so I've not managed to reproduce this yet.

A thought occurs to me however; what do you get for the following command?

rpm -qa | grep -i h264

@hauihau
Copy link
Author

hauihau commented Jan 24, 2025

I'm having a few problems with the debuffer on Alma 9, so I've not managed to reproduce this yet.

A thought occurs to me however; what do you get for the following command?

rpm -qa | grep -i h264

noopenh264-0.1.0~openh264_2.4.1-2.el9.x86_64

@thomaspmorgan
Copy link

I have the same issue on RHEL 9.5 with latest updates. Was working before latest update.

@matt335672
Copy link
Member

I think I can see what's going on here.

With noopenh264 installed:-

$ rpm -qi noopenh264-0.1.0~openh264_2.4.1-2.el9.x86_64
Name        : noopenh264
. . .
Summary     : Fake implementation of the OpenH264 library
Description :
Fake implementation of the OpenH264 library we can link from
regardless of the actual library being available.

You should both have openh264 installed:-

$ rpm -qi openh264-2.3.1-1.el9.x86_64
Name        : openh264
. . .

Summary     : H.264 codec library
Description :
OpenH264 is a codec library which supports H.264 encoding and decoding. It is
suitable for use in real time applications such as WebRTC.

My (possibly incorrect) understanding is that there are some concerns over H264 patents which is why the noopenh264 package exists.

I'll need to look into why this has happened with a fresh Alma install.

The following command might fix things, but I'd only try it on a test system for the time being until I've looked in more detail. Please don't run this in production yet!

sudo dnf install openh264 --allowerasing

@matt335672
Copy link
Member

I can see where noopenh264 could come from. It's a bit complicated, and the explanation is correspondingly long:-

I've made a vanilla installation of Alma 9.5 with EPEL using this procedure:-

  1. Install base system from iso
  2. sudo dnf update
  3. sudo dnf install epel-release
  4. sudo dnf groupinstall Xfce base-x
  5. sudo systemctl set-default graphical
  6. sudo reboot

Then:-

$ sudo dnf install xrdp
Last metadata expiration check: 1:55:56 ago on Sat Jan 25 14:36:39 2025.
Dependencies resolved.
================================================================================
 Package                  Arch    Version                      Repository  Size
================================================================================
Installing:
 xrdp                     x86_64  1:0.10.2-7.el9               epel       593 k
Installing dependencies:
 dbus-x11                 x86_64  1:1.12.20-8.el9              appstream   24 k
 fuse3-libs               x86_64  3.10.2-9.el9                 appstream   90 k
 imlib2                   x86_64  1.7.4-1.el9                  epel       225 k
 noopenh264               x86_64  0.1.0~openh264_2.4.1-2.el9   epel        22 k
 tigervnc-license         noarch  1.14.1-1.el9_5               appstream   17 k
 tigervnc-server-minimal  x86_64  1.14.1-1.el9_5               appstream  1.2 M
Installing weak dependencies:
 xrdp-selinux             x86_64  1:0.10.2-7.el9               epel        13 k

Transaction Summary
================================================================================
Install  8 Packages

Total download size: 2.1 M
Installed size: 7.4 M
Is this ok [y/N]: 

There is no sign of the openh264 package which is actually needed by xrdp:-

$ sudo dnf search openh264
Last metadata expiration check: 0:00:07 ago on Sat Jan 25 16:52:04 2025.
======================= Name & Summary Matched: openh264 =======================
noopenh264.x86_64 : Fake implementation of the OpenH264 library
noopenh264-devel.x86_64 : Development files for noopenh264

$ sudo dnf repolist --all
repo id                    repo name                                    status
appstream                  AlmaLinux 9 - AppStream                      enabled
appstream-debuginfo        AlmaLinux 9 - AppStream - Debug              disabled
appstream-source           AlmaLinux 9 - AppStream - Source             disabled
baseos                     AlmaLinux 9 - BaseOS                         enabled
baseos-debuginfo           AlmaLinux 9 - BaseOS - Debug                 disabled
baseos-source              AlmaLinux 9 - BaseOS - Source                disabled
crb                        AlmaLinux 9 - CRB                            disabled
crb-debuginfo              AlmaLinux 9 - CRB - Debug                    disabled
crb-source                 AlmaLinux 9 - CRB - Source                   disabled
epel                       Extra Packages for Enterprise Linux 9 - x86_ enabled
epel-debuginfo             Extra Packages for Enterprise Linux 9 - x86_ disabled
epel-source                Extra Packages for Enterprise Linux 9 - x86_ disabled
epel-testing               Extra Packages for Enterprise Linux 9 - Test disabled
epel-testing-debuginfo     Extra Packages for Enterprise Linux 9 - Test disabled
epel-testing-source        Extra Packages for Enterprise Linux 9 - Test disabled
extras                     AlmaLinux 9 - Extras                         enabled
extras-debuginfo           AlmaLinux 9 - Extras - Debug                 disabled
extras-source              AlmaLinux 9 - Extras - Source                disabled
highavailability           AlmaLinux 9 - HighAvailability               disabled
highavailability-debuginfo AlmaLinux 9 - HighAvailability - Debug       disabled
highavailability-source    AlmaLinux 9 - HighAvailability - Source      disabled
nfv                        AlmaLinux 9 - NFV                            disabled
nfv-debuginfo              AlmaLinux 9 - NFV - Debug                    disabled
nfv-source                 AlmaLinux 9 - NFV - Source                   disabled
plus                       AlmaLinux 9 - Plus                           disabled
plus-debuginfo             AlmaLinux 9 - Plus - Debug                   disabled
plus-source                AlmaLinux 9 - Plus - Source                  disabled
resilientstorage           AlmaLinux 9 - ResilientStorage               disabled
resilientstorage-debuginfo AlmaLinux 9 - ResilientStorage - Debug       disabled
resilientstorage-source    AlmaLinux 9 - ResilientStorage - Source      disabled
rt                         AlmaLinux 9 - RT                             disabled
rt-debuginfo               AlmaLinux 9 - RT - Debug                     disabled
rt-source                  AlmaLinux 9 - RT - Source                    disabled
sap                        AlmaLinux 9 - SAP                            disabled
sap-debuginfo              AlmaLinux 9 - SAP - Debug                    disabled
sap-source                 AlmaLinux 9 - SAP - Source                   disabled
saphana                    AlmaLinux 9 - SAPHANA                        disabled
saphana-debuginfo          AlmaLinux 9 - SAPHANA - Debug                disabled
saphana-source             AlmaLinux 9 - SAPHANA - Source               disabled

However, if at this point, I run a dnf-update, the version of the epel-release package gets updated from 9.5 to 9.9:-

Name        : epel-release
Version     : 9
Release     : 5.el9
Architecture: noarch
Install Date: Sat Jan 25 13:44:18 2025
Group       : Unspecified
Size        : 25723
License     : GPLv2
Signature   : RSA/SHA256, Mon Aug 21 14:35:39 2023, Key ID d36cb86cb86b3716
Source RPM  : epel-release-9-5.el9.src.rpm
Build Date  : Mon Aug 21 13:56:23 2023
Build Host  : s390x-builder01.almalinux.org
Packager    : AlmaLinux Packaging Team <[email protected]>
Vendor      : AlmaLinux
URL         : http://download.fedoraproject.org/pub/epel
Summary     : Extra Packages for Enterprise Linux repository configuration
Description :
This package contains the Extra Packages for Enterprise Linux (EPEL) repository
GPG key as well as configuration for yum.

$ sudo dnf update
Last metadata expiration check: 0:02:50 ago on Sat Jan 25 16:52:04 2025.
Dependencies resolved.
================================================================================
 Package                Architecture     Version           Repository      Size
================================================================================
Upgrading:
 epel-release           noarch           9-9.el9           epel            19 k

Transaction Summary
================================================================================
Upgrade  1 Package

Total download size: 19 k
Is this ok [y/N]: y
Downloading Packages:
epel-release-9-9.el9.noarch.rpm                  94 kB/s |  19 kB     00:00    
--------------------------------------------------------------------------------
Total                                            39 kB/s |  19 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Upgrading        : epel-release-9-9.el9.noarch                            1/2 
  Running scriptlet: epel-release-9-9.el9.noarch                            1/2 
  Cleanup          : epel-release-9-5.el9.noarch                            2/2 
  Running scriptlet: epel-release-9-5.el9.noarch                            2/2 
  Verifying        : epel-release-9-9.el9.noarch                            1/2 
  Verifying        : epel-release-9-5.el9.noarch                            2/2 

Upgraded:
  epel-release-9-9.el9.noarch                                                   

Complete!

$ rpm -qi epel-release
Name        : epel-release
Version     : 9
Release     : 9.el9
Architecture: noarch
Install Date: Sat Jan 25 16:54:57 2025
Group       : Unspecified
Size        : 26551
License     : GPLv2
Signature   : RSA/SHA256, Tue Dec  3 01:04:39 2024, Key ID 8a3872bf3228467c
Source RPM  : epel-release-9-9.el9.src.rpm
Build Date  : Mon Dec  2 23:59:07 2024
Build Host  : buildvm-x86-05.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://download.fedoraproject.org/pub/epel
Bug URL     : https://bugz.fedoraproject.org/epel-release
Summary     : Extra Packages for Enterprise Linux repository configuration
Description :
This package contains the Extra Packages for Enterprise Linux (EPEL) repository
GPG key as well as configuration for yum.

Now I can see the openh264 package:-

$ sudo dnf search openh264
Extra Packages for Enterprise Linux 9 openh264  4.7 kB/s | 2.5 kB     00:00    
Last metadata expiration check: 0:00:01 ago on Sat Jan 25 16:56:06 2025.
======================== Name Exactly Matched: openh264 ========================
openh264.x86_64 : H.264 codec library
======================= Name & Summary Matched: openh264 =======================
noopenh264.x86_64 : Fake implementation of the OpenH264 library
noopenh264-devel.x86_64 : Development files for noopenh264
openh264-devel.x86_64 : Development files for openh264
============================ Name Matched: openh264 ============================
gstreamer1-plugin-openh264.x86_64 : GStreamer H.264 plugin
mozilla-openh264.x86_64 : H.264 codec support for Mozilla browsers

$ sudo dnf repolist --all
repo id                       repo name                                 status
appstream                     AlmaLinux 9 - AppStream                   enabled
appstream-debuginfo           AlmaLinux 9 - AppStream - Debug           disabled
appstream-source              AlmaLinux 9 - AppStream - Source          disabled
baseos                        AlmaLinux 9 - BaseOS                      enabled
baseos-debuginfo              AlmaLinux 9 - BaseOS - Debug              disabled
baseos-source                 AlmaLinux 9 - BaseOS - Source             disabled
crb                           AlmaLinux 9 - CRB                         disabled
crb-debuginfo                 AlmaLinux 9 - CRB - Debug                 disabled
crb-source                    AlmaLinux 9 - CRB - Source                disabled
epel                          Extra Packages for Enterprise Linux 9 - x enabled
epel-cisco-openh264           Extra Packages for Enterprise Linux 9 ope enabled
epel-cisco-openh264-debuginfo Extra Packages for Enterprise Linux 9 ope disabled
epel-cisco-openh264-source    Extra Packages for Enterprise Linux 9 ope disabled
epel-debuginfo                Extra Packages for Enterprise Linux 9 - x disabled
epel-source                   Extra Packages for Enterprise Linux 9 - x disabled
epel-testing                  Extra Packages for Enterprise Linux 9 - T disabled
epel-testing-debuginfo        Extra Packages for Enterprise Linux 9 - T disabled
epel-testing-source           Extra Packages for Enterprise Linux 9 - T disabled
extras                        AlmaLinux 9 - Extras                      enabled
extras-debuginfo              AlmaLinux 9 - Extras - Debug              disabled
extras-source                 AlmaLinux 9 - Extras - Source             disabled
highavailability              AlmaLinux 9 - HighAvailability            disabled
highavailability-debuginfo    AlmaLinux 9 - HighAvailability - Debug    disabled
highavailability-source       AlmaLinux 9 - HighAvailability - Source   disabled
nfv                           AlmaLinux 9 - NFV                         disabled
nfv-debuginfo                 AlmaLinux 9 - NFV - Debug                 disabled
nfv-source                    AlmaLinux 9 - NFV - Source                disabled
plus                          AlmaLinux 9 - Plus                        disabled
plus-debuginfo                AlmaLinux 9 - Plus - Debug                disabled
plus-source                   AlmaLinux 9 - Plus - Source               disabled
resilientstorage              AlmaLinux 9 - ResilientStorage            disabled
resilientstorage-debuginfo    AlmaLinux 9 - ResilientStorage - Debug    disabled
resilientstorage-source       AlmaLinux 9 - ResilientStorage - Source   disabled
rt                            AlmaLinux 9 - RT                          disabled
rt-debuginfo                  AlmaLinux 9 - RT - Debug                  disabled
rt-source                     AlmaLinux 9 - RT - Source                 disabled
sap                           AlmaLinux 9 - SAP                         disabled
sap-debuginfo                 AlmaLinux 9 - SAP - Debug                 disabled
sap-source                    AlmaLinux 9 - SAP - Source                disabled
saphana                       AlmaLinux 9 - SAPHANA                     disabled
saphana-debuginfo             AlmaLinux 9 - SAPHANA - Debug             disabled
saphana-source                AlmaLinux 9 - SAPHANA - Source            disabled

and, more importantly:-

$ sudo dnf install xrdp
Last metadata expiration check: 0:01:19 ago on Sat Jan 25 16:56:06 2025.
Dependencies resolved.
================================================================================
 Package                  Arch    Version            Repository            Size
================================================================================
Installing:
 xrdp                     x86_64  1:0.10.2-7.el9     epel                 593 k
Installing dependencies:
 dbus-x11                 x86_64  1:1.12.20-8.el9    appstream             24 k
 fuse3-libs               x86_64  3.10.2-9.el9       appstream             90 k
 imlib2                   x86_64  1.7.4-1.el9        epel                 225 k
 openh264                 x86_64  2.3.1-1.el9        epel-cisco-openh264  425 k
 tigervnc-license         noarch  1.14.1-1.el9_5     appstream             17 k
 tigervnc-server-minimal  x86_64  1.14.1-1.el9_5     appstream            1.2 M
Installing weak dependencies:
 xrdp-selinux             x86_64  1:0.10.2-7.el9     epel                  13 k

Transaction Summary
================================================================================
Install  8 Packages

Total download size: 2.5 M
Installed size: 8.5 M
Is this ok [y/N]: 

Carrying on with the installation works perfectly.

What is happening here, is that the epel-release which comes with AlmaLinux (9.5) does not have the file /etc/yum.repos.d/epel-cisco-openh264.repo in it. There is an update to this package within EPEL itself. To get this, once you have installed the epel-release package, you must update it before installing xrdp.

If you are in the situation where xrdp has been installed along with noopenh264, proceed as follows:-

  1. Check what depends on noopenh264 by trying to remove it:-

    $ sudo dnf --assumeno remove noopenh264
    Dependencies resolved.
    ================================================================================
     Package                  Arch    Version                     Repository   Size
    ================================================================================
    Removing:
     noopenh264               x86_64  0.1.0~openh264_2.4.1-2.el9  @epel        43 k
    Removing dependent packages:
     xorgxrdp                 x86_64  0.10.3-5.el9                @epel       178 k
     xrdp                     x86_64  1:0.10.2-7.el9              @epel       3.4 M
    Removing unused dependencies:
     dbus-x11                 x86_64  1:1.12.20-8.el9             @appstream   36 k
     fuse3-libs               x86_64  3.10.2-9.el9                @appstream  281 k
     imlib2                   x86_64  1.7.4-1.el9                 @epel       744 k
     tigervnc-license         noarch  1.14.1-1.el9_5              @appstream   18 k
     tigervnc-server-minimal  x86_64  1.14.1-1.el9_5              @appstream  2.8 M
     xrdp-selinux             x86_64  1:0.10.2-7.el9              @epel       9.0 k
    
    Transaction Summary
    ================================================================================
    Remove  9 Packages
    
    Freed space: 7.5 M
    Operation aborted.
    

    The above example shows only xrdp and xorgxrdp are affected. So it's safe to proceed.

  2. Manually kill any active xrdp sessions.

  3. Make sure epel-release is up to date with sudo dnf update epel-release ; sudo dnf update

  4. Replace noopenh264 with openh264 with sudo dnf install openh264 --allowerasing

  5. Restart xrdp with sudo systemctl restart xrdp

@matt335672 matt335672 changed the title Issue with H.264 when client request color depth 32 bit AlmaLinux: Issue with H.264 when client request color depth 32 bit (xorgxrdp) Jan 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants