Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stable-20250126.1 is unbootable on Nvidia systems #2157

Open
Malix-Labs opened this issue Jan 26, 2025 · 19 comments
Open

stable-20250126.1 is unbootable on Nvidia systems #2157

Malix-Labs opened this issue Jan 26, 2025 · 19 comments
Labels
bug Something isn't working

Comments

@Malix-Labs
Copy link

Malix-Labs commented Jan 26, 2025

Describe the bug

stable-20250126.1 seems unbootable (tested with Bluefin-DX-Nvidia on a Lenovo Legion 5)

My booting screen (personally, with a Lenovo and a Bluefin logo plus a spinning loading icon) shows up, then seems to freeze after a few seconds, then the screen turns full black forever

What did you expect to happen?

A working stable release

Output of rpm-ostree status

❯ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:3a2bd576580c679d83174e8c2391108d26920cdeea44ffd8bede85ebedd071db
                  Version: 41.20250126.1 (2025-01-26T06:07:45Z)
            SecAdvisories: 1 low, 2 important, 1 critical
                     Diff: 162 upgraded, 1 removed, 1 added
          LayeredPackages: google-chrome-beta waydroid

● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:dfdbfb5acd4390ed61d024aaa356568db90c20aff5c75ae76724a344f2f95cae
                  Version: 41.20250119.1 (2025-01-19T06:13:33Z)
          LayeredPackages: google-chrome-beta waydroid

Output of bootc status

❯ bootc status
ERROR Status: Initializing storage: Acquiring sysroot: Preparing for write: Querying root privilege: This command requires root privileges

Output of sudo bootc status

❯ sudo bootc status
No staged image present
Current booted state is native ostree
Current rollback state is native ostree

Output of fastfetch

❯ fastfetch
                                               ▖      malix@malix-pc
                                   ▃▃▄▃▂▃▄▅▅▅▂▄▆
                                ▗▄▄▅╱▅▅▆┯▘┳▇▗▄┱╴    󱋩  bluefin-dx-nvidia:stable 🔐
  ▂▃▄▄▃                       ▘▁▄▄╸▂▖╾┖┸┘▖▄╊╱┵▆▅    󰣛  Bluefin (Version: 41.20250119.1 / FROM Fedora Silverblue 41)
 ▘▇▗┗▃┮▆▅▃▂▃                ▘▘▝╵┍▘▌▘━▚▇┆▗▝▅▊╲▃▅▄      Linux 6.11.8-300.fc41.x86_64
▋▝▖▄▆▁━━━▁▁▅▆▄▂▆┊▝▂        ▘▗▚▚▚▘╏▄▁▃▂▅┛▅▅▃▚▄▅▝▄    󰅐  4 mins
 ▝▊▝▃▃▄▖▃┄┄╵╶└┄▇┗▂▆▅▃▂▁▃▄▃▃▂▃▆▅▆▂┓▂▖▃▄▆▆▂▄▄▝▝▁▇▘    󰔠  Forged on May 19 2024
    ▃▖▃▃▄▖▉▁┒▝▗▆▖▃▆▅▄▃▂╼┒▆▅▄╼╸▂▄▄▂▃▄▄▃▄▅▅▅▅▄▝▚▃▗
     ▝▆┘▅╴▂▁▁▆▂▍┯▘▂▄▆▂▝╾▃▃▂▂▄▃▃▄▅▅▄▄▃▅▄▂▄▆▃▂▎▖▄▝    󰾰  82JU (LEGION 5 15ACH6H)
        ▁▂┯▝▖▝▂┗▄▃▚▃▃▚▇╾┓▅▄▝▂▃▆▄▗▅▊▆▅╹▄▆▇╌▃▆▅▃▅▅    󰻠  AMD Ryzen 7 5800H (16) @ 4.46 GHz
     ▂▄▆▇▇╴▝▇▎▄▅▖┊▅▍▆╹▄▄▆▄▃▂▂▂▃▅▅┕│┋╴▆▂▁▅╴▖▌▄╲▃▃    󰍛  NVIDIA GeForce RTX 3060 Mobile / Max-Q [Discrete]
    ▗╵▄▂▂▂▆━┗▂▄▘▅┉┖▅▁▃▂▃▅▄▅▄▄▂▂╾▁▄▘▄┭▂▎▍▆┪▇┉▗┈▅┓    󰍛  AMD Radeon Vega Series / Radeon Vega Mobile Series [Integra]
       ╾╻▝▗▅╁▇▝▏▃▃▁▅▄▂▅▃▂▂─▄┙╎▅▘▃╌╷▂▃▌▄┒╶▆┍▅▍▋╺╲    󰧑  3.07 GiB / 13.49 GiB (23%)
    ▂▄▝▂▅▅╌┠╳▁▁▂━▄▅▗▃▆▁┄┊▁╺▅╸╴▂┛▇▇┊▌╱╱╶┉▄▌▗▂▝▖╲┓      140.10 GiB / 952.60 GiB (15%) - btrfs [Read-only]
  ▃╶▋▅▝▂▅▁▃▆▂▇▎▂┡▄▁▂╼╲┴╳┵▄▆▂▃▄▄▄▃▂▅▂╅▃▄▁┋╲▗▖▃▅┓▌    󰍹  1920x1080 @ 165 Hz (as 1536x864) in 15" [Built-in]
▗╵▅▝▃▅▅▆▝▖▅▇▅━▖━▝╺┱▄▘┯▆▁▄▄▄▆▁▘╶▁▇▘▆▇▎▋┊▝▖┓╲┻▌▊▃┬    󰍹  2560x1440 @ 144 Hz (as 2048x1152) in 27" [External] *
  ▅▆        ▆▖▘▁▄▃▃▅▅▂▃▄▄  ▗╸╎▝▗▆▗▃▘▎▊▝▘┃▌▚▄▊▊▎       100% [AC Connected]
              ▝▗▖▆▅▅▅    ▗▆▄▇▄╸▃▊▍▘▅▖▍▏▎▊▎▅┒╍╵  
                            ▄▘   ▅  ▌▗▋▝▄▄▃▘▄▘      󰕮  GNOME 47.2
                                    ▅▆                Mutter (Wayland)
                                                      bash 5.2.32
                                                      Ptyxis 47.6
                                                    󰏖  2583 (rpm), 78 (flatpak-system), 7 (flatpak-user), 49 (brew)

                                                                              
                                                                              

Extra information or context

Need to rollback to previous deploy (last week's stable - stable-20250119.1)

@dosubot dosubot bot added the bug Something isn't working label Jan 26, 2025
@francoism90
Copy link

Same here, it seems to be failing on the sb key.

@francoism90
Copy link

This turns out to be a nvidia kernel module issue for me.

API mismatch: the client has the version 570.86.15, but this kernel module has the version 565.77

I think the NV drivers are not bumped somewhere?

@dylanmtaylor
Copy link
Contributor

dylanmtaylor commented Jan 28, 2025

I'm also experiencing this issue on builds newer than 1/24 of Bluefin.

➜  ~ rpm-ostree status -v
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:latest (index: 0)
                   Digest: sha256:60da328330d34441e34304188148a50d774a8a259762f2079a39bc019c0cfe42
                  Version: latest-41.20250128 (2025-01-28T05:08:06Z)
               BaseCommit: 9df0bdaa9ed0412e79169249fe43a266a377962d33a8d26ea2fadbabb2f4f835
                   Commit: 0bc58ca60903020f26ea44b678e935ce26ac65853777f60b07c1db226bf1fac6
                           ├─ fedora (2024-10-24T13:55:59Z)
                           ├─ updates (2025-01-28T02:55:16Z)
                           └─ updates-archive (2025-01-28T03:14:54Z)
                   Staged: no
                StateRoot: default
            SecAdvisories: FEDORA-2025-497995b101  Important  glibc-2.40-21.fc41.i686
                           FEDORA-2025-497995b101  Important  glibc-2.40-21.fc41.x86_64
                           FEDORA-2025-497995b101  Important  glibc-all-langpacks-2.40-21.fc41.x86_64
                           FEDORA-2025-497995b101  Important  glibc-common-2.40-21.fc41.x86_64
                           FEDORA-2025-497995b101  Important  glibc-devel-2.40-21.fc41.x86_64
                           FEDORA-2025-497995b101  Important  glibc-gconv-extra-2.40-21.fc41.i686
                           FEDORA-2025-497995b101  Important  glibc-gconv-extra-2.40-21.fc41.x86_64
                             glibc: getrandom, arc4random can return predictable data after fork (CVE-2025-0577)
                             https://bugzilla.redhat.com/show_bug.cgi?id=2338960
                             CVE-2025-0395 glibc: buffer overflow in the GNU C Library's assert() [fedora-41]
                             https://bugzilla.redhat.com/show_bug.cgi?id=2341631
                           FEDORA-2025-908dfe95f6  Important  buildah-2:1.38.1-1.fc41.x86_64
                           FEDORA-2025-908dfe95f6  Important  containers-common-5:0.61.1-1.fc41.noarch
                           FEDORA-2025-908dfe95f6  Important  containers-common-extra-5:0.61.1-1.fc41.noarch
                           FEDORA-2025-908dfe95f6  Important  podman-5:5.3.2-1.fc41.x86_64
                           FEDORA-2025-908dfe95f6  Important  podman-machine-5:5.3.2-1.fc41.x86_64
                           FEDORA-2025-908dfe95f6  Important  podmansh-5:5.3.2-1.fc41.x86_64
                             CVE-2024-11218 podman: buildah: Container breakout by using --jobs=2 and a race condition when building a malicious Containerfile
                             https://bugzilla.redhat.com/show_bug.cgi?id=2326231
                 Upgraded: bootc 1.1.2-2.fc41 -> 1.1.4-6.fc41
                           buildah 2:1.38.0-2.fc41 -> 2:1.38.1-1.fc41
                           clang-libs 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           clang-resource-filesystem 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           compiler-rt 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           containers-common 5:0.61.0-1.fc41 -> 5:0.61.1-1.fc41
                           containers-common-extra 5:0.61.0-1.fc41 -> 5:0.61.1-1.fc41
                           glibc 2.40-17.fc41 -> 2.40-21.fc41
                           glibc 2.40-17.fc41 -> 2.40-21.fc41
                           glibc-all-langpacks 2.40-17.fc41 -> 2.40-21.fc41
                           glibc-common 2.40-17.fc41 -> 2.40-21.fc41
                           glibc-devel 2.40-17.fc41 -> 2.40-21.fc41
                           glibc-gconv-extra 2.40-17.fc41 -> 2.40-21.fc41
                           glibc-gconv-extra 2.40-17.fc41 -> 2.40-21.fc41
                           gnome-classic-session 47.3-1.fc41 -> 47.3-2.fc41
                           gnome-shell-extension-apps-menu 47.3-1.fc41 -> 47.3-2.fc41
                           gnome-shell-extension-common 47.3-1.fc41 -> 47.3-2.fc41
                           gnome-shell-extension-launch-new-instance 47.3-1.fc41 -> 47.3-2.fc41
                           gnome-shell-extension-places-menu 47.3-1.fc41 -> 47.3-2.fc41
                           gnome-shell-extension-search-light 0.0.git.167.e245254e-1.fc41 -> 0.0.git.171.4e51fbda-1.fc41
                           gnome-shell-extension-user-theme 47.3-1.fc41 -> 47.3-2.fc41
                           gnome-shell-extension-window-list 47.3-1.fc41 -> 47.3-2.fc41
                           httpd 2.4.62-2.fc41 -> 2.4.63-1.fc41
                           httpd-core 2.4.62-2.fc41 -> 2.4.63-1.fc41
                           httpd-filesystem 2.4.62-2.fc41 -> 2.4.63-1.fc41
                           httpd-tools 2.4.62-2.fc41 -> 2.4.63-1.fc41
                           kcli 99.0.0.git.202501231904.df4c73f-0.fc41 -> 99.0.0.git.202501271815.88ced64-0.fc41
                           kernel 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-core 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-devel 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-devel-matched 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-modules 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-modules-core 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-modules-extra 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-tools 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           kernel-tools-libs 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           libchromaprint 1.5.1-22.fc41 -> 1.5.1-24.fc41
                           libldb 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           libnetapi 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           libnvidia-cfg 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           libnvidia-fbc 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           libnvidia-ml 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           libnvidia-ml 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           libomp 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           libomp-devel 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           libsmbclient 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           libwbclient 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           libwnck3 43.1-1.fc41 -> 43.2-1.fc41
                           lldb 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           llvm-libs 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           llvm-libs 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           mod_lua 2.4.62-2.fc41 -> 2.4.63-1.fc41
                           nautilus-open-any-terminal 0.0.git.320.4177091c-1.fc41 -> 0.0.git.324.305bac0b-1.fc41
                           nvidia-driver 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-driver-cuda 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-driver-cuda-libs 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-driver-cuda-libs 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-driver-libs 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-driver-libs 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-kmod-common 3:565.77-2.fc41 -> 3:570.86.15-1.fc41
                           nvidia-libXNVCtrl 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-modprobe 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-persistenced 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           nvidia-settings 3:565.77-1.fc41 -> 3:570.86.15-1.fc41
                           openjpeg 2.5.3-4.fc41 -> 2.5.3-6.fc41
                           passt 0^20241211.g09478d5-1.fc41 -> 0^20250121.g4f2c8e7-2.fc41
                           passt-selinux 0^20241211.g09478d5-1.fc41 -> 0^20250121.g4f2c8e7-2.fc41
                           perl-AutoLoader 5.74-512.fc41 -> 5.74-514.fc41
                           perl-B 1.89-512.fc41 -> 1.89-514.fc41
                           perl-Class-Struct 0.68-512.fc41 -> 0.68-514.fc41
                           perl-DynaLoader 1.56-512.fc41 -> 1.56-514.fc41
                           perl-Errno 1.38-512.fc41 -> 1.38-514.fc41
                           perl-Fcntl 1.18-512.fc41 -> 1.18-514.fc41
                           perl-File-Basename 2.86-512.fc41 -> 2.86-514.fc41
                           perl-File-stat 1.14-512.fc41 -> 1.14-514.fc41
                           perl-FileHandle 2.05-512.fc41 -> 2.05-514.fc41
                           perl-Getopt-Std 1.14-512.fc41 -> 1.14-514.fc41
                           perl-IO 1.55-512.fc41 -> 1.55-514.fc41
                           perl-IPC-Open3 1.22-512.fc41 -> 1.22-514.fc41
                           perl-NDBM_File 1.17-512.fc41 -> 1.17-514.fc41
                           perl-POSIX 2.20-512.fc41 -> 2.20-514.fc41
                           perl-PathTools 3.91-511.fc41 -> 3.91-512.fc41
                           perl-SelectSaver 1.02-512.fc41 -> 1.02-514.fc41
                           perl-Symbol 1.09-512.fc41 -> 1.09-514.fc41
                           perl-base 2.27-512.fc41 -> 2.27-514.fc41
                           perl-if 0.61.000-512.fc41 -> 0.61.000-514.fc41
                           perl-interpreter 4:5.40.0-512.fc41 -> 4:5.40.1-514.fc41
                           perl-libs 4:5.40.0-512.fc41 -> 4:5.40.1-514.fc41
                           perl-locale 1.12-512.fc41 -> 1.12-514.fc41
                           perl-mro 1.29-512.fc41 -> 1.29-514.fc41
                           perl-overload 1.37-512.fc41 -> 1.37-514.fc41
                           perl-overloading 0.02-512.fc41 -> 0.02-514.fc41
                           perl-vars 1.05-512.fc41 -> 1.05-514.fc41
                           podman 5:5.3.1-1.fc41 -> 5:5.3.2-1.fc41
                           podman-machine 5:5.3.1-1.fc41 -> 5:5.3.2-1.fc41
                           podmansh 5:5.3.1-1.fc41 -> 5:5.3.2-1.fc41
                           python3-lldb 19.1.5-1.fc41 -> 19.1.7-1.fc41
                           python3-perf 6.12.9-200.fc41 -> 6.12.10-200.fc41
                           samba 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-client 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-client-libs 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-common 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-common-libs 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-common-tools 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-dcerpc 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-ldb-ldap-modules 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-libs 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-winbind 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-winbind-clients 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                           samba-winbind-modules 2:4.21.3-3.fc41 -> 2:4.21.3-5.fc41
                    Added: add-determinism-0.3.6-3.fc41.x86_64
                           akmod-nvidia-3:570.86.15-1.fc41.x86_64
                           akmods-0.6.0-8.fc41.noarch
                           annobin-docs-12.69-1.fc41.noarch
                           annobin-plugin-gcc-12.69-1.fc41.x86_64
                           ansible-srpm-macros-1-16.fc41.noarch
                           build-reproducibility-srpm-macros-0.3.6-3.fc41.noarch
                           efi-srpm-macros-5-13.fc41.noarch
                           fakeroot-1.36.2-1.fc41.x86_64
                           fakeroot-libs-1.36.2-1.fc41.x86_64
                           fonts-srpm-macros-1:2.0.5-17.fc41.noarch
                           forge-srpm-macros-0.4.0-1.fc41.noarch
                           fpc-srpm-macros-1.3-13.fc41.noarch
                           gcc-plugin-annobin-14.2.1-7.fc41.x86_64
                           ghc-srpm-macros-1.9.1-2.fc41.noarch
                           gnat-srpm-macros-6-6.fc41.noarch
                           go-srpm-macros-3.6.0-5.fc41.noarch
                           grubby-8.40-78.fc41.x86_64
                           kernel-srpm-macros-1.0-24.fc41.noarch
                           kmodtool-1.1-11.fc41.noarch
                           libgit2-1.9.0-4.fc41.x86_64
                           llhttp-9.2.1-2.fc41.x86_64
                           lua-srpm-macros-1-14.fc41.noarch
                           ocaml-srpm-macros-10-3.fc41.noarch
                           openblas-srpm-macros-2-18.fc41.noarch
                           package-notes-srpm-macros-0.5-12.fc41.noarch
                           perl-srpm-macros-1-56.fc41.noarch
                           pyproject-srpm-macros-1.16.4-1.fc41.noarch
                           python-srpm-macros-3.13-3.fc41.noarch
                           python3-babel-2.16.0-1.fc41.noarch
                           python3-click-plugins-1.1.1-22.fc41.noarch
                           python3-progressbar2-3.53.2-13.fc41.noarch
                           python3-pygit2-1.17.0-1.fc41.x86_64
                           python3-rpmautospec-0.7.3-1.fc41.noarch
                           python3-rpmautospec-core-0.1.5-2.fc41.noarch
                           python3-utils-3.7.0-6.fc41.noarch
                           qt5-srpm-macros-5.15.15-1.fc41.noarch
                           qt6-srpm-macros-6.8.1-4.fc41.noarch
                           redhat-rpm-config-293-1.fc41.noarch
                           rpm-build-4.20.0-1.fc41.x86_64
                           rpmdevtools-9.6-8.fc41.noarch
                           rust-srpm-macros-26.3-3.fc41.noarch
                           systemd-rpm-macros-256.11-1.fc41.noarch
                           zig-srpm-macros-1-3.fc41.noarch
          LayeredPackages: f3 parallel

● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:latest (index: 1)
                   Digest: sha256:8f6e1295c59450ae0a9f7530a7620812077ba1a3795d73b2280517c303538095
                  Version: latest-41.20250124.2 (2025-01-24T05:08:38Z)
               BaseCommit: 37ea6af207715a9ff71ebfecfeca000e4944ecc992a36081484f951a5965e17d
                   Commit: 750f540f1199a9c3a93ae1ec388aafe88dd539f535178b79e426b0809c8af0c6
                           ├─ fedora (2024-10-24T13:55:59Z)
                           ├─ updates (2025-01-24T01:20:52Z)
                           └─ updates-archive (2025-01-24T01:47:23Z)
                StateRoot: default
          LayeredPackages: f3 parallel

@castrojo
Copy link
Member

@francoism90 @dylanmtaylor please edit your posts and toss in an rpm-ostree status if you don't mind.

@castrojo castrojo changed the title stable-20250126.1 is unbootable stable-20250126.1 is unbootable on Nvidia systems Jan 28, 2025
@castrojo
Copy link
Member

p5 has pushed the last known good -nvidia images to the tags, so if you do an update you should be good. Can ya'll confirm if that is indeed the case?

Then when the team fixes it we'll just republish the new images.

@Malix-Labs
Copy link
Author

Malix-Labs commented Jan 28, 2025

The latest bluefin/aurora-nvidia (normal and dx) builds seem to date from 12h ago

Running ujust upgrade and topgrade (and letting them 100% go through) then rebooting (systemctl reboot) didn't worked

Did you mean something else by

do an update

?

@francoism90
Copy link

@castrojo Everything after 26 Jan doesn't work.

I don't fully understand, do you mean you have pushed an update? And do you mean updating using ujust?

@dylanmtaylor
Copy link
Contributor

p5 has pushed the last known good -nvidia images to the tags, so if you do an update you should be good. Can ya'll confirm if that is indeed the case?

Then when the team fixes it we'll just republish the new images.
error: While checking against deployment timestamp: Upgrade target revision 'f7604b57117593cd48d5dec82a1eab9d445c4a2a2595389bbfdb58b738abfa16' with timestamp 'Sun 26 Jan 2025 05:08:21 AM UTC' is chronologically older than current revision '9df0bdaa9ed0412e79169249fe43a266a377962d33a8d26ea2fadbabb2f4f835' with timestamp 'Tue 28 Jan 2025 05:08:06 AM UTC'; use --allow-downgrade to permit

That said, the 570 driver is in both repos now so there should be no more mismatch

@francoism90
Copy link

francoism90 commented Jan 29, 2025

@dylanmtaylor @bsherman (hopefully tagging the correct persons) - it works again. :)

$ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:latest
                   Digest: sha256:4838d6eefbe781581a27f216815e02de0bc900821caf7ff3a4baa9ca06df8644
                  Version: latest-41.20250129.2 (2025-01-29T05:16:06Z)

Thanks for the fix! Let's hope the new (opensource) nvidia driver will prevent this in the future. I had this issues in the past on Arch a couple of times.

@Malix-Labs
Copy link
Author

Malix-Labs commented Jan 29, 2025

The new images work!

However, as of now, the currently tagged "stable" for *-nvidia images are still not pointing to the fixed images (meaning that stable users updating their OS will still run into this issue)

Would there be a way to hotfix this @castrojo?

@dylanmtaylor
Copy link
Contributor

This is fixed for me on the new image versions

@bsherman
Copy link
Contributor

The new images work!

However, as of now, the currently tagged "stable" for *-nvidia images are still not pointing to the fixed images (meaning that stable users updating their OS will still run into this issue)

Would there be a way to hotfix this @castrojo?

@Malix-Labs I've just checked ghcr.io/ublue-os/bluefin-nvidia:stable and ghcr.io/ublue-os/bluefin-dx-nvidia:stable and i see...

# rpm -qa |grep 570

nothing 570 is in those stable images... I believe @p5 mitigated this yesterday around 12:00 EST by re-tagging stable to known good images.

New builds are looking good. I think we can close this issue now.

@Malix-Labs
Copy link
Author

Malix-Labs commented Jan 29, 2025

@bsherman, I just ran ujust update and it pulled the exact same borked image, couldn't boot

I am back at my previous deployment again

❯ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:3a2bd576580c679d83174e8c2391108d26920cdeea44ffd8bede85ebedd071db
                  Version: 41.20250126.1 (2025-01-26T06:07:45Z)
            SecAdvisories: 1 low, 2 important, 1 critical
                     Diff: 163 upgraded, 1 removed, 1 added
          LayeredPackages: google-chrome-beta waydroid

● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:dfdbfb5acd4390ed61d024aaa356568db90c20aff5c75ae76724a344f2f95cae
                  Version: 41.20250119.1 (2025-01-19T06:13:33Z)
          LayeredPackages: google-chrome-beta waydroid

https://paste.centos.org/view/3242a436

Note that the deployment was using the cache for sure

cleanup -p, cleanup -r, and cleanup -m clear out any deployment in status

❯ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:dfdbfb5acd4390ed61d024aaa356568db90c20aff5c75ae76724a344f2f95cae
                  Version: 41.20250119.1 (2025-01-19T06:13:33Z)
          LayeredPackages: google-chrome-beta waydroid

but when I upgrade again, the deployment is way too fast to not be using cache

@castrojo
Copy link
Member

Ok let's just push a new stable, try again when this turns green, thanks!

https://github.com/ublue-os/bluefin/actions/runs/13039922089

@bsherman
Copy link
Contributor

❯ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:3a2bd576580c679d83174e8c2391108d26920cdeea44ffd8bede85ebedd071db
                  Version: 41.20250126.1 (2025-01-26T06:07:45Z)
            SecAdvisories: 1 low, 2 important, 1 critical
                     Diff: 163 upgraded, 1 removed, 1 added
          LayeredPackages: google-chrome-beta waydroid

`cleanup -p`, `cleanup -r`, and `cleanup -m` clear out any deployment in `status`

but when I upgrade again, the deployment is way too fast to not be using cache

This is odd.

Here's a snippet of me rebasing to downgrade to this image.

$ sudo rpm-ostree rebase ostree-unverified-registry:ghcr.io/ublue-os/bluefin-dx-nvidia:stable
Pulling manifest: ostree-unverified-registry:ghcr.io/ublue-os/bluefin-dx-nvidia:stable
Staging deployment... done
Freed: 531.6 MB (pkgcache branches: 0)
Downgraded:
--- snip ---
  kernel 6.12.9-200.fc41 -> 6.11.8-300.fc41
  kernel-core 6.12.9-200.fc41 -> 6.11.8-300.fc41
  kernel-modules 6.12.9-200.fc41 -> 6.11.8-300.fc41
  kernel-modules-core 6.12.9-200.fc41 -> 6.11.8-300.fc41
  kernel-modules-extra 6.12.9-200.fc41 -> 6.11.8-300.fc41
  kmod-nvidia 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
 --- snip ---
 libnvidia-cfg 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  libnvidia-fbc 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  libnvidia-ml 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  libnvidia-ml 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  mod_lua 2.4.63-1.fc41 -> 2.4.62-2.fc41
  nvidia-driver 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-driver-cuda 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-driver-cuda-libs 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-driver-cuda-libs 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-driver-libs 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-driver-libs 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-kmod-common 3:570.86.15-1.fc41 -> 3:565.77-2.fc41
  nvidia-libXNVCtrl 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-modprobe 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-persistenced 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
  nvidia-settings 3:570.86.15-1.fc41 -> 3:565.77-1.fc41
--- snip ---

$ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 13h ago
Deployments:
  ostree-unverified-registry:ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:3a2bd576580c679d83174e8c2391108d26920cdeea44ffd8bede85ebedd071db
                  Version: 41.20250126.1 (2025-01-26T06:07:45Z)
                     Diff: 38 downgraded, 306 removed, 153 added

This is the same digest as what you report, but it has me moving to a consistent Nvidia 565 build.

Not sure what the problem is on your system

Regardless, with Jorge building new releases of stable, I expct that will resolve things.

@castrojo
Copy link
Member

Image

@Malix-Labs
Copy link
Author

Malix-Labs commented Jan 29, 2025

It...

took 45 minutes

...Worked

pixelated pepe exhausted

GG for everyone involved except my slow ass internet <3

@Malix-Labs
Copy link
Author

Could there be any added unit tests to prevent as much causes of unbootable system as possible?

@Malix-Labs
Copy link
Author

Malix-Labs commented Jan 30, 2025

I tried many things but the new image seem to not work either

It is probable than yesterday night I didn't booted to the new deployment by mistake, sorry for the false resolved status

❯ rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:cfdecfd35bc47326868df2f7d71bd4b7677de8ab7486c337b829f5f6422f6ac5
                  Version: 41.20250129.4 (2025-01-29T20:50:43Z)
            SecAdvisories: 1 low, 3 important, 1 critical
                     Diff: 208 upgraded, 45 removed, 1 added
          LayeredPackages: google-chrome-beta waydroid

● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
                   Digest: sha256:dfdbfb5acd4390ed61d024aaa356568db90c20aff5c75ae76724a344f2f95cae
                  Version: 41.20250119.1 (2025-01-19T06:13:33Z)
          LayeredPackages: google-chrome-beta waydroid

Back on 41.20250119.1 again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants