logrotate.service failed due to bad permissions on /var/log/sssd/*.log #1798

Nemric · 2024-09-18T08:45:36Z

Describe the bug

Fedora CoreOS 41.20240916.1.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Tue Sep 17 19:17:17 2024 from 192.168.10.58
[systemd]
Failed Units: 1
  logrotate.service

core@Turing:~$ journalctl -eu logrotate.service 
Sep 18 00:38:22 Turing systemd[1]: Starting logrotate.service - Rotate log files...
Sep 18 00:38:22 Turing logrotate[330652]: error: skipping "/var/log/sssd/*.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
Sep 18 00:38:22 Turing systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 00:38:22 Turing systemd[1]: logrotate.service: Failed with result 'exit-code'.
Sep 18 00:38:22 Turing systemd[1]: Failed to start logrotate.service - Rotate log files.

Reproduction steps

Well, just upgrade and wait or manualy trigger logrotate.service
I run FCOS on this machine since months/years so I don't know when these files get their first permissions that seems to be a problem now

Expected behavior

No failed units

Actual behavior

Having a failed unit after logrotate

System details

Baremetal PXE booted FCOS with /var mounted on HDD for data persistance

Butane or Ignition config

not relevant

Additional information

/var/log/sssd is empty and is owned by sssd:sssd

root@Turing:~# ls -lah /var/log/sssd/
total 4.0K
drwxrwx---.  2 sssd sssd    6 Apr 21  2023 .
drwxr-xr-x. 12 root root 4.0K Sep  1 00:00 ..

The text was updated successfully, but these errors were encountered:

tazihad · 2024-09-18T09:11:24Z

Similar issue for rebasing Fedora kinoite 40 to kinoite 41
https://discussion.fedoraproject.org/t/rebase-to-fedora-41-kinoite-gives-error/

travier · 2024-09-18T16:50:02Z

Likely https://bugzilla.redhat.com/show_bug.cgi?id=2308428

dustymabe · 2024-10-18T20:56:14Z

Landed upstream in SSSD/sssd#7610 and in Fedora in https://bodhi.fedoraproject.org/updates/FEDORA-2024-73827b9035

dustymabe · 2024-10-18T20:56:37Z

Fast-track PR was in coreos/fedora-coreos-config#3208

aaradhak · 2024-10-21T17:10:47Z

This issue seems to be resolved when tested with build 41.20241017.10.0 that has the fix sssd-2.10.0-1.fc41

Verification steps:

aaradhak@fedora ~/fcos2 $ cosac run --qemu-image=fedora-coreos-40.20241006.2.1-qemu.x86_64.qcow2
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/home/aaradhak/fcos2:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa quay.io/coreos-assembler/coreos-assembler:latest run --qemu-image=fedora-coreos-40.20241006.2.1-qemu.x86_64.qcow2
Fedora CoreOS 40.20241006.2.1
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

[core@cosa-devsh ~]$ rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Mon 2024-10-21 14:40:06 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/testing
                  Version: 40.20241006.2.1 (2024-10-07T23:08:52Z)
                   Commit: 33f3ff2d0be25991e48980ab1983ce3e2406d3c897e705ce940f7089c4ab5133
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
[core@cosa-devsh ~]$ sudo systemctl start logrotate.service 
[core@cosa-devsh ~]$ systemctl status logrotate.service 
○ logrotate.service - Rotate log files
     Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead) since Mon 2024-10-21 14:43:45 UTC; 10s ago
TriggeredBy: ● logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
    Process: 2413 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=0/SUCCESS)
   Main PID: 2413 (code=exited, status=0/SUCCESS)
        CPU: 20ms

Oct 21 14:43:45 cosa-devsh systemd[1]: Starting logrotate.service - Rotate log files...
Oct 21 14:43:45 cosa-devsh systemd[1]: logrotate.service: Deactivated successfully.
Oct 21 14:43:45 cosa-devsh systemd[1]: Finished logrotate.service - Rotate log files.
[core@cosa-devsh ~]$ ARCH="x86_64"
[core@cosa-devsh ~]$ STREAM="next-devel"
[core@cosa-devsh ~]$ sudo systemctl stop zincati.service
[core@cosa-devsh ~]$ sudo rpm-ostree rebase "fedora-compose:fedora/${ARCH}/coreos/${STREAM}"

[core@cosa-devsh ~]$ sudo systemctl reboot

Broadcast message from root@localhost on pts/1 (Mon 2024-10-21 14:46:45 UTC):

The system will reboot now!

Fedora CoreOS 41.20241017.10.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Last login: Mon Oct 21 14:30:01 2024
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
Deployments:
● fedora-compose:fedora/x86_64/coreos/next-devel
                  Version: 41.20241017.10.0 (2024-10-18T01:59:58Z)
                   Commit: ec7e70f49951eabc52ebed6f50ae2804d6510c90e6b9ba28e67c125950feec66
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1

  fedora:fedora/x86_64/coreos/testing
                  Version: 40.20241006.2.1 (2024-10-07T23:08:52Z)
                   Commit: 33f3ff2d0be25991e48980ab1983ce3e2406d3c897e705ce940f7089c4ab5133
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ sudo systemctl start logrotate.service 
[core@cosa-devsh ~]$ sudo systemctl status logrotate.service 
○ logrotate.service - Rotate log files
     Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead) since Mon 2024-10-21 14:47:22 UTC; 6s ago
 Invocation: 6a842525b5bf4a799b270d4cbf2ad53f
TriggeredBy: ● logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
    Process: 1853 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=0/SUCCESS)
   Main PID: 1853 (code=exited, status=0/SUCCESS)
   Mem peak: 4M
        CPU: 47ms

Oct 21 14:47:22 cosa-devsh systemd[1]: Starting logrotate.service - Rotate log files...
Oct 21 14:47:22 cosa-devsh systemd[1]: logrotate.service: Deactivated successfully.
Oct 21 14:47:22 cosa-devsh systemd[1]: Finished logrotate.service - Rotate log files.

Nemric · 2024-10-22T18:26:33Z

Yeah ! did run systemctl start logrotate and everything went fine !

dustymabe · 2024-10-28T19:39:10Z

The fix for this went into next stream release 41.20241020.1.0. Please try out the new release and report issues.

dustymabe · 2024-10-28T19:39:59Z

This issue never affected testing or stable streams.

markuslindenberg · 2024-11-15T08:33:42Z

I just had this happen to a bunch of machines that got upgraded to 41 stable two nights ago:

root@stun-dev:~# systemctl status logrotate.service 
× logrotate.service - Rotate log files
     Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Fri 2024-11-15 00:37:15 CET; 8h ago
 Invocation: cb5e63e53a074a61a943ecdf3c974611
TriggeredBy: ● logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
    Process: 2322 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
   Main PID: 2322 (code=exited, status=1/FAILURE)
   Mem peak: 3.8M
        CPU: 105ms

Nov 15 00:37:15 stun-dev.example.com systemd[1]: Starting logrotate.service - Rotate log files...
Nov 15 00:37:15 stun-dev.example.com logrotate[2322]: error: unable to open /var/log/sssd/p11_child.log-20241023 (read-only) for compression: Permission denied
Nov 15 00:37:15 stun-dev.example.com systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Nov 15 00:37:15 stun-dev.example.com systemd[1]: logrotate.service: Failed with result 'exit-code'.
Nov 15 00:37:15 stun-dev.example.com systemd[1]: Failed to start logrotate.service - Rotate log files.
root@stun-dev:~# rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Fri 2024-11-15 08:28:52 UTC)
Deployments:
● fedora:fedora/x86_64/coreos/stable
                  Version: 41.20241027.3.0 (2024-11-08T21:22:46Z)
               BaseCommit: 8caf08dc8e25f389ade284ad2ad3ef64763e30a63038459d89a616b8725d12a6
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
          LayeredPackages: htop oddjob-mkhomedir python3 qemu-guest-agent tcpdump vim

  fedora:fedora/x86_64/coreos/stable
                  Version: 40.20241019.3.0 (2024-10-26T12:34:27Z)
               BaseCommit: 6df70065620571076f242857b9080d747891e2279dff3ed1756270f6889731ce
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
          LayeredPackages: htop oddjob-mkhomedir python3 qemu-guest-agent tcpdump vim
root@stun-dev:~#

DemonicTutor · 2024-11-15T13:45:13Z

logrotate failed on all of our EC2 instances updating to f41.

systemctl stop logrotate
systemctl start logrotate

seems to have fixed it

Nov 15 00:11:46 ip-10-104-0-79 systemd[1]: Starting logrotate.service - Rotate log files...
Nov 15 00:11:46 ip-10-104-0-79 logrotate[10262]: error: unable to open /var/log/sssd/sssd_nss.log-20241031 (read-only) for compression: Permission denied
Nov 15 00:11:46 ip-10-104-0-79 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Nov 15 00:11:46 ip-10-104-0-79 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Nov 15 00:11:46 ip-10-104-0-79 systemd[1]: Failed to start logrotate.service - Rotate log files.
Nov 15 13:39:07 ip-10-104-0-79 systemd[1]: Starting logrotate.service - Rotate log files...
Nov 15 13:39:07 ip-10-104-0-79 logrotate[28434]: error: unable to open /var/log/sssd/sssd_ssh.log-20241031 (read-only) for compression: Permission denied
Nov 15 13:39:07 ip-10-104-0-79 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Nov 15 13:39:07 ip-10-104-0-79 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Nov 15 13:39:07 ip-10-104-0-79 systemd[1]: Failed to start logrotate.service - Rotate log files.
Nov 15 13:39:20 ip-10-104-0-79 systemd[1]: Starting logrotate.service - Rotate log files...
Nov 15 13:39:21 ip-10-104-0-79 systemd[1]: logrotate.service: Deactivated successfully.
Nov 15 13:39:21 ip-10-104-0-79 systemd[1]: Finished logrotate.service - Rotate log files.

dustymabe · 2024-11-15T14:28:11Z

@alexey-tikhonov

Maybe a few more directories we need to add like we did in SSSD/sssd#7610 ?

alexey-tikhonov · 2024-11-15T14:39:04Z

@alexey-tikhonov

Maybe a few more directories we need to add like we did in SSSD/sssd#7610 ?

No, that's not relevant.

See above - I guess logrotate needs to be restarted to pickup new config.

dustymabe · 2024-11-15T15:29:50Z

See above - I guess logrotate needs to be restarted to pickup new config.

In other words.. for the boot where the chown happens as part of ExecStartPre for sssd.service logrotate could have started first and hit this error prior to the permissions being fixed?

alexey-tikhonov · 2024-11-15T15:36:47Z

See above - I guess logrotate needs to be restarted to pickup new config.

In other words.. for the boot where the chown happens as part of ExecStartPre for sssd.service logrotate could have started first and hit this error prior to the permissions being fixed?

The fix for this issue - SSSD/sssd@e4ae4d6 - doesn't require any chown-s.

jlebon · 2024-11-15T16:58:36Z

This should've been a new issue. The error message is different. It's now:

error: unable to open /var/log/sssd/sssd_nss.log-20241031 (read-only) for compression: Permission denied

Hmm, sssd.service already has:

ExecStartPre=+-/bin/sh -c "/bin/chown -f @SSSD_USER@:@SSSD_USER@ @logpath@/*.log"

Seems like that should be e.g. @logpath@/*.log* ? (Or just @logpath@/*.)

alexey-tikhonov · 2024-11-15T17:27:14Z

Seems like that should be e.g. @logpath@/*.log* ? (Or just @logpath@/*.)

Maybe. Please feel free to open a PR upstream.
But frankly this is one-time event, maybe easier to chown manually once.

markuslindenberg · 2024-11-15T18:11:22Z

Maybe not helping in solving the problem, but I have now rolled out DEBUG_LOGGER=--logger=journald in /etc/sysconfig/sssd and deleted all log files on my systems. Seems like sssd and auditd are the only services left (at least on my coreos systems) that still log to /var/log "manually". I'd rather have everything log to journald.

Nemric added the kind/bug label Sep 18, 2024

travier added F41 status/pending-upstream-release Fixed upstream. Waiting on an upstream component source code release. labels Sep 18, 2024

dustymabe changed the title ~~[f41] logrotate.service failed due to bad permissions on /var/log/sssd/*.log~~ logrotate.service failed due to bad permissions on /var/log/sssd/*.log Sep 20, 2024

dustymabe added the fallout/f41 label Sep 20, 2024

aaradhak added the jira for syncing to jira label Oct 17, 2024

dustymabe added status/pending-next-release Fixed upstream. Waiting on a next release. and removed status/pending-upstream-release Fixed upstream. Waiting on an upstream component source code release. labels Oct 18, 2024

Nemric closed this as completed Oct 22, 2024

dustymabe removed the status/pending-next-release Fixed upstream. Waiting on a next release. label Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

logrotate.service failed due to bad permissions on /var/log/sssd/*.log #1798

logrotate.service failed due to bad permissions on /var/log/sssd/*.log #1798

Nemric commented Sep 18, 2024 •

edited

Loading

tazihad commented Sep 18, 2024

travier commented Sep 18, 2024

dustymabe commented Oct 18, 2024

dustymabe commented Oct 18, 2024

aaradhak commented Oct 21, 2024 •

edited

Loading

Nemric commented Oct 22, 2024

dustymabe commented Oct 28, 2024

dustymabe commented Oct 28, 2024

markuslindenberg commented Nov 15, 2024

DemonicTutor commented Nov 15, 2024

dustymabe commented Nov 15, 2024

alexey-tikhonov commented Nov 15, 2024

dustymabe commented Nov 15, 2024

alexey-tikhonov commented Nov 15, 2024

jlebon commented Nov 15, 2024

alexey-tikhonov commented Nov 15, 2024

markuslindenberg commented Nov 15, 2024

logrotate.service failed due to bad permissions on /var/log/sssd/*.log #1798

logrotate.service failed due to bad permissions on /var/log/sssd/*.log #1798

Comments

Nemric commented Sep 18, 2024 • edited Loading

Describe the bug

Reproduction steps

Expected behavior

Actual behavior

System details

Butane or Ignition config

Additional information

tazihad commented Sep 18, 2024

travier commented Sep 18, 2024

dustymabe commented Oct 18, 2024

dustymabe commented Oct 18, 2024

aaradhak commented Oct 21, 2024 • edited Loading

Nemric commented Oct 22, 2024

dustymabe commented Oct 28, 2024

dustymabe commented Oct 28, 2024

markuslindenberg commented Nov 15, 2024

DemonicTutor commented Nov 15, 2024

dustymabe commented Nov 15, 2024

alexey-tikhonov commented Nov 15, 2024

dustymabe commented Nov 15, 2024

alexey-tikhonov commented Nov 15, 2024

jlebon commented Nov 15, 2024

alexey-tikhonov commented Nov 15, 2024

markuslindenberg commented Nov 15, 2024

Nemric commented Sep 18, 2024 •

edited

Loading

aaradhak commented Oct 21, 2024 •

edited

Loading