Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full System Policy (FSP) #252

Open
11 of 14 tasks
roddhjav opened this issue Nov 19, 2023 · 57 comments
Open
11 of 14 tasks

Full System Policy (FSP) #252

roddhjav opened this issue Nov 19, 2023 · 57 comments

Comments

@roddhjav
Copy link
Owner

roddhjav commented Nov 19, 2023

Following #249, this issue aims to keep tracks on progress on full system policy (FSP) integration.

The architecture overview can be found on the documentation: https://apparmor.pujol.io/full-system-policy/

Minimal feature set for FSP

  • systemd profile using early policy load.
  • Selection of program to run inherited (ix) from the systemd profile.
  • List of drop in systemd service file disabling no new privileges flag for some services: systemd/full
  • Stacked profiles when nnp is not an option
  • New @{systemd} variable used in rules such as signal and ptrace set to unconfined or systemd on prebuild time
  • New profiles group: service, to confine some systemd unit service (only)
  • Fallback profile: default / default-user
  • New profiles for all programs in:
    /usr/lib/systemd/system-generators/*
    /usr/lib/systemd/system-environment-generators/*
    /usr/lib/systemd/system/*.service
    
  • Same for user services:
    /usr/lib/systemd/user-generators/*
    /usr/lib/systemd/user-environment-generators/*
    /usr/lib/systemd/user/*.service
    
  • Test on all supported distribution & DE

Security Enhancement

  • Prefer stacking instead of disabling no new privileges
  • Document the full policy architecture, security model, and requirement
  • systemd-user prevents systemd --user to start on desktop (works fine on server).
  • User & context dependant profile thanks to PAM
@monsieuremre
Copy link
Contributor

This looks very good! You actually improved my amateur profile massively.

There are some minor details I do not understand. This section:

  # Required by stacked profiles
  capability net_admin,
  capability net_bind_service,
  capability net_raw,
  capability setfcap,
  capability setgid,
  capability setpcap,
  capability setuid,
  capability sys_nice,
  capability sys_ptrace,
  capability sys_time,

Why did you add this back? What do you mean by "Required by stacked profiles"? Phasing these out worked pretty well for me, normally. It seems you changed the approach of changing profiles. This, I presume, is to prevent the problems that might occur with nonewprivs. How does this stacking solve the problems with nonewpriv? That I do not understand unfortunately. It seems to be, reading the canonical documentation, that this should still be problematic, but I guess it is not.

Also the default and default-user profiles. I am assuming we would call these whenever we are normally using pUx or something similar. So maybe phasing out more things is more suited in these profiles and not in the systemd profile? I am assuming.

So my understanding is, the 'actual' full system policy is going to be these default and default-user profiles. But again, it is not clear to me how we would fallback to these profiles. We can fallback to inherit or unconfined, but there is no fallback to some other profile option. How will this be implemeted?

@roddhjav
Copy link
Owner Author

roddhjav commented Nov 20, 2023

A lot of service are configured with nnp options. Sometime, I use the stacking as workaround.

Stacking the parent profile (systemd) with a child (see systemd//& in the profile). For example, it means that the program polkitd is confined using the profile systemd//&polkitd (so the intersection of the profile systemd and polkitd). The direct consequence is that to work without issue, the systemd profile needs to include all rules of the polkitd profile. It explain why some access from systemd are only needed because of stacking. (Example for polkit: /etc/polkit*/** r, & sys_ptrace)

It is also good to mention that the systemd profile has pivot_root and change_profile (to every profile) rules. It is part of systemd feature set, so we need to have them. However, this effectively allow any program running under the systemd profile to change its profile to any other.

It may evolve, but the current plan for the fallback (and therefore, the full policy) should work as follows:

  • systemd / systemd-user should not fallback at all. I am working on some new profiles to ensure that 99% of program started by system have a profile.
  • Replace all Ux to Px -> default for system programs (yes, as you said, in some setup it might be hard - I have an idea, but it is ugly)
  • Replace all Ux to Px -> default-user for user programs (like GUI programs that don't have profile defined). We may also need some generic specialised profile such a default-flatpak profile for apps sandboxed under flatpak. I mention it as some sandbox manager (flatpak) is well-known to not have integration for apparmor.
  • Allow user connection under PAM rules set such as a random shell interactively opened by a user (as user or as root). This is already documented and just need to be implemented and tested (they are some bases already).

The security benefic of such a structure is multiple:

  • We don't fallback to one of the largest profile in the system (systemd)
  • We confine systemd in multiple part (system & user)
  • We provide multiple default profiles depending of the context (system, user, UI...)
  • That gives some sort of context based security. As in you start a console/program for either: admin, classic userland, dev...

As for right now, I am going to have a look at our main default profile. Once it is ready we will effectively have a full policy system, that will get improved later with time.

roddhjav added a commit that referenced this issue Nov 22, 2023
@roddhjav
Copy link
Owner Author

roddhjav commented Nov 22, 2023

Default profiles added (mostly wip, expect some issue). It should be good enough for GUI programs. More advanced privileged programs are expected to have a profile. In conjunction, I am currently adding about 100 new profiles to ensure neither the systemd nor the default profile get polluted with rule from other activity.

In a similar way, a flatpak integration is in progress.

But again, it is not clear to me how we would fallback to these profiles.

The solution is quite simple. The default profile attachment is: /**. It works as apparmor selects the best/closest match, so @{bin}/foo rPx will always transition to the foo profile if it exist, to default otherwise.

@monsieuremre
Copy link
Contributor

I see what you are doing and I don't understand how we can possibly manage this many default profiles. Two of them can be managable, for root and non root. But since the default profiles are only going to be called from other profiles, how do we know when to call default-app or default-bwrap or just default?

@monsieuremre
Copy link
Contributor

Depending on users with pam is doable, but how to distinguish default-app from default?

@roddhjav
Copy link
Owner Author

roddhjav commented Nov 23, 2023

For now, we don't have default-user. As you guessed, it may not be possible to have it with the /** attachment.
Currently the main default profile is: default. This is the one that has /** as attachment. The other default are simply transitioned from this one:

In default:

   @{bin}/bwrap           rPx -> default-bwrap,
   @{bin}/sudo            rPx -> default-sudo,

In default-bwrap:

  @{bin}/**                Px -> default-bwrap//&default-app,

Pam is different. On user logging, the pam role would transition the user under a given profile. See pam_roles & mappings

roddhjav added a commit that referenced this issue Nov 26, 2023
On  full system policy, use the new bwrap profile (and bwrap-app) to confine sandboxed application.
It is not enabled by default as the sandbox profile is quite large.

Also integrate with the gnome app that use bwrap as sandbox manager.

Update other related profiles

See Full system policy #252
roddhjav added a commit that referenced this issue Nov 26, 2023
- Add flatpak profile
- Add flatpak-bwrap subprofile: it manage the sandbox creation & has some larger access.
- Add flatpak-app, default profile for sandboxed app.

See Full system policy #252
@roddhjav
Copy link
Owner Author

@monsieuremre full system policy should be better now. Can you test it on Whonix and report any issue?

@monsieuremre
Copy link
Contributor

Debian and Kicksecure results in no unexpected behavior on Gnome with almost all the profiles enforcing even. I can do casual browsing, send emails, install stuff and run them. Xfce and KDE seem to break completely when enforcing most stuff, but I don't think this is unexpected anyway. All the testing I did was using KVM and not VirtualBox, so there is a slight chance Whonix might behave differently. I did the testing with the manual installation with make install and not by building the whonix deb package, so that will need seperate testing.

I suspect the bigger problem here would be on Whonix on Qubes tho. I am in no position to test this. But I think this is looking very very good and is ready for testing by more people. @adrelanos what do you think is the way to start testing? I think you should have a look at this for yourself first. It might be a little too early to add anything to the official testing repositories of Whonix, but I think you asking people in the forums would still help massively.

@adrelanos
Copy link

Adding a package (that is supposedly easily built) to the Kicksecure (and thereby by extension also to the Whonix) repository is doable but might be too early. I can probably do that quite soon if requested.

Unfortunately, I don't have time to initially test this on Whonix, Qubes. I suppose issues could be as big as an unbootable system, broken networking and whatnot.

Then me copying here apparmor denied messages, try new profiles, rinse and repeat a few times doesn't seem to be an efficient process either.

General thoughts:

  • Since Kicksecure and Whonix are two different Linux distributions and Kicksecure being independent from Whonix, a full system policy only being functional in Kicksecure but not in Whonix would be acceptable. I don't have the right to demand anything but even less so to demand Kicksecure + Whonix being functional at the same time.
  • You could say you only want to support Kicksecure, Gnome (on hardware +) in KVM. It could be documented as is, added to the Kicksecure repository but then this would messy and not be suitable for installation by default.
  • Xfce is a must or Kicksecure (and Whonix) as long as that's the default desktop before considering default installation obviously.

What other options are there?

I suspect the bigger problem here would be on Whonix on Qubes tho. I am in no position to test this.

Fixable? Might be too much of a personal question.

No interest in Whonix? No problem.

There's also Kicksecure for Qubes. Interested, usable?

If not usable because...? Lack of hardware? I might be able to monetarily donate hardware.

Otherwise would a CI server be useful? Qubes is already running a OpenQA server. Might be doable for Qubes to run OpenQA builds with full system profile. If the boot breaks, there would be logs. Then the package could be improved, merged, uploaded and check CI result again.

Github Actions (CI) an option?

chroot / systemd-nspawn a suitable environment to do testing?

Could also rent a server in the cloud that could be used for Qubes / Whonix testing.

@monsieuremre
Copy link
Contributor

Adding a package (that is supposedly easily built) to the Kicksecure (and thereby by extension also to the Whonix) repository is doable but might be too early. I can probably do that quite soon if requested.

Yes it is too early for that, but that should be the goal at some point I think.

Xfce is a must or Kicksecure (and Whonix) as long as that's the default desktop before considering default installation obviously.

When not enforcing some select profiles xfce works. So some profiles have to be in complain mode, which they actually are by default unless the user manually enforces them.

Then me copying here apparmor denied messages, try new profiles, rinse and repeat a few times doesn't seem to be an efficient process either.

Actually I quite disagree. It would not be efficient if you were the only one doing it. But if several people could try this out, I am sure most of the possible errors would be directly apparent very fast and could then be fixed. So even if only a handful of people installed these all in complain mode and provided their apparmor logs, I am very positive this would help @roddhjav quite a lot in further developing/adjusting the full system policy.

I can theoretically test for qubes with the current hardware I have, but this will require some preparation on my end to back up my current system, set up and install qubes, extensively test things and restore my old system afterwards and so on. So realistically I can't do this kind of testing any time soon due to time limitations unfortunately.

@osevan
Copy link

osevan commented Nov 29, 2023

I loove to see full system policy inside your project.

And hope it will be stable and more restricted than before for enforcers like us .-)

Thanks and i wish everyone else good luck.

@monsieuremre
Copy link
Contributor

I agree.

Once again, your work is amazing here @roddhjav. I have a literal full-system MAC policy in my system that actually functions. There is no desktop operating system that can achieve this, apart from Chrome OS, which also does it still in a rather uncomplete way compared to something like this or selinux on android.

It is so amazing that we are now more close than ever to having a fine-grained, fully functioning and non-debiltating full-system MAC policy for desktop GNU/Linux. This project is the only accessible way to achieve this on any desktop os ever and it is the only way to achieve this with apparmor anywhere ever (because android does this on mobile with selinux already). This is quite literally massive.

I would hope to see this implemented by general linux distributions by default also for non techy users in general with time. I am sure Whonix/Kicksecure will be the pioneer in this as they are the primary distribution focused on security.

I will continue testing and try to contribute to the best of my ability.

@osevan
Copy link

osevan commented Nov 30, 2023

I hope someone can write in future method for other init systems than systemd.

Because, security ppl like me dont using systemd .

Dinit is btw fastest init system with less LOC than systemd 2.1 million LOC.
Example artix dinit

Or devuan runit.

I hope it will possible to confine these init systems.

Thanks and

Best regards

@roddhjav
Copy link
Owner Author

roddhjav commented Nov 30, 2023

It is too early to add the package to Kicksecure/Whonix. It needs testing first.

  1. First we need to ensure that no logs are raised on classic mode (without full system policy), and that not too many major profiles are missing.
  2. Then, we can test the full policy mode and ensure all needed profiles are implemented

As Kicksecure/Whonix is pretty much a Debian, it should not be too hard to do knowing that Debian already works fine. My concern is more regarding the DE.

Then me copying here apparmor denied messages, try new profiles, rinse and repeat a few times doesn't seem to be an efficient process either.

On the contrary, as long as you are testing on complain mode and report log with aa-log -R. Logs provided by different people on different platform have been really useful to make this project more stable. This is thanks to this that KDE works fine (see #208).

Xfce is a must or Kicksecure (and Whonix) as long as that's the default desktop before considering default installation obviously.

I haven't tested this project with Xfce. I doubt it will even boot (even in complain mode) as too many profiles/rules would be missing. At minimum, this will require adding a lot of new profiles.

It seems you are considering moving to KDE (Kicksecure/security-misc#168). If you are serious with this, it would be easier to only apply apparmor.d once the move is effective as KDE is already supported.

If not usable because...? Lack of hardware? I might be able to monetarily donate hardware.

Biggest limitation is lack of time (as in: this project does not pay my rent, so the time I can spend in it is limited)

Otherwise would a CI server be useful?

There is already a full CI pipeline that build the package and do basic test on both Gitlab & Github. The difficulties are that:

  1. Beyond basic check like ensuring the profiles compile, we need a real test suite for apparmor profile. That's a wip, but right now, it is not looking great.
  2. Apparmor is a LSM, so we need a VM with apparmor enabled, not a container to test anything

When not enforcing some select profiles xfce works. So some profiles have to be in complain mode, which they actually are by default unless the user manually enforces them.

@monsieuremre can you share the log you got on xfce?

@osevan beware, it is only supposed to work on Gnome or KDE under a distribution fully supported in this project. By design, the default profile is not large enough to be used as fallback for privileged programs or for some DE shell.

I hope someone can write in future method for other init systems than systemd.

I won't have time to support any other init system than systemd. However, there is a solution to have similar setup outside systemd. In apparmor 4 (in alpha) there is a new aa-load tool that does the same thing than systemd: load a pre-compiled profile cache. So if you add something like aa-load /etc/apparmor/earlypolicy early in your boot script, it should work in the same way than with systemd. You will still need another profile than systemd as profile for your PID 1, but is is a start...

@monsieuremre
Copy link
Contributor

I want to test and debug the systemd-user profile to find out what stops it from functioning with DE's. Can not conduct tests anymore, cannot build with full and after install it.

root@debian12-uni-2:/home/user/Downloads/apparmor.d-main# sudo make full
Building apparmor.d profiles for debian.
 ✓ Initialize a new clean apparmor.d build directory
 ✓ Ignore profiles and files from:
    ⋅ dists/ignore/main.ignore
    ⋅ dists/ignore/debian.ignore
 ✓ Merge all profiles
 ✓ Set distribution specificities
 !! Profile bwrap not found
 !! Profile bwrap-app not found
 !! Profile default not found
 !! Profile default-sudo not found
 !! Profile systemd not found
 !! Profile systemd-user not found
 !! Profile aa-load not found
 !! Profile appimagelauncherd not found
 !! Profile apport not found
 !! Profile atril-previewer not found
 !! Profile chronyd not found
 !! Profile code not found
 !! Profile ctop not found
 !! Profile flatpak-validate-icon not found
 !! Profile gcr-ssh-agent not found
 !! Profile man not found
 !! Profile netplan not found
 !! Profile package-data-downloader not found
 !! Profile pam_kwallet_init not found
 !! Profile passim not found
 !! Profile plasma-discover not found
 !! Profile qdbus not found
 !! Profile realmd not found
 !! Profile snap not found
 !! Profile steam not found
 !! Profile steam-fossilize not found
 !! Profile steam-game not found
 !! Profile steam-gameoverlayui not found
 !! Profile steam-reaper not found
 !! Profile systemd-battery-check not found
 !! Profile systemd-bsod not found
 !! Profile systemd-firstboot not found
 !! Profile systemd-generator-user-environment-flatpak not found
 !! Profile systemd-network-generator not found
 !! Profile systemd-pcrphase not found
 !! Profile uuidd not found
 !! Profile WebKitNetworkProcess not found
 !! Profile dpkg-status not found
 ✓ Set flags on some profiles
    ⋅ dists/flags/main.flags
    ⋅ dists/flags/debian.flags
 ✓ Configure AppArmor for full system policy
 ✓ Bypass userspace tools restriction
 ✓ Build all profiles for full system policy mode
 ✓ Set complain flag on all profiles
 ✓ DBus directive applied
root@debian12-uni-2:/home/user/Downloads/apparmor.d-main# sudo make install
install: cannot stat '.build/systemd/user/*': No such file or directory
make: *** [Makefile:33: install] Error 1
root@debian12-uni-2:/home/user/Downloads/apparmor.d-main# 
  • Debian 12 latest on Gnome, virtualized with KVM
  • Golang backported or not, makes no difference

@roddhjav
Copy link
Owner Author

Good point, I forgot to push some files. Should be fixed with 18dbc60

@monsieuremre
Copy link
Contributor

Ok initial test results. First test was to make sure that the problem was caused because of the systemd-user profile indeed. It was the case but I had my doubts that this was not the only problem. Anyway, intuitively I thought maybe it's a dbus thing, so did the following test.

Adding the following lines to the profile:

dbus,

Result: Successfully boots on DE, slow boot.
Problems that still persist: Apparmor service fails at first, profiles still load afterwards somehow.

Jan 21 14:58:46 debian12-uni-2 apparmor.systemd[726]: ERROR processing regexs for profile bwrap, failed to load
Jan 21 14:58:47 debian12-uni-2 apparmor.systemd[2378]: profile has merged rule with conflicting x modifiers
Jan 21 14:58:47 debian12-uni-2 apparmor.systemd[2378]: ERROR processing regexs for profile bwrap, failed to load
Jan 21 14:58:47 debian12-uni-2 apparmor.systemd[2905]: profile has merged rule with conflicting x modifiers
Jan 21 14:58:47 debian12-uni-2 apparmor.systemd[2905]: ERROR processing regexs for profile default, failed to load
Jan 21 14:58:48 debian12-uni-2 apparmor.systemd[581]: Error: At least one profile failed to load
Jan 21 14:58:48 debian12-uni-2 systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
Jan 21 14:58:48 debian12-uni-2 systemd[1]: apparmor.service: Failed with result 'exit-code'.
Jan 21 14:58:48 debian12-uni-2 systemd[1]: Failed to start apparmor.service - Load AppArmor profiles.
Jan 21 14:58:48 debian12-uni-2 systemd[1]: apparmor.service: Consumed 18.385s CPU time.

Seems like an issue with the other profiles.

@monsieuremre
Copy link
Contributor

systemd-user prevents systemd --user to start on desktop (works fine on server).

@roddhjav Can not reproduce everything working without any desktop environment, if this is what you meant. The same profiles, bubblewrap and so on, break the boot sequence still. Apparmor service still seems to not start properly on boot. Profiles are still enforced after the fact (somehow, no idea how). So the same behavior as with a desktop environment. Tested on debian stable, so might be version related of some package.

@roddhjav
Copy link
Owner Author

roddhjav commented Jan 25, 2024

Problems that still persist: Apparmor service fails at first, profiles still load afterwards somehow.

This should be fixed now. I added a test in CI to ensure FSP mode always compile without issue.
Please keep in my, it is still pretty much in alpha quality.

Profiles are still enforced after the fact (somehow, no idea how).

That's because the fallback profile was not loaded (as it had an issue). And Apparmor breaks when it cannot find the profile to transition (this is a feature, not a bug)

@roddhjav
Copy link
Owner Author

Also, to test it with the systemd-user profile, you need to uncomment the lines in systemd/full/system/[email protected]

@monsieuremre
Copy link
Contributor

I have done some more tests and I can report the following.

The problem is not in systemd-user. The following is the results on debian 12 gnome desktop.

  • All profiles in enforce mode -> breaks
  • All profiles in complain mode -> does not break
  • All profiles in complain mode, but the default profile in enforce mode -> breaks
  • All profiles in enforce mode, but ``default``` in complain mode -> does not break

Following these results I draw tthe conclusion that the problem lies within the default profile. After having booted wiht all profiles in enforce mode and default in complain mode, running ```sudo aa-status returns these as the complain mode paths:

   /usr/sbin/avahi-daemon (1861) default
   /usr/sbin/avahi-daemon (1880) default
   /usr/sbin/spice-webdavd (1919) default
   /usr/lib/systemd/systemd (2484) default
   /usr/bin/pipewire (2500) default
   /usr/bin/wireplumber (2502) default
   /usr/bin/pipewire (2503) default
   /usr/libexec/gcr-ssh-agent (2593) default
   /usr/bin/dash (2809) default
   /usr/bin/bash (3522) default
   /usr/bin/bash (3528) default

These units are run under default, eventhough they should not be doing that. This might the root. In fact, the line

/usr/lib/systemd/systemd (2484) default

suggests the profile systemd-user is not used at all. The problem is with the transition from default. You tell me what might be the reason these select programs do not run in their profiles.

@monsieuremre
Copy link
Contributor

I also have to mention, when everything is enforced but the default profile: true that it doesn't break and boots. But during shutdown, gnome session thing hangs and does not shut down properly, we wait for timeout. It boots without any issue tho.

@roddhjav
Copy link
Owner Author

roddhjav commented Jan 26, 2024

Do not try it in enforce mode yet, it is not ready for this (most of the profiles should stay in complain mode anyway).

It seems that everything is run in the default profile, this is not supposed to work this way at all. On what system (distribution &
DE) are you testing it? Did you enable early policy load?

@monsieuremre
Copy link
Contributor

It seems that everything is run in the default profile, this is not supposed to work this way at all.

No, not everything. Everything runs in their own profile with no problems. Only the ones I sent are running under default. These are the problematic profiles. This is tested in Debian 12 gnome wayland session on kvm. Default installation on everything.

All I did was I downloaded the repo, then i did a make full and make install. Did not build the deb package or anything. Then I enforced all profiles, except default. Then I rebooted.

@monsieuremre
Copy link
Contributor

No actually never mind. It also broke after second reboot even with default in complain mode.

@roddhjav
Copy link
Owner Author

You will have to share some logs here...

@monsieuremre
Copy link
Contributor

Yessir of course. But before that I want to also share this

   /usr/sbin/avahi-daemon (1849) default
   /usr/sbin/avahi-daemon (1863) default
   /usr/sbin/spice-webdavd (1920) default
   /usr/libexec/gcr-ssh-agent (2044) default
   /usr/libexec/realmd (2321) default
   /usr/libexec/gcr-ssh-agent (2560) default
   /usr/bin/bash (3232) default
   /usr/bin/bash (3339) default

Pipewire and systemd is now fixed doing what you suggested, but these programs still run under default. Doesn't seem right.

@monsieuremre
Copy link
Contributor

Yeah so doing the same thing, I can report the following.

  • Everything is in complain -> nothing breaks ever
  • Everything enforce, default and systemd.service in complain, nothing breaks.
  • In addition default in enforce -> can't use coreutils or like cat ls etc to read logs.
  • In addition systemd.service in enforce mode -> mounts fail and does not boot. Also if enforced after boot, gnome session does not shut down.

Problems I can tell for certain:

  • systemd.service -> causes mount problems in boot.
  • Also causes problems with coreutils, but default causes this as well.
  • Avahi daemon and some other stuff are still run under default no matter what.

@roddhjav roddhjav changed the title Full system policy Full System Policy (FSP) Feb 1, 2024
@monsieuremre
Copy link
Contributor

Ok I am sorry seems like everything works just fine, because there is no profile for avahi daemon. What does remove avahi daemon as it already exists mean? Where does it exist upstream? Why was avahi daemon deleted from this project?

@monsieuremre
Copy link
Contributor

When all profiles are in complain and only the systemd profile is in enforce mode, boot fails no matter what. Specifically, this happens:

Failed to read configured hostname: Permission denied
Hostname set to <localhost>
Failed to read AF_UNIX diagram queue length, ignoring: Permission denied
Queued start job for default target graphical.target.
[UNSUPP] Starting of proc-sys-fs-binfmt_misc.automount ...
something something
...
[FAILED] Failed to start systemd-binfmt.service...
[FAILED] Failed to start accounts-daemon.service...
[FAILED] Failed to start e2scrub_reap.service...
[FAILED] Failed to start systemd-hostnamed.service...
[FAILED] Failed to start accounts-daemon.service...
[FAILED] Failed to start low-memory-monitor.service...

Everything else starts with no problems everything is green. Only these ones fail. Enforcing systemd profile only is enough. Giving all permissions apart from file execution paths won't change the result. Conclusion: the problem lies within the path and profile transmission rules in the systemd profile. It may also be due to something else, but then why does it persist even with everything allowed, apart from execute rules? No idea.

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 5, 2024

When all profiles are in complain and only the systemd profile is in enforce mode, boot fails no matter what.

Please, do not enforce anything yet on FSP. That is totally expected for now.

Also, when you report issue, please profile aa-log too, otherwise there is nothing I can do.

@monsieuremre
Copy link
Contributor

Here are some logs of what the systemd profile wants to deny normally.

ALLOWED systemd-service signal comm=sh requested_mask=send denied_mask=send signal=usr1 peer=ibus-daemon
ALLOWED ibus-daemon signal comm=sh requested_mask=receive denied_mask=receive signal=usr1 peer=systemd-service
ALLOWED goa-daemon signal comm=systemd requested_mask=receive denied_mask=receive signal=kill peer=systemd
ALLOWED systemd open /dev/snd/ comm=wireplumber requested_mask=r denied_mask=r
ALLOWED plymouthd signal comm=plymouthd requested_mask=send denied_mask=send signal=rtmin+22 peer=systemd
ALLOWED systemd signal comm=plymouthd requested_mask=receive denied_mask=receive signal=rtmin+22 peer=plymouthd
ALLOWED systemd-service mknod owner @{run}/network/restart-hotplug comm=touch requested_mask=c denied_mask=c
ALLOWED systemd-service open owner @{run}/network/restart-hotplug comm=touch requested_mask=wc denied_mask=wc
ALLOWED systemd open owner /etc/adjtime comm=systemd requested_mask=r denied_mask=r
ALLOWED systemd open owner @{sys}/class/power_supply/ comm=systemd requested_mask=r denied_mask=r
ALLOWED systemd file_lock comm=(s-daemon) family=unix sock_type=dgram protocol=0 requested_mask=send
ALLOWED systemd file_lock comm=(crub_all) family=unix sock_type=dgram protocol=0 requested_mask=send
ALLOWED systemd file_lock comm=(-monitor) family=unix sock_type=dgram protocol=0 requested_mask=send
ALLOWED systemd open owner /dev/snd/ comm=systemd requested_mask=r denied_mask=r
ALLOWED systemd file_lock comm=(ostnamed) family=unix sock_type=dgram protocol=0 requested_mask=send
ALLOWED systemd open owner @{PROC}/@{pid}/limits comm=(systemd) requested_mask=r denied_mask=r
ALLOWED systemd open owner /etc/security/limits.d/ comm=(systemd) requested_mask=r denied_mask=r
ALLOWED systemd open owner /etc/security/limits.d/95-pipewire.conf comm=(systemd) requested_mask=r denied_mask=r
ALLOWED systemd file_lock comm=(t-daemon) family=unix sock_type=dgram protocol=0 requested_mask=send
ALLOWED systemd open owner /var/lib/gdm3/.local/state/wireplumber/restore-stream comm=wireplumber requested_mask=r denied_mask=r
ALLOWED systemd open owner /var/lib/gdm3/.local/state/wireplumber/default-routes comm=wireplumber requested_mask=r denied_mask=r
ALLOWED systemd open /dev/snd/ comm=wireplumber requested_mask=r denied_mask=r
ALLOWED systemd file_lock comm=(-localed) family=unix sock_type=dgram protocol=0 requested_mask=send
ALLOWED systemd mknod owner /var/lib/gdm3/.local/state/wireplumber/restore-stream.ICRNI2 comm=wireplumber requested_mask=c denied_mask=c
ALLOWED systemd open owner /var/lib/gdm3/.local/state/wireplumber/restore-stream.ICRNI2 comm=wireplumber requested_mask=wrc denied_mask=wrc
ALLOWED systemd rename_src owner /var/lib/gdm3/.local/state/wireplumber/restore-stream.ICRNI2 comm=wireplumber requested_mask=wrd denied_mask=wrd
ALLOWED systemd rename_dest owner /var/lib/gdm3/.local/state/wireplumber/restore-stream comm=wireplumber requested_mask=wc denied_mask=wc
ALLOWED systemd open owner /var/lib/gdm3/.local/state/wireplumber/ comm=wireplumber requested_mask=r denied_mask=r
ALLOWED systemd mknod owner /var/lib/gdm3/.local/state/wireplumber/default-routes.1QQOI2 comm=wireplumber requested_mask=c denied_mask=c
ALLOWED systemd open owner /var/lib/gdm3/.local/state/wireplumber/default-routes.1QQOI2 comm=wireplumber requested_mask=wrc denied_mask=wrc
ALLOWED systemd rename_src owner /var/lib/gdm3/.local/state/wireplumber/default-routes.1QQOI2 comm=wireplumber requested_mask=wrd denied_mask=wrd
ALLOWED systemd rename_dest owner /var/lib/gdm3/.local/state/wireplumber/default-routes comm=wireplumber requested_mask=wc denied_mask=wc
ALLOWED systemd open owner @{user_state_dirs}/wireplumber/default-routes comm=wireplumber requested_mask=r denied_mask=r
ALLOWED systemd open owner @{user_state_dirs}/wireplumber/restore-stream comm=wireplumber requested_mask=r denied_mask=r
ALLOWED systemd mknod owner /var/lib/gdm3/.local/state/wireplumber/restore-stream.BB6OI2 comm=wireplumber requested_mask=c denied_mask=c
ALLOWED systemd open owner /var/lib/gdm3/.local/state/wireplumber/restore-stream.BB6OI2 comm=wireplumber requested_mask=wrc denied_mask=wrc
ALLOWED systemd rename_src owner /var/lib/gdm3/.local/state/wireplumber/restore-stream.BB6OI2 comm=wireplumber requested_mask=wrd denied_mask=wrd
ALLOWED systemd mknod owner @{user_state_dirs}/wireplumber/restore-stream.J5FWI2 comm=wireplumber requested_mask=c denied_mask=c
ALLOWED systemd open owner @{user_state_dirs}/wireplumber/restore-stream.J5FWI2 comm=wireplumber requested_mask=wrc denied_mask=wrc
ALLOWED systemd rename_src owner @{user_state_dirs}/wireplumber/restore-stream.J5FWI2 comm=wireplumber requested_mask=wrd denied_mask=wrd
ALLOWED systemd rename_dest owner @{user_state_dirs}/wireplumber/restore-stream comm=wireplumber requested_mask=wc denied_mask=wc
ALLOWED systemd open owner @{user_state_dirs}/wireplumber/ comm=wireplumber requested_mask=r denied_mask=r
ALLOWED systemd mknod owner @{user_state_dirs}/wireplumber/default-routes.X76VI2 comm=wireplumber requested_mask=c denied_mask=c
ALLOWED systemd open owner @{user_state_dirs}/wireplumber/default-routes.X76VI2 comm=wireplumber requested_mask=wrc denied_mask=wrc
ALLOWED systemd rename_src owner @{user_state_dirs}/wireplumber/default-routes.X76VI2 comm=wireplumber requested_mask=wrd denied_mask=wrd
ALLOWED systemd rename_dest owner @{user_state_dirs}/wireplumber/default-routes comm=wireplumber requested_mask=wc denied_mask=wc
ALLOWED ibus-memconf open /var/lib/dbus/machine-id comm=ibus-memconf requested_mask=r denied_mask=r
ALLOWED systemd open owner /var/cache/fwupd/ comm=(fwupd) requested_mask=r denied_mask=r

@monsieuremre
Copy link
Contributor

Also, I do not want to create a separate issue just to ask this. Why was the profile for avahi daemon removed?

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 6, 2024

Thanks, I found the solution regarding system-user. Without big surprise, it was dbus... it is always dbus (same than in #74, #80 & #235). The other log concern audio stacking, that going to be moved in systemd-user soon.

Why was the profile for avahi daemon removed?

It is part of the default apparmor profile set. You should have it in /etc/apparmor.d/usr.sbin.avahi-daemon

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 6, 2024

The systemd-user profile now works fine... if it is used without FSP... Otherwise systemd --user fail to start with:

× [email protected] - User Manager for UID 120
     Loaded: loaded (/usr/lib/systemd/system/[email protected]; static)
    Drop-In: /usr/lib/systemd/system/[email protected]
             └─10-login-barrier.conf, apparmor.conf
     Active: failed (Result: exit-code) since Tue 2024-02-06 21:59:12 UTC; 7min ago
       Docs: man:[email protected](5)
    Process: 2227 ExecStart=/usr/lib/systemd/systemd --user (code=exited, status=1/FAILURE)
   Main PID: 2227 (code=exited, status=1/FAILURE)
      Error: 6 (No such device or address)
        CPU: 16ms

Feb 06 21:59:12 aa-archlinux-gnome systemd[1]: Starting User Manager for UID 120...
Feb 06 21:59:12 aa-archlinux-gnome (systemd)[2227]: pam_warn(systemd-user:setcred): function=[pam_sm_setcred] flags=0x8002 service=[systemd-user] terminal=[] user=[gdm] ruser=[<unknown>] rhost=[<unknown>]
Feb 06 21:59:12 aa-archlinux-gnome (systemd)[2227]: pam_unix(systemd-user:session): session opened for user gdm(uid=120) by gdm(uid=0)
Feb 06 21:59:12 aa-archlinux-gnome systemd[2227]: Failed to lookup RuntimeDirectory path: No such device or address
Feb 06 21:59:12 aa-archlinux-gnome systemd[2227]: Failed to allocate manager object: No such device or address
Feb 06 21:59:12 aa-archlinux-gnome systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Feb 06 21:59:12 aa-archlinux-gnome systemd[1]: [email protected]: Failed with result 'exit-code'.
Feb 06 21:59:12 aa-archlinux-gnome systemd[1]: Failed to start User Manager for UID 120.

@monsieuremre
Copy link
Contributor

Well that sounds about right. I have done some testing myself. The full system policy still works without the systemd-user profile still. So, what does that mean? The two are fine but they just don't work when together?

How do we know that systemd user.service drop in config file is enough to make user services run under that profile? We enforce the FSP with a hook. So maybe it somehow takes prescedence? Like how else would this work?

I do not think the problem lies within the systemd-user profile. No matter what changes in that, it still blocks boot. Also, it breaks on complain mode, so what does that mean?

Also, why do we have a path for the sytemd-user profile? Isn't that like problematic because it is the same path as the normal systemd profile is supposed to have?

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 7, 2024

I do agree that systemd-user works fine, the issue lies is that systemd failed to start systemd-user (cf log).

The systemd profile is directly loaded by system at a very initial stage (it is loaded directly because the profile name is systemd). Then systemd starts [email protected] under a different profile thanks to the drop in file /usr/lib/systemd/system/[email protected]/apparmor.conf:

[Service]
AppArmorProfile=systemd-user

I don't think anything more should be needed here.

systemd-user does not have profile attachment (no @{exec_path} after profile systemd-user).

@monsieuremre
Copy link
Contributor

I can tell you this much, the problem very specifically lies in the line @{lib}/systemd/systemd rix,. Now forget about everything, forget systemd-user. Just deleting this line from systemd breaks the system. And adding that drop-in profile definition is essentially equivalent to ignoring this line. If you change the line rPx, rUx, it still breaks.

But good news. There is a way. If you change the line to @{lib}/systemd/systemd rpx -> systemd-user,, suddenly everything works. User is confined, everything starts, everything is fine. So delete the config file, and instead we do the transition directly within systemd in this manner. So the breakage is the scure profile transition. Scrubbing the environment messes up the system. Don't why, can't tell, but this is the reason, and this is the fix.

Also the avahi daemon profile is not included by default in debian, you need to install the apparmor profiles or whatever package. So you might consider restoring that profile just to be included in debian builds, until the next stable comes out.

@monsieuremre
Copy link
Contributor

Is it possible to extend the base abstraction for the whonix build? Because I would like to add a whonix specific abstraction that explicitly denies and forbids certain activities. There will be a file whonix under /etc/apparmor.d/abstractions. This will only be installed when the target is selected as whonix when building. The abstraction will explicitly deny writing to hardware, reading raw data, reading from devices directly, reading or changing kernel parameters, executing anything under /tmp and some other directories, restrictricts access to sys and proc and their sensitive subdirectories explicitly, hides various hardware information by explicitly denying access to various paths, modifying boot parameters, among other things. This will ensure that the full system policy still protects various things even when unconfined. This is meant for whonix. Whonix implements these measures manually by other means that are much weaker. This will enable whonix to implement the measures in a mac policy, even when some parts are in complain mode. I have draft of the abstraction. I will create a pull based on your feedback. As I said, this is for the whonix target only. If there is a better way for this than universally altering the base abstraction, please let me know. I think @adrelanos would support this too.

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 7, 2024

But good news. There is a way. If you change the line to @{lib}/systemd/systemd rpx -> systemd-user

Dam, it seems you are right. The solution seems so simple... and I have wasted so many time on it. Thanks !

Also the avahi daemon profile is not included by default in debian, you need to install the apparmor profiles or whatever package. So you might consider restoring that profile just to be included in debian builds, until the next stable comes out.

usr.sbin.avahi-daemon is part of apparmor-profiles. apparmor-profiles is a required dependency of apparmor.d.

Is it possible to extend the base abstraction for the whonix build?

Yes, that is possible (and purposely easy):

  1. Add you whonix specific base addition in: apparmor.d/abstractions/base.d/whonix
  2. Add this path in the ignore file of other distribution than whonix:
    In dists/ignore/<dist>.ignore (where <dist> is arch, debian, opensuse and ubuntu) add: apparmor.d/abstractions/base.d/whonix at the end of the existing Whonix specific definition section.

Now, let me warn you: most of the stuff you want to block would most likely break everything if you do it for every profile. Only testing can says, what will work and what won't. So good luck I guess...

@monsieuremre
Copy link
Contributor

So for the systemd profile. We have mount and umount. Which is a little too slack. So can we replace it with the following?

mount fstype=proc proc -> /proc,
mount fstype=sysfs sysfs -> /sys
mount fstype=devtmpfs udev -> /dev
mount fstype=devpts devpts -> /dev/pts
mount fstype=tmpfs tmpfs -> /run/**
mount fstype=ramfs ramfs -> /run/credentials/**
mount fstype=tmpfs tmpfs -> /dev/shm
mount fstype=tmpfs tmpfs -> /tmp
mount fstype=mqueue mqueue -> /dev/mqueue
mount fstype=tmpfs tmpfs -> /tmp
mount fstype=binfmt_misc binfmt_misc -> /proc/sys/fs/binfmt_misc
mount fstype=hugetlbfs hugetlbfs -> /dev/hugepages
mount fstype=configfs configfs -> /sys/kernel/config
mount fstype=fusectl fusectl -> /sys/fs/fuse/connections
mount fstype=cgroup cgroup -> /sys/fs/cgroup/net_cls
mount fstype=fuse.gvfsd-fuse gvfsd-fuse -> /run/user/1000/gvfs
mount fstype=fuse.portal portal -> /run/user/1000/doc
mount fstype=tracefs tracefs -> /sys/kernel/tracing
mount fstype=autofs systemd-1 -> /proc/sys/fs/binfmt_misc
mount fstype=bpf bpf -> /sys/fs/bpf
mount fstype=efivars efivars -> /sys/firmware/efi/efivars
mount fstype=pstore pstore -> /sys/fs/pstore
mount fstype=cgroup2 cgroup2 -> /sys/fs/cgroup
mount fstype=securityfs securityfs -> /sys/kernel/security
mount fstype=devtmpfs udev -> /dev

mount /dev/{s,h,v}d** # disk
mount /dev/{s,h,v}r** # cd rom
mount /dev/nvme** # nvme
mount /dev/mapper/** # luks stuff
mount /media/** # media

mount ** -> /mnt/** # anything can be mounted to /mnt

A little long but covers everything a system needs. Also covers 99% of all legit user specific use cases. It still blocks a lot of things compared to allowing mount all together tho. So it is a giant improvement. I am pretty sure some of these rules are not needed here but rather some other profile optimally. If that is the case point them out.

@adrelanos
Copy link

adrelanos commented Feb 9, 2024 via email

@monsieuremre
Copy link
Contributor

Possible. Anyone can choose to include the abstraction with a one liner makefile modification. Having it default for the whonix target is ultimately the goal tho.

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 9, 2024

So for the systemd profile. We have mount and umount. Which is a little too slack. So can we replace it with the following?

100% agree, current mount, is temporary, PR is welcome.

@monsieuremre
Copy link
Contributor

Also the current default profile is too strict. At least read privileges are necessary for some system directories. For the root user it should be even further expanded. How does the pam integration stuff work actually? There are no solid examples and the wiki is just like, old. What can we achieve here? There is already pam_binareis and other files among your profiles but these are really not worked on.

@monsieuremre
Copy link
Contributor

You can check this one from the list by the way:

systemd-user prevents systemd --user to start on desktop (works fine on server).

And also I want to ask,

Same for user services:
/usr/lib/systemd/user-generators/*
/usr/lib/systemd/user-environment-generators/*
/usr/lib/systemd/user/*.service

which specific profiles and profiles do we need to add manually? Can you create the dummy empty profiles so that I can test & post the logs here?

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 9, 2024

I need to expand the documentation, security model, architecture of FSP (See: https://apparmor.pujol.io/full-system-policy/#fallback). However the basic idea is that, in FSP mode:

  1. All privileged applications must have a profile. It possibly means we need to add a few profiles, but that's not a problem...
  2. default should only be used for user application. Allowing it to be used for root app would make it useless (way too large).

I still need to integrate pam into FSP. But, this is closely related to the security model and what kind of security you want to get. See https://fedoraproject.org/wiki/SIGs/ConfinedUsers

@monsieuremre
Copy link
Contributor

Last addition. So the drop in files to disable no new privileges are https://github.com/roddhjav/apparmor.d/tree/main/systemd/full/system.

You say we can stack profiles and not disable no new privs. Well, we do profile stacking for polkitd but we still disable no new privileges. So what's up with that? What exactly do we need to have so that we won't need to disable no new privileges?

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 9, 2024

I was testing stuff. Disabling nnp in system drop in for a lot of programs seems a bad idea (it may require disabling much more than nnp as a lot of options imply it, and thus it is a pain to test/maintain). Meanwhile stacking is not possible for all profiles (but fine for systemd one) as it require to add all content of the profile into the parent one.

I am currently cleaning this a bit...

@adrelanos
Copy link

adrelanos commented Feb 10, 2024 via email

@monsieuremre
Copy link
Contributor

Yes. The derivative specific opt-in then is hopefully then just a one liner with all the complexities and features available to all non-derivative users who wish to use it too.

Well you don't say. I'm on it, will have a pull very soon.

But before that, imma have a pull with the mount stuff. @roddhjav did you test what you have pushed? It seems a little missing to me. First of all i think it is better to add the fstype constraints to most stuff. There is no legit reason to mount /dev with something other than udev. Same applies for a lot of things. I think /media and /mnt stuff should be added for more user convenience. To cover most anything. I do not understand the namespace thing as the target but I will after I test it. Seems like a better option than what I had for constraints.

Also, I wanna ask again @roddhjav, what profiles are missing for user services? Can you at least make a list to start working on. I think these will be easier to profile than system services, which you have already done. I just want to get everything ready on the checklist's essential list before trying to test and learn integrating the pam stuff.

@monsieuremre
Copy link
Contributor

Also big time problem:

After installing with the full option, why does the early policy not get configured automatically? How does this make any sense? this can be automated in the makefile easily. Should create a pull request or is there a specific reason you keep it this way?

@roddhjav
Copy link
Owner Author

roddhjav commented Feb 10, 2024

But even better if neither specific to Whonix nor Kicksecure but a
generic feature that other users are free to re-use.

Best to keep specificity to any projects as low as possible. Possible?

As much as I understand you... it is more complex than this. There are multiple version of apparmor and apparmor-profiles arround. apparmor.d uses as reference the latest available (and upstream some directly into apparmor-profiles). However, as Debian is always a bit behind, apparmor.d needs to manually include some resources that are already upstreamed (see https://github.com/roddhjav/apparmor.d/tree/main/dists/ubuntu/abstractions) as well as patch the profiles for older version of apparmor. This is actually going to be a big step as apparmor 4.0 is going to be released soon.

To sum up they will always be some sort of patch for some distribution.

But before that, imma have a pull with the mount stuff. @roddhjav did you test what you have pushed?

Yes, this is tested... there is no fstype or src for some part, because the rules were not working with it. I still need to add rules regarding the @{MOUNTS}.

What profiles are missing for user services?

I still need to make the list, but you can have a look yourself with

grep ExecStart /usr/lib/systemd/system/*.service

and:

grep ExecStart /usr/lib/systemd/user/*.service

As it changes depending of the distribution (and the program installed), for Whonix, it will be your job to ensure everything is handled with a profile.

We might be about 95% here in term of profile (and it might take more time to ensure we have everything than to write the missing - usually very simple - profiles).

After installing with the full option, why does the early policy not get configured automatically?

early policy needs to be configured in /etc/apparmor/parser.conf witch is not a file that apparmor.d own, therefore it is never going to be automatic (make does not have control over it).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants