WIP try to move our pods to nonroot-v2 profile #605
Conversation
gibizer
commented
Nov 24, 2023
gibizer
commented
- Kolla needs sudo, sudo needs uid 0.
- Manually mapping uid 0 via annotation does not seem to work.
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: gibizer The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
Build failed (check pipeline). Post https://review.rdoproject.org/zuul/buildset/e2de8f29f7514ef386f8c4be35d31b62 ✔️ nova-operator-content-provider SUCCESS in 1h 35m 38s |
pkg/novaconductor/dbsync.go
Outdated
| @@ -59,6 +59,11 @@ func CellDBSyncJob( | |||
| }, | |||
| Spec: batchv1.JobSpec{ | |||
| Template: corev1.PodTemplateSpec{ | |||
| ObjectMeta: metav1.ObjectMeta{ | |||
| Annotations: map[string]string{ | |||
| "io.kubernetes.cri-o.userns-mode": "private:uidmapping=0:100000:90000;gidmapping=0:100000:90000", | |||
There was a problem hiding this comment.
looking into rhcos crc host's /etc/crio/crio.conf.d/00-default, there is:
[crio.runtime.workloads.openshift-builder]
activation_annotation = "io.openshift.builder"
allowed_annotations = [
"io.kubernetes.cri-o.userns-mode",
"io.kubernetes.cri-o.Devices"
]
which looks a bit different to https://github.com/cri-o/cri-o/blob/main/tutorials/userns.md#cri-o-configuration
we need to reach out OCP folks to clarify the support status of this
There was a problem hiding this comment.
My naive guess, we also need the activation annotation specified so that the allowed annotations became treated as needed
There was a problem hiding this comment.
There was a problem hiding this comment.
to summarize:
- we might need /etc/subuid and /etc/subgid tweaks like that blog post provides.
# on my crc host:
[core@crc-lz7xw-master-0 ~]$ cat /etc/subgid
core:100000:65536
containers:200000:16000000
[core@crc-lz7xw-master-0 ~]$ cat /etc/subuid
core:100000:65536
containers:200000:16000000
# proposed to patch for MCO in the blogpost:
[core@crc-lz7xw-master-0 ~]$ echo Y29yZToxMDAwMDA6NjU1MzYKY29udGFpbmVyczoyMDAwMDA6MjY4NDM1NDU2Cg== | base64 -d
core:100000:65536
containers:200000:268435456
[core@crc-lz7xw-master-0 ~]$ echo Y29yZToxMDAwMDA6NjU1MzYKY29udGFpbmVyczoyMDAwMDA6MjY4NDM1NDU2Cg== | base64 -d
core:100000:65536
containers:200000:268435456
- we need to use io.openshift.builder annotations as well
- we need map-to-root https://frasertweedale.github.io/blog-redhat/posts/2022-02-02-openshift-user-ns-without-anyuid.html#solution
- we can omit unrelated things mentioned in the blogpost, like machne config tweaks for cgroups v2
There was a problem hiding this comment.
The doc talks about changing a conf file /etc/crio/crio.conf.d/01-userns-workload.conf so that is out of scope of our possibilitities.
There was a problem hiding this comment.
this is not needed, see the summarized steps
There was a problem hiding this comment.
we cant modify any config on the CRC host.
we cannot require any host config bar the default provided by openshfit.
so unless the /etc/subuid and /etc/subgid are purely in the container we cannot make those changes
There was a problem hiding this comment.
i have not fully read the blog you linked https://frasertweedale.github.io/blog-redhat/posts/2022-02-02-openshift-user-ns-without-anyuid.html#solution
but yes we might be able to use some of those steps
so i take it your suggestion is just to add
Annotations: map[string]string{
io.openshift.userns: "true",
io.kubernetes.cri-o.userns-mode: "auto:size=65536;map-to-root=true"
}
or perhaps
Annotations: map[string]string{
io.openshift.userns: "true",
"io.kubernetes.cri-o.userns-mode": "private:uidmapping=0:100000:90000;gidmapping=0:100000:90000;map-to-root=true"
}
There was a problem hiding this comment.
we cant modify any config on the CRC host. we cannot require any host config bar the default provided by openshfit.
so unless the /etc/subuid and /etc/subgid are purely in the container we cannot make those changes
I think we can omit those MCO tunings, it should just work (see the diff I provided, it looks not so much important)
There was a problem hiding this comment.
see resulting change that I tested bogdando@0bd0e77#diff-9d28b78ae63978beb363c084081d603e3566e8827c948d735f83f5868e6c6fca
|
FYI use |
|
This test job has worked for me it logs |
| ObjectMeta: metav1.ObjectMeta{ | ||
| Annotations: map[string]string{ | ||
| "io.openshift.builder": "true", | ||
| "io.kubernetes.cri-o.userns-mode": "auto:size=65536;map-to-root=false", |
There was a problem hiding this comment.
In my local tests this still result in:
❯ oc logs nova-cell0-conductor-db-sync-wf8md --follow
+ sudo -E kolla_set_configs
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?
There was a problem hiding this comment.
yes, I can confirm that:
[nova@nova-cell0-conductor-db-sync-djkdt /]$ sudo whoami
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?
so this PoC needs more work then...
There was a problem hiding this comment.
btw, the sudo-less way to test this also works with map-to-root=true
|
Build failed (check pipeline). Post https://review.rdoproject.org/zuul/buildset/83a1e22d24eb41d490e77ce6eb9289fa ✔️ nova-operator-content-provider SUCCESS in 1h 23m 26s |
|
In CI I see a possibly different error.
|
* Kolla needs sudo, sudo needs uid 0. * Manually mapping uid 0 via annotation does not seem to work.
This was a missing / unaligned RBAC annotation for nonroot-v2, fixed now. |
|
Build failed (check pipeline). Post https://review.rdoproject.org/zuul/buildset/6346d0c06b2d4385b63ce70425efa144 ✔️ nova-operator-content-provider SUCCESS in 59m 02s |
|
If the use of sudo by kolla blocks this, I'd say let's no longer use kolla and sudo |
|
For what I see, using kolla implies that
What's the goal of having kolla at all? Config file management seems to be as easy with or without it (arguably simpler without?) What else? (Config file management becomes a moot point for services like OVN that don't even have config files / INI files through oslo.config). Anyone have an idea behind kolla beyond historical reasons? |
note that we don't use upstream Kolla, but its forked version as of tripleo. |
|
we provide the kolla image abi to our partenrs to integrate against we can define our own but we need to do so carefully if we do @booxter we also tried to document this in https://github.com/openstack-k8s-operators/docs/blob/main/service_config.md |
As I noted earlier, formally we do not follow it as of 2020 |
we follow the API we forked the implementation there is a diffence |
We do not track Kolla API changes to reflect them into the fork. My point is that we could change the sudo requirements in the forked implementation w/o breaking the APIs |
