Bottlerocket support

[rcgeorge23]

Hi sysbox team,

I was wondering whether there are any plans to support Bottlerocket in the future?

Thanks

[ctalledo]

Hi @rcgeorge23, thanks for opening this discussion.

We've not had a chance yet at looking into running Sysbox on Bottlerrocket, but it would be a natural fit. The upcoming Sysbox v0.5.0 release will make it possible to run Sysbox on most distros assuming kernel >= 5.12 (i.e., to leverage a new feature called ID-mapped mounts that Sysbox uses to expose host files into the rootless container with proper permissions). Thus, if Bottlerocket brings in kernel >= 5.12, it's likely Sysbox v0.5.0 will work on it, although we would need to try and test (devil is in the details always :) ).

[ctalledo]

Hi @StevenPJ, apologies for the belated reply (the message notification got buried in my email).

We've not yet had a chance to try Sysbox on bottlerocket, but since it includes kernel 5.15, I don't expect too many issues to pop up. Have you had a chance to try it?

One thing that would help us support it would be to have a container image based on Botterocket, such that we can run Sysbox inside of it and catch any issues (similar to what we have for other Linux distros (see Dockerfiles here).

[stmcginnis]

to have a container image based on Botterocket

One point of clarification - Bottlerocket is an OS for running containers, but as a base for container images themselves.

[ctalledo]

Hi @stmcginnis, thanks for the clarification. My point there was that some Linux distros offer a containerized image of the distro itself (just the user-space portion, not the kernel). For example, the Ubuntu or Alpine images. I am hoping there is something similar for Bottlerocket, as that helps us test that Sysbox works in the distro, with the caveat that the kernel in an actual Bottlerocket installation would be different.

[stmcginnis]

Unfortunately no. Bottlerocket is a container optimized OS (think CoreOS and Flatcar), so it's not something that can be containerized itself.

[ctalledo]

Oh that's too bad. FYI there is one such container image for Flatcar, we used it about a year ago to test Sysbox on it.

[eugenejen]

is there any roadmap on running sysbox on bottlerocket os now?

[ctalledo]

Hi @eugenejen, unfortunately we don't have concrete plans to support Sysbox on Bottlerocket, but I would be surprised if it does not work already. I say so because the latest version of Sysbox (v0.6.2) works on several Linux distros, assuming kernel >= 5.19. We don't have a package installer for distros other than Ubuntu/Debian, but it's pretty easy to build the binaries from source and then run them on any distro.

[eugenejen]

Hi @ctalledo ,

I think the current amazon bottlerocket is configured to have /usr/local as read-only by default. Therefore sysbox-install will failed as

Warning Failed 10s kubelet Error: failed to generate container "134d3472b419b5a66c495d7297c45f6b248a42105672a27f0053c5d4cbc60cda" spec: failed to apply OCI options: failed to mkdir "/usr/local/bin/": mkdir /usr/local: read-only file system

[ctalledo]

Thanks @eugenejen; what would be the right directory to install binaries on Bottlerocket then?

BTW, the sysbox-deploy-k8s daemonset source code is here, and the main installation script is here. We can adjust it for bottlerocket as needed.

[eugenejen]

@ctalledo I think i probably need to submit and MR. For the moment i just start to create an EKS with ubuntu.

I found a different issue now for ubutun but that will be different discussion topic.

[eugenejen]

just a note. the root cause for bottlerocket filesystem lockdown is
due to this feature.

https://github.com/bottlerocket-os/bottlerocket/blob/develop/SECURITY_FEATURES.md#immutable-rootfs-backed-by-dm-verity

[eugenejen]

@ctalledo from what i read from bottlerocket. it seems like unless we add sysbox into bottlerocket build pipeline, I don't see a good way. We may need to install all packages on data devices, but any mutation on systemd's configuration is probably impossible.