exec /opt/_slim/bin/slim-sensor: no such file or directory

[MartinLoeper]

Expected Behavior

Successful slim build command execution.
I am trying to build an alpine image.

---

Actual Behavior

Error: exec /opt/_slim/bin/slim-sensor: no such file or directory

cmd=build info=image.users all='node' exe='node'
cmd=build info=image.stack index='0' name='' id='sha256:ec8e87cebbfae6d326ac8fc0fa2094a97be254dc2b52c85b6b145980ee8b04a3'
cmd=build info=image.exposed_ports list='8080/tcp'
cmd=build state=image.inspection.done
cmd=build state=container.inspection.start
time="2025-01-03T02:11:35+01:00" level=debug msg="FatContainerCmd - [node apps/server/bin/server.js]" app=slim cmd=build component=container.inspector
time="2025-01-03T02:11:35+01:00" level=info msg="starting instrumented 'fat' container..." app=slim cmd=build
time="2025-01-03T02:11:35+01:00" level=debug msg="sensor.EnsureLocalBinary: sensor (/nix/store/wznxq2wq23aw1bp5fq5f937ng51fxfi6-docker-slim-1.40.11/bin/slim-sensor) perms => 0555" app=slim cmd=build component=container.inspector
time="2025-01-03T02:11:35+01:00" level=debug msg="ensureSensorVolume: already have volume = slim-sensor.1.40.11" app=slim cmd=build component=container.inspector
time="2025-01-03T02:11:35+01:00" level=debug msg="RunContainer: default exposed ports => map[docker.Port]struct {}{\"65501/tcp\":struct {}{}, \"65502/tcp\":struct {}{}}" app=slim cmd=build component=container.inspector
time="2025-01-03T02:11:35+01:00" level=debug msg="RunContainer: HostConfig.PublishAllPorts => true" app=slim cmd=build component=container.inspector
cmd=build info=container status='created' name='slimk_2887647_20250103011135' id='0298b30d25986ade64f44445c2b3775a2f650a936707910eccabcd0d13236aad'
cmd=build info=container status='crashed' id='0298b30d25986ade64f44445c2b3775a2f650a936707910eccabcd0d13236aad'
time="2025-01-03T02:11:35+01:00" level=debug msg="getting container logs => 0298b30d25986ade64f44445c2b3775a2f650a936707910eccabcd0d13236aad" app=slim cmd=build component=container.inspector
time="2025-01-03T02:11:35+01:00" level=debug msg="container HostConfig.NetworkMode => bridge len(ports)=0" app=slim cmd=build component=container.inspector op=container.Inspector.RunContainer
time="2025-01-03T02:11:35+01:00" level=debug msg="getting container logs => 0298b30d25986ade64f44445c2b3775a2f650a936707910eccabcd0d13236aad" app=slim cmd=build component=container.inspector
slim: container stdout:
slim: container stderr:
exec /opt/_slim/bin/slim-sensor: no such file or directory
slim: end of container logs =============
slim: container stdout:
slim: container stderr:
exec /opt/_slim/bin/slim-sensor: no such file or directory
slim: end of container logs =============
cmd=build state=exited code=-123 version=linux/amd64|Transformer|1.40.11|1.40.11|latest
time="2025-01-03T02:11:35+01:00" level=info msg="can't stop the slim container (container is not running)..." app=slim cmd=build component=container.inspector op=container.Inspector.ShutdownContainer

---

Specifications

• Version: slim version linux/amd64|Transformer|1.40.11|1.40.11|latest
• Platform: NixOS 24.11

[kcq]

Thank you for opening the issue @MartinLoeper !

TLDR on the error (exec /opt/_slim/bin/slim-sensor: no such file or directory): The tool has two binaries where the sensor binary is responsible for collecting telemetry in a temporary container that gets created and executed. This sensor is added to the temporary container using a Docker volume. This error you are getting means the sensor binary wasn't successfully copied to the Docker volume and when the temporary container started running it failed because the sensor binary isn't there (which caused the no such file or directory error).

Quick fix to try: try removing the sensor volume, so the main app tries to recreate the Docker volume the next time you want to slim a container image. If you do docker volume ls you'll see one or more volumes prefixed with slim-sensor.. Delete those. Note that the latest version of the tool has a built-in command to clean up the volumes ( mint app remove-sensor-volumes). Make sure you are using the latest version/release.

Do you mind sharing a bit about your setup. How did you install the tool in your Nix environment?

The Docker volume creation logic might have a problem if the sensor binary ends up being located where it can't find it, which might explain why the Docker volume is missing slim-sensor. Is there any easy way to repro your environment or, at least, enough of it for the SlimToolkit installation? The log message shows that you have the sensor binary as /nix/store/wznxq2wq23aw1bp5fq5f937ng51fxfi6-docker-slim-1.40.11/bin/slim-sensor... Where is the main binary located? is it /nix/store/wznxq2wq23aw1bp5fq5f937ng51fxfi6-docker-slim-1.40.11/bin/slim?

[MartinLoeper]

Hi @kcq. Thanks for your detailed explanation!

I have a standard nix setup and install docker-sim via the official nixos package:
https://github.com/NixOS/nixpkgs/blob/nixos-24.11/pkgs/by-name/do/docker-slim/package.nix#L46

The slim and slim-sensor binaries are located in the same nix store path under the bin folder.
I tried to recreate the volume as you suggested but it did not work. I also tried to mount the slim-sensor binary manually but could not make it work.

Can you help me bring the slim-sensor binary into the container mount? What is the program argument to do that?

[kcq]

Thanks for the additional context @MartinLoeper ! Can you also enable trace logging to see if we can catch any extra info when the Docker volume create logic for the sensor runs. This can be done with the global --log-level flag, so your command would look like this (run it with trace logging after you remove the existing sensor volume): slim --log-level=trace build the_rest_of_flags_here.

Can you also try the latest/mint version. There was a symlink related problem a while back, which might be relevant here... because Nix does a lot of its magic with symlinks :-)

How is Docker installed/configured in your Nix environment? Do you have a special user group for it? Do you need to run the Docker commands with 'sudo'?

[MartinLoeper]

Thanks for the instructions @kcq!
I changed the nixos package and built the docker-slim master. Running the latest binary with trace log-level yields the same error:

time="2025-01-03T21:30:14+01:00" level=debug msg="FatContainerCmd - [node apps/server/bin/server.js]" app=slim cmd=build component=container.inspector
time="2025-01-03T21:30:14+01:00" level=info msg="starting instrumented 'fat' container..." app=slim cmd=build
time="2025-01-03T21:30:14+01:00" level=debug msg="sensor.EnsureLocalBinary: sensor (/nix/store/xqlj584x2m27aj0xgm64gjhll5wwp0jf-docker-slim-master/bin/slim-sensor) perms => 0555" app=slim cmd=build component=container.inspector
time="2025-01-03T21:30:14+01:00" level=debug msg="dockerutil.HasVolume: volume not found - slim-sensor.master"
time="2025-01-03T21:30:14+01:00" level=debug msg="ensureSensorVolume: no volume yet = slim-sensor.master" app=slim cmd=build component=container.inspector
time="2025-01-03T21:30:14+01:00" level=debug msg="dockerutil.CreateVolumeWithData: volumeInfo = &{Name:slim-sensor.master Driver:local Mountpoint:/var/lib/docker/volumes/slim-sensor.master/_data Labels:map[] Options:map[] CreatedAt:2025-01-03 21:30:14 +0100 CET}"
time="2025-01-03T21:30:14+01:00" level=debug msg="dockerutil.CopyToVolume: containerID - 93f9a7f328af5432c82c0907ccea6ae27a5b8b7c525ebba681ed65209876c845"
time="2025-01-03T21:30:14+01:00" level=debug msg="RunContainer: default exposed ports => map[docker.Port]struct {}{\"65501/tcp\":struct {}{}, \"65502/tcp\":struct {}{}}" app=slim cmd=build component=container.inspector
time="2025-01-03T21:30:14+01:00" level=debug msg="RunContainer: HostConfig.PublishAllPorts => true" app=slim cmd=build component=container.inspector
cmd=build info=container status='created' name='slimk_219152_20250103203014' id='3adb78c9a0b6ca8c11e7c34d2ce025d46499dac03eadae686b7ff7a228630711' 
cmd=build info=container status='crashed' id='3adb78c9a0b6ca8c11e7c34d2ce025d46499dac03eadae686b7ff7a228630711' exit.code='1' 
time="2025-01-03T21:30:15+01:00" level=debug msg="getting container logs => 3adb78c9a0b6ca8c11e7c34d2ce025d46499dac03eadae686b7ff7a228630711" app=slim cmd=build component=container.inspector
time="2025-01-03T21:30:15+01:00" level=debug msg="container HostConfig.NetworkMode => bridge len(ports)=0" app=slim cmd=build component=container.inspector op=container.Inspector.RunContainer
time="2025-01-03T21:30:15+01:00" level=debug msg="getting container logs => 3adb78c9a0b6ca8c11e7c34d2ce025d46499dac03eadae686b7ff7a228630711" app=slim cmd=build component=container.inspector
slim: container stdout:
slim: container stdout:
slim: container stderr:
exec /opt/_slim/bin/slim-sensor: no such file or directory
slim: end of container logs =============
slim: container stderr:
exec /opt/_slim/bin/slim-sensor: no such file or directory
slim: end of container logs =============
cmd=build state=exited code=-999 sensor.filemode=-r-xr-xr-x sensor.volume=slim-sensor.master version=linux/amd64|Transformer|master|master|latest location.exe=/nix/store/xqlj584x2m27aj0xgm64gjhll5wwp0jf-docker-slim-master/bin location.sensor=/nix/store/xqlj584x2m27aj0xgm64gjhll5wwp0jf-docker-slim-master/bin/slim-sensor 
time="2025-01-03T21:30:15+01:00" level=info msg="can't stop the slim container (container is not running)..." app=slim cmd=build component=container.inspector op=container.Inspector.ShutdownContainer

Is there something in the error message to proceed debugging. Thanks again :)

[kcq]

@MartinLoeper the trace output confirms that the volume creation logic is ok. That's good to eliminate this as the source of failure. We also have the volume name here (slim-sensor.master), which is a bit different from it's usual name because it usually includes the release version instead of a branch name (e.g., slim-sensor.1.40.11).

Let's list files in the Docker volume next to see what's inside (assuming the volume name is slim-sensor.master): docker run -it --rm -v slim-sensor.master:/sensorvol alpine ls -lh /sensorvol

The goal is to see if there are any files there and if the sensor file is there the goal is to see the file size, which should indicate if we have a symlink or an actual file.

[MartinLoeper]

 docker run -it --rm -v slim-sensor.master:/sensorvol alpine ls -lh /sensorvol
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
38a8310d387e: Already exists
Digest: sha256:21dc6063fd678b478f57c0e13f47560d0ea4eeba26dfc947b2a4f81f686b9f45
Status: Downloaded newer image for alpine:latest
total 5M
-r-xr-xr-x    1 root     root        5.2M Jan  1  1970 slim-sensor

=)

[kcq]

 docker run -it --rm -v slim-sensor.master:/sensorvol alpine ls -lh /sensorvol
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
38a8310d387e: Already exists
Digest: sha256:21dc6063fd678b478f57c0e13f47560d0ea4eeba26dfc947b2a4f81f686b9f45
Status: Downloaded newer image for alpine:latest
total 5M
-r-xr-xr-x    1 root     root        5.2M Jan  1  1970 slim-sensor

=)

That's interesting... 5MB sensor means that it's definitely not a symlink :)

[kcq]

@MartinLoeper the next thing we can try is the --use-sensor-volume build command flag. This is the flag to use to specify the Docker volume to use as the sensor volume. This way you can control what's used. You can start by pointing at slim-sensor.master first. Then you can try creating your own volume and copying the sensor binary to it.

[MartinLoeper]

Hi @kcq, Thanks again for your help! Unfortunately I cannot make the --use-sensor-volume flag work as it says it cannot find such a flag.

I noticed another thing though: I inspected the slim-sensor binary using readelf. It shows a program interpreter of /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/ld-linux-x86-64.so.2 and a library runpath of /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib.

I do not understand how this is supposed to work and how the official NixOS package ever worked. I think will have to go and open an upstream issue. Since nixos does not adhere to Linux FSH, I cannot imagine how copying a nixos executable into a non-nixos docker image is supposed to work. What do you think @kcq?

It looks like the binary is there in the docker container, it is also executed but the loader fails as it is pointing at a path on the host machine which is not available in the container. Maybe I could patch the loader manually though to make it work until it is fixed upstream...

[kcq]

@MartinLoeper are you passing the --use-sensor-volume as a global flag or as a build command flag (after the build command keyword on the command line)? The CLI library used in the app is very strict with where the flags are specified. Either way, I recommend using the latest release / mint code base otherwise you'll be missing the imagebuild command completely and most of the debug command as well as numerous build and xray command enhancements.

Sounds like you are onto something... It appears that the Nix installer nixified the sensor binary and it broke it because the sensor needs to run in a regular non-nix environment.

A workaround for it would be downloading the sensor binary yourself and creating a Docker volume yourself with that clean non-nixified binary and then naming the volume what the slim app expects its to be.

[MartinLoeper]

Either way, I recommend using the latest release / mint code base otherwise you'll be missing the imagebuild command completely and most of the debug command as well as numerous build and xray command enhancements.

Wait so mint is the successor of this repository or how are they connected?

Sounds like you are onto something... It appears that the Nix installer nixified the sensor binary and it broke it because the sensor needs to run in a regular non-nix environment.

I'll ask the maintainers and link to this issue upstream.

A workaround for it would be downloading the sensor binary yourself and creating a Docker volume yourself with that clean non-nixified binary and then naming the volume what the slim app expects its to be.

Will do that!

[MartinLoeper]

@kcq I managed to copy one of the latest gh releases into the volume manually and it worked.

I think a solution would be to statically compile the slim-sensor binary. Was that the case in the past? I asked on nixpkgs upstream how the nix package ever worked. One answer could be that previous versions were statically linked.

Could you give me a hint how to enable static linking? Is it just CGO_ENABLED=0 stuff or is there something more to it?

[kcq]

@kcq I managed to copy one of the latest gh releases into the volume manually and it worked.

I think a solution would be to statically compile the slim-sensor binary. Was that the case in the past? I asked on nixpkgs upstream how the nix package ever worked. One answer could be that previous versions were statically linked.

Could you give me a hint how to enable static linking? Is it just CGO_ENABLED=0 stuff or is there something more to it?

Thank you for putting in the extra effort @MartinLoeper ! Great to hear that the solution worked!

The sensor in the official builds has always been statically compiled (e.g., the file util output for the sensor binary: /usr/local/bin/mint-sensor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=t-7c9rL2mugwO4TG2Bw6/fSd7eW6vqNQa0hW4jH1a/PvcrANnkZTvk7tbEttNC/F0DBfbfByB0xQZGruBKZ, stripped and the ldd output is not a dynamic executable ). The docker-slim/package.nix config doesn't appear to have CGO_ENABLED=0 in it, so it's a good idea to add it there.

I've been thinking about embedding the sensor binary in the main app. The overall UX will be nicer too and the Nix problem would be avoided. There's been a number of support requests that were the result of the sensor not being in the right place, the sensor binary being outdated and the sensor volume creation problems (due to symlinks and other gotchas). The build process becomes messier and more complicated though (but still doable :-)).

The other option I've been considering is publishing pre-created volumes to a container registry either as non-image artifacts or as images that have only sensors in them. In a way, the second option is sort of there because for each release there's a containerized version published to DockerHub and the sensor binaries can be extracted from those container images.

[MartinLoeper]

Thank you for putting in the extra effort @MartinLoeper ! Great to hear that the solution worked!

I have to thank you for pointing me into the right direction!

The sensor in the official builds has always been statically compiled.

That is interesting! It is apparently not the case for the NixOS binary, see:

/nix/store/wznxq2wq23aw1bp5fq5f937ng51fxfi6-docker-slim-1.40.11/bin/slim: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/ld-linux-x86-64.so.2, for GNU/Linux 3.10.0, not stripped

I am going the create a PR to fix that!

The build process becomes messier and more complicated though.

It is good to see that you have ideas to make the whole usage experience smoother! I cannot comment on that since I have no idea how you could go about embedding the sensor binary. :D

The other option I've been considering is publishing pre-created volumes to a container registry.

I did not know that is possible either. Definitely keen to see this setup. =)

[MartinLoeper]

Upstream NixOS PR: NixOS/nixpkgs#373029

I gonna close this issue soon when I get approval for the PR and they confirm that my change works.

[kcq]

Upstream NixOS PR: NixOS/nixpkgs#373029

I gonna close this issue soon when I get approval for the PR and they confirm that my change works.

This is great! Thank you for doing the PR there!

I'd also recommend updating the repo it's pointing to so it picks up the latest/newest enhancements (quite a few of those :-))

[MartinLoeper]

Good point. Going to open a separate PR for the version bump.
Has something changed with regards to the build process or build flags?

[kcq]

Good point. Going to open a separate PR for the version bump.
Has something changed with regards to the build process or build flags?

Thank you for doing that! No changes with the flags.

[MartinLoeper]

merged upstream

[MartinLoeper]

@kcq You mentioned the new mint codebase. Should I update the code references for the NixOS package to use the mint repository instead of this one? Is there something with regards to naming that must be adjusted or is it effectively just the repository location that changed?

I need your statement here to justify my NixOS Pull Request if I gonne change the whole repository URL instead of only the tag.

[MartinLoeper]

@kcq I just saw that the whole binary name changed to mint. That is probably an issue since the package is not backward compatible due to that. I also cannot create a new package called mint since there is already an existing one.