x86_64-unknown-linux-musl with -Ctarget-feature=-crt-static links to glibc

[LunarLambda]

tested with rustc stable 1.83.0, on Arch Linux (kernel 6.6.69 LTS)

rustup install stable-x86_64-unknown-linux-gnu
rustup target add x86_64-unknown-linux-musl
cargo new musl-hm
cd musl-hm
cargo rustc --target x86_64-unknown-linux-musl -- -Ctarget-feature=-crt-static
ldd target/x86_64-unknown-linux-musl/debug/musl-hm
        linux-vdso.so.1 (0x00007ffc176a7000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x0000755e74d12000)
        libc.so.6 => /usr/lib/libc.so.6 (0x0000755e74b21000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x0000755e74dc4000)

The same thing happens with cargo build and RUSTFLAGS instead of cargo rustc.
This happens irrespective of whether a the musl system package is installed or not.

If you set target.x86_64-unknown-linux-musl.linker = "musl-gcc" then linking fails with -Ctarget-feature=-crt-static (complains about missing libgcc_s), and without -Ctarget-feature, the resulting binary crashes instantly (that's a duplicate of #95926), but is dynamically linked to the musl library and loader.

I think rustc should error out on musl targets if attempting to disable crt-static, since anything else produces wrong or broken binaries. If the idea is that -crt-static works on targets where musl is the system libc (e.g. Alpine), then perhaps there should be something that detects whether the system libc is musl, and errors (or ignores crt-static) if not.

[bjorn3]

(complains about missing libgcc_s)

You need to install libgcc_s for musl. Rust needs libgcc_s.so for the unwinder, but musl doesn't install it by default. When statically linking without musl-gcc, rustc's bundled copy of libgcc.a is used instead.

[LunarLambda]

musl oesn't use the system GCC libgcc_s, and doesn't use its own. There is no way for me to install one. Either way it's broken.

[bjorn3]

Musl itself doesn't use any unwinder. Programs that are compiled for musl however can use libgcc_s as unwinder if they want to. In that case you need to install a libgcc_s version compiled for musl. For example Alpine linux has the libgcc package for this: https://pkgs.alpinelinux.org/package/edge/main/x86_64/libgcc It seems like Archlinux doesn't have a package with a precompiled libgcc_s however. It doesn't have a package for any other unwinder that can be used either.

By the way I was wrong about rustc using libgcc by default on musl when statically linking. Libunwind is used instead. Archlinux doesn't have libunwind compiled for musl either however.

tl;dr: You will either have to statically link against rustc's bundled libunwind, you have to compile an unwinder for musl yourself or you have to compile on alpine with the libgcc package installed.

[LunarLambda]

Ultimately it doesn't matter as long as long as linking with musl-gcc is broken anyway (the missing libgcc_s only happens with musl-gcc as linker)

Still, I think either rustc should detect systems/configurations where -crt-static (without musl-gcc) is not supported instead of silently linking to the completely wrong libc, or it should be clearly documented that -crt-static is only supported on specific systems/configurations.

[workingjubilee]

musl-gcc?

[LunarLambda]

wrapper script around gcc shipped with musl that passes in a .specs file that sets up the musl libraries, interpreter and crt.

[workingjubilee]

@richfelker Okay, I know we already have busted compilation semantics for musl atm, but what exactly should I be doing here for the unwinder on musl? Should I just ship a Rust unwinding library for musl targets?

[bjorn3]

We already ship libunwind for musl when statically linking. When dynamically linking we can't safely use this however as there may only be a single unwinder in the whole program and one of the dynamic libraries the program depends on may already be linking against libgcc_s.so as unwinder, so we are forced to dynamically link against libgcc_s.so too as at least on Alpine linux that is the system unwinder that is used. Shipping our own libgcc_s.so copy would be an option, but that would only fix compilation. Not running it on Archlinux which should work if libgcc_s.so is part of LD_LIBRARY_PATH as the musl package also installs the correct dynamic linker and libc.so in the location where the executable would expect it. The correct solution I think would be for Archlinux to provide libgcc_s.so as part of their musl package or as separate libgcc-musl package.

[LunarLambda]

The correct solution I think would be for Archlinux to provide libgcc_s.so as part of their musl package or as separate libgcc-musl package.

I can take this to their bug tracker tomorrow, I think this is solvable on their side.

Where does that leave Rust? is musl-gcc the "correct" way to link to a dynamic musl & loader with -crt-static on systems where musl is not the system libc (and therefore -crt-static does not work out of the box)?

Also, as a side note, musl-gcc seems to want to pull in libgcc_eh if it exists, not libgcc_s. I'm not sure whether these are the same or at least compatible things.

[bjorn3]

I think libgcc_s.so is a dynamic library made by linking libgcc.a and libgcc_eh.a together. Statically linking libgcc_eh.a when dynamically linking musl may result in multiple unwinders in the same program, which is UB.

[richfelker]

I'm confused by this whole issue, especially text like:

I think rustc should error out on musl targets if attempting to disable crt-static, since anything else produces wrong or broken binaries.

A non-static musl target binary is only usable on a musl-based host or one where musl is installed alongside something else in /lib/ld-musl-$(ARCH).so.1. But in this context it should work fine regardless of how libgcc was linked, assuming the musl-based host/target-libs include a suitable libgcc.

is musl-gcc the "correct" way to link to a dynamic musl

No, musl-gcc is a wrapper for evaluation purposes and compiling/linking simple C-only programs on non-musl hosts. It's not a proper cross compiler and not the right way (not even present) on musl-based hosts. I think this whole issue is confusing 3 different scenarios:

1. Building native programs on a musl-based host.
2. Cross-compiling for a musl-based target, static or dynamic, from a non-musl-based host.
3. Building static binaries against musl to run on the non-musl-based host.

Case 3 seems to be what a lot of people are assuming when they hear "musl", but it shouldn't be considered exclusively at the expense of breaking the others.

Statically linking libgcc_eh.a when dynamically linking musl may result in multiple unwinders in the same program, which is UB.

This is not supposed to be the case. There is no good reason you can't have multiple unwinders in the same program, as long as they're all capable of processing the read-only unwind tables mapped at execution time. We do not intend to support the legacy "register eh frame" way of doing things; the unwinder is supposed to call dl_iterate_phdr to find the unwind tables statelessly without any registration. Only the legacy registration-based approach has issues with multiple instances.

If this is not working right, we should try to get it fixed.

In any case I don't see how it's relevant. If you're static linking musl, you have to static link everything. There is no such thing (in the context of musl) as a binary with static libc but dynamic linked to other things. If you try to make such a thing, it will always come out broken, and that seems to be what's going on in the original issue report here. If you dynamic link musl, you must have dynamic target libs and a dynamic musl execution environment (ld-musl in /lib, path config file in /etc if not the default, libgcc_s if you're running C++ binaries or other stuff with unwinding, etc.) on the machine you will run the binary on.

[bjorn3]

This is not supposed to be the case. There is no good reason you can't have multiple unwinders in the same program, as long as they're all capable of processing the read-only unwind tables mapped at execution time. We do not intend to support the legacy "register eh frame" way of doing things; the unwinder is supposed to call dl_iterate_phdr to find the unwind tables statelessly without any registration. Only the legacy registration-based approach has issues with multiple instances.

JIT's need to use __register_frame which only registers unwind tables with a single unwinder. Also AFAIK the unwinder state between different unwinder implementations is not ABI compatible, so unwinding from Rust code linked against one unwinder into C++ code linked against another unwinder would call _Unwind_Resume for the wrong unwinder and thus cause UB, right? Unwinding from Rust into C++ and vice versa is officially supported with the extern "C-unwind" ABI.