Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DllMain is never run when given stdcall decorated export #135188

Open
PARTYMANX opened this issue Jan 7, 2025 · 0 comments
Open

DllMain is never run when given stdcall decorated export #135188

PARTYMANX opened this issue Jan 7, 2025 · 0 comments
Labels
C-bug Category: This is a bug. needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. O-windows Operating system: Windows

Comments

@PARTYMANX
Copy link

PARTYMANX commented Jan 7, 2025

I'm creating a Windows DLL in an admittedly goofy fashion: I have an executable that links a DLL and imports DllMain, using stdcall decorations (so in reality it's looking for _DllMain@12). This causes the DllMain function to not actually run when the DLL is linked. I made a dummy DllMain function that just returns false to prevent the linking program from running, making it obvious that my code is running, but the program is starting successfully. It's worth noting that I'm running into this when testing out replacing a C DLL with a new, Rust-based DLL. My C version natively exports _DllMain@12

Using the common wisdom of using #[no_mangle] on DllMain, it is failing to link with my executable because it does not export _DllMain@12. When modifying the executable to import DllMain, my own DllMain function runs, causing the program to fail to start (as intended, in this test case).

When testing with a program that instead uses LoadLibrary(), my code will run in both cases, indicating that there's different behavior for a DLL being linked and loaded programmatically. I'm not an expert on how DLLs are loaded, but it seems like there's an entrypoint function that is run when the DLL is linked, and the entrypoint will only run DllMain when its symbol is specifically DllMain instead of the also valid _DllMain@12.

I tried this code:

#[unsafe(export_name = "_DllMain@12")]
pub extern "stdcall" fn DllMain(_hinst_dll: usize, _fdw_reason: u32, _lp_reserved: usize) -> i32 {
    0
}

(crate-type is set to cdylib)

I expected to see this happen: When the DLL is linked, the program fails due to DllMain returning false

Instead, this happened: The DLL links successfully, with DllMain seemingly returning true

Meta

rustc --version --verbose:

$ rustc --version --verbose
rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: x86_64-pc-windows-msvc
release: 1.83.0
LLVM version: 19.1.1
@PARTYMANX PARTYMANX added the C-bug Category: This is a bug. label Jan 7, 2025
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Jan 7, 2025
@bjorn3 bjorn3 added the O-windows Operating system: Windows label Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug. needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. O-windows Operating system: Windows
Projects
None yet
Development

No branches or pull requests

3 participants