Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multithread tests fail on master #2302

Closed
mhauru opened this issue Aug 13, 2024 · 6 comments
Closed

Multithread tests fail on master #2302

mhauru opened this issue Aug 13, 2024 · 6 comments

Comments

@mhauru
Copy link
Member

mhauru commented Aug 13, 2024

This happened recently:

We seem to have an unfortunate case here where a PR that passed tests (#2291) was merged, and then immediately after merging tests on master started to fail (https://github.com/TuringLang/Turing.jl/actions/runs/10281463329/job/28451089388). It's a concurrency problem of some kind, only appears on multithreaded. Haven't dug into it yet.

That's still the case on master. Looking into this now.

@devmotion
Copy link
Member

Was curious and checked the logs: The final culprit seems to be the write access in https://github.com/JuliaLang/julia/blob/96fd25a764c316e9e09dfd7de8e84d854a73b2cd/base/logging/ConsoleLogger.jl#L112 but AFAICT it is only triggered by the deprecation warning of ADTypes triggered in LogDensityProblemsAD.

@devmotion
Copy link
Member

tpapp/LogDensityProblemsAD.jl#34 should fix the deprecation warning

@mhauru
Copy link
Member Author

mhauru commented Aug 13, 2024

Thanks @devmotion! I was looking at the ConsoleLogger bit as well, and I find it alarming that calling depwarn (or probably other logging calls would do as well) can cause a concurrency error. I just posted on Julia's Slack to see if anyone can tell me more about thread safety of logging.

@devmotion
Copy link
Member

It seems fixing the ADTypes deprecations resolved the test error: https://github.com/TuringLang/Turing.jl/actions/runs/10322735917/job/28712240978

@mhauru
Copy link
Member Author

mhauru commented Aug 13, 2024

Great, thanks!

@mhauru mhauru closed this as completed Aug 13, 2024
@mhauru
Copy link
Member Author

mhauru commented Aug 14, 2024

The root cause is this long standing Julia issue: JuliaLang/julia#34600

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants