FEX: Allocate a VMA allocator when running on a 48-bit VA #4299
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sonicadvance1
force-pushed
the
fix_48bit_wine
branch
2 times, most recently
from
9ba206a to
8c349d4
Compare
|
Had to do a tertiary bug fix in our Allocator that we hadn't hit before. This never occurred on the 32-bit side where we reserve the upper 32-bits because the first VMA region would be hundreds of gigabytes. Since we're using the regions in the top 128TB on 48-bit systems now, due to how ASLR is laid out (roughly starts in the middle of the VA) there was a pretty high chance of the /first/ VMA region being small enough to hit the bug. |
Sonicadvance1
force-pushed
the
fix_48bit_wine
branch
2 times, most recently
from
668a702 to
fab8506
Compare
When running on a system with a 48-bit VA, if FEX does any allocations between us reserving the upper 128TB and the application running, then /technically/ we are intersecting with the application's memory region in the lower 47-bits. This didn't typically result in any problems due to how ASLR works, but if we did any large allocations (like FEX-Emu#4291 wants with 128MB VMA region) then these typically get pushed higher in the VA space. Again not usually a problem, but if you happen to be running an application that is using MAP_FIXED with hardcoded addresses then this can stomp over FEX-Emu memory causing problems. This is what happens with Wine, it reserves the upper-32MB of its 47-bit VA space, which is /highly/ likely to stomp on FEX memory. In-fact it likely occurs all the time, we just got lucky with whatever it was clobbering wasn't used at the time. On 39-bit VA systems this isn't a problem because the mmap fails outright with a warning message from WINE. Because we are already reserving the upper 128TB of VA space, instead just always enable our allocator and use the regions that were reserved. We need to be a little bit careful to ensure we don't accidentally allocate more memory post-reservation but that just requires a small adjustment to our unique_ptr and constructor for the 64BitAllocator. This means /all/ FEX-Emu allocations will be in the upper 128TB VA space when running 64-bit applications on a 48-bit VA system. Which is kind of nice. Fixes WINE in FEX-Emu#4291 when the allocator stats are bumped to 128MB per process.
Sonicadvance1
force-pushed
the
fix_48bit_wine
branch
2 times, most recently
from
942838e to
d2036f1
Compare
…gions When small regions were being used for VMA allocations (less than 64 pages), this function was truncating the result to zero. Resulting in incorrect `LiveVMARegion` size calculations. It would calculate that the FlexBitSet consumes zero bits of space, even though it needs to use at least 2, or 3 if we actually want to allocate anything from that LiveVMARegion. This was noticed in this PR because our VMA region tracking is being used more, which has a more likely chance to have small VMA regions for allocating from. Cause a 1page allocation to try and use a 3 page VMA region for allocation, but failing because the FlexBitSet size wasn't calculated correctly. - Page layout: - [0x0, 0x1000): struct LiveVMARegion - [0x1000, 0x2000): FlexBitSet<uint64_t> UsedPages - ^ This space wasn't allocated/mprotected due to the size not calculating correctly. - [0x2000, 0x3000): Memory for allocation
Anything less than three pages can't be used for FEX allocations due to VMA implementation details. Plus we may have reduced a single page reservation to zero with the prior ObjectAlloc size reservation.
To ensure correctness
Sonicadvance1
force-pushed
the
fix_48bit_wine
branch
from
d2036f1 to
a6674ed
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running on a system with a 48-bit VA, if FEX does any allocations between us reserving the upper 128TB and the application running, then /technically/ we are intersecting with the application's memory region in the lower 47-bits.
This didn't typically result in any problems due to how ASLR works, but if we did any large allocations (like #4291 wants with 128MB VMA region) then these typically get pushed higher in the VA space.
Again not usually a problem, but if you happen to be running an application that is using MAP_FIXED with hardcoded addresses then this can stomp over FEX-Emu memory causing problems.
This is what happens with Wine, it reserves the upper-32MB of its 47-bit VA space, which is /highly/ likely to stomp on FEX memory. In-fact it likely occurs all the time, we just got lucky with whatever it was clobbering wasn't used at the time.
On 39-bit VA systems this isn't a problem because the mmap fails outright with a warning message from WINE.
Because we are already reserving the upper 128TB of VA space, instead just always enable our allocator and use the regions that were reserved. We need to be a little bit careful to ensure we don't accidentally allocate more memory post-reservation but that just requires a small adjustment to our unique_ptr and constructor for the 64BitAllocator.
This means /all/ FEX-Emu allocations will be in the upper 128TB VA space when running 64-bit applications on a 48-bit VA system. Which is kind of nice.
Fixes WINE in #4291 when the allocator stats are bumped to 128MB per process.