Check for Windows NLS vs. ICU behavior differences in Globalization tests

[lbussell]

Nano Server 2025 does not include ICU by default.

When ICU is missing on Linux, Globalization Invariant Mode is disabled, and you try to use globalization features, the .NET Runtime throws an exception.

In the same scenario on Windows, instead of throwing an error, the Runtime falls back to using Windows' NLS (National Language Support) feature. This has some breaking changes which are documented here: https://learn.microsoft.com/en-us/dotnet/core/extensions/globalization-icu

We should check for those breaking changes in the recently added globalization test scenario.

---

Related Issues: #5944, #5635

The following table summarizes my best understanding of the .NET Runtime's behavior under different globalization scenarios:

OS	ICU Present	Invariant Mode	Behavior when using globalization features
Linux	✅	✅	Exception (expected)
Linux	✅	❌	Works as expected
Linux	❌	✅	Exception (expected)
Linux	❌	❌	Exception - I consider this scenario a misconfiguration
---	---	---	---
Windows	✅	✅	Exception (expected)
Windows	✅	❌	Works as expected
Windows	❌	✅	Exception (expected)
Windows	❌	❌	Falls back to NLS behavior!

[ericstj]

@tarekgh to weigh in,

This is intentional. The windows build of .NET works on older OSes where ICU is absent and our customers expect it to use NLS.

I guess you're saying you'd prefer that we would have made it more breaking by not falling back to NLS on an OS version where we expect ICU to be present, unless the customer specifically told us to use NLS with the switch?

[MichaelSimons]

I guess you're saying you'd prefer that we would have made it more breaking by not falling back to NLS on an OS version where we expect NLS to be present, unless the customer specifically told us to use NLS with the switch?

Can you clarify if this is even a supported/tested scenario? e.g. NLS fallback on an modern OS like nanoserver 2025 where the globalization FOD is not installed by default?

[tarekgh]

Comments on the table above:

• Behavior when using globalization features: This title needs some clarification, particularly when Globalization Invariant mode is enabled.

  • When using the Invariant culture and the Invariant mode is on, most scenarios will not throw exceptions but will exhibit specific behaviors. For instance, string comparisons will be ordinal rather than linguistic. Additional details about Invariant mode can be found in the documentation.
  • When other cultures are used when Invariant mode is on, an exception is typically expected by default. However, this behavior can be adjusted with a configuration switch that allows the creation of any culture, though all created cultures will still behave like the Invariant culture.

• On Windows systems: Even when ICU is available and Invariant mode is turned off, users can switch to NLS by setting the appropriate configuration switch.

• App-local ICU: Apps have the option to use ICU app-local, enabling them to manage globalization independently from the OS by carrying their own version of ICU.

The only scenario that is not supported which you called it misconfiguration is when ICU is missing on non-Windows platforms and the Invariant mode is not enabled.

Can you clarify if this is even a supported/tested scenario? e.g. NLS fallback on an modern OS like nanoserver 2025 where the globalization FOD is not installed by default?

.NET ensures that applications follow OS behavior. So, if ICU is available on Windows, the app will use ICU by default. If ICU is not present, the app will revert to the default globalization behavior, similar to native applications.

[tarekgh]

Some clarification, on nanoserver 2025 I didn't check if NLS APIs will still be available to call and only trimming the data. I'll try to find out that.

[lbussell]

@ericstj I think you meant:

I guess you're saying you'd prefer that we would have made it more breaking by not falling back to NLS on an OS version where we expect ICU to be present, unless the customer specifically told us to use NLS with the switch?

If that's the case, then yes that's the question we're asking. If ICU is the forward-looking globalization approach for Windows, then we are wondering if we should reconsider the Runtime's behavior for this scenario (for new Windows versions). I think all other Windows versions since ICU was added all have ICU. Except now NanoServer 2025.

We are inclined to turn globalization invariant mode on by default for this image, so that we avoid users sliently encountering breaking changes when migrating from NanoServer 2022 to 2025 images and not realizing they're using NLS. Then we would provide instructions for installing ICU and enabling globalization.

[richlander]

Let's assume we ship images w/o ICU installed and w/o GIVM enabled. Do we need to warn or just inform users about this choice? That's the key question.

[tarekgh]

Do we have the option to install ICU on such image?

[richlander]

Yes! That's one of the options and is my preference. We need to find out the size impact for that.

It would be good to get an answer to the question I asked. Some of this thread is operating on the premise that NLS behavior is different/worse and that we'd need to warn people about that. I want to determine if that is true.

[tarekgh]

    Directory: C:\windows\System32

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-ar--           9/30/2024  9:51 PM        2765784 icu.dll
-ar--           9/30/2024  9:51 PM          32768 icuin.dll
-ar--           9/30/2024  9:51 PM          36864 icuuc.dll

[tarekgh]

Do we need to warn or just inform users about this choice? That's the key question.

I think it will be good to inform/warn about that. The reason is there are some features that are not going to work at all when switching back to NLS. For example, creating time zone objects using IANA names will not work.

[richlander]

I meant "warn" and "inform" as two different options. I hear you saying "warn", which is what I was looking for.

OK. This leads me to believe that we should not ship images with NLS as the default. We need to decide if we're going to switch to shipping nanoserver images with the -extra plan like we do for Linux or ship ICU and be done. I'm still more compelling by the latter.

A good option is to ship with ICU in-box for .NET 8/9 and then revisit for .NET 10 if we have any feedback that motivates a different choice.

Compressed size is the key aspect, but that size looks very familiar w/rt Linux, so I assume it will add about 12MB compressed.

[tarekgh]

I agree with @richlander. If there is no strong reason prevent getting ICU installed on that image.

Compressed size is the key aspect, but that size looks very familiar w/rt Linux, so I assume it will add about 12MB compressed.

ICU libraries on my machine look like they are under 3MB. What am I missing? maybe I missed the data file which should be big. Somehow this file is not showing next to the ICU libraries.

Another option (I am not necessarily preferring) is to make the image behave as Linux. i.e, setting the environment variable DOTNET_SYSTEM_GLOBALIZATION_USENLS = 0 which will disable NLS but the apps will not start at all at that time.

[lbussell]

Let me get back to you on the compressed size difference.

[tarekgh]

 Directory of C:\Windows\Globalization\ICU

03/31/2024  11:26 PM    <DIR>          .
04/01/2024  12:03 AM    <DIR>          ..
03/31/2024  11:22 PM        31,079,968 icudtl.dat
03/31/2024  11:22 PM            43,200 metaZones.res
03/31/2024  11:22 PM            20,176 timezoneTypes.res
03/31/2024  11:22 PM            22,592 windowsZones.res
03/31/2024  11:22 PM           148,352 zoneinfo64.res

    Directory: C:\windows\System32

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-ar--           9/30/2024  9:51 PM        2765784 icu.dll
-ar--           9/30/2024  9:51 PM          32768 icuin.dll
-ar--           9/30/2024  9:51 PM          36864 icuuc.dll

[lbussell]

The Nano Server Globalization FOD is not available yet. The expected availability date is TBD. However I did build and calculate sizes for some .NET Runtime 9.0.0 images today for comparison:

Image	.NET Runtime Version	Size (on disk)	Size (compressed)
Nano Server 2022	9.0.0	372 MB	149 MB
Nano Server 2025	9.0.0	565 MB	225 MB

To get compressed size, I built and pushed the images to a test ACR. ACR does not show compressed size in the Azure UI, but I computed it by summing the size of the individual compressed layers like so:

PS> [math]::Round($(oras manifest fetch dotnetdockerdev.azurecr.io/dotnet/runtime/9.0:nanoserver-ltsc2025-amd64 | jq '.layers[].size' | Measure-Object -Sum).Sum / 1MB, 2)
224.63

[tarekgh]

@lbussell Is the Nano-Server 2025 image size calculated without including ICU? I'm trying to determine how much the ICU library contributes to the compressed image size.

[lbussell]

@tarekgh Nano Server 2025 does not contain an icu.dll that I was able to find. However, after inspecting the Nano Server 2022 image closer myself, I also wasn't able to find any ICU files there either. This is really violating my assumptions... Here is what I observed:

File	Nano Server 2022	Nano Server 2025	Server Core 2022	Server Core 2025
Windows.Globalization.dll	Yes	No	Has .winmd file	Has .winmd file
BCP47*	Yes	No	Yes	Yes
icu.dll, icuin.dll, icuuc.dll, icudtl.dat	No	No	Yes	Yes

I used images mcr.microsoft.com/windows/(nanoserver|servercore):ltsc(2022|2025)-amd64 and searched for files using dir /s.

[lbussell]

But ultimately yes @tarekgh, the above table was calculated without adding ICU and with adding the .NET Runtime. The Dockerfiles I used to build them are in my draft PR for adding the Server 2025 images.

[tarekgh]

Interesting, this means nothing changed between nano-server 2022 and 2025 regarding globalization. So, why are we worried about this scenario? Users of 2022 are already using NLS and if they upgrade to 2025 will continue using NLS if we don't do anything. Right?

[richlander]

So, why are we worried about this scenario?

We started the discussion with an incorrect set of assumptions.

We would like your team to do a test pass on Nano Server 2025 to validate that there have been no changes of behavior or regressions that we need to address or document.

/cc @ericstj

[ericstj]

@tarekgh -- do you think we have some tests from dotnet/runtime that we can run specifically on these SKUs and diff the results? Probably they aren't part of our regular test matrix so we wouldn't know if we have failures or not.

[tarekgh]

I am talking to Windows team to know exactly what the globalization differences between nano-server is 2022 and 2025. This is simpler than trying to figure it out ourselves.

[richlander]

Reading a spec and validating an implementation are two different activities.

[ericstj]

@richlander is the ask here then to more generally validate the entire runtime on Nano Server 2025? I've lost track of the next steps since this started as a specific globalization issue, but now it seems broader. I'd like to make sure we're on the same page and not repeating efforts others are already doing, or putting in more resources to this specific case than we do for other platforms.

The best way to scout these things is if we have a helix queue to add work to - we can then schedule a build sending work to that queue.

[lbussell]

@richlander is the ask here then to more generally validate the entire runtime on Nano Server 2025? I've lost track of the next steps since this started as a specific globalization issue, but now it seems broader. I'd like to make sure we're on the same page and not repeating efforts others are already doing, or putting in more resources to this specific case than we do for other platforms.

The best way to scout these things is if we have a helix queue to add work to - we can then schedule a build sending work to that queue.

This will be blocked on Add Windows 2025 support to Helix (dotnet/dnceng#4170).

We can produce Nano Server 2025 Helix container images from buildtools-prereqs, but it won't matter if we don't have a Server Core 2025 queue to run them on, if I understand correctly how Helix works.

/cc @ilyas1974

[ericstj]

How are folks testing on Server Core 2025 right now? Just building locally and running tests? VMs? Any instructions that could be shared - maybe we can get someone to scout the library tests.

[lbussell]

We are running the .NET Docker tests which aren't very comprehensive. I don't know any other teams that are running tests on 2025. There is one 2025 ARM queue for some reason. But no AMD64: dotnet/dnceng#3343.

I proposed in my PR that we ship the nightly .NET images now to better facilitate testing while we wait on Helix queues. These images are functional, but we just want to understand the nuances of the globalization changes better. #5944

[tarekgh]

but we just want to understand the nuances of the globalization changes better.

I already talked to Windows team and the behavior should be clear now. nano-server 2025 will work as nano-server-2022 which will use NLS. Both are not installing ICU till now. Only changes can be noticed when Windows updates the locale data, which usually happens almost with every release. Let me know if you have any more questions regarding globalization I can help with.

[richlander]

Thanks. I think that is good enough. We have been so used to ICU at this point, that an NLS scenario seemed unique and different. Seems like we should move forward treating this as just another version upgrade.

Thanks for reaching out to the Windows team and helping us better understand the situation.

[lbussell]

@tarekgh, so the presence or absence of Windows.Globalization.dll should not affect .NET's behavior?

[tarekgh]

so the presence or absence of Windows.Globalization.dll should not affect .NET's behavior?

That is correct, .NET don't depend on this library. This library is used by WinRT which we are not calling for globalization.

[lbussell]

.NET images for Server 2025 are now published from our nightly branch (not supported for production yet). Can @tarekgh or @ericstj do any testing with these images to validate @tarekgh's claims here?

• mcr.microsoft.com/dotnet/nightly/runtime:9.0-nanoserver-ltsc2025
• mcr.microsoft.com/dotnet/nightly/aspnet:9.0-nanoserver-ltsc2025
• mcr.microsoft.com/dotnet/nightly/sdk:9.0-nanoserver-ltsc2025

[tarekgh]

@lbussell I smoke tested the SDK image and I can confirm NLS is used successfully there and behave as expected.

[richlander]

Thanks much!

[lbussell]

Tests were added in #6120.