Skip to content

CultureNotFoundException thrown when a keyboard with language tag und is activated #98543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mcdurdin opened this issue Feb 16, 2024 · 7 comments · Fixed by #115166
Closed

CultureNotFoundException thrown when a keyboard with language tag und is activated #98543

mcdurdin opened this issue Feb 16, 2024 · 7 comments · Fixed by #115166
Labels
area-System.Globalization in-pr There is an active PR which will close this issue when it is merged
Milestone
10.0.0

Comments

@mcdurdin
Copy link

mcdurdin commented Feb 16, 2024 •
edited
Loading

Description

If a user installs and activates an input method with the custom locale und-Latn, a .NET application will instantly crash with CultureNotFoundException thrown ultimately by System.Globalization.CultureData.GetCultureData(Int32, Boolean).

Reproduction Steps

  1. Install the custom language und-Latn using e.g. PowerShell: 
    $1 = Get-WinUserLanguageList
$1.Add("und-Latn")
Set-WinUserLanguageList $1 -Force
  2. Create a new WPF Application in Visual Studio from the New Project Wizard, targeting .NET 7.0 or later.
  3. Add a TextBox to MainWindow.xaml: 
    <TextBox x:Name="textBox" HorizontalAlignment="Left" Margin="138,123,0,0" TextWrapping="Wrap" Text="TextBox" VerticalAlignment="Top" Width="120"/>
  4. Run the application, focus the text field.
  5. Select the UND language from the Windows language picker.
  6. The app will instantly crash.

Expected behavior

The language should be selected and the app should not crash.

Actual behavior

The application crashes with the exception CultureNotFoundException

Call stack from a crash dump given me by a user experiencing this problem:

0:000> !CLRStack -a
OS Thread Id: 0x7f88 (0)
        Child SP               IP Call Site
000000FD163ACBF8 00007ffc2c36fec4 [HelperMethodFrame: 000000fd163acbf8] 
000000FD163ACCF0 00007ffb62b5ec5b System.Globalization.CultureData.GetCultureData(Int32, Boolean)
    PARAMETERS:
000000FD163ACD30 00007ffb62b64f41 System.Globalization.CultureInfo..ctor(Int32, Boolean)
    PARAMETERS:
        this (<CLR reg>) = 0x000002ab8e7dda08
000000FD163ACD70 00007ffb0ee515f1 System.Windows.Input.InputLanguageSource.get_CurrentInputLanguage()
000000FD163ACDB0 00007ffb0ee51538 System.Windows.Input.InputLanguageManager.get_CurrentInputLanguage()
000000FD163ACE00 00007ffb0ee50fb2 System.Windows.Documents.TextSelection.EnsureCaret()
000000FD163ACE50 00007ffb100bf2d8 System.Windows.Documents.TextSelection.System.Windows.Documents.ITextSelection.UpdateCaretAndHighlight()
000000FD163ACEB0 00007ffb0ee46fe9 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF00 00007ffb0ee46e74 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF30 00007ffb0ee46e2b System.Windows.Controls.Primitives.TextBoxBase.OnGotKeyboardFocus()
000000FD163ACF60 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163ACFA0 00007ffb0f595359 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD090 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD100 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD140 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD220 00007ffb100ad87f System.Windows.Input.KeyboardDevice.ChangeFocus()
000000FD163AD2B0 00007ffb100463e9 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD320 00007ffb10046241 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD370 00007ffb10093c9e System.Windows.UIElement.Focus()
000000FD163AD3B0 00007ffb1065af1d Libronix.Utility.Windows.TextBoxUtility.AutoSelectAllTextBox_PreviewMouseDown()
000000FD163AD410 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163AD450 00007ffb0f595725 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD540 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD5B0 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD5F0 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD6D0 00007ffb0f8f742a System.Windows.Input.InputProviderSite.ReportInput()
000000FD163AD720 00007ffb0f917e65 System.Windows.Interop.HwndMouseInputProvider.ReportInput()
000000FD163AD820 00007ffb0f629999 System.Windows.Interop.HwndMouseInputProvider.FilterMessage()
000000FD163AD8C0 00007ffb0f6295ed System.Windows.Interop.HwndSource.InputFilterMessage()
000000FD163AD930 00007ffb0efa5919 MS.Win32.HwndWrapper.WndProc()
000000FD163AD9C0 00007ffb0efa5837 MS.Win32.HwndSubclass.DispatcherCallbackOperation()
000000FD163ADA10 00007ffb0cac4d43 System.Windows.Threading.ExceptionWrapper.InternalRealCall()
000000FD163ADA70 00007ffb0cac4b52 System.Windows.Threading.ExceptionWrapper.TryCatchWhen()
000000FD163ADAC0 00007ffb0ef78f4f System.Windows.Threading.Dispatcher.LegacyInvokeImpl()
000000FD163ADB50 00007ffb0efa5602 MS.Win32.HwndSubclass.SubclassWndProc()
000000FD163ADC40 00007ffb09b83ea0 ILStubClass.IL_STUB_ReversePInvoke(Int64, Int32, Int64, Int64)
    PARAMETERS:
000000FD163ADEC8 00007ffc2ab98241 [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEC8 00007ffb0bbf9ffb [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEA0 00007ffb0bbf9ffb ILStubClass.IL_STUB_PInvoke
    PARAMETERS:
000000FD163ADF60 00007ffb10080564 System.Windows.Threading.Dispatcher.PushFrameImpl()
000000FD163AE080 00007ffb0b1d7149 System.Windows.Threading.Dispatcher.PushFrame()
000000FD163AE100 00007ffb0b1d7277 System.Windows.Threading.Dispatcher.Run()
000000FD163AE140 00007ffb0b1d73b7 System.Windows.Application.RunDispatcher()
000000FD163AE180 00007ffb0b1d7b44 System.Windows.Application.RunInternal()
000000FD163AE270 00007ffb0b1d7baa System.Windows.Application.Run()
000000FD163AE2A0 00007ffb0b1d7bfa System.Windows.Application.Run()
000000FD163AE2D0 00007ffb09b8b715 LDLS4.OurApp.InitializeAndRun()
000000FD163AE3C0 00007ffb09b2fedf LDLS4.OurApp.Main()

Regression?

Unclear, although it appears that 4.x was working okay when I attempted a repro there.

Known Workarounds

Don't use the und language tag.

Configuration

  • .NET Framework 7.0.10+a6dbb800a47735bde43187350fd3aff4071c7f9c (7.0.1023.36312)
  • Windows 10 Pro 22H2 and others
  • x64

Other information

It appears that this is related to the use of the ICU API uloc_canonicalize() in pal_locale.c:GetLocale(), which treats und as a default or "no language" and strips it off, returning _Latn when passed und-Latn.

FixupLocaleName() in pal_locale.c will convert _ back to - to bring the tag back into the format expected by .NET framework. However, the edge case of und being removed entirely is not handled. It may be possible to replace FixupLocaleName() with a call to uloc_toLanguageTag(): uloc_toLanguageTag("_Latn") yields und-Latn (but I have not verified that this is an exact match for requirements).

ICU bug reports:

Keyman bug report -- original report:

Other issues that may be related:

Many thanks to @srl295 for his assistance in tracing this.
@ghost ghost added needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners untriaged New issue has not been triaged by the area owner labels Feb 16, 2024
@mcdurdin mcdurdin mentioned this issue Feb 16, 2024
Open
@ghost
Copy link

ghost commented Feb 16, 2024

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

If a user installs and activates an input method with the custom locale und-Latn, a .NET application will instantly crash with CultureNotFoundException thrown ultimately by System.Globalization.CultureData.GetCultureData(Int32, Boolean).

Reproduction Steps

  1. Install the custom language und-Latn using e.g. PowerShell: 
    $1 = Get-WinUserLanguageList
$1.Add("und-Latn")
Set-WinUserLanguageList $1 -Force
  2. Create a new WPF Application in Visual Studio from the New Project Wizard, targeting .NET 7.0 or later.
  3. Add a TextBox to MainWindow.xaml: 
    <TextBox x:Name="textBox" HorizontalAlignment="Left" Margin="138,123,0,0" TextWrapping="Wrap" Text="TextBox" VerticalAlignment="Top" Width="120"/>
  4. Run the application, focus the text field.
  5. Select the UND language from the Windows language picker.
  6. The app will instantly crash.

Expected behavior

The language should be selected and the app should not crash.

Actual behavior

The application crashes with the exception CultureNotFoundException

Call stack from a crash dump given me by a user experiencing this problem:

0:000> !CLRStack -a
OS Thread Id: 0x7f88 (0)
        Child SP               IP Call Site
000000FD163ACBF8 00007ffc2c36fec4 [HelperMethodFrame: 000000fd163acbf8] 
000000FD163ACCF0 00007ffb62b5ec5b System.Globalization.CultureData.GetCultureData(Int32, Boolean)
    PARAMETERS:
000000FD163ACD30 00007ffb62b64f41 System.Globalization.CultureInfo..ctor(Int32, Boolean)
    PARAMETERS:
        this (<CLR reg>) = 0x000002ab8e7dda08
000000FD163ACD70 00007ffb0ee515f1 System.Windows.Input.InputLanguageSource.get_CurrentInputLanguage()
000000FD163ACDB0 00007ffb0ee51538 System.Windows.Input.InputLanguageManager.get_CurrentInputLanguage()
000000FD163ACE00 00007ffb0ee50fb2 System.Windows.Documents.TextSelection.EnsureCaret()
000000FD163ACE50 00007ffb100bf2d8 System.Windows.Documents.TextSelection.System.Windows.Documents.ITextSelection.UpdateCaretAndHighlight()
000000FD163ACEB0 00007ffb0ee46fe9 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF00 00007ffb0ee46e74 System.Windows.Documents.TextEditor.OnGotKeyboardFocus()
000000FD163ACF30 00007ffb0ee46e2b System.Windows.Controls.Primitives.TextBoxBase.OnGotKeyboardFocus()
000000FD163ACF60 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163ACFA0 00007ffb0f595359 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD090 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD100 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD140 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD220 00007ffb100ad87f System.Windows.Input.KeyboardDevice.ChangeFocus()
000000FD163AD2B0 00007ffb100463e9 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD320 00007ffb10046241 System.Windows.Input.KeyboardDevice.Focus()
000000FD163AD370 00007ffb10093c9e System.Windows.UIElement.Focus()
000000FD163AD3B0 00007ffb1065af1d Libronix.Utility.Windows.TextBoxUtility.AutoSelectAllTextBox_PreviewMouseDown()
000000FD163AD410 00007ffb0f5b2237 System.Windows.RoutedEventArgs.InvokeHandler()
000000FD163AD450 00007ffb0f595725 System.Windows.EventRoute.InvokeHandlersImpl()
000000FD163AD540 00007ffb0f594853 System.Windows.UIElement.RaiseEventImpl()
000000FD163AD5B0 00007ffb0f616a38 System.Windows.UIElement.RaiseTrustedEvent()
000000FD163AD5F0 00007ffb0f643a5c System.Windows.Input.InputManager.ProcessStagingArea()
000000FD163AD6D0 00007ffb0f8f742a System.Windows.Input.InputProviderSite.ReportInput()
000000FD163AD720 00007ffb0f917e65 System.Windows.Interop.HwndMouseInputProvider.ReportInput()
000000FD163AD820 00007ffb0f629999 System.Windows.Interop.HwndMouseInputProvider.FilterMessage()
000000FD163AD8C0 00007ffb0f6295ed System.Windows.Interop.HwndSource.InputFilterMessage()
000000FD163AD930 00007ffb0efa5919 MS.Win32.HwndWrapper.WndProc()
000000FD163AD9C0 00007ffb0efa5837 MS.Win32.HwndSubclass.DispatcherCallbackOperation()
000000FD163ADA10 00007ffb0cac4d43 System.Windows.Threading.ExceptionWrapper.InternalRealCall()
000000FD163ADA70 00007ffb0cac4b52 System.Windows.Threading.ExceptionWrapper.TryCatchWhen()
000000FD163ADAC0 00007ffb0ef78f4f System.Windows.Threading.Dispatcher.LegacyInvokeImpl()
000000FD163ADB50 00007ffb0efa5602 MS.Win32.HwndSubclass.SubclassWndProc()
000000FD163ADC40 00007ffb09b83ea0 ILStubClass.IL_STUB_ReversePInvoke(Int64, Int32, Int64, Int64)
    PARAMETERS:
000000FD163ADEC8 00007ffc2ab98241 [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEC8 00007ffb0bbf9ffb [InlinedCallFrame: 000000fd163adec8] MS.Win32.UnsafeNativeMethods.DispatchMessage()
000000FD163ADEA0 00007ffb0bbf9ffb ILStubClass.IL_STUB_PInvoke
    PARAMETERS:
000000FD163ADF60 00007ffb10080564 System.Windows.Threading.Dispatcher.PushFrameImpl()
000000FD163AE080 00007ffb0b1d7149 System.Windows.Threading.Dispatcher.PushFrame()
000000FD163AE100 00007ffb0b1d7277 System.Windows.Threading.Dispatcher.Run()
000000FD163AE140 00007ffb0b1d73b7 System.Windows.Application.RunDispatcher()
000000FD163AE180 00007ffb0b1d7b44 System.Windows.Application.RunInternal()
000000FD163AE270 00007ffb0b1d7baa System.Windows.Application.Run()
000000FD163AE2A0 00007ffb0b1d7bfa System.Windows.Application.Run()
000000FD163AE2D0 00007ffb09b8b715 LDLS4.OurApp.InitializeAndRun()
000000FD163AE3C0 00007ffb09b2fedf LDLS4.OurApp.Main()

Regression?

Unclear, although it appears that 4.x was working okay when I attempted a repro there.

Known Workarounds

Don't use the und language tag.

Configuration

  • .NET Framework 7.0.10+a6dbb800a47735bde43187350fd3aff4071c7f9c (7.0.1023.36312)
  • Windows 10 Pro 22H2 and others
  • x64

Other information

It appears that this is related to the use of the ICU API uloc_canonicalize() in pal_locale.c:GetLocale(), which treats und as a default or "no language" and strips it off, returning _Latn when passed und-Latn.

FixupLocaleName() in pal_locale.c will convert _ back to - to bring the tag back into the format expected by .NET framework. However, the edge case of und being removed entirely is not handled. It may be possible to replace FixupLocaleName() with a call to uloc_toLanguageTag(): uloc_toLanguageTag("_Latn") yields und-Latn (but I have not verified that this is an exact match for requirements).

ICU bug reports:

Keyman bug report -- original report:

Other issues that may be related:

Author: mcdurdin
Assignees: -
Labels:

area-System.Globalization, untriaged, needs-area-label
Milestone: -

@jkotas jkotas removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Feb 16, 2024
@tarekgh
Copy link
Member

tarekgh commented Feb 16, 2024 •
edited
Loading

I think there is some confusion here. The thrown exception is not because .NET couldn't handle the name und-Latn but because WPF is trying to create the culture using invalid LCID. Look at the stack and you will see System.Globalization.CultureInfo..ctor(Int32, Boolean). If you have the full exception message, it should tell what LCID value was used. My guess here is WPF trying to get LCID of the input keyboard from Windows and then try to create the culture using that LCID. It is possible the LCID returned from Windows is 0x1000 which is a custom LCID which cannot create a culture using such LCID.

This doesn't mean .NET will succeed creating a culture with the name und-Latn, that is because we use ICU API uloc_getName which truncate the language part. We already discussed this issue with the ICU guys in the ticket https://unicode-org.atlassian.net/browse/ICU-22137 and we are considering switch using uloc_toLanguageTag instead and ensure canonicalize the culture name too. I'll keep this issue open to track that work.

@tarekgh tarekgh added this to the Future milestone Feb 16, 2024
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Feb 16, 2024
@mcdurdin
Copy link
Author

mcdurdin commented Feb 17, 2024 •
edited
Loading

@tarekgh that was my initial assumption too (see #37789 which was precisely that). However, when I debugged my sample application, I found that .NET is translating from the transient locale 0x2000 correctly to get und-Latn, which it then passes in to ICU.

To prove this, I added another transient locale, this time picking tpi-PG (Tok Pisin, Papua New Guinea), which was assigned LCID 0x2400:

PS C:\Users\mcdurdin> $1 = Get-WinUserLanguageList
PS C:\Users\mcdurdin> $1.add("tpi-pg")
PS C:\Users\mcdurdin> Set-WinUserLanguageList $1 -force
PS C:\Users\mcdurdin> Get-WinUserLanguageList


LanguageTag     : en-AU
Autonym         : English (Australia)
EnglishName     : English
LocalizedName   : English (Australia)
ScriptName      : Latin
InputMethodTips : {0C09:00000409}
Spellchecking   : True
Handwriting     : False

LanguageTag     : km
Autonym         : ភាសាខ្មែរ
EnglishName     : Khmer
LocalizedName   : Khmer
ScriptName      : Khmer
InputMethodTips : {0453:{FE0420F1-38D1-4B4C-96BF-E7E20A74CFB7}{99A91333-C033-428B-A076-D44C9FE0F450}}
Spellchecking   : True
Handwriting     : False

LanguageTag     : und-Latn
Autonym         :
EnglishName     : Undetermined (und-Latn)
LocalizedName   :
ScriptName      : Latin
InputMethodTips : {2000:00000409}
Spellchecking   : True
Handwriting     : False

LanguageTag     : tpi-PG
Autonym         : Tok Pisin (tpi-PG)
EnglishName     : Tok Pisin (tpi-PG)
LocalizedName   : Tok Pisin (tpi-PG)
ScriptName      : Latin
InputMethodTips : {2400:00000409}
Spellchecking   : True
Handwriting     : False

Selecting and using tpi-PG never caused the exception, whereas selecting und-Latn consistently did (note: if the text field was already focused when I switched languages, then the first keystroke would throw).

I debugged GetCultureData(int,bool) in CultureData.cs:

            // Convert the lcid to a name, then use that
            string? localeName = LCIDToLocaleName(culture);

            if (!string.IsNullOrEmpty(localeName))
            {
                // Valid name, use it
                retVal = GetCultureData(localeName, bUseUserOverride);
            }

LCIDToLocaleName is passed 0x2000 and returns und-Latn. GetCultureData('und-Latn',true) returns null.

  • Parameter bUseUserOverride == true per the call from System.Windows.Input.InputLanguageSource.get_CurrentInputLanguage().

GlobalizationMode flags are all default and recommended settings:

  • GlobalizationMode.PredefinedCulturesOnly == false
  • GlobalizationMode.Invariant == false
  • GlobalizationMode.UseNls == false

Rough sequence of events:

  1. GetCultureData(string,bool) calls CreateCultureData(cultureName, useUserOverride) (because the language is not yet in the cached cultures hash table)
  2. That then calls CultureData.InitCultureDataCore().
  3. Which calls InitIcuCultureDataCore().
  4. InitIcuCultureDataCore() ends up with _sRealName set to "-Latn" as a result of the call to GetLocaleName() in the ICU wrapper pal_locale.c.
  5. InitCultureDataCore() then passes this bogus string to Windows API GetLocaleInfoEx() (which of course balks at that).

So I think my initial analysis is correct.

Side note: as an experiment, in the debugger, I overrode the _sWindowsName value returned from GetLocaleName() and set it back to 'und-Latn'. That avoided the crash -- the culture data was successfully initialized and added to the cached culture hash table. No other issues were encountered.

@tarekgh
Copy link
Member

tarekgh commented Feb 17, 2024

@mcdurdin thanks for your investigation and analysis. I agree with you now 😄. I didn't know that WPF was getting a transient LCID 0x2000 and passing it to the .NET. This explains why this worked on the .NET Framework too.

By the way, you can enable Using NLS instead of ICU which should make it work without hacking values under the debugger.

@mcdurdin
Copy link
Author

Thanks-- I will document the globalization nls flag as a workaround for our users as well. It was useful to verify the root cause in the debugger though without other potential side effects 😀

@srl295
Copy link

srl295 commented Feb 17, 2024

we are considering switch using uloc_toLanguageTag instead and ensure canonicalize the culture name too. I'll keep this issue to track that work.

If you want bcp47 you should definitely be using uloc_toLanguageTag. We wrote the other apis long before bcp47 existed, now they should have warnings on them.

@tarekgh tarekgh modified the milestones: Future, 10.0.0 Sep 4, 2024
@miloush
Copy link
Contributor

miloush commented Sep 25, 2024

There is two issues here, one is CultureInfo incompatibility with .NET FW, second one on InputLanguageSource.CurrentInputLanguage using LCID rather than a language tag to instantiate it. I suggest an issue is filed at WPF as well to switch to language tags.

@tarekgh tarekgh mentioned this issue Mar 14, 2025
Open
14 tasks
@mcdurdin mcdurdin mentioned this issue Mar 31, 2025
Closed
@tarekgh tarekgh mentioned this issue Apr 29, 2025
Merged
@dotnet-policy-service dotnet-policy-service bot added the in-pr There is an active PR which will close this issue when it is merged label Apr 29, 2025
@matouskozak matouskozak mentioned this issue May 2, 2025
Closed
@github-actions github-actions bot locked and limited conversation to collaborators May 31, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Globalization in-pr There is an active PR which will close this issue when it is merged
Projects
None yet
Milestone
10.0.0
Development

Successfully merging a pull request may close this issue.

Fix culture creation with undetermined lang tag tarekgh/runtime
5 participants
@srl295 @mcdurdin @jkotas @miloush @tarekgh