Faied to input emoji in windows terminal (and vscode)

[codehz]

Environment

PSReadLine: 2.1.0
PowerShell: 7.1.3
OS: Microsoft Windows 10.0.21354
BufferWidth: 120
BufferHeight: 30

Last 36 Keys

 c d Spacebar D : | Backspace \ E x p Tab Enter
 c d Spacebar m i n i Tab Enter
 g i t Spacebar i n i t Spacebar - b Spacebar � Enter

(real command line: git init -b 💡 (Don't paste directly, input with windows emoji panel (win + .)), windows experimental unicode mode is NOT enabled)

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 12 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass83_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)

[ninmonkey]

@codehz Did you use the hotkey windows+. to paste the emoji? If yes I can reliably recreate that.

Otherwise copying and pasting emoji works normally. The only problem is when I use that hotkey.

Unable to translate Unicode character \uD83D

those codepoints are a surrogate key. It looks like it's encoding as UTF-16BE, which seems weird, powershell defaults to UTF-16LE for WinPS and Utf8 for Ps

I tried each of the following, then re-tried win+. They all reproduce the error.

$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [Text.Encoding]::GetEncoding('utf-16')
$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [Text.Encoding]::GetEncoding('utf-16le')
$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [Text.Encoding]::GetEncoding('utf-8')

Working: From clipboard

🐒> pwsh -nop -nol                              
PS C:\Users\cppmo_000\Documents\2020\powershell>
>> '🐒' * 3                                     
🐒🐒🐒     

Crashes: Using win+. popup menu

### Environment                                                                                                                                                                                                     
PSReadLine: 2.2.0-beta2                                                                                                                                                                                             
PowerShell: 7.1.2                                                                                                                                                                                                   
OS: Microsoft Windows 10.0.19041                                                                                                                                                                                    
BufferWidth: 120                                                                                                                                                                                                    
BufferHeight: 44                                                                                                                                                                                                    

Last 200 Keys

 Ctrl+v Ctrl+Backspace Ctrl+Backspace Ctrl+Spacebar p r e d RightArrow LeftArrow Enter                                                                                                                              
 Spacebar Ctrl+Spacebar RightArrow RightArrow RightArrow LeftArrow LeftArrow Enter                                                                                                                                  
 Enter                                                                                                                                                                                                              
 UpArrow Ctrl+Backspace Ctrl+Spacebar Enter                                                                                                                                                                         
 Enter                                                                                                                                                                                                              
 UpArrow UpArrow Shift+Home Ctrl+x g i t Spacebar s t a t u s Enter                                                                                                                                                 
 g i t Spacebar a d d d Spacebar Backspace Backspace Spacebar . Enter                                                                                                                                               
 g i t Spacebar c o m Ctrl+a g i t Spacebar d i f f Enter                                                                                                                                                           
 g i t Spacebar c o m m i t Spacebar - m Spacebar ' u p d a t e d Spacebar c o m m a n d s ' Enter                                                                                                                  
 g i t Spacebar p u s h Enter                                                                                                                                                                                       
 UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow UpArrow DownArrow DownArrow LeftArrow DownArrow UpArrow UpArrow UpArrow UpArrow Ctrl+a Backspace Ctrl+v Home Backspace Ctrl+l Enter                
 Ctrl+v Enter                                                                                                                                                                                                       
 UpArrow Spacebar Backspace Backspace Shift+Spacebar | Shift+Spacebar s e l Tab Spacebar - f Tab Spacebar 1 Backspace 3 Enter                                                                                       
 UpArrow Ctrl+Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace Backspace ' Enter 
 Alt+. Escape Ctrl+Spacebar Ctrl+a Alt+, Escape Alt+, Backspace Backspace Backspace Alt+, Backspace Backspace � LeftArrow LeftArrow RightArrow RightArrow Enter                                                     
                                                                                                                                                                                                                    

Exception

System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83D at index 0 to specified code page.                                                                                              
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)                                                                                                                            
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)                                                                                                                
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)                                                       
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)                                                                                               
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)                                                                                 
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)                                                                                                                                       
   at System.IO.StreamWriter.Dispose(Boolean disposing)                                                                                                                                                             
   at System.IO.TextWriter.Dispose()                                                                                                                                                                                
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass98_0.<WriteHistoryRange>b__0()                                                                                                                        
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)                                                                                                                   
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)                                                                                                         
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)                                             
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()                                                                                                                                                            
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken)                                                                    

[codehz]

@ninmonkey yes, I use win+. to input emoji, it seems only works in wsl(with Windows Terminal)(cmd.exe is also failed to receive emoji)

[Neptune650]

This bug also seems to happen when pasting emojis.

[tsukumijima]

I've seen the discussion. The essence of this problem is that PSReadline cannot correctly recognize surrogate pair characters.

Surrogate pair characters are famous for emoji, but they are also used to represent lesser-known kanji in Japanese (kanji).
Most of them are kanji that we Japanese don't even use. However, for example, the kanji 𩸽 (U+29E3D), which represents the fish"atka mackerel", causes exactly the same error as the emoji.

An example of a list of surrogate pair kanji can be found at https://www.magata.net/memo/index.php?%A5%B5%A5%ED%A5%B2%A1%BC%A5%C8%A5%DA%A5%A2. (Japanese)

I used Windows Terminal and typed echo 🤔 using three terminals: PowerShell, Windows PowerShell, and cmd.exe.
I attach the log and screenshot.
In this example, I tried with 🤔 , but even if 🤔 is 𩸽, the problem does not change.

PowerShell (7.1.3, PSReadLine 2.2.0-beta3)

C:\Users\User> echo �

Oops, something went wrong.
Please report this bug with ALL the details below, including both the 'Environment' and 'Exception' sections.
Please report on GitHub: https://github.com/PowerShell/PSReadLine/issues/new?template=Bug_Report.md
Thank you!

### Environment
PSReadLine: 2.2.0-beta3
PowerShell: 7.1.3
OS: Microsoft Windows 10.0.19042
BufferWidth: 88
BufferHeight: 22

Last 12 Keys
\```
 w e Backspace Backspace Backspace e c h o Spacebar � Enter

\```

### Exception
\```
System.Text.EncoderFallbackException: Unable to translate Unicode character \\uD83E at index 5 to specified code page.
   at System.Text.EncoderExceptionFallbackBuffer.Fallback(Char charUnknown, Int32 index)
   at System.Text.EncoderFallbackBuffer.InternalFallback(ReadOnlySpan`1 chars, Int32& charsConsumed)
   at System.Text.Encoding.GetBytesWithFallback(ReadOnlySpan`1 chars, Int32 originalCharsLength, Span`1 bytes, Int32 originalBytesLength, EncoderNLS encoder)
   at System.Text.Encoding.GetBytesWithFallback(Char* pOriginalChars, Int32 originalCharCount, Byte* pOriginalBytes, Int32 originalByteCount, Int32 charsConsumedSoFar, Int32 bytesWrittenSoFar, EncoderNLS encoder)
   at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
   at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.PowerShell.PSConsoleReadLine.<>c__DisplayClass98_0.<WriteHistoryRange>b__0()
   at Microsoft.PowerShell.PSConsoleReadLine.WithHistoryFileMutexDo(Int32 timeout, Action action)
   at Microsoft.PowerShell.PSConsoleReadLine.WriteHistoryRange(Int32 start, Int32 end, Boolean overwritten)
   at Microsoft.PowerShell.PSConsoleReadLine.IncrementalHistoryWrite()
   at Microsoft.PowerShell.PSConsoleReadLine.MaybeAddToHistory(String result, List`1 edits, Int32 undoEditIndex, Boolean fromDifferentSession, Boolean fromInitialRead)
   at Microsoft.PowerShell.PSConsoleReadLine.InputLoop()
   at Microsoft.PowerShell.PSConsoleReadLine.ReadLine(Runspace runspace, EngineIntrinsics engineIntrinsics, CancellationToken cancellationToken, Nullable`1 lastRunStatus)
   
C:\Users\User> echo �

The error is exactly the same as the log attached by the other person.

However, if you enter echo 🤔 before PowerShell initialization is complete, you will see 🤔 successfully. Even if you execute it again from the command history, it is normal.
Even then, if you type or paste 🤔 yourself after initialization, you will get the same error.

Since only one � is displayed, it is presumed that only one of the two character codes of the surrogate pair can be recognized.

PowerShell (7.1.3, remove PSReadLine)

C:\Users\User> remove-module PSReadline
C:\Users\User> echo ��
🤔
C:\Users\User>

The characters are garbled in the input, but they are displayed normally in the output.
If you try to re-execute from the command history, the input will be displayed without garbled characters.

From this, it can be seen that the problem of garbled input is a problem of PowerShell itself, and it is a problem of PSReadline that an error occurs when reading surrogate pair characters.

Windows PowerShell (5.1, PSReadLine 2.0.0-beta2)

PS C:\Users\User> echo ??
🤔
PS C:\Users\User>

The input shows ??, but the output shows it normally.
If you try to re-execute from the command history, the input will be displayed as ?? as well.

If you run remove-module in advance, it will be in the same state as PowerShell (7.1.3, remove PSReadline).

cmd.exe

C:\Users\User>echo ��
🤔

C:\Users\User>

It is in the same state as PowerShell (7.1.3, remove PSReadline).

The characters are garbled in the input, but they are displayed normally in the output.
If you try to re-execute from the command history, the input will be displayed without garbled characters.

Since I am using translation, the text may be unnatural.
I strongly hope that this problem will be resolved as soon as possible.

[tsukumijima]

When I looked it up, I found something strange.

First of all, this issue does not occur at all in PSReadline 1.2. In addition, PSReadline 1.2 does not cause garbled characters when entering surrogate pair characters. This is the same for both Windows PowerShell 5.1 and PowerShell 7.

While using PSReadline 1.2, I ran Remove-Module PSReadLine to temporary remove PSReadline. Then, although the output display is normal, the input display is garbled.
From this, it is considered that PSReadline 1.2 implemented a measure to prevent the input display from being garbled, but PSReadline 2.x has unknowingly disabled it for some reason.

---

🔼 PSReadline 2.0.0-beta2 bundled with Windows 10 2004
🔽 PSReadline 2.0.0-beta2 distributed by PowerShellGallery

What's more, strangely, PSReadline 2.0.0-beta2 bundled with Windows 10 2004 seems to be different from PSReadline 2.0.0-beta2 distributed by PowerShellGallery.
The file structure and modification date are significantly different, and the dll file name is Microsoft.PowerShell.PSReadline.dll instead of Microsoft.PowerShell.PSReadLine2.dll.

And I found that only PSReadline 2.0.0-beta2 bundled with Windows 10 2004 did not give an error.
However, as mentioned above, garbled characters will occur when you enter surrogate pair characters.
Using PSReadline 2.0.0-beta2 distributed by PowerShellGallery causes exactly the same issues as PSReadline 2.2.0-beta3.

---

So there are currently two workarounds for this issue in PowerShell 7.
One is to uninstall PSReadline 2.x and install PSReadline 1.2. The other is to copy PSReadline 2.0.0-beta2 bundled with Windows 10 2004 from C:\Program Files\WindowsPowerShell\Modules\PSReadLine\2.0.0 to the Modules folder in PowerShell 7.

I'd like to avoid downgrading as much as possible, but I can't help it because the previous version solves the problem. If this problem is left unattended, it's a shame.

@daxian-dbw