session freeze during linux nano edit of a file

[DougSmythies]

I communicate with various linux computers via OpenSSH_for_Windows_9.5p1 from my windows 11 computer. I edit files on the linux computers using the nano editor.
Sometimes after a copy and paste, the nano session freezes. I have to close the terminal session in windows and start again, often loosing the edits done prior to the freeze. I have tried every keystroke I can think of to try to get the session unfrozen, but so far nothing has worked. As far as I can determine the nano task seems to still be running.
How do I know the problem is on the OpenSSH_for_Windows? I don’t, and am pursuing the issue from the Ubuntu linux side also. Regardless, if there is some way to “kick” things out from being frozen it would very useful.

A couple of screen shots are attached.

EDIT:
For the same file:
nano 7.2 on Ubuntu server 24.04.1 fails.
nano 3.2 on Raspberry Pi passes.
nano 5.4 on Debian server 11 passes. <<< Correction: Fails.
nano 8.1 on Ubuntu desktop 24.10 VM guest on the Debian server host worked but pastes “10000~” instead of the copied “10000”.
nano 7.2 on Ubuntu desktop 24.04.2 VM quest on the Debian server host fails.

EDIT: same issue using "OpenSSH_for_Windows_9.8p1 Win32-OpenSSH-GitHub, LibreSSL 3.9.2"
Link to this same issue posted on Ubuntu discourse

EDIT: The root issue seems to be the breaking up of "bracketed pastes" in the middle of the "End of pasted text" sequence. Similar to this bug report , however in my example the paste buffer is only 5 characters.

[DougSmythies]

As far as I can determine it is the OpenSSH_for_Windows program that is introducing the broken up "bracketed pastes" sequences in the character stream that is being sent to the SSH server.
I compared debug prints outs from the nano program for 4 different failing paste buffer lengths, 5, 85, 197, and 293 characters.
Using OpenSSH_for_Windows all of those paste buffer sizes split the trailing "end of pasted text" escape sequence, putting the trailing "~" character on a separate chunck.
Using PuTTY instead for those same paste buffer sizes worked fine.
The debug prints out examples are attached.
examples.txt

[DougSmythies]

This broken "bracketed pastes" issue seems to be related to this pull request, which will reduce the probability of the issue but not eliminate it. (Thanks to the maintainer of nano for finding that pull request.)

[lhecker]

I would suggest testing this hypothesis by compiling this project locally. It's thankfully very easy to do. If you don't want to install Visual Studio, I can provide you an .exe that I compiled myself (but that carries the theoretical risk of me being a bad actor of course).

Thanks to the maintainer of nano for finding that pull request.

If this is indeed due to the chunking, I would consider this a bug in nano actually.
My intention of the PR was to improve performance and to reduce the chance of encountering bugs in applications that don't correctly handle trailing escapes in the input.
The way I see it, any terminal application should poll for more input when the current input ends in an escape character as even on UNIX these issues should occur randomly (particularly via SSH).
In other words, I believe nano (and fish, etc.) should work correctly even without my PR, unless I'm mistaken.

[DougSmythies]

It has been about 13 years since I have compiled code on a windows system. If you make an .exe for this I'll try it. I understand the risk. I do the same with the linux kernel for those unable, or unwilling, to compile it for themselves, along with the same disclaimer that I might secretly be evil.
The main maintainer of nano has been spending a great deal of time on this. The related email thread and the related bug report.

[lhecker]

Here's the version I built for myself after opening the PR: openssh_771.zip
(It includes debug symbols, just in case you need them.)

I'm happy to see the escape detection being discussed in the bug report. It's pretty much exactly what I meant.

[DougSmythies]

As a side note: It will be interesting to see how much the PR change improves paste performance. For a 93,298 character paste, currently Win32-OpenSSH is 7.75 times slower than PuTTY and takes 0.63 seconds. (the 93,298 number was arbitrary.) A couple of graphs are attached.

[DougSmythies]

Using the PR version from @lhecker now it is only 1.66 times slower than PuTTY for the same 93,298 character paste, taking 0.135 seconds to paste. Attached is a new graph with the "leonard" data added, from this version.
I also tried to paste 1025 (1013 characters + 2X6 char ESC sequences) and 4097 (4085 characters + 2X6 char ESC sequences) buffers many many times and never got the trailing "~" of the "end of pasted text" ESCAPE sequence delayed from the previous chunk's last character, "0" by more than 900 uSec. If the windows computer was ridiculously overloaded, then delays of 200 mSec would occur. Delays of 15 to 40 mSec was very common with the 16 character sized chunks.

[lhecker]

now it is only 1.66 times slower than PuTTY for the same 93,298 character paste, taking 0.135 seconds to paste.

I don't know its source code, but I suspect PuTTY doesn't use a "console server" per-se, so its builtin SSH client is more like a ssh client <> VT parser <> text viewer directly strapped together, right? That would explain its excellent performance.

As I briefly touched on in my PR, the console IO overhead on Windows is atrocious ¹ at ~30us on average, but often more. So, assuming a per-byte I/O overhead of somewhere in the neighborhood of ~5ns (+/- lots, because it depends on the exact operation), even at 4KiB reads you'd probably have ~2/3rds call overhead. (IIRC, this is also generally true for TTYs on Linux/macOS, but to a much lesser extent.) My PR only bumps it up to 1KiB, which would explain the ~2x performance difference to PuTTY, and I'd expect that to go away at e.g. typical 128KiB reads. But I was hesitant to bump it by that much all in one PR, particularly without having something like a cheap bump allocator or similar and without an idea how the project overall works.

Apropos, if you benchmark console apps on Windows, please give the latest Windows Terminal a try. It contains major performance improvements (in some areas in the order of magnitudes). 🙂

[...the first image...]

There's still a significant difference to the PuTTY latency, right? It's a little hard to tell, as the chart is vertically compressed. I suppose this is also due to the call overhead? I have not actually looked deeper into this project's use of the console APIs, but hopefully it only does 1 console call per packet.

Footnotes

1. In my defense of my choice of words: I co-maintain the console subsystem on Windows. 😄 ↩

[DougSmythies]

I don't know its source code, but I suspect PuTTY doesn't use a "console server" per-se, so its builtin SSH client is more like a ssh client <> VT parser <> text viewer directly strapped together, right?

I don't know, but it makes sense.

As I briefly touched on in my PR, the console IO overhead on Windows is atrocious

Yes, I saw, and agree.

There's still a significant difference to the PuTTY latency, right? It's a little hard to tell, as the chart is vertically compressed. I suppose this is also due to the call overhead?

Yes, I think so. I did re-do one of the other two graphs, but didn't include it earlier. I will now.
If we ignore the extremely long system calls of the first 9 chunk boundaries, that are very roughly 3000 times the nominal system call time of 2500 nanoseconds, then This test WIN-SSH code is only 2.7% slower than PuTTY for the rest of the paste. Whereas the original WIN-SSH program is still 7 times slower than PuTTY.

Note 1: I did not include a new "detail b" graph because there nothing new to show at that y-axis scaling.
Note 2: The lower limit of the key-stroke timing program is unknown, but likely a significant portion of the lowest times in the 600 nSec range.
EDIT: The minimum loop time of the key-stroke monitoring and timing program, for redirected output, is about 410 nSec. Therefore it is not the limiting factor here. As a side note, when not redirecting the output, the minimum loop time is about 1000 nSec.

[DougSmythies]

@lhecker , or anybody, does this code even know about "bracketed paste" mode? Neither the nano maintainer or I have been able to find any related code. If I understand correctly, nano expects the related escape sequences not to split across chunks and WIN-SSH is expecting whomever is on the other end to deal with it. I do not know if there is any specification for this or if something in between the two is supposed to deal with it.

[lhecker]

Maybe @tgauth knows?

My own understanding so far was that SSH is conceptually entirely unaware about VT sequences (server and client). It just sits in the middle between a remote PTY and a local terminal and passes text through. As such it would not be aware about the bracketed paste sequences generated by the terminal. It's unrealistic to expect the sequences to not be split because of this, as between the terminal and nano there could be pretty much anything that may end up chunking the text stream in any arbitrary way. SSH is presumably the most popular cause for chunking (TCP packet size is limited + MTU if you use something like mosh).

---

In my own text editor, I'm doing the following: I always read stdin in chunks (currently 1KiB; = reader). On top of that I've got a stateful, classic VT parser (= vt-parser). And on top of that in turn I have my VT-to-input-actions parser (= input-parser). It looks somewhat like this:

reader = { stdin };
vt_parser = {};
input_parser = { vt_parser }; // wraps the VT parser

loop {
    timeout = vt_parser.get_ideal_read_timeout(); // explained below
    chunk = reader.read(timeout); // reads 1 chunk of input from stdin

    do {
        chunk, input = input_parser.parse(chunk); // calls the VT parser until it gets an input
        /* do it */
    } while (chunk not empty);
}

If the vt-parser is left in the "escape" state, because the last char in the chunk was an escape character, get_ideal_read_timeout will return a finite timeout (currently 100ms).
During the next read I'll then use ppoll to wait on the input handle. If the read times out, it'll return an empty chunk, which the vt-parser recognizes and flush out the pending escape. This solves the is-escape-an-escape-sequence-or-escape-key-press ambiguity in terminals.

For incomplete VT sequences I currently don't use any timeout, but I did plan to make it finite in that case as well. (Maybe 1s? should be an edge case after all.)

Since the vt-parser retains state across chunked reads, this won't run into any issues with bracketed paste sequences getting split up.

[DougSmythies]

There has been a fix committed on the nano bug report.
As an side test, and more related to the PR over here, I tested times to do a large paste of 16,528,512 characters:
before PR: paste took 103 seconds
after PR: paste took 23 seconds, or 4.5 X improvement
PuTTY: paste took 16 seconds

Thanks for the help and explanations herein.

[lhecker]

Thank you so much for your investigations! I hope we can get that down to 0.23s in the future, be it Linux, Windows, this project, PuTTY, nano, or whatever else! Nothing is better than stuff being fast. 😄

If you ever have any terminal related performance differences like that (Windows Terminal or anything involved with conhost/OpenConsole), please feel free to drop by at https://github.com/microsoft/terminal and report it as a bug. I just did a test and noticed that conhost/OpenConsole is extremely slow for pasting large amounts of text and will keep that in mind for my future refactors. Late last year I implemented stdout passthrough for 20x perf improvements and I was hoping to do stdin passthrough for similar improvements sometime this year or early next year. 🙂