drivers/serial: Modify serial/pty to allow NSH/Telnet line edit

[acassis]

Summary

Both the UART and PTY serial drivers previously assumed all VT100/ANSI escape sequences were fixed 3-byte CSI sequences, causing longer CSI and SS3 key sequences (such as Home, End, Delete, and modified keys) to leak stray characters into the terminal when local echo was enabled. This patch replaces the fixed-length logic with a state machine that correctly recognizes and suppresses escape sequences of any length, while preserving the data delivered to applications. The change only affects local echo behavior, is fully backward compatible, and has been validated with both interactive NSH sessions and automated PTY tests.

Assisted-by: Claude Code

Impact

Users will be able to use keys arrows, Home, End, etc to edit the nsh> command line (patch to nuttx-apps will come)

Testing

Tested on sim:nsh and stm32f103-minimum:nsh

Note the cursor moving to the left:

[github-actions]

MemBrowse Memory Report

arduino-mega2560

• flash: .text +60 B (+0.1%, 67,314 B / 262,144 B, total: 26% used)

esp32-devkitc

• ROM: .flash.text +44 B (+0.0%, 122,748 B / 4,194,272 B, total: 3% used)
• irom0_0_seg: .flash.text +44 B (+0.1%, 87,116 B / 3,342,304 B, total: 3% used)

mirtoo

• kseg0_progmem: .text +72 B (+0.1%, 67,192 B / 131,072 B, total: 51% used)

qemu-armv8a

• Code: .text.uart_readv +64 B (+0.0%, 315,414 B)

qemu-intel64

• Code: .text +48 B (+0.0%, 8,655,476 B)

rx65n-rsk2mb

• ROM: .text +48 B (+0.1%, 85,704 B / 2,097,152 B, total: 4% used)

s698pm-dkit

• Code: .text +144 B (+0.0%, 358,208 B)

stm32-nucleo-f103rb

• flash: .text +24 B (+0.1%, 33,804 B / 131,072 B, total: 26% used)
  No memory changes detected for:
• hifive1-revb

[acassis]

@xiaoxiang781216 @jerpelea @cederom @linguini1 please take a look

[linguini1]

I wonder if this will fix the weird 'K' artifact on the Picos

[acassis]

I wonder if this will fix the weird 'K' artifact on the Picos

Yes, probably it will too

[linguini1]

I will try with the Pico today if I have some time and let you know. That would be awesome!

[acassis]

I will try with the Pico today if I have some time and let you know. That would be awesome!

I think some terminals that are not VT102 compatible also will display the [^K, please double check if your serial console terminal supports it.

More info: https://stackoverflow.com/questions/21384040/why-does-the-terminal-show-a-b-c-d-when-pressing-the-arrow-k

[acassis]

@linguini1 a curiosity: if you press: Ctrl+V the next key you type will be printed as raw key, so you will see how the key is send to the application (the terminal in this case). Try: Ctrl+V+ENTER, or other special key, like arrows keys, etc.

[linguini1]

Haha just like Vim! I'll try that as well.

[linguini1]

I will try with the Pico today if I have some time and let you know. That would be awesome!

I think some terminals that are not VT102 compatible also will display the [^K, please double check if your serial console terminal supports it.

More info: https://stackoverflow.com/questions/21384040/why-does-the-terminal-show-a-b-c-d-when-pressing-the-arrow-k

This might have been it, I noticed it I think mostly with a pyserial shell but almost never in Minicom.

[acassis]

I will try with the Pico today if I have some time and let you know. That would be awesome!

I think some terminals that are not VT102 compatible also will display the [^K, please double check if your serial console terminal supports it.
More info: https://stackoverflow.com/questions/21384040/why-does-the-terminal-show-a-b-c-d-when-pressing-the-arrow-k

This might have been it, I noticed it I think mostly with a pyserial shell but almost never in Minicom.

Yes, pyserial will not parser these codes, unless you add some VT102 filter

[linguini1]

Ah, so that was the issue after all then. Anyways, good improvement!