Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support combining sequences that don't normalize #2616

Open
tig opened this issue May 9, 2023 · 0 comments
Open

Support combining sequences that don't normalize #2616

tig opened this issue May 9, 2023 · 0 comments
Labels
bug
Milestone
V2 Release

Comments

@tig
Copy link
Collaborator

tig commented May 9, 2023 •
edited
Loading

See this discussion for more: #2939

What's broken today (v2):

  • Combining Marks that don't normalize are not supported
    • /u0301 at column 0 renders ́ even though /u0301 is a zero-width codepoint.
    • a/u0301 works because ConsoleDriver normalizes it á
    • a/u301/u0301 doesn't work. ConsoleDriver normalizes it to á and ignores the 2nd /u0301.
    • TextField/TextView` strip out CMs on paste.
  • non-BMP characters
    • WindowsDriver
      • Not supported by Windows Terminal when using the WriteConsoleOutput (what WindowsDriver currently does). WindowsDriver currently outputs non-BMP characters as Rune.ReplacementChar
    • CursesDriver
      • Single-width non-BMP chars output as a space
      • Wide non-BMP characters are output as *
    • NetDriver
      • Single-width non-BMP chars correctly
      • Wide non-BMP characters chars output correctly

Note, TextView/TextField and other input views do not handle any of this stuff correctly (mostly).

Related:
@tig tig added the bug label May 9, 2023
@tig tig mentioned this issue May 9, 2023
Merged
22 tasks
@tig tig mentioned this issue May 23, 2023
Merged
8 tasks
@tig tig moved this to 🆕 New in Terminal.Gui V2 Beta Aug 7, 2023
BDisp added a commit to BDisp/Terminal.Gui that referenced this issue Oct 24, 2023
@BDisp
Fixes gui-cs#2616. Support combining sequences that don't normalize
0436019
@BDisp BDisp mentioned this issue Oct 24, 2023
Merged
8 tasks
@tig tig mentioned this issue Oct 29, 2023
Open
tig added a commit that referenced this issue Oct 29, 2023
@BDisp @tig
Partially Addresses #2616. Support combining sequences that don't nor…
aa8b952 
…malize (#2932)

* Fixes #2616. Support combining sequences that don't normalize

* Decouples Application from ConsoleDriver in TestHelpers

* Updates driver tests to match new arch

* Start on making all driver tests test all drivers

* Improves handling if combining marks.

* Fix unit tests fails.

* Fix unit tests fails.

* Handling combining mask.

* Tying to fix this unit test that sometimes fail.

* Add support for combining mask on NetDriver.

* Enable CombiningMarks as List<Rune>.

* Prevents combining marks on invalid runes default and space.

* Formatting for CI tests.

* Fix non-normalized combining mark to add 1 to Col.

* Reformatting for retest the CI.

* Forces non-normalized CMs to be ignored.

---------

Co-authored-by: Tig <[email protected]>
tig added a commit that referenced this issue Jan 5, 2024
@tig @BDisp
Robustness improvements in prep for implementing Virtual Terminal Seq…
7af54f3 
…uences (#3094)

* Fixes #2616. Support combining sequences that don't normalize

* Decouples Application from ConsoleDriver in TestHelpers

* Updates driver tests to match new arch

* Start on making all driver tests test all drivers

* Improves handling if combining marks.

* Fix unit tests fails.

* Fix unit tests fails.

* Handling combining mask.

* Tying to fix this unit test that sometimes fail.

* Add support for combining mask on NetDriver.

* Enable CombiningMarks as List<Rune>.

* Prevents combining marks on invalid runes default and space.

* Formatting for CI tests.

* Fix non-normalized combining mark to add 1 to Col.

* Reformatting for retest the CI.

* Forces non-normalized CMs to be ignored.

* Initial experiment

* Created ANSiDriver. Updated UI Catalog command line handling

* Fixed ForceDriver logic

* Fixed ForceDriver logic

* Updating P/Invoke

* Force16 colors WIP

* Fixed 16 colo mode

* Updated unit tests

* UI catalog tweak

* Added chinese scenario from bdisp

* Disabled AnsiDriver unit tests for now.

* Code cleanup

* Initial commit (fork from v2_fixes_2610_WT_VTS)

* Code cleanup

* Removed nativemethods.txt

* Removed not needed native stuff

* Code cleanup

* Ensures command line handler doesn't eat exceptions

---------

Co-authored-by: BDisp <[email protected]>
BDisp added a commit to BDisp/Terminal.Gui that referenced this issue Jan 5, 2024
@tig @BDisp
Robustness improvements in prep for implementing Virtual Terminal Seq…
91f47a2 
…uences (gui-cs#3094)

* Fixes gui-cs#2616. Support combining sequences that don't normalize

* Decouples Application from ConsoleDriver in TestHelpers

* Updates driver tests to match new arch

* Start on making all driver tests test all drivers

* Improves handling if combining marks.

* Fix unit tests fails.

* Fix unit tests fails.

* Handling combining mask.

* Tying to fix this unit test that sometimes fail.

* Add support for combining mask on NetDriver.

* Enable CombiningMarks as List<Rune>.

* Prevents combining marks on invalid runes default and space.

* Formatting for CI tests.

* Fix non-normalized combining mark to add 1 to Col.

* Reformatting for retest the CI.

* Forces non-normalized CMs to be ignored.

* Initial experiment

* Created ANSiDriver. Updated UI Catalog command line handling

* Fixed ForceDriver logic

* Fixed ForceDriver logic

* Updating P/Invoke

* Force16 colors WIP

* Fixed 16 colo mode

* Updated unit tests

* UI catalog tweak

* Added chinese scenario from bdisp

* Disabled AnsiDriver unit tests for now.

* Code cleanup

* Initial commit (fork from v2_fixes_2610_WT_VTS)

* Code cleanup

* Removed nativemethods.txt

* Removed not needed native stuff

* Code cleanup

* Ensures command line handler doesn't eat exceptions

---------

Co-authored-by: BDisp <[email protected]>
@tig tig added this to the v2 Release milestone Jul 11, 2024
@tig tig mentioned this issue Aug 19, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
Terminal.Gui V2 Initial Release
Status: No status
Terminal.Gui V2 Beta
Status: 🆕 Not Triaged
Milestone
V2 Release
Development

No branches or pull requests
1 participant
@tig