Releases: LanguageMachines/ucto
Releases · LanguageMachines/ucto
v0.9.2
v0.9.1
v0.9.0
Major update
- now use uctodata for language specific information
ucto itself only supports a generic tokenizer - interactive use now uses readline library
- accept long options --help and --version
- UTF16BE now works
- better support for crooked Windows files in general
- added a --normalize option to map tokens in a certain TokenClass
to it's generic name