Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support python 3 non-ascii identifiers #128

Open
tyrylu opened this issue Dec 7, 2017 · 1 comment
Open

Support python 3 non-ascii identifiers #128

tyrylu opened this issue Dec 7, 2017 · 1 comment

Comments

@tyrylu
Copy link

tyrylu commented Dec 7, 2017

Hello.
It would be helpful for one of my usecases of baron if it supported non ascii identifiers.
Currently (in 0.6.6 but probably also in the latest master) the following code:
>>> baron.parse("fňuk=42")
Raises the following:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python3\lib\site-packages\baron\baron.py", line 49, in parse
    tokens = tokenize(source_code, False)
  File "C:\Python3\lib\site-packages\baron\baron.py", line 70, in tokenize
    return mark_indentation(inner_group(space_group(_tokenize(group(split(pouet)), print_function))))
  File "C:\Python3\lib\site-packages\baron\spliter.py", line 6, in split
    return list(split_generator(sequence))
  File "C:\Python3\lib\site-packages\baron\spliter.py", line 76, in split_generator
    raise UntreatedError("Untreated elements: %s" % iterator.rest_of_the_sequence().__repr__()[:50])
baron.spliter.UntreatedError: Untreated elements: 'ňuk=42\n'

The reason looks quite clear, during the splitting the condition block with the identifier chars uses ascii only.
It would be probably easy to fix this error without taking into account any python 2.x compatibility, but if it is desired, then it is more complicated.
But anyway, could something be done with it?

@asterbini
Copy link

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants