Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ISO-8859-1 encoding for testcase json files #100

Open
Eorlariel opened this issue Oct 17, 2024 · 1 comment
Open

Support ISO-8859-1 encoding for testcase json files #100

Eorlariel opened this issue Oct 17, 2024 · 1 comment
Labels
lobster-json Affects JSON integration

Comments

@Eorlariel
Copy link

Eorlariel commented Oct 17, 2024

Currently umlaute as "ä", "ö" a.s.o are failing in lobster-json if the json file is saved with encoding ISO-8859-1, because lobster-json is trying to read it with utf-8 encoding. lobster-json should not fail in these cases, but should support also other encodings.

This may be a solution on how to detect the encoding:
https://www.powershellgallery.com/packages/poshfunctions/2.2.1.1/content/functions/get-fileencoding.ps1

Acceptance Criterias:
lobster doesn't fail when using "Umlaute" in testcase json files in common encodings like:

  • UTF8
  • ISO-8859-1
  • UTF16 LE
  • UTF16 BE
  • Windows 1252
@phiwuu phiwuu added the lobster-json Affects JSON integration label Oct 17, 2024
@phiwuu
Copy link
Member

phiwuu commented Oct 17, 2024

One possibility is to use this code snippet to guess the encoding with a certain confidence:

import chardet

with open('example.txt', 'rb') as file:
    result = chardet.detect(file.read())
    encoding = result['encoding']
    confidence = result['confidence']

print(f"The file is encoded in '{encoding}' with confidence {confidence * 100:.2f}%.")

If the confidence is above a threshold, we could take it as granted. We could add a command line flag like --detect-encoding=80 to specify that the encoding shall be detected, and that the confidence level must be at least 80%.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lobster-json Affects JSON integration
Projects
None yet
Development

No branches or pull requests

2 participants