Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add UTF-8 conformance tests #3

Open
nlohmann opened this issue Jul 25, 2020 · 3 comments
Open

Add UTF-8 conformance tests #3

nlohmann opened this issue Jul 25, 2020 · 3 comments
Labels
help wanted Extra attention is needed

Comments

@nlohmann
Copy link

When implementing nlohmann/json, I struggled a lot with the UTF-8 validation. I'm now pretty sure that I will only accept correctly encoded inputs and reject anything else. It would be great to add such tests to the conformance tests, because it makes a difference whether a parser just copies everything between quotes or actually checks whether the strings make sense. This also includes decoding of \uxxxx escapes.

@Loki-Astari
Copy link
Owner

Loki-Astari commented Jul 29, 2020

There is a specific test suite making sure \uxxxx is decoded correctly.

See: https://github.com/Loki-Astari/JsonBenchmark/tree/master/data/validate_string

Each test file has the format:

{Json Array Containing one String}<{whitespace}>{UTF-8 character string expected result}<

There is a separate set of tests the should fail parsing:

See: https://github.com/Loki-Astari/JsonBenchmark/tree/master/data/jsonchecker_fail

Each test here is a JSON object.
This can be an invalid string that should not parse correctly.

@nlohmann
Copy link
Author

I saw these tests - I think they are still rather cursory. I was more thinking about tests like those in http://seriot.ch/parsing_json.php (see Sect. 2.5).

@Loki-Astari
Copy link
Owner

This links to this github repository with 300 tests:
https://github.com/nst/JSONTestSuite

If anybody wants to add a way to run these test automatically that would be great.

@Loki-Astari Loki-Astari added the help wanted Extra attention is needed label Jul 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants