(Try to) fix Utf8

[bynect]

The aim of this pr is to add a way to validate utf8 and to handle "gracefully" invalid sequences, by replacing them with a placeholder character.

[bynect]

I was not sure on how to handle high/low surrogates so I added a config flag, but in most cases they should be interpreted as invalid

[bynect]

This may also be related with #40

[arthurbacci]

Why to check twice instead of checking just on utf8ToMultibyte and replacing it if it's invalid?

[bynect]

Why to check twice instead of checking just on utf8ToMultibyte and replacing it if it's invalid?

Where?

[arthurbacci]

You can detect it only when displaying.

[bynect]

You can detect it only when displaying.

You have to check when reading because the next characters shouldn't be consumed if the codepoint is malformed. The check is just used to ungetc the characters. In this way malformed utf8 can be stored without losing its original value, at least for what I have tried.

[bynect]

Now I am adding validation for user input