Introduce HTML5 mode

[bamtor]

(originally reported by [email protected])

When used in conjunction with Chromium, Blink only supports ISO-2022-JP. The detection of other 7-bit encodings and other non-HTML5 encodings should be disabled in CED. We can handle it by introducing HTML5 mode.

[bamtor]

9012c0a handles 7-bit encodings other than ISO-20220-JP so that the result will be ASCII-7BIT instead.

[jungshik]

Sorry that I made a misleading/confusing bug report. Other than UTF-7, ISO-2022-{KR,CN} and HZ-GB are treated as replacement encoding per WHATWG encoding spec. So, if they're detected by CED, Blink will convert the whole input to a single character U+FFFD.

Given this, I think we'd better leave alone the detection of those encodings and let Blink deal with them (convert to U+FFFD).

UTF-7 is a bit tricky. I'm filing a bug against WHATWG encoding spec so that it's treated the same way as ISO-2022-{KR,CN}, HZ-GB. See whatwg/encoding#68

[bamtor]

Let me revert the change. I believe, however, HTML5_MODE is still valid for sanitizing encoding names as filed in #1 I'll keep it and use it for that purpose.

[JinsukKim]

e21eb6a kinds of handled this issue on CED side by returning LATIN if the detected encoding is not supported by WHATWG. This is to make the behavior conform to the standard as well as leave the document intact in such situation.