# -*- coding: utf-8 -*-
Mojibake occurs in English most frequently due to misinterpreting and bad-transcoding between Windows-1252, ISO-8859-1, and UTF-8. This module provides a mojibake sequence to original character mapping table, and utility to recover mojibake’d text.
Testing has been with English but other Latin based languages, where Windows-1252 is in the wild, should also benefit.
Mojibake mappings generation requires the String encoding support in ruby 1.9 as provided by:
-
ruby 1.9.2+ (tested 1.9.2p290, Linux)
-
jruby 1.6.5+ (tested 1.6.5, Linux)
-
ruby 2.0.0 (tested 2.0.0p0, Linux)
Recovery of text with default settings is supported by using the pre-generated table.json when in ruby 1.8 mode.
gem install mojibake require 'mojibake' mapper = MojiBake::Mapper.new mapper.recover( '“quotedâ€�' ) #=> '“quoted”'
Or via cli:
mojibake -h
List the mojibake mapping table (output in UTF-8):
mojibake -t
Recover from a text file:
mojibake input.txt
Copyright © 2013 David Kellum, Maciej Kowalski
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at:
www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.