-
Notifications
You must be signed in to change notification settings - Fork 1
/
README
188 lines (133 loc) · 7.44 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
Emoji SMS Fix for the N9/N950 and N900
--------------------------------------
Web: http://thp.io/2014/emojifix/
The Emoji SMS Fix consists of a patcher utility and a drop-in library that can
be used to fix a bug in libsms-utils, causing incoming SMS containing Emoji
characters to be silently dropped.
Thanks to Jonni for the initial inspiration and some pointers.
This fixes delivery of SMS to the SMS application. For actually displaying
Emoji, you need to install Harmoji, which includes suitable fonts.
THE BUG
=======
When an incoming SMS contains a Emoji outside of the Unicode range that is
covered by UCS-2 is received, it's silently dropped by csd:
csd[...]: SMS-lib utils_tpdu_data_parse():
Error decoding TPDUs. Discarding message
Digging deeper into the system libraries, utils_tpdu_data_parse() is found in
libsms.so.0 and from there, we look for UCS-2 related functions:
$ objdump -T /usr/lib/libsms.so.0 | grep -i 'ucs.*2'
000168f8 g DF .text 00000054 Base isi_sms_conv_utf8_to_ucs2be
0001694c g DF .text 00000054 Base isi_sms_conv_ucs2be_to_utf8
Given that the SMS encoding is UCS-2, and the middleware probably works with
UTF-8, isi_sms_conv_ucs2be_to_utf8() seems to be the interesting function
(ucs2be = UCS-2 Big Endian), but a quick search in /usr/lib/ doesn't turn up
anything useful. Digging a little further gets us to libsms-utils.so.0, which
has an interesting function sms_conv_ucs2_to_utf8(), and that one is actually
called when the TPDU error happens.
Breaking on this function in gdb and stepping through it reveals that iconv(3)
is used to convert from UCS-2 to UTF-8, by using iconv_open(3) with "UTF-8" and
"UCS-2" as parameters. This works great when the SMS is encoded in UTF-16, and
all characters are in the unicode code point range 0x0000-0xFFFF. Some Emojis
even are, but not all of them - those that are outside couldn't be encoded by
UCS-2, but they can be encoded by UTF-16 (which is why iOS and Android use it).
THE FIX
=======
The simple fix is to call iconv_open with "UTF-16" as the second argument
instead of "UCS-2" in sms_conv_ucs2_to_utf8(). This could be done with
LD_PRELOAD and wrapping iconv_open(3), but as we don't know all locations where
the library libsms-utils.so.0 is used, we can't really do that reliably (I
tried, and the SMS then failed in telepathy-ring instead, which apparently also
does some parsing using libsms-utils).
So the "easier" way is to modify libsms-utils.so.0 directly in such a way that
when an application is linked against it, our custom iconv_open(3) function is
used (but only for libsms-utils.so.0). Here's how it works normally:
<some application> -> libsms-utils.so.0 -> iconv_open() from glibc
| ^
\------------------------------------------------/
What we want to do is this (xconv_open is a wrapper we write for iconv_open):
<some application> -> libsms-utils.so.0 -> xconv_open()
| |
\-----------> iconv_open() from glibc <---------/
The way we achieve this is by actually creating a drop-in replacement for
libsms-utils.so.0 that just defines our custom iconv_open(3) function, and
then links against a modified version of the original libsms-utils.so.0
(we call the modified version libemojitils.so.0):
<some application> -> drop-in libsms-utils.so.0 -> libemojitils.so.0
(contains only xconv_open) |
^ |
\--------------/
Our custom libsms-utils.so.0 is linked against libemojitils.so.0, so that all
symbols exported by the original libsms-utils.so.0 will be available to the
application at link time (otherwise the app would have unresolved symbols, and
it would not load at all).
Creating libemojitils.so.0 from the original libsms-utils.so.0 is relatively
straightforward:
1. Change the SONAME from libsms-utils.so.0 to libemojitils.so.0
(this is needed, as we'll link against it later on)
2. Change the dynamic symbol name iconv_open() to xconv_open()
(so our wrapper is called every time iconv_open() is called)
3. Remove the symbol version information for iconv_open/xconv_open
The reason for step (3) is that the symbol iconv_open is versioned to the
symbol version GLIBC_2.4, and we have to remove that, otherwise loading it
will fail, as it tries to look for xconv_open() in libc.so.6 instead of
in the global symbols.
The "patcher" utility takes care of those three steps.
The remaining step is to create the drop-in replacement for libsms-utils.so.0
with the following properties:
1. Contains a function xconv_open() with the same signature as iconv_open(3)
2. Links against libemojitils.so.0 (to pull in the real functions)
The wrapper function is straightforward - just replace fromcode with "UTF-16"
when when function is called with tocode="UTF-8" and fromcode="UCS-2":
iconv_t
xconv_open(const char *tocode, const char *fromcode)
{
if (strcmp(tocode, "UTF-8") == 0 && strcmp(fromcode, "UCS-2") == 0) {
fromcode = "UTF-16";
}
return iconv_open(tocode, fromcode);
}
To link against libemojitils.so, we need to make sure the linker includes it
even if it figures out that it's not really needed (our library doesn't use
any of the functions in libemojitils.so.0 directly, so the linker does what
would normally be the "right" thing and avoid the dependency, but we actually
want to have the dependency here):
-Wl,--no-as-needed -L. -lemojitils
After this, we are left with two files:
1. libemojitils.so.0: This is the original libsms-utils.so.0, but it calls
xconv_open() instead of iconv_open(3)
2. libsms-utils.so.0: Contains xconv_open() and pulls in libemojitils.so.0
Deploy those two files to /usr/lib/ on the device (make sure you are using an
aegis-neutered kernel, or otherwise put those files into a libsms-utils0 .deb
and install using any of the "com.nokia.maemo"-origin hacks to avoid MALF).
Reboot the device, and from now on, Emoji SMS should be arriving. To also see
the Emojis (and not just rectangular boxes), install Harmoji, which includes
the right fonts.
SHA1SUMS
========
Original and patched hashes for the N9 and N950 / MeeGo 1.2 "Harmattan":
48bcd471a8f99b0bcb5502eb7d2af32b279778c0 libsms-utils.so.0.0.0
1f6b0ae0af7cb4644da3092e2b7a85d874ed3979 libemojitils.so
Original and patched hashes for the N900 / Maemo 5 "Fremantle":
48bcd471a8f99b0bcb5502eb7d2af32b279778c0 libsms-utils.so.0.0.0
300b7058aacbfa554bb43d3bb0c1bf9d63bf57e4 libemojitils.so
LINKS
=====
Threads on TMO:
http://talk.maemo.org/showthread.php?t=82210
http://talk.maemo.org/showthread.php?t=85257
http://talk.maemo.org/showthread.php?t=93427
http://talk.maemo.org/showthread.php?t=93578
Symbol Versioning:
https://refspecs.linuxfoundation.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/symversion.html
ELF file format header:
http://linux.die.net/man/5/elf
UTF-16, UCS-2 and the BMP:
https://en.wikipedia.org/wiki/UTF-16
https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane
SMS encoding:
http://www.unicode.org/mail-arch/unicode-ml/y2011-m08/0429.html
https://www.twilio.com/engineering/2012/11/08/adventures-in-unicode-sms
http://android.stackexchange.com/questions/29962/what-encoding-is-text-messaging-in
Harmoji:
http://talk.maemo.org/showthread.php?t=86704
https://code.google.com/p/harmoji/