[$4000] OnxyData is being dropped from the notification payload

[kavimuru]

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!

---

Action Performed:

1. Open mweb chrome from User A account
2. Go to chat section with User B
3. Type in multi lines (6-7 lines) of emoji within a single message and send it to User B (Make sure to login with user B on android)
4. Notice that multilines of emojis sent from user A doesn’t show on notification on user B android

Expected Result:

Sending multi lines of emoji within a single message should be shown in the notification section like texts

Actual Result:

Sending multi lines of emoji within a single message is not shown on the notification

Workaround:

unknown

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android / native

Version Number: 1.2.63-0
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos:

emojivideo.mp4

az_recorder_20230131_170058.1.mp4

Expensify/Expensify Issue URL:
Issue reported by: @priya-zha
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1675166101726659

View all open jobs on GitHub

Tasks

• User created room notification has incorrect title #18071
• https://github.com/Expensify/Expensify/issues/286640
• https://github.com/Expensify/Expensify/issues/286641
• https://github.com/Expensify/Expensify/issues/286643

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~0145bce78b355295bd
• Upwork Job ID: 1621105271543308288
• Last Price Increase: 2023-02-16

[melvin-bot]

Triggered auto assignment to @dylanexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[dylanexpensify]

reviewing

[dylanexpensify]

Confirmed reproduction

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~0145bce78b355295bd

[melvin-bot]

Current assignee @dylanexpensify is eligible for the External assigner, not assigning anyone new.

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @eVoloshchak (External)

[melvin-bot]

Triggered auto assignment to @cead22 (External), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[priyeshshah11]

This could be due to some size limit in notifications' data.

[alexxxwork]

Proposal

Please re-state the problem that we are trying to solve in this issue.

We are trying to resolve an issue in push messages when messages which consist of uncode symbols only gets truncated or just don't come at all

What is the root cause of that problem?

I've made some investigations but I haven't got full clear view of the root cause. I suggest that the problem is on the server side.
There's native android code for custom airship listener and provider, but I can't see any message length restrictions:

App/android/app/src/main/java/com/expensify/chat/customairshipextender/CustomNotificationProvider.java

Line 160 in aea38c5

private void applyMessageStyle(@NonNull Context context, NotificationCompat.Builder builder, JsonMap payload, int notificationID) {

So I think it's a problem in airship service.

Expensify uses urbanairship lib to handle notifications

App/src/libs/Notification/PushNotification/index.native.js

Line 59 in aea38c5

UrbanAirship.addListener(EventType.PushReceived, (notification) => {

It seems to truncate ordinary text to 112 ascii symbols like this:
⁦1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012..."
but can't truncate emojis
if a line ends with other symobls - it will be displayed in notification
but if there's only emojis - you can recieve only 28 emojis in notification which is two lines of 14 symobls (may be device specific)

It seems to be the limits of urbanairship implementation as documented here:
https://support.airship.com/hc/en-us/articles/213491643-What-are-the-Maximum-Characters-for-Push-Notifications-

What changes do you think we should make in order to solve the problem?

I would make a service request in airship service using Expensify airship agreement credentials. Not shure this could be done by an external contributor

[dylanexpensify]

No proposals

[alexxxwork]

No proposals

I think the problem is on the server side (airship push server).

[cead22]

Sorry I haven't looked into this but I'll prioritize it tomorrow

[alexxxwork]

@dylanexpensify updated my proposal with the template #14715 (comment). Maybe this makes more sense now :)

[cead22]

I think we should confirm if this is in fact an airship limitation. I think it's possible that we're exceeding 37-44 maximum character limit when sending more than 45 emojis, each of which has a string length > 2 because they're unicode characters

Once we confirm that's the case by finding the limit for an android device, and confirming a message with 1 less character makes the notification work, then I think we can discuss whether it's worth truncating these messages ourselves, and how to do it

[cead22]

I wrote the previous comment assuming the column notification drawer SystemNotification was the right one, but is that the one to look at or Notification drawer DefaultNotification Big Text Style? Lock Screen Portrait has the same values as notification drawer SystemNotification

[alexxxwork]

@cead22 oh, thanks for this addition, I just didn't realized that there are more than 2-byte emojis. I'll make additional tests and post the results.

[cead22]

I thought I remembered there being some with length = 4, but I was very surprised by the one that returns 11

[alexxxwork]

I thought I remembered there being some with length = 4, but I was very surprised by the one that returns 11

And there could be more when you change the skin color.
Here's 15-bytes: `👩🏿‍❤️‍💋‍👨🏾`.length (skin color - black)

[Julesssss]

Okay, back on this today.

Setup steps for debugging Android notifications

• Enable debugging
• Replace the production Airship keys with debug
• Create an Android release build from Android studio

Notes
Small messages are correctly retrieved, handled, and threaded.

At a certain point (once the message payload exceeds the current ~2kb limit, we strip onyxData from the payload data, and we fall back to the default notification styling (no threading).

It is extremely easy to hit the current 2kb limit, a message of just 15 characters triggers the onyx data to be stripped from a group chat message.

Message	payload size (b)
1 char	1899
15 char	1994
50 chars	2242
500 chars	4977

[Julesssss]

While this is quite alarming, we can make some pretty vast improvements:

1. Stop removing the onyxData once the payload size hits 2kb. FirebaseCloudMessaging has a limit of 4kb, and Ryan (from Airship) said that we need to leave some allowance for their metadata, so maybe we can stretch to ~3.5kb
2. Use gzip to compress the payload. Ryan previously looked into this for us, so i will follow up
3. Remove duplicate data from the payload. We're sending the message (which makes up the majority of the data) twice. Once for the notification title, and once within the onyxData. By switching to onyxData for the source, we can vastly reduce the payload size

• I have broken down the individual key sizes and will share the breakdown of this tomorrow
• Very early PR for the App changes

[Julesssss]

What is causing us to exceed the payload limit?

Here's a breakdown of a few example notifications. As a reminder, any fullsize over 2000 causes the onyxData to be removed from the notification payload

Example	payload size (b)
10 chars
10 emoji
100 chars
500 chars
1000 chars

• Notice that the reportAction is roughly 40+% of the payload, this is not used in NewDot!
• We are supposed to first try to remove the reportAction, but this doesn't appear to be working

[Julesssss]

I went a bit further and sized up the individual onyxData fields:

• For newDot, we can completely drop reportAction for an immediate 40% increase as it is a duplicate of data already included in the onyxUpdate
• Within the Onyx update, message and originalMessage are the largest fields by far, followed by person, lastMessageText, and avatar

Example	payload size (b)
100 chars
500 chars
1000 chars

[Julesssss]

Okay, so what's the solution?

I think we should do the following, in this order:

0. Build the notification from onyxData, to free up payload space (complete ✅ )
1. Increase the payload size limit from 2kb to 3.5kb (WIP PR 🚧 )
2. Strip the 'reportAction' data from newDot notifications, strip onyxData from oldDot notifications (WIP PR 🚧)
3. Get Airship to implement a payload compression feature into their API (I have reached out to Ryan from Airship to see if the feature is ready for us to use 🚧)
4. Pause to review. ^ This should prevent the majority of messages from having their onyxData removed, but we can make further improvements...
5. Introduce a max message length of 500-1500 chars depending on the new payload size after implementing 1, 2, and 3
6. Introduce an API error when the payload exceeds our 3.5kb limit to help us track occurrences and adjust the max message length
7. If reducing the max message length isn't enough, strip unnecessary fields from onyx payload and show the message as pending until the openReport request completes (avatar and person are likely already stored locally)

[Julesssss]

Here's a WIP PR for solving tasks 1 and 2.

[Julesssss]

I didn't have much time for this after the mobile deployer duty today, but I was able to resolve some style/psalm issues and applied some change suggestions form the WIP PR.

I also prepared custom iOS and Android builds so that I can verify the Web-E changes tomorrow.

[Julesssss]

Awesome, I'm seeing great results from solutions 0-2 listed above... once we start zipping the payload we'll have an even larger message length limit:

onyxPayload-appBackgrounded.mp4

[Julesssss]

The first PR is in review.

[Julesssss]

Here are the results of our first improvements. Notifications for messages up to roughly 980 chars in length will now persist the message successfully into onyx, up from 25!

Example	old size (b)	old wasPersisted (x < 2000)	new size (b)	new wasPersisted (x < 3500)
10 chars	1869	✅	1256	✅
10 emoji	2639	❌	1794	✅
100 chars	2490	❌	1600	✅
500 chars	4884	❌	2803	✅
1000 chars	7890	❌	4303	❌

[melvin-bot]

@Julesssss, @jliexpensify, @dylanexpensify Whoops! This issue is 2 days overdue. Let's get this updated quick!

[Julesssss]

The first PR resolved issues https://github.com/Expensify/Expensify/issues/286640 &
https://github.com/Expensify/Expensify/issues/286641

Continuing with improved logging next

[melvin-bot]

@Julesssss, @jliexpensify, @dylanexpensify Uh oh! This issue is overdue by 2 days. Don't forget to update your issues!

[Julesssss]

I think we could close this one now. We'll create follow up issue for further improvements, but the major issue has been resolved.

[alexxxwork]

@alexxxwork thanks for your help so far. We're going to take this internal and work with Callstack and Airship to resolve this given that it has gone 4+ weeks, but we'll be compensating you for your help so far

@cead22 @dylanexpensify It this issue still eligible for compensation?

[priya-zha]

@cead22 @dylanexpensify I'm the external reporter for this issue.

[Julesssss]

@dylanexpensify could you please pay out the reporting bonus and 'compensation' based on this comment please.

Sorry all, I have been working on similar issues for a while and didn't realise it was initially raised externally.

[dylanexpensify]

Upwork job here!

@priya-zha please apply for reporting!
@alexxxwork please apply for the work you've done for $500!

[priya-zha]

@dylanexpensify Proposal submitted. Thanks.

[alexxxwork]

@dylanexpensify applied, thanks

[dylanexpensify]

@alexxxwork offer sent!

[Julesssss]

@dylanexpensify is this paid out? If so can we close it please

[dylanexpensify]

Done!