Synchronization fails when for instance DESCRIPTION is too large

[SergeyPetrov98]

Problem scope

• I'm sure that this is a DAVx⁵ problem.

App version

• I'm using the latest available DAVx⁵ version.

Android version and device/firmware type

Android 14/AP2A.240905.003.F1.ova.2024101100

Steps to reproduce

1. Attach DAVx5 to some sync server(for example, Exchange ActiveSync)
2. Create an event which contains a big picture in the description field (if you have no such a picture at hand, use this: 4k-pc-wallpaper-ai-art.webp)
3. Create a couple of ordinary events.
4. Sync the calendar on your device.

Actual result

DAVx5 silently stops the synchronization at the event with the big picture. As a result, neither the event with the big picture, nor the events follow it does not arrive to the device.

Expected result

At least DAVx5 should throw an exception, as it did before 4.4.3.2-ose (see the log in the Further info).

Further info

Here is, a patch which cuts large messages. Cutting the messages considered as an alternative to throw an exception, as on the one hand, the patch shows the user that something went wrong, but on the other hand, it allows DAVx5 to finish the synchronization even if it encounters malformed messages.

Here is the log:

10-29 12:20:02.593 27714 28694 I davx5   : , duration=null, exDates=[], exRules=[], exceptions=[], lastModified=null, location=ВКС, opaque=true, organizer=null, rDates=[], rRules=[], recurrenceId=null, status=null, summary=TEST DAVx5, unknownProperties=[PRIORITY:5
10-29 12:20:02.593 27714 28694 I davx5   : , X-MICROSOFT-CDO-BUSYSTATUS:BUSY
10-29 12:20:02.593 27714 28694 I davx5   : ], url=null, sequence=2, uid=ebbd2784-0e42-4f17-b16b-039ca046843b, userAgents=[], shadow$_klass_=class at.bitfire.ical4android.Event, shadow$_monitor_=0]
10-29 12:20:02.608 27714 28694 W BpBinder: Large outgoing transaction of 9508120 bytes, interface descriptor , code 20
10-29 12:20:02.609 27714 28694 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 9508120)
--------- beginning of kernel
10-29 12:20:02.634 28694 28694 I binder_alloc: 27782: binder_alloc_buf size 9508128 failed, no address space
10-29 12:20:02.634 28694 28694 I binder_alloc: allocated: 0 (num: 0 largest: 0), free: 1040384 (num: 1 largest: 1040384)
10-29 12:20:02.634 28694 28694 I binder  : 27714:28694 transaction failed 29201/-28, size 9508120-8 line 3456
10-29 12:20:02.611 27714 28694 W ical4android: [BatchOperation] Transaction too large, splitting (losing atomicity)
10-29 12:20:02.678 28694 28694 I binder_alloc: 27782: binder_alloc_buf size 9506872 failed, no address space
10-29 12:20:02.652 27714 28694 W BpBinder: Large outgoing transaction of 9506864 bytes, interface descriptor , code 20
10-29 12:20:02.678 28694 28694 I binder_alloc: allocated: 0 (num: 0 largest: 0), free: 1040384 (num: 1 largest: 1040384)
10-29 12:20:02.653 27714 28694 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 9506864)
10-29 12:20:02.678 28694 28694 I binder  : 27714:28694 transaction failed 29201/-28, size 9506864-8 line 3456
10-29 12:20:02.656 27714 28694 W ical4android: [BatchOperation] Transaction too large, splitting (losing atomicity)
10-29 12:20:02.661 27714 28694 W BpBinder: Large outgoing transaction of 9505936 bytes, interface descriptor , code 20
10-29 12:20:02.662 27714 28694 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 9505936)
10-29 12:20:02.664 27714 28694 W ical4android: [BatchOperation] Transaction too large, splitting (losing atomicity)
10-29 12:20:02.687 28694 28694 I binder_alloc: 27782: binder_alloc_buf size 9505944 failed, no address space
10-29 12:20:02.687 28694 28694 I binder_alloc: allocated: 0 (num: 0 largest: 0), free: 1040384 (num: 1 largest: 1040384)
10-29 12:20:02.687 28694 28694 I binder  : 27714:28694 transaction failed 29201/-28, size 9505936-8 line 3456
10-29 12:20:02.669 27714 28694 W BpBinder: Large outgoing transaction of 9505644 bytes, interface descriptor , code 20
10-29 12:20:02.669 27714 28694 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 9505644)
10-29 12:20:02.695 28694 28694 I binder_alloc: 27782: binder_alloc_buf size 9505656 failed, no address space
10-29 12:20:02.695 28694 28694 I binder_alloc: allocated: 0 (num: 0 largest: 0), free: 1040384 (num: 1 largest: 1040384)
10-29 12:20:02.695 28694 28694 I binder  : 27714:28694 transaction failed 29201/-28, size 9505644-8 line 3456
10-29 12:20:02.671 27714 28694 E davx5   : [syncadapter.SyncManager] Couldn't access local storage
10-29 12:20:02.671 27714 28694 E davx5   : EXCEPTION at.bitfire.ical4android.CalendarStorageException: Can't transfer data to content provider (too large data row can't be split)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:155)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:144)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:144)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:147)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.ical4android.BatchOperation.commit(BatchOperation.kt:132)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.ical4android.AndroidEvent.update(AndroidEvent.kt:221)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.processVEvent$lambda$14(CalendarSyncManager.kt:58)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.$r8$lambda$yYaBcOPOfnfThZ0SSno9xPavRpk(CalendarSyncManager.kt:1)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager$$ExternalSyntheticLambda8.invoke(R8$$SyntheticClass:14)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.localExceptionContext(SyncManager.kt:8)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.processVEvent(CalendarSyncManager.kt:146)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.downloadRemote$lambda$11$lambda$10$lambda$9(CalendarSyncManager.kt:114)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.$r8$lambda$oUmR5-GxgmED6lAOvm6yA_DQLKA(CalendarSyncManager.kt:1)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager$$ExternalSyntheticLambda5.invoke(R8$$SyntheticClass:7)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.responseExceptionContext(SyncManager.kt:13)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.downloadRemote$lambda$11$lambda$10(CalendarSyncManager.kt:22)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.$r8$lambda$Li77wWzMaCPpoaV87DiqsWgoOsA(CalendarSyncManager.kt:1)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager$$ExternalSyntheticLambda3.onResponse(R8$$SyntheticClass:3)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.dav4jvm.Response$Companion.parse(Response.kt:311)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.dav4jvm.DavResource.processMultiStatus$parseMultiStatus(DavResource.kt:51)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.dav4jvm.DavResource.processMultiStatus(DavResource.kt:16)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.dav4jvm.DavResource.processMultiStatus(DavResource.kt:9)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.dav4jvm.DavCalendar.multiget(DavCalendar.kt:69)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.dav4jvm.DavCalendar.multiget$default(DavCalendar.kt:12)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.downloadRemote$lambda$11(CalendarSyncManager.kt:28)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.$r8$lambda$zO7xd3T8fuN76tJe1J-6n13zUxo(CalendarSyncManager.kt:1)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager$$ExternalSyntheticLambda1.invoke(R8$$SyntheticClass:7)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.remoteExceptionContext(SyncManager.kt:1)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.remoteExceptionContext(SyncManager.kt:5)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncManager.downloadRemote(CalendarSyn
10-29 12:20:02.671 27714 28694 E davx5   : cManager.kt:46)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager$syncRemote$1$download$1.invokeSuspend(SyncManager.kt:14)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:9)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:111)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:143)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source:105)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source:3)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.syncRemote(SyncManager.kt:39)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.performSync$lambda$3(SyncManager.kt:398)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.$r8$lambda$MpzTclS4zDl6HCoNNJ-PSpp6k_Y(SyncManager.kt:1)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager$$ExternalSyntheticLambda10.invoke(R8$$SyntheticClass:3)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.unwrapExceptions(SyncManager.kt:2)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.SyncManager.performSync(SyncManager.kt:20)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.CalendarSyncer.sync(CalendarSyncer.kt:175)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.Syncer.onPerformSync(Syncer.kt:125)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.BaseSyncWorker$doSyncWork$2.invokeSuspend$lambda$1$lambda$0(BaseSyncWorker.kt:16)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.BaseSyncWorker$doSyncWork$2.$r8$lambda$15kebWhEyOIttFHdUJpWmlcwh04(BaseSyncWorker.kt:1)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.BaseSyncWorker$doSyncWork$2$$ExternalSyntheticLambda0.invoke(R8$$SyntheticClass:13)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.InterruptibleKt$runInterruptible$2.invokeSuspend(Interruptible.kt:75)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.InterruptibleKt$runInterruptible$2.invoke(Interruptible.kt:13)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:16)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.BuildersKt.withContext(Unknown Source:41)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.InterruptibleKt.runInterruptible$default(Interruptible.kt:9)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.davdroid.syncadapter.BaseSyncWorker$doSyncWork$2.invokeSuspend(BaseSyncWorker.kt:496)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:9)
10-29 12:20:02.671 27714 28694 E davx5   : 	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:111)
10-29 12:20:02.671 27714 28694 E davx5   : 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
10-29 12:20:02.671 27714 28694 E davx5   : 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
10-29 12:20:02.671 27714 28694 E davx5   : 	at java.lang.Thread.run(Thread.java:1012)
10-29 12:20:02.671 27714 28694 E davx5   : Caused by: android.os.TransactionTooLargeException: data parcel size 9505644 bytes
10-29 12:20:02.671 27714 28694 E davx5   : 	at android.os.BinderProxy.transactNative(Native Method)
10-29 12:20:02.671 27714 28694 E davx5   : 	at android.os.BinderProxy.transact(BinderProxy.java:586)
10-29 12:20:02.671 27714 28694 E davx5   : 	at android.content.ContentProviderProxy.applyBatch(ContentProviderNative.java:635)
10-29 12:20:02.671 27714 28694 E davx5   : 	at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:574)
10-29 12:20:02.671 27714 28694 E davx5   : 	at android.content.ContentProviderClient.applyBatch(ContentProviderClient.java:562)
10-29 12:20:02.671 27714 28694 E davx5   : 	at at.bitfire.ical4android.BatchOperation.runBatch(BatchOperation.kt:62)
10-29 12:20:02.671 27714 28694 E davx5   : 	... 57 more

[rfc2822]

Thanks for the report!

Here is, a patch which cuts large messages.

This will only apply to the DESCRIPTION. Unfortunately the Android IPC limit of 1 MB applies to the whole iCalendar. So if we have for instance a 500 kB DESCRIPTION and a 500 kB ATTACHment, it would still be a problem.

So we have two problems:

1. the actual problem that we can't overcome the 1 MB IPC limit when talking to the calendar provider (I don't know a solution for that),
2. that the TransactionTooLargeException is not shown anymore, as you say.

The latter shouldn't happen, at least there was no intention to change it and if it's true, the error should be shown again.

[SergeyPetrov98]

Hi! Having done some tests on iCalendar, I found that ATTACHments are always removed regardless of their sizes so that we may focus on the DESCRIPTION field only.

As it was indicated, there are two problems and, as far as I understand, the second one is going to be fixed.

Let us proceed to the first indicated problem. You said that you didn't know how to overcome the 1 MB IPC limit when talking to the calendar provider. It appears to me that, if we cut the DESCRIPTION field so that its contents fit into the IPC buffer, we can overcome the 1 MB IPC limit. Of course, there must be a way to notify the user that some events were cut. I would suggest to put the notification right in the DESCRIPTION field. Here is the patch which implements such behavior. Let me provide you with examples of its work:

1. A too large DESCRIPTION (big_text_input.txt) was shortened (text_input_output.txt) and at the end the notification was put:

...
Integer suscipit dignissim ligula. Aenean in odio nec ipsum ultrices euismod non non ipsum. Nullam lacus sapien, placerat eu risus ac, feugiat maximus erat. Suspendisse elit neque, maximus in libero sit amet, lacinia finibus erat. Donec semper felis nisi. Ut mattis nisl justo, sit amet fermentum leo semper placerat. Aenean fermentum arcu vitae neque faucibus, vel elementum sem malesuada. Pellentesque eu libero dolor. Sed ullamcorper congue felis ullamcorper fermentum. Nulla id sem vitae sapien consectetur viverra. In placerat est sit amet leo tempor porta. Aliquam sagittis maximus eleifend. Ut lacus tellus, eleifend ac felis vitae, maximus rutrum to

===== Base64 sipped by DavX =====

2. In a DESCRIPTION containing a mixture of Base64 and ordinary text (base64_input.txt), all the Base64 was substituted with the notification:

This is begin of a event description

===== Base64 sipped by DavX =====

This is end of a message

As a result, both the events although being modified reached the device.

Could you share your opinion on the approach to overcome the 1 MB IPC limit and the patch implementing it?

[rfc2822]

As it was indicated, there are two problems and, as far as I understand, the second one is going to be fixed.

This problem is now recorded in #1138.

Hi! Having done some tests on iCalendar, I found that ATTACHments are always removed regardless of their sizes so that we may focus on the DESCRIPTION field only.

It was just an example. You can for instance take a 500 kB SUMMARY and a 500 kB LOCATION, even without a DESCRIPTION and it will still cause a TransactionTooLargeException.