Deadlock in web sync agent

[radulacatus]

Setup:

• DownloadOnly using web orchestrator
• Server using SqlServer database:
  • Dotmim.Sync.Core 1.0.2
  • Dotmim.Sync.SqlServer.ChangeTracking 1.0.2
  • Dotmim.Sync.Web.Server 1.0.2
• Client using SqlServer database:
  • Dotmim.Sync.Core 1.0.2
  • Dotmim.Sync.Sqlite 1.0.2
  • Dotmim.Sync.SqlServer.ChangeTracking 1.0.2
  • Dotmim.Sync.Web.Client 1.0.0

Our client runs in an on premise network and we need to be resilient to internet downtime.
Calls to SynchronizeAsync are executed sequentially per scope inside a scheduled message handler.
The message handle has a timeout and cancels the cancellation token source if the sync call takes too long.

var progress = new SynchronousProgress<ProgressArgs>(args =>
logger.LogDebug($"{args.ProgressPercentage:p}:  \t[{args.Source[..Math.Min(4, args.Source.Length)]}] {args.TypeName}: {args.Message}"));
var res = await syncAgent.SynchronizeAsync(scopeName, null, syncType, null, cancellationToken, progress);

Most of the time everything works as expected. Even, cancelling the sync and retrying with a backoff.
Sometimes, the progress get's stuck at 0%, cancelling the token doesn't result in an interruption and the sync can only be unlocked by restarting the process.
This looks like a deadlock in the SyncAgent when using WebRemoteOrchestrator.
The same process does not hang when we used LocalOrchestrator, but this is not an option for us anymore.
We could not reproduce this issue in a POC, so maybe it's also related to the bigger volume of data in our production environment.

We noticed that SynchronousProgress is using SynchronizationContext.
Could it help if we don't use it anymore?
Do you have any other suggestions?

[Mimetis]

Dont use SynchronousProgress if it's not in a console application.
Prefer use your own IProgress object

[radulacatus]

We tried it out. We used our own implementation of IProgress, both serverside and clientside.
The issue still reproduces.

[Mimetis]

Okay, can you share a simple application, like a github repository, where you can share a simple application reproducing the error ?
I see that this error is not happening every time, so I guess it can be difficult to reproduce.

The idea is to have something I can work on.

See an example where a github repo has been shared with an sql script to create the environment : #1286 (comment)

[radulacatus]

I created a repo HERE.
Unfortunately with a POC like this i was not able to reproduce the issue.

[BogdanGeorge]

Another thing maybe worth mentioning, we create a singleton SyncAgent when we initialize our hosting service.
Over the entire lifecycle of the service we do multiple syncronizations and all of them are using the same SyncAgent with the same WebRemoteOrchestrator and the same HttpClient used by the web orchestrator since there is no way to provide a HttpClientFactory.
Are the SyncAgent and WebRemoteOrchestrator meant to be used as singletons, and reused over multiple syncronizations ? Could this be the reason leading to deadlock ?