Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RunAsync in service abstractions doesn't handle failing actions correctly #564

Open
K-Cully opened this issue Apr 25, 2023 · 0 comments
Open
Labels
bug Something isn't working

Comments

@K-Cully
Copy link
Member

K-Cully commented Apr 25, 2023

The use of WhenAll in https://github.com/microsoft/Omex/blob/main/src/Hosting.Services/OmexStatelessService.cs and it's Stateful equivalent means that if an action throws, the action ends but the service replica is not restarted, since WhenAll waits for all other tasks to end also.
This prevents registering of more than one action which should cause a service replica restart on failure and leaves the developer with only two options in such cases:

  1. Swallow exceptions, log failures and rely passive monitoring to alert on the service failures. This would require manual intervention to recover.
  2. Add complex task management logic to the service code, wrapping multiple actions into a single action for registration with the service instance.
    Neither of these options is a good solution.

A better approach would be to update these service classes to apply WhenAny to the enumeration of tasks.
If any task throws a non-recoverable exception, an appropriate exception would be thrown here to cause the service replica to recycle and the remaining tasks would be cancelled/ended.
If a single task ends gracefully, remaining tasks are waited using WhenAny again.

This would allow for a single action throwing to trigger service recovery, while also allowing for graceful ending of actions.

Relevant reference docs:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicefabric.services.runtime.statelessservice.runasync?view=azure-dotnet#remarks
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicefabric.services.runtime.statelessservice.runasync?view=azure-dotnet#microsoft-servicefabric-services-runtime-statelessservice-runasync(system-threading-cancellationtoken)

@K-Cully K-Cully added the bug Something isn't working label Apr 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant