Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop python process from exiting too early with async tracker still having data to send #1214

Open
skrawcz opened this issue Nov 3, 2024 · 0 comments
Labels
async bug Something isn't working SDK Related to hamilton SDK for th UI

Comments

@skrawcz
Copy link
Collaborator

skrawcz commented Nov 3, 2024

Current behavior

If you're in aws lamdba, the lambda can kill the event loop before we manage to send all events.
This means that things complete, but we're missing data for a few nodes/functions.

Steps to replicate behavior

This is hard to recreate locally unless you kill the process immediately after the driver finished executing.

  1. run async driver logging to tracker
  2. immediate kill process

Library & System Information

latest

Expected behavior

The tracker should be able to register something with python to say "hey run this before the process closes" to ensure the events get sent -- we do this with registering stop() with the sync client.

Additional context

Asyncio and atexit don't seem to play well together. There seems to be a library that could help. But yeah could be easy, or could be digging into the internals of python to figure this out.

@skrawcz skrawcz added bug Something isn't working SDK Related to hamilton SDK for th UI async labels Nov 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
async bug Something isn't working SDK Related to hamilton SDK for th UI
Projects
None yet
Development

No branches or pull requests

1 participant