[Feautre Reuqest] [Bug] Async + RAGAS

[logan-markewich]

Describe the Feature
As far as I can tell, it seems like there is a lack of async functions in ragas, which leads to a lot of nesting async issues (since sync functions are calling async functions somewhere in the stack). Furthermore, nest_asyncio.apply() is automatically called in the import stack, which breaks a lot of applications

File "/usr/local/lib/python3.10/site-packages/ragas/executor.py", line 15, in <module>
2025-01-07 23:06:08     nest_asyncio.apply()

It would be awesome to have async functions like aevaluate instead of evaluate -- the approach I might take is make everything async first, and then get/create an event loop and run it there for the sync versions to avoid code duplication.

At the very least, removing the automatic nest_asyncio.apply() might be a good idea?

Why is the feature important for you?
Async code is the backbone of production systems -- without it, ragas will block the event loop of the server

Additional context
I THINK the workaround here for now is doing something like

def run_ragas(...):
  from app.evaluation.ragas import generate_dev_set, generate_dev_set_csv
  # other ragas imports

# Then, later on in your code
result = await asyncio.to_thread(run_ragas(...))

Which delays the ragas imports and runs it behind an isolated thread, which should be safe. But obviously not ideal

[jjmachan]

hey @logan-markewich 🙂 👋🏽

like you suggested the main culprit is the

File "/usr/local/lib/python3.10/site-packages/ragas/executor.py", line 15, in <module>
2025-01-07 23:06:08     nest_asyncio.apply()

now we added this as a dirty fix for the executor thread which was responsible for handling the logic "he approach I might take is make everything async first, and then get/create an event loop and run it there for the sync versions to avoid code duplication."

I'll do a bit more experiments and testing to remove the need for nesting_asyncio call.

[jjmachan]

OpenHands started fixing the issue! You can monitor the progress here.

[jjmachan]

OpenHands started fixing the issue! You can monitor the progress here.

[jjmachan]

OpenHands started fixing the issue! You can monitor the progress here.