Deadlocks when using @Transactional on top of Kotlin's suspend function

[krystofrezac]

Expected Behavior

If I use @Transactional annotation on top of suspend function, no deadlock occurs.

Actual Behaviour

If I run a suspend function annotated with @Transactional with enough parallel processing(>= number of coroutine pool threads + size of db pool), then the application gets deadlocked. After some time(datasources.default.connection-timeout) the deadlock is released.

Hypothesis:
The function annotated with @Transactional runs N times, which depletes the connection pool. Following calls to this function get "blocked" in TransactionalInterceptor, which results in blocking the thread.

The functions that were able to run get suspended, which frees the coroutine thread pool and another request starts to be processed. The request processing calls the suspend function again and gets "blocked" in TransactionalInterceptor because the connection pool is still depleted.

And because all threads are being "blocked" by the TransactionalInterceptor the suspended function cannot resume execution and cannot free the db connection.

Potential fix
We were able to mitigate this problem by switching to Dispatchers.IO before the TransactionalInterceptor, you can see that in the example repository(AntiDeadlockTransactionalInterceptor), but we observe problems from time to time with releasing the db connectionv - Got unhandled exception: Cannot activate transaction synchronization - already active" micronaut.application.name=xxx stack_trace="java.lang.IllegalStateException: Cannot activate transaction synchronization - already active

Video
https://github.com/user-attachments/assets/c80499bb-9198-46be-81b2-e5376a915ece

Video with the potential fix
https://github.com/user-attachments/assets/8d025d78-d628-429c-bdbb-5d516cc869f8

Steps To Reproduce

1. Clone the example repository (https://github.com/krystofrezac/micronaut-deadlock-example)
2. Run k6 script - k6 run k6-script.js
3. Watch the logs, they will freeze for 30 seconds, and then you will see timeout errors

Environment Information

• OS: MacOS
• java: openjdk 21.0.5 2024-10-15 LTS

Example Application

https://github.com/krystofrezac/micronaut-deadlock-example

Version

4.7.1