Deadlocks when using @Transactional on top of Kotlin's suspend function

[krystofrezac]

Expected Behavior

If I use @Transactional annotation on top of suspend function, no deadlock occurs.

Actual Behaviour

If I run a suspend function annotated with @Transactional with enough parallel processing(>= number of coroutine pool threads + size of db pool), then the application gets deadlocked. After some time(datasources.default.connection-timeout) the deadlock is released.

Hypothesis:
The function annotated with @Transactional runs N times, which depletes the connection pool. Following calls to this function get "blocked" in TransactionalInterceptor, which results in blocking the thread.

The functions that were able to run get suspended, which frees the coroutine thread pool and another request starts to be processed. The request processing calls the suspend function again and gets "blocked" in TransactionalInterceptor because the connection pool is still depleted.

And because all threads are being "blocked" by the TransactionalInterceptor the suspended function cannot resume execution and cannot free the db connection.

Potential fix
We were able to mitigate this problem by switching to Dispatchers.IO before the TransactionalInterceptor, you can see that in the example repository(AntiDeadlockTransactionalInterceptor), but we observe problems from time to time with releasing the db connectionv - Got unhandled exception: Cannot activate transaction synchronization - already active" micronaut.application.name=xxx stack_trace="java.lang.IllegalStateException: Cannot activate transaction synchronization - already active

Video
https://github.com/user-attachments/assets/c80499bb-9198-46be-81b2-e5376a915ece

Video with the potential fix
https://github.com/user-attachments/assets/8d025d78-d628-429c-bdbb-5d516cc869f8

Steps To Reproduce

1. Clone the example repository (https://github.com/krystofrezac/micronaut-deadlock-example)
2. Run k6 script - k6 run k6-script.js
3. Watch the logs, they will freeze for 30 seconds, and then you will see timeout errors

Environment Information

• OS: MacOS
• java: openjdk 21.0.5 2024-10-15 LTS

Example Application

https://github.com/krystofrezac/micronaut-deadlock-example

Version

4.7.1

[distinctdan]

I'm also seeing behavior that looks like this issue. If my connection pool size is 10, the first 10 requests work fine, but then they start taking a very long time to return. For my use case, I'm not using @Transactional, but I'm still seeing this issue. It's acting like micronaut-data isn't freeing DB connections when a suspend function finishes. Here's my setup:

open class FooController {
    @Get("/foo")
    suspend fun getFoo(ids: List<Long>) {
        return myRepository.getFoo(ids)
    }
}

@JdbcRepository(dialect = Dialect.POSTGRES)
interface FooRepository : GenericRepository<Foo, Long> {
    @Query(
        """
        SELECT *
        FROM foo
        WHERE id in (:ids)
    """, nativeQuery = true
    )
    suspend fun getFoo(ids: List<Long>): List<Foo>
}

[distinctdan]

Update: actually my controller method is calling a @Transactional method before it calls the repository method. I'm not sure if that should affect the repository call, since the transaction should be finished by the time it gets to it, but it may be relevant.

[dstepanov]

Can you please reproduce it with a test and integrated test resources?

[krystofrezac]

Yes I can, here it is

https://github.com/krystofrezac/micronaut-deadlock-example/blob/d6d78b5402ec07612f9b6236803c6f123c776a3f/src/test/kotlin/com/example/DemoTest.kt#L28