Concurrency error in HandleTransactionCompleted (RelationalConnection)

[FlipEX]

Bug description

We are "randomly" experiencing the exception "System.InvalidOperationException: A root ambient transaction was completed before the nested transaction. The nested transactions should be completed first.". When looking at our code, we don't have any root ambient transaction, being completed before a nested transaction. The problem, however, is when using distributed transactions and having the MSTDC "fire" an exception.
As it doesn't happen all the time (it's somewhat of a timings issue), I am not able to create a repo that reproduces the problem. However, I think I have tracked down where the error is. The error occurs in Microsoft.EntityFrameworkCore.Storage.RelationalConnection https://github.com/dotnet/efcore/blob/4b0d9dc416c57368b1cf76e9d76de365cc5e6d9f/src/EFCore.Relational/Storage/RelationalConnection.cs
The problem occurs when HandleTransactionCompleted is called at a time when ClearTransactions has just exactly popped the givet ambientTransaction from _ambientTransactions (line 697); but not yet called ambientTransaction.TransactionCompleted -= HandleTransactionCompleted (line 697) which makes sure that HandleTransactionCompleted is not called;
As stated in HandleTransactionCompleted: This could be invoked on a different thread at arbitrary time after the transaction completes we need to have some kind of concurrency handling in ClearTransactions and HandleTransactionCompleted. I don't know what the best solution to the problem is.
Let me know if you have any questions.

@dshuvaev is tagged as I believe he introduced the error in this commit: daa1ed2

Your code

As stated above the error only happens in very specific timing cases, which is why I haven't been able to create code that reproduces the exception on a regular basis.

Stack traces

Verbose output

EF Core version

8.0.10

Database provider

No response

Target framework

.NET 8

Operating system

Windows Server 2022

IDE

No response

[roji]

@AndriySvyryd this looks like PR #20012.

[FlipEX]

@AndriySvyryd this looks like PR #20012.

If you mean that the error was introduced in that PR, yes you are right.

[bluebaroncanada]

I am also having the same issue in Azure 100% of the time. Works locally. Does not work remotely.

[ThreePinkApples]

This sounds like the same issue I've been struggling with for some months, without having any idea of the root cause. From time to time the NServiceBus transactions throws System.InvalidOperationException: A root ambient transaction was completed before the nested transaction. The nested transactions should be completed first., but the database transaction is completed and committed, so the NServiceBus retry handling can't recover as the database changes are not rolled back.

[ThreePinkApples]

I'm wondering if someone might have some tips to workaround our specific usecase, but maybe asking Particular would be better.

We use NServiceBus with SqlServerTransport. The message receive/sending is done inside a TransactionScope (SendsAtomicWithReceive), which is the outer/root TransactionScope in this scenario. Then we use NServiceBus' "Unit of Work" with handlers wrapped in a TransactionScope, which becomes the inner/nested scope in this scenario.
Our issue is that this nested scope has completed and committed to the database, but then the NServiceBus messages are not sent due to the System.InvalidOperationException: A root ambient transaction was completed before the nested transaction. The nested transactions should be completed first. error. NServiceBus automatically retries the message correctly, but since the DB transaction was completed, the message is discarded as a duplicate.
I'm not sure how to work around this, if it is even possible. Right now we just have to manually fix every case we encounter, typically a few cases every day.