Concurrency error in HandleTransactionCompleted (RelationalConnection)

[FlipEX]

Bug description

We are "randomly" experiencing the exception "System.InvalidOperationException: A root ambient transaction was completed before the nested transaction. The nested transactions should be completed first.". When looking at our code, we don't have any root ambient transaction, being completed before a nested transaction. The problem, however, is when using distributed transactions and having the MSTDC "fire" an exception.
As it doesn't happen all the time (it's somewhat of a timings issue), I am not able to create a repo that reproduces the problem. However, I think I have tracked down where the error is. The error occurs in Microsoft.EntityFrameworkCore.Storage.RelationalConnection https://github.com/dotnet/efcore/blob/4b0d9dc416c57368b1cf76e9d76de365cc5e6d9f/src/EFCore.Relational/Storage/RelationalConnection.cs
The problem occurs when HandleTransactionCompleted is called at a time when ClearTransactions has just exactly popped the givet ambientTransaction from _ambientTransactions (line 697); but not yet called ambientTransaction.TransactionCompleted -= HandleTransactionCompleted (line 697) which makes sure that HandleTransactionCompleted is not called;
As stated in HandleTransactionCompleted: This could be invoked on a different thread at arbitrary time after the transaction completes we need to have some kind of concurrency handling in ClearTransactions and HandleTransactionCompleted. I don't know what the best solution to the problem is.
Let me know if you have any questions.

@dshuvaev is tagged as I believe he introduced the error in this commit: daa1ed2

Your code

As stated above the error only happens in very specific timing cases, which is why I haven't been able to create code that reproduces the exception on a regular basis.

Stack traces

Verbose output

EF Core version

8.0.10

Database provider

No response

Target framework

.NET 8

Operating system

Windows Server 2022

IDE

No response

[roji]

@AndriySvyryd this looks like PR #20012.

[FlipEX]

@AndriySvyryd this looks like PR #20012.

If you mean that the error was introduced in that PR, yes you are right.

[bluebaroncanada]

I am also having the same issue in Azure 100% of the time. Works locally. Does not work remotely.