[RuntimeAsync] A few tests fail with runtime async due to unexpected use of sync context

[VSadov]

Observed while enabling rt async in Libraires partition - #119432

Note: the issue may be not completely deterministic. I think I saw different number of failures in different runs. The runs had other changes, but none specifically for this failure, so it is possible that there is dependency on whether code suspends or takes fast path.

The failure looks like:

        Sync Ctx used:    at System.Environment.get_StackTrace() in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Environment.cs:line 239
           at System.Threading.Tests.TrackingSynchronizationContext.Post(SendOrPostCallback d, Object state)
           at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 697
           at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 3486
           at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Future.cs:line 385
           at System.Threading.Tasks.TaskCompletionSource`1.TrySetResult(TResult result) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskCompletionSource_T.cs:line 218

These are complicated scenarios involving HTTP server/client talking to each other asynchronously.
The tests deliberately set a synchronization context on the client's thread that tracks its use and subsequently fail if it ends up used.
Somehow we may end up using the context when runtime async is enabled.

An additional care may need to be taken to ensure that this is a real issue and not just artefact of particular way to test.
In a way it would still be an observable change and thus a compat concern.

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

There is also a failure in the yield test: System.Threading.Tasks.Tests.YieldAwaitableTests.AsyncMethod_Yields_ReturnsToCorrectSynchronizationContext

That one is pretty manageable size, so I think a better place to start. I will investigate that.

[jakobbotsch]

I think the issue is somewhat obvious and is in the task-returning thunks. They look approximately like

ExecutionAndSyncBlockStore store = default;
store.Push();
Continuation cont = null;
T result = default;
try
{
  try
  {
    result = Inner(args);
    cont = StubHelpers.AsyncCallContinuation();
  }
  catch (Exception ex)
  {
    return Task.FromException(ex);
  }
}
finally
{
  store.Pop();
}

if (cont == null)
  return Task.FromResult(result);

return FinalizeTaskReturningThunk(cont);

We end up restoring the synchronization context too early in the suspension case, before FinalizeTaskReturningThunk is called. FinalizeTaskReturningThunk calls into user code (UnsafeOnCompleted), which expectedly might be looking at the sync context.

[VSadov]

We end up restoring the synchronization context too early in the suspension case

That could also explain why it is not reproducing consistently in scenarios that may or may not suspend.

[VSadov]

Also the following tests fail due to hangs or timeouts:

• `System.IO.Pipelines.Tests.SchedulerFacts.UseSynchronizationContextFalseIgnoresSyncContextForReaderScheduler'
• 'AddHttpClient_GetAwaiterAndResult_InSingleThreadedSynchronizationContext_ShouldNotHangs'

I see them failing both locally and in the CI. It may be related to the Sync Ctx used: failures, or could be something else.

[VSadov]

The debug/debug stack is:
(may be more informative at how we get to the unexpected Post)

      System.Net.Http.Functional.Tests.PlatformHandler_HttpClientHandler_Asynchrony_Test.ResponseHeadersRead_SynchronizationContextNotUsedByHandler(responseHeadersRead: False, contentMode: ContentLength) [FAIL]
        Sync Ctx used:    at System.Environment.get_StackTrace() in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Environment.cs:line 239
           at System.Threading.Tests.TrackingSynchronizationContext.Post(SendOrPostCallback d, Object state) in E:\dotnet004\runtime\src\libraries\Common\tests\System\Threading\TrackingSynchronizationContext.cs:line 17
           at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.PostAction(Object state) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 428
           at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 697
           at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.Run(Task task, Boolean canInlineContinuationTask) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 409
           at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 3487
           at System.Threading.Tasks.Task.FinishContinuations() in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 3451
           at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Future.cs:line 385
           at System.Threading.Tasks.TaskCompletionSource`1.TrySetResult(TResult result) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskCompletionSource_T.cs:line 218
           at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state) in E:\dotnet004\runtime\src\libraries\System.Net.Http.WinHttpHandler\src\System\Net\Http\WinHttpHandler.cs:line 1007
           at System.Runtime.CompilerServices.AsyncHelpers.ThunkTaskCore.MoveNext[T,TOps](T task) in E:\dotnet004\runtime\src\coreclr\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncHelpers.CoreCLR.cs:line 354
           at System.Runtime.CompilerServices.AsyncHelpers.ThunkTask.MoveNext() in E:\dotnet004\runtime\src\coreclr\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncHelpers.CoreCLR.cs:line 306
           at System.Threading.Tasks.Task.InnerInvoke() in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 2397
           at System.Threading.Tasks.Task.<>c.<.cctor>b__288_0(Object obj) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 2385
           at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 264
           at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 2347
           at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 2281
           at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread) in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 2272
           at System.Threading.ThreadPoolWorkQueue.Dispatch() in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ThreadPoolWorkQueue.cs:line 1154
           at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in E:\dotnet004\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\PortableThreadPool.WorkerThread.cs:line 118
           at System.Threading.Thread.StartCallback() in E:\dotnet004\runtime\src\coreclr\System.Private.CoreLib\src\System\Threading\Thread.CoreCLR.cs:line 108