InProcessNoEmitToolchain + NativeAOT Support?

[eliphatfs]

I have the same use cases as #2694 to benchmark the same binary on different machines.

I use this config object:

    public class BenchmarkConfig : ManualConfig
    {
        public BenchmarkConfig()
        {
            AddJob(Job.Default.WithToolchain(InProcessNoEmitToolchain.Instance));
        }
    }

However, when running a published NativeAOT (.NET 9.0.2) executable, I get:

System.InvalidOperationException: Bug: type BenchmarkDotNet.Toolchains.InProcess.NoEmit.InProcessNoEmitRunner+Runnable not found.
   at BenchmarkDotNet.Toolchains.InProcess.NoEmit.InProcessNoEmitRunner.Run(IHost, BenchmarkCase) + 0x370
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at BenchmarkDotNet.Toolchains.Results.ExecuteResult.LogIssues(ILogger, BuildResult) + 0x1f
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunExecute(ILogger, BenchmarkCase, BenchmarkId, IToolchain, BuildResult, IResolver, IDiagnoser, Int32) + 0xeb
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger, BenchmarkCase, BenchmarkId, IToolchain, BuildResult, IResolver) + 0x572
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase, BenchmarkId, ILogger, IResolver, BuildResult) + 0xa3
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo, Dictionary`2, IResolver, ILogger, EventProcessor, List`1, String, String, Int32, StartedClock&, Int32&, TaskbarProgress) + 0x7f6
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[]) + 0x982
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithDirtyAssemblyResolveHelper(Type, IConfig, String[]) + 0x50
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithExceptionHandling(Func`1) + 0x20
   at BenchmarkDotNet.Running.BenchmarkRunner.Run[T](IConfig, String[]) + 0x77

I see BenchmarkDotNet.Toolchains.InProcess.NoEmit.InProcessNoEmitRunner+Runnable is a private class https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet/Toolchains/InProcess/NoEmit/InProcessNoEmitRunner.cs so I can't
use DynamicDependencyAttribute to keep the class from being trimmed.

[timcassell]

Can you use the string-based ctor of the attribute? https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.codeanalysis.dynamicdependencyattribute.-ctor?view=net-9.0#system-diagnostics-codeanalysis-dynamicdependencyattribute-ctor(system-string-system-string-system-string)

Either way, it would be good to have all of the types for that toolchain properly annotated for AOT consumption. Accepting PRs.

[eliphatfs]

It doesn't seem to work. I don't know if I am doing it correctly.

[DynamicDependency(DynamicallyAccessedMemberTypes.All, "BenchmarkDotNet.Toolchains.InProcess.NoEmit.InProcessNoEmitRunner+Runnable", "BenchmarkDotNet")]

I also tried another workaround, to copy code from InProcessNoEmitToolchain into my own project, mark the dynamic dependency to Runnable, and use that toolchain instead. I am now hitting another error:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.NotSupportedException: 'BenchmarkDotNet.Toolchains.InProcess.NoEmit.BenchmarkActionFactory+BenchmarkAction`1[System.UInt32]' is missing native code or metadata. This can happen for code that is not compatible with trimming or AOT. Inspect and fix trimming and AOT related warnings that were generated when the app was published. For more information see https://aka.ms/nativeaot-compatibility
   at System.Reflection.Runtime.General.TypeUnifier.WithVerifiedTypeHandle(RuntimeConstructedGenericTypeInfo, RuntimeTypeInfo[]) + 0x7e
   at System.Reflection.Runtime.TypeInfos.RuntimeTypeInfo.MakeGenericType(Type[]) + 0x207
   at BenchmarkDotNet.Toolchains.InProcess.NoEmit.BenchmarkActionFactory.CreateCore(Object instance, MethodInfo targetMethod, MethodInfo fallbackIdleSignature, Int32 unrollFactor) + 0x154
   at Benchmark.InProcessNoEmitRunner.Runnable.RunCore(IHost host, BenchmarkCase benchmarkCase) + 0x

[timcassell]

You could also try to disable trimming.

[eliphatfs]

NativeAOT doesn't allow to disable trimming.
The BenchmarkAction<T> types seem tricky to mark.
By examining code, if I modify my benchmarks to return void instead of byte and uint previously, I can avoid running into MakeGenericType calls and run benchmarks successfully.

[eliphatfs]

In my case my functions are not in the realm of nanoseconds so I can wrap around my results with DeadCodeEliminationHelper.KeepAliveWithoutBoxingReadonly to prevent DCE, and still obtaining rather accurate measurements.