DotNetSdkValidator failed when using CsProjClassicNetToolchain

[filzrev]

Steps to reproduce problem

1. Create simple benchmark dotnet project
2. Modify project to use <TargetFrameworks>net48;net8.0</TargetFrameworks>
3. Modify Program.cs with following content

   internal class Program
   {
       static void Main(string[] args)
       {
           var config = DefaultConfig.Instance
               .AddJob(Job.Dry.WithToolchain(CsProjClassicNetToolchain.Net48));
               // .AddJob(Job.Dry.WithToolchain(CsProjCoreToolchain.NetCoreApp80));
   
           BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);
       }
   }
   
   public class Benchmarks
   {
       [Benchmark]
       public void Benchmark1() { };
   }
   

4. Run program by using net8.0 framework.
5. Following error is shown.

   // Validating benchmarks:
   // Benchmark Benchmark.IRunEveryWhere: Dry(Toolchain=.NET Framework 4.8, IterationCount=1, LaunchCount=1, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=1)
   //    * The required .NET Framework SDK version 8.0 or higher is not installed.
   

---

This problem is not occurred when running benchmark host project with net48.
And it seems not occurred when using BenchmarkDotNet v0.14.0

[timcassell]

I cannot repro. I suspect it's a machine-specific error. If you can find the cause and send a fix, that'd be great!

[filzrev]

I've confirmed it can be reproduced on GitHub actions environment also.

Logs
https://github.com/filzrev/BenchmarkDotNet/actions/runs/19913622689/job/57087688449

Note: Returned summary is not validated and process exit with 0.

As far as I've confirmed related source code.
DotNetSdkValidator is executed when using CsProjClassicNetToolchain and It's expected always failed when running on .NET 8.

[timcassell]

I don't see failures in that link.

CsProjClassicNetToolchain runs a different validation method than CsProjCoreToolchain that checks only for .Net Framework sdks.

[timcassell]

Sorry I misspoke. CsProjClassicNetToolchain does check if the cli is installed because it's required, but it doesn't care what version it is.

[filzrev]

I don't see failures in that link.

It failed at "Run benchmark" steps on test-windows-core with following message.

// * The required .NET Framework SDK version 8.0 or higher is not installed.

It's shown as success on CI.
Because BenchmarkDotNet.Samples.Program.cs using void Main
I've created PR #2898 that validate result summaries for BenchmarkDotNet.Samples.

It might be better similar code on BenchmarkDotNet side in future.
https://github.com/dotnet/BenchmarkDotNet/blob/master/templates/templates/BenchmarkDotNet.BenchmarkProjectTemplate.CSharp/Program.cs

---

CsProjClassicNetToolchain does check if the cli is installed because it's required, but it doesn't care what version it is.

As far as I've confirmed, following code is not works as expected. (IsFullFramework is false when running benchmark host on ,NET 8)

BenchmarkDotNet/src/BenchmarkDotNet/Portability/RuntimeInformation.cs

Lines 173 to 176 in a62d90a

	internal static Runtime GetTargetOrCurrentRuntime(Assembly? assembly)
	=> !IsMono && !IsWasm && IsFullFramework // Match order of checks in GetCurrentRuntime().
	? ClrRuntime.GetTargetOrCurrentVersion(assembly)
	: GetCurrentRuntime();

[timcassell]

I see it's another symptom of #2609