EvaluateOverhead command line option and docs

Author: timcassell

docs/articles/configs/jobs.md

@@ -70,7 +70,7 @@ If you want to change the accuracy level, you should use the following character
 * `MaxRelativeError`, `MaxAbsoluteError`: Maximum acceptable error for a benchmark (by default, BenchmarkDotNet continue iterations until the actual error is less than the specified error). *In these two characteristics*, the error means half of 99.9% confidence interval. `MaxAbsoluteError` is an absolute `TimeInterval`; doesn't have a default value. `MaxRelativeError` defines max acceptable (`(<half of CI 99.9%>) / Mean`).
 * `MinIterationTime`: Minimum time of a single iteration. Unlike `Run.IterationTime`, this characteristic specifies only the lower limit. In case of need, BenchmarkDotNet can increase this value.
 * `MinInvokeCount`:  Minimum about of target method invocation. Default value if `4` but you can decrease this value for cases when single invocations takes a lot of time.
-* `EvaluateOverhead`: if your benchmark method takes nanoseconds, BenchmarkDotNet overhead can significantly affect measurements. If this characteristic is enabled, the overhead will be evaluated and subtracted from the result measurements. Default value is `true`.
+* `EvaluateOverhead`: if your benchmark method takes nanoseconds, BenchmarkDotNet overhead can significantly affect measurements. If this characteristic is enabled, the overhead will be evaluated and subtracted from the result measurements. Default value is `false`.
 * `WithOutlierMode`: sometimes you could have outliers in your measurements. Usually these are unexpected outliers which arose because of other processes activities. By default (`OutlierMode.RemoveUpper`), all upper outliers (which is larger than Q3) will be removed from the result measurements. However, some of benchmarks have *expected* outliers. In these situation, you expect that some of invocation can produce outliers measurements (e.g. in case of network activities, cache operations, and so on). If you want to see result statistics with these outliers, you should use `OutlierMode.DontRemove`. If you can also choose `OutlierMode.RemoveLower` (outliers which are smaller than Q1 will be removed) or `OutlierMode.RemoveAll` (all outliers will be removed). See also: @BenchmarkDotNet.Mathematics.OutlierMode
 * `AnalyzeLaunchVariance`: this characteristic makes sense only if `Run.LaunchCount` is default. If this mode is enabled and, BenchmarkDotNet will try to perform several launches and detect if there is a variance between launches. If this mode is disable, only one launch will be performed.
 

docs/articles/guides/console-args.md

@@ -321,7 +321,7 @@ dotnet run -c Release -- --filter * --runtimes net6.0 net8.0 --statisticalTest 5
 * `--wasmDataDir`             Wasm data directory
 * `--wasmCoreCLR`             (Default: false) Use CoreCLR runtime pack (Microsoft.NETCore.App.Runtime.browser-wasm) instead of the Mono runtime pack for WASM benchmarks.
 * `--noForcedGCs`             Specifying would not forcefully induce any GCs.
-* `--noOverheadEvaluation`    Specifying would not run the evaluation overhead iterations.
+* `--evaluateOverhead`        Specifies whether to run and evaluate overhead iterations.
 * `--resume`                  (Default: false) Continue the execution if the last run was stopped.
 * `--help`                    Display this help screen.
 * `--version`                 Display version information.

src/BenchmarkDotNet/ConsoleArguments/CommandLineOptions.cs

@@ -228,8 +228,8 @@ public bool UseDisassemblyDiagnoser
         [Option("noForcedGCs", Required = false, HelpText = "Specifying would not forcefully induce any GCs.")]
         public bool NoForcedGCs { get; set; }
 
-        [Option("noOverheadEvaluation", Required = false, HelpText = "Specifying would not run the evaluation overhead iterations.")]
-        public bool NoEvaluationOverhead { get; set; }
+        [Option("evaluateOverhead", Required = false, HelpText = "Specifies whether to run and evaluate overhead iterations.")]
+        public bool? EvaluateOverhead { get; set; }
 
         [Option("resume", Required = false, Default = false, HelpText = "Continue the execution if the last run was stopped.")]
         public bool Resume { get; set; }

src/BenchmarkDotNet/ConsoleArguments/ConfigParser.cs

@@ -443,9 +443,9 @@ private static Job GetBaseJob(CommandLineOptions options, IConfig? globalConfig)
                 baseJob = baseJob.WithMemoryRandomization();
             if (options.NoForcedGCs)
                 baseJob = baseJob.WithGcForce(false);
-            if (options.NoEvaluationOverhead)
+            if (options.EvaluateOverhead is bool evaluateOverhead)
 #pragma warning disable CS0618 // Type or member is obsolete
-                baseJob = baseJob.WithEvaluateOverhead(false);
+                baseJob = baseJob.WithEvaluateOverhead(evaluateOverhead);
 #pragma warning restore CS0618 // Type or member is obsolete
 
             if (options.EnvironmentVariables.Any())

tests/BenchmarkDotNet.Tests/ConfigParserTests.cs

@@ -683,15 +683,15 @@ public void UserCanSpecifyNoForceGCs()
         }
 
         [Fact]
-        public void UsersCanSpecifyWithoutOverheadEvalution()
+        public void UsersCanSpecifyEvaluateOverhead()
         {
-            var parsedConfiguration = ConfigParser.Parse(["--noOverheadEvaluation"], new OutputLogger(Output));
+            var parsedConfiguration = ConfigParser.Parse(["--evaluateOverhead", "true"], new OutputLogger(Output));
             Assert.NotNull(parsedConfiguration.config);
             Assert.True(parsedConfiguration.isSuccess);
 
             foreach (var job in parsedConfiguration.config.GetJobs())
             {
-                Assert.False(job.Accuracy.EvaluateOverhead);
+                Assert.True(job.Accuracy.EvaluateOverhead);
             }
         }