Skip to content

Benchmarks for #116057 (EgorBo) #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
EgorBot opened this issue May 28, 2025 · 11 comments
Open

Benchmarks for #116057 (EgorBo) #363

EgorBot opened this issue May 28, 2025 · 11 comments

Comments

@EgorBot
Copy link
Owner

EgorBot commented May 28, 2025

Processing dotnet/runtime#116057 (comment) command:

Command

-amd -intel -arm

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    private static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 10, 20, 30, 40, 50, 75, 100, 1000, 10000])
            yield return new string('a', len);
    }
}

(EgorBot will reply in this issue)

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

❌ Benchmark validation failed

cc @EgorBo (logs). EgorBot manual: link.

@EgorBo
Copy link

EgorBo commented May 28, 2025

@EgorBot -arm -amd -intel

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    public static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 10, 20, 30, 40, 50, 75, 100, 1000, 10000])
            yield return new string('a', len);
    }
}

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

Benchmark results on linux_azure_genoa

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
AMD EPYC 9V74, 1 CPU, 16 logical and 8 physical cores
  Job-UFGQFL : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-SGSYBA : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
Method Toolchain key Mean Error Ratio
Contains Main a 5.898 ns 0.0013 ns 1.00
Contains PR a 6.156 ns 0.0019 ns 1.04
Contains Main aaaaaaaaaa 6.433 ns 0.0050 ns 1.00
Contains PR aaaaaaaaaa 6.443 ns 0.0011 ns 1.00
Contains Main aaaaaaaaaaaaaaaaaaaa 8.645 ns 0.0067 ns 1.00
Contains PR aaaaaaaaaaaaaaaaaaaa 8.635 ns 0.0011 ns 1.00
Contains Main aaaaa(...)aaaaa [30] 9.249 ns 0.0108 ns 1.00
Contains PR aaaaa(...)aaaaa [30] 9.430 ns 0.0020 ns 1.02
Contains Main aaaaa(...)aaaaa [40] 12.269 ns 0.0047 ns 1.00
Contains PR aaaaa(...)aaaaa [40] 12.753 ns 0.0010 ns 1.04
Contains Main aaaaa(...)aaaaa [50] 13.095 ns 0.0058 ns 1.00
Contains PR aaaaa(...)aaaaa [50] 14.758 ns 0.0045 ns 1.13
Contains Main aaaaa(...)aaaaa [75] 19.227 ns 0.0065 ns 1.00
Contains PR aaaaa(...)aaaaa [75] 15.741 ns 0.0036 ns 0.82
Contains Main aaaa(...)aaaa [100] 22.772 ns 0.0268 ns 1.00
Contains PR aaaa(...)aaaa [100] 18.676 ns 0.0050 ns 0.82
Contains Main aaaa(...)aaaa [1000] 206.707 ns 0.0958 ns 1.00
Contains PR aaaa(...)aaaa [1000] 55.444 ns 0.1823 ns 0.27
Contains Main aaa(...)aaa [10000] 2,050.498 ns 0.3053 ns 1.00
Contains PR aaa(...)aaa [10000] 389.271 ns 0.3163 ns 0.19

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

Benchmark results on linux_azure_cobalt100

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
azure_cobalt100
  Job-PCUMCK : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
  Job-VDCGEX : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
Method Toolchain key Mean Error Ratio
Contains Main a 7.136 ns 0.0013 ns 1.00
Contains PR a 6.834 ns 0.0021 ns 0.96
Contains Main aaaaaaaaaa 7.001 ns 0.0009 ns 1.00
Contains PR aaaaaaaaaa 6.968 ns 0.0010 ns 1.00
Contains Main aaaaaaaaaaaaaaaaaaaa 10.881 ns 0.0006 ns 1.00
Contains PR aaaaaaaaaaaaaaaaaaaa 10.395 ns 0.0013 ns 0.96
Contains Main aaaaa(...)aaaaa [30] 10.999 ns 0.0014 ns 1.00
Contains PR aaaaa(...)aaaaa [30] 10.839 ns 0.0007 ns 0.99
Contains Main aaaaa(...)aaaaa [40] 14.434 ns 0.0019 ns 1.00
Contains PR aaaaa(...)aaaaa [40] 19.117 ns 0.0011 ns 1.32
Contains Main aaaaa(...)aaaaa [50] 14.793 ns 0.0009 ns 1.00
Contains PR aaaaa(...)aaaaa [50] 22.769 ns 0.0013 ns 1.54
Contains Main aaaaa(...)aaaaa [75] 22.219 ns 0.0017 ns 1.00
Contains PR aaaaa(...)aaaaa [75] 23.945 ns 0.0012 ns 1.08
Contains Main aaaa(...)aaaa [100] 26.025 ns 0.0036 ns 1.00
Contains PR aaaa(...)aaaa [100] 28.577 ns 0.0014 ns 1.10
Contains Main aaaa(...)aaaa [1000] 250.081 ns 0.0271 ns 1.00
Contains PR aaaa(...)aaaa [1000] 169.024 ns 0.0062 ns 0.68
Contains Main aaa(...)aaa [10000] 2,492.973 ns 0.2946 ns 1.00
Contains PR aaa(...)aaa [10000] 1,595.048 ns 0.0941 ns 0.64

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

Benchmark results on linux_azure_cascadelake

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
Intel Xeon Platinum 8370C CPU 2.80GHz, 1 CPU, 16 logical and 8 physical cores
  Job-PUCUGS : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-ONGJWX : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
Method Toolchain key Mean Error Ratio
Contains Main a 6.817 ns 0.0150 ns 1.00
Contains PR a 6.508 ns 0.0151 ns 0.95
Contains Main aaaaaaaaaa 6.263 ns 0.0608 ns 1.00
Contains PR aaaaaaaaaa 6.547 ns 0.0278 ns 1.05
Contains Main aaaaaaaaaaaaaaaaaaaa 10.183 ns 0.0774 ns 1.00
Contains PR aaaaaaaaaaaaaaaaaaaa 9.659 ns 0.0219 ns 0.95
Contains Main aaaaa(...)aaaaa [30] 10.965 ns 0.0011 ns 1.00
Contains PR aaaaa(...)aaaaa [30] 10.780 ns 0.0017 ns 0.98
Contains Main aaaaa(...)aaaaa [40] 14.253 ns 0.0559 ns 1.00
Contains PR aaaaa(...)aaaaa [40] 14.910 ns 0.0067 ns 1.05
Contains Main aaaaa(...)aaaaa [50] 15.068 ns 0.0139 ns 1.00
Contains PR aaaaa(...)aaaaa [50] 16.843 ns 0.0024 ns 1.12
Contains Main aaaaa(...)aaaaa [75] 22.663 ns 0.0077 ns 1.00
Contains PR aaaaa(...)aaaaa [75] 17.879 ns 0.0061 ns 0.79
Contains Main aaaa(...)aaaa [100] 26.029 ns 0.0125 ns 1.00
Contains PR aaaa(...)aaaa [100] 20.844 ns 0.0093 ns 0.80
Contains Main aaaa(...)aaaa [1000] 261.494 ns 0.0259 ns 1.00
Contains PR aaaa(...)aaaa [1000] 71.795 ns 0.0257 ns 0.27
Contains Main aaa(...)aaa [10000] 2,584.381 ns 0.2983 ns 1.00
Contains PR aaa(...)aaa [10000] 560.577 ns 0.1273 ns 0.22

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

@EgorBo
Copy link

EgorBo commented May 28, 2025

@EgorBot -amd -intel -arm

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    private static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 2, 3, 10, 20, 30, 40, 50, 63, 64, 70, 100, 120, 128, 1000, 10000])
            yield return new string('a', len);
    }
}

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

❌ Benchmark validation failed

cc @EgorBo (logs). EgorBot manual: link.

@EgorBo
Copy link

EgorBo commented May 28, 2025

@EgorBot -amd -intel -arm

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Bench).Assembly).Run(args);

public class Bench
{
    private static Dictionary<string, int> Dictionary
        // We don't need to fill the dictionary as we're only benchmarking the hashcode operation.
        = new() {{ "1", 2 }};

    [Benchmark]
    [ArgumentsSource(nameof(GetKeys))]
    public bool Contains(string key) => Dictionary.ContainsKey(key);

    public static IEnumerable<string> GetKeys()
    {
        foreach (int len in (int[])[1, 2, 3, 10, 20, 30, 40, 50, 63, 64, 70, 100, 120, 128, 1000, 10000])
            yield return new string('a', len);
    }
}

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

Benchmark results on linux_azure_genoa

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
AMD EPYC 9V74, 1 CPU, 16 logical and 8 physical cores
  Job-BJLMKI : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-NTYPJN : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
Method Toolchain key Mean Error Ratio
Contains Main a 5.968 ns 0.0024 ns 1.00
Contains PR a 5.880 ns 0.0009 ns 0.99
Contains Main aa 5.333 ns 0.0007 ns 1.00
Contains PR aa 5.079 ns 0.0011 ns 0.95
Contains Main aaa 5.609 ns 0.0012 ns 1.00
Contains PR aaa 5.351 ns 0.0010 ns 0.95
Contains Main aaaaaaaaaa 6.424 ns 0.0011 ns 1.00
Contains PR aaaaaaaaaa 6.698 ns 0.0028 ns 1.04
Contains Main aaaaaaaaaaaaaaaaaaaa 8.646 ns 0.0069 ns 1.00
Contains PR aaaaaaaaaaaaaaaaaaaa 8.915 ns 0.0139 ns 1.03
Contains Main aaaaa(...)aaaaa [30] 9.255 ns 0.0163 ns 1.00
Contains PR aaaaa(...)aaaaa [30] 9.726 ns 0.0088 ns 1.05
Contains Main aaaaa(...)aaaaa [40] 14.139 ns 0.0214 ns 1.00
Contains PR aaaaa(...)aaaaa [40] 12.000 ns 0.0219 ns 0.85
Contains Main aaaaa(...)aaaaa [50] 13.092 ns 0.0102 ns 1.00
Contains PR aaaaa(...)aaaaa [50] 12.682 ns 0.0134 ns 0.97
Contains Main aaaaa(...)aaaaa [63] 15.800 ns 0.0065 ns 1.00
Contains PR aaaaa(...)aaaaa [63] 14.784 ns 0.0504 ns 0.94
Contains Main aaaaa(...)aaaaa [64] 15.808 ns 0.0107 ns 1.00
Contains PR aaaaa(...)aaaaa [64] 14.067 ns 0.0030 ns 0.89
Contains Main aaaaa(...)aaaaa [70] 17.053 ns 0.0069 ns 1.00
Contains PR aaaaa(...)aaaaa [70] 14.695 ns 0.0024 ns 0.86
Contains Main aaaa(...)aaaa [100] 22.778 ns 0.0188 ns 1.00
Contains PR aaaa(...)aaaa [100] 19.337 ns 0.0040 ns 0.85
Contains Main aaaa(...)aaaa [120] 26.708 ns 0.0307 ns 1.00
Contains PR aaaa(...)aaaa [120] 22.553 ns 0.0041 ns 0.84
Contains Main aaaa(...)aaaa [128] 28.358 ns 0.0558 ns 1.00
Contains PR aaaa(...)aaaa [128] 20.538 ns 0.0428 ns 0.72
Contains Main aaaa(...)aaaa [1000] 206.625 ns 0.0587 ns 1.00
Contains PR aaaa(...)aaaa [1000] 55.756 ns 0.1391 ns 0.27
Contains Main aaa(...)aaa [10000] 2,051.798 ns 0.3873 ns 1.00
Contains PR aaa(...)aaa [10000] 389.687 ns 0.3023 ns 0.19

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

Benchmark results on linux_azure_cobalt100

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
azure_cobalt100
  Job-DZSNTL : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
  Job-XTOYGC : .NET 10.0.0 (42.42.42.42424), Arm64 RyuJIT AdvSIMD
Method Toolchain key Mean Error Ratio
Contains Main a 7.040 ns 0.0009 ns 1.00
Contains PR a 5.955 ns 0.0020 ns 0.85
Contains Main aa 5.804 ns 0.0011 ns 1.00
Contains PR aa 5.319 ns 0.0008 ns 0.92
Contains Main aaa 6.214 ns 0.0010 ns 1.00
Contains PR aaa 5.276 ns 0.0018 ns 0.85
Contains Main aaaaaaaaaa 7.124 ns 0.0009 ns 1.00
Contains PR aaaaaaaaaa 6.442 ns 0.0011 ns 0.90
Contains Main aaaaaaaaaaaaaaaaaaaa 10.815 ns 0.0047 ns 1.00
Contains PR aaaaaaaaaaaaaaaaaaaa 9.480 ns 0.0016 ns 0.88
Contains Main aaaaa(...)aaaaa [30] 11.357 ns 0.0008 ns 1.00
Contains PR aaaaa(...)aaaaa [30] 10.960 ns 0.0084 ns 0.96
Contains Main aaaaa(...)aaaaa [40] 14.012 ns 0.0019 ns 1.00
Contains PR aaaaa(...)aaaaa [40] 13.665 ns 0.0017 ns 0.98
Contains Main aaaaa(...)aaaaa [50] 14.705 ns 0.0023 ns 1.00
Contains PR aaaaa(...)aaaaa [50] 14.230 ns 0.0016 ns 0.97
Contains Main aaaaa(...)aaaaa [63] 17.666 ns 0.0016 ns 1.00
Contains PR aaaaa(...)aaaaa [63] 16.976 ns 0.0032 ns 0.96
Contains Main aaaaa(...)aaaaa [64] 17.686 ns 0.0016 ns 1.00
Contains PR aaaaa(...)aaaaa [64] 17.130 ns 0.0030 ns 0.97
Contains Main aaaaa(...)aaaaa [70] 19.661 ns 0.0018 ns 1.00
Contains PR aaaaa(...)aaaaa [70] 18.691 ns 0.0021 ns 0.95
Contains Main aaaa(...)aaaa [100] 25.934 ns 0.0039 ns 1.00
Contains PR aaaa(...)aaaa [100] 25.669 ns 0.0120 ns 0.99
Contains Main aaaa(...)aaaa [120] 30.781 ns 0.0057 ns 1.00
Contains PR aaaa(...)aaaa [120] 31.218 ns 0.0047 ns 1.01
Contains Main aaaa(...)aaaa [128] 33.385 ns 0.0035 ns 1.00
Contains PR aaaa(...)aaaa [128] 34.263 ns 0.0069 ns 1.03
Contains Main aaaa(...)aaaa [1000] 250.878 ns 0.0455 ns 1.00
Contains PR aaaa(...)aaaa [1000] 169.736 ns 0.0136 ns 0.68
Contains Main aaa(...)aaa [10000] 2,493.119 ns 0.8691 ns 1.00
Contains PR aaa(...)aaa [10000] 1,594.650 ns 0.0617 ns 0.64

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

@EgorBot
Copy link
Owner Author

EgorBot commented May 28, 2025

Benchmark results on linux_azure_cascadelake

BenchmarkDotNet v0.14.0, Ubuntu 24.04.2 LTS (Noble Numbat)
Intel Xeon Platinum 8370C CPU 2.80GHz, 1 CPU, 16 logical and 8 physical cores
  Job-BCXAWP : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-LZKWTZ : .NET 10.0.0 (42.42.42.42424), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
Method Toolchain key Mean Error Ratio
Contains Main a 6.822 ns 0.0653 ns 1.00
Contains PR a 6.248 ns 0.0048 ns 0.92
Contains Main aa 5.717 ns 0.0221 ns 1.00
Contains PR aa 4.654 ns 0.0112 ns 0.81
Contains Main aaa 5.574 ns 0.0013 ns 1.00
Contains PR aaa 4.800 ns 0.0154 ns 0.86
Contains Main aaaaaaaaaa 6.129 ns 0.0652 ns 1.00
Contains PR aaaaaaaaaa 6.526 ns 0.0070 ns 1.06
Contains Main aaaaaaaaaaaaaaaaaaaa 9.842 ns 0.0406 ns 1.00
Contains PR aaaaaaaaaaaaaaaaaaaa 9.820 ns 0.0386 ns 1.00
Contains Main aaaaa(...)aaaaa [30] 10.937 ns 0.0024 ns 1.00
Contains PR aaaaa(...)aaaaa [30] 10.207 ns 0.0468 ns 0.93
Contains Main aaaaa(...)aaaaa [40] 14.688 ns 0.0208 ns 1.00
Contains PR aaaaa(...)aaaaa [40] 13.944 ns 0.0185 ns 0.95
Contains Main aaaaa(...)aaaaa [50] 15.118 ns 0.0137 ns 1.00
Contains PR aaaaa(...)aaaaa [50] 14.537 ns 0.0151 ns 0.96
Contains Main aaaaa(...)aaaaa [63] 18.436 ns 0.0236 ns 1.00
Contains PR aaaaa(...)aaaaa [63] 17.422 ns 0.0061 ns 0.94
Contains Main aaaaa(...)aaaaa [64] 18.440 ns 0.0071 ns 1.00
Contains PR aaaaa(...)aaaaa [64] 15.066 ns 0.0075 ns 0.82
Contains Main aaaaa(...)aaaaa [70] 19.719 ns 0.0053 ns 1.00
Contains PR aaaaa(...)aaaaa [70] 16.251 ns 0.0042 ns 0.82
Contains Main aaaa(...)aaaa [100] 26.027 ns 0.0100 ns 1.00
Contains PR aaaa(...)aaaa [100] 20.761 ns 0.0038 ns 0.80
Contains Main aaaa(...)aaaa [120] 30.702 ns 0.0191 ns 1.00
Contains PR aaaa(...)aaaa [120] 25.226 ns 0.0052 ns 0.82
Contains Main aaaa(...)aaaa [128] 32.759 ns 0.0266 ns 1.00
Contains PR aaaa(...)aaaa [128] 21.959 ns 0.0275 ns 0.67
Contains Main aaaa(...)aaaa [1000] 261.421 ns 0.0154 ns 1.00
Contains PR aaaa(...)aaaa [1000] 67.318 ns 0.0258 ns 0.26
Contains Main aaa(...)aaa [10000] 2,583.893 ns 0.1799 ns 1.00
Contains PR aaa(...)aaa [10000] 558.398 ns 0.0300 ns 0.22

BDN_Artifacts.zip

cc @EgorBo (agent_logs.txt). EgorBot manual: link.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants