Provide a way to run benchmarks under Unity Mono and IL2CPP

[MichalPetryka]

Currently, there is no official (but there have been attempts it seems, looking at #1838 and #759) way to use BenchmarkDotNet under Unity nor to benchmark code running under Unity. This has resulted in many improper benchmarks with incorrect results being shared around the internet. Providing an option to use Unity Mono and IL2CPP as runtimes would be highly beneficial for game development, since that is an usecase where perfomance often matters.

I thought about how the support should look like and I've came up with a few things:

• BenchmarkDotNet would contain a project template that would contain the code required to run the benchmarks, where the benchmark code would be copied on build.
• Since Unity allows only a single instance of a project to be open at once, BenchmarkDotNet would need to create a separate copy of the project generated from the template or to compile them sequentially.
• The app would be a full Unity app for the Mono runtime, instead of just running the code with their fork of Mono to also measure the engines real impact on performance and also since that would not be possible with IL2CPP.
• The app would use Unitys headless mode to reduce the performance overhead and to allow issue-less measurement under headless server environments.
• Nuget support would have to be realized by downloading the dependencies and copying them into the created project.
• Native libraries referenced in the project would need to be similarly copied.
• Unlike the Mono runtime target, Unity runtimes would be versioned like .Net.
• The engine would need to be installed manually by the user.
• In order to allow benchmarking of the Unity API, BenchmarkDotNet would need to provide reference assemblies allowing access to them. This could be realised by providing a separate BenchmarkDotNet.Unity Nuget package. This would need to be versioned accordingly to Unity
• The IL2CPP target, being a Full AOT runtime like NativeAOT (formerly CoreRT), would face the same restrictions.

Open questions:

• Should BenchmarkDotNet build a code binary with dotnet and then include that in the project or copy the code directly?
• Unlike dotnet, Unity doesn't add itself to PATH nor provide an easy way to access its install location. This could make specifing the Engine path by the user required.
• Should the template project provide optional Unity features like DOTS in order to make it possible to benchmark them?
• I'm not sure if all diagnosers like MemoryDiagnoser or DisassemblyDiagnoser (this project could provide some insight with this one for Mono) would be possible to implement.
• AFAIK Unitys Terms of Service wouldn't allow the redistribution of referenece assemblies generated against their code, so this would require some cooperation from Unity and their legal team.

There has been recent talk about this on Unitys forum, which tells us that Unity staff would be interested in this too, at least according to @joshpeterson.

[timcassell]

Thanks for opening this issue. I was the one who asked about it on that thread.

One thing that I'm hesitant about is actually supporting the full engine, since BDN currently does not support any applications except console (it used to support others, but dropped support). Of course, in order to do that, Unity would need to make its IL2CPP runtime available separately from the Unity engine, which I'm not sure if they have any interest in doing.

Since the IL2CPP runtime comes bundled with the engine, that means if we want to add support for that without Unity's cooperation, a full engine build is required.

I'm not sure how we could compile with Unity dependencies from BenchmarkDotNet.Unity inside the console app and then translate them over to package dependencies in the generated project, maybe you know how that could be done?

In any case, ignoring Unity dependencies, project generation can be done like so:

• create project folder BDNUnityProject_BenchmarkName (separate folder for each benchmark)
• add the code BDN already generates to /Assets/GeneratedCode.cs
• add /Assets/BDNBehaviour.cs with its Awake() function calling GeneratedCode.Main(System.Environment.GetCommandLineArgs()), then Application.Quit()
• add /Assets/BDNBehaviour.cs.meta with its GUID created by BDN
• add a /Assets/Startup.unity scene file with a single gameobject and the BDNBehaviour attached (using the GUID from previous step)
• add a /Assets/Startup.unity.meta with its GUID created by BDN
• add /ProjectSettings/EditorBuildSettings.asset with the Startup scene's path and GUID.
• copy any nuget packages and native references to /Assets/Plugins/ (yay magic folders!)

[Edit] A template can be created for all this to more easily tweak the necessary values.

Then it can be built like this:

<unity-editor-path> -batchmode -nographics -quit -projectPath BDNUnityProject_BenchmarkName -build<Windows/Linux/OSX>64Player BDNUnityProject_BenchmarkName/Build/BenchmarkName.exe

I'm not sure if the runtime Mono/IL2CPP can be configured from command line, might need to manually configure the ProjectSettings.asset for it. Also, the API compatibility level will need to be configurable (.Net 4.X/.Net Standard 2.0/.Net Standard 2.1).

And finally executed like this:

BDNUnityProject_BenchmarkName/Build/BenchmarkName.exe -batchmode -nographics <bdn arguments here>

Another thing I'm unsure of is how the Unity process will communicate with the BDN console process. I haven't dug in too deeply how it currently works with spawned benchmark processes.

And, of course, all this requires the user to have a licensed copy of Unity (free license is enough).

---

Should BenchmarkDotNet build a code binary with dotnet and then include that in the project or copy the code directly?

In this case, I don't think it really matters either way. But BDN already has a C# source generator for external-process, so may as well re-use it (in-process wouldn't even be possible here anyway).

Though, another question comes out of this... how would we support code with #if ENABLE_IL2CPP preprocessor symbols and the like? Or separate dependencies for Unity or full DotNet (like MessagePack). But maybe that's outside the scope of this issue? This may be related to #1773.

Unlike dotnet, Unity doesn't add itself to PATH nor provide an easy way to access its install location. This could make specifing the Engine path by the user required.

Yes, I don't think there's any way around that. And since that is the case, I also don't think that BDN should care about the Unity version (unless/until Unity ships IL2CPP runtime separately).

Should the template project provide optional Unity features like DOTS in order to make it possible to benchmark them?

I would say no. The template can be just the same as the .Net 5/6 template, just with UnityMono or UnityIL2CPP as the runtime.

I'm not sure if all diagnosers like MemoryDiagnoser or DisassemblyDiagnoser (this project could provide some insight with this one for Mono) would be possible to implement.

MemoryDiagnoser depends on AppDomain.MonitoringIsEnabled, which mono doesn't support already, and I'm not sure if IL2CPP supports it. Nice idea for the mono dissassembler, and I also don't know how it could be done with IL2CPP, unfortunately.

[Edit] Perhaps there is something useful here? https://github.com/djkaty/Il2CppInspector

AFAIK Unitys Terms of Service wouldn't allow the redistribution of referenece assemblies generated against their code, so this would require some cooperation from Unity and their legal team.

With my method of project generation above, this shouldn't be a concern. It has zero Unity dependencies. [Edit] I guess this is only for BenchmarkDotNet.Unity.

[MichalPetryka]

I think that avoiding the usage of the full engine here isn't a good idea, after all without the overhead it introduces, this won't result in testing in a realistic usecase, since nobody would use just their Mono.

I'm not sure how we could compile with Unity dependencies from BenchmarkDotNet.Unity inside the console app and then translate them over to package dependencies in the generated project, maybe you know how that could be done?

The BenchmarkDotNet.Unity would just be ref assemblies to compile against, the actual implementation of them would be provided by the engine.

I'm not sure if the runtime Mono/IL2CPP can be configured from command line, might need to manually configure the ProjectSettings.asset for it.

This could be handled by using -executeMethod instead, which would then set the build configuiration and start a build.

Also, the API compatibility level will need to be configurable (.Net 4.X/.Net Standard 2.0/.Net Standard 2.1).

The non framework compatibility levels only limit the available API surface, so the framework ones can be used here in all cases. But settings like IL2CPP build configuration or Incremental GC should be configurable since they have runtime impact.

And since that is the case, I also don't think that BDN should care about the Unity version (unless/until Unity ships IL2CPP runtime separately).

That's a fair point, I guess it being non versioned might actually be better. But the reference assemblies would still be versioned.

MemoryDiagnoser depends on AppDomain.MonitoringIsEnabled, which mono doesn't support already, and I'm not sure if IL2CPP supports it.

Unity profiler API could possibly be used for this

[timcassell]

All good points.

The non framework compatibility levels only limit the available API surface, so the framework ones can be used here in all cases. But settings like IL2CPP build configuration or Incremental GC should be configurable since they have runtime impact.

Net 4.X and NetStandard 2.1 actually diverge in their API surfaces, because Net 4.X does not implement NetStandard 2.1. NetStandard 2.1 has some APIs that Net 4.X does not, and vice-versa.

[timcassell]

Another thing to think about concerning benchmarking Unity APIs is, as it is currently implemented, BDN wouldn't be able to measure any of Unity's AsyncOperations, because those all rely on main thread frame updates for their completions. BDN's engine implementation would need to be adjusted to be able to measure across multiple frames. The current Task measurement would not suffice, because it blocks the calling thread.

[MichalPetryka]

The non framework compatibility levels only limit the available API surface, so the framework ones can be used here in all cases. But settings like IL2CPP build configuration or Incremental GC should be configurable since they have runtime impact.

Net 4.X and NetStandard 2.1 actually diverge in their API surfaces, because Net 4.X does not implement NetStandard 2.1. NetStandard 2.1 has some APIs that Net 4.X does not, and vice-versa.

In Unity they actually made the ref assemblies for .Net 4.x include everything.

[timcassell]

Well there are a lot of build options, and even different options between unity versions. So maybe BDN could allow the user to specify the path to a project settings file to be copied into the benchmark project.

[adamsitnik]

Does Unity support C# Source Generators? I wonder if we could implement the support for it (#1770) instead of introducing a new Toolchain.

cc @xoofx who might be interested in having the support as well

[joshpeterson]

Does Unity support C# Source Generators? I wonder if we could implement the support for it (#1770) instead of introducing a new Toolchain.

Yes, Unity supports in version 2021.2 and later.

[neuecc]

@joshpeterson
Are you sure you're saying Unity officialy support SourceGenerator?
Indeed, Unity 2021.2 adds support for C# 9.0.
However, some functions will not work.
For example, SkipLocalsInit, ModuleInitilizer, etc.
Also, record, init will not compile as-is due to lack of IsExternalInitAttribute.

I also tried to load SourceGenerator using the same procedure as the Analyzer support, but I don't think it works.
Also, the csproj generated by Unity is still in legacy format, is the IDE support ok?

[joshpeterson]

Are you sure you're saying Unity officialy support SourceGenerator?

Yes! I'm not too familiar with the support myself, but you can find Unity's official documentation about them here: https://docs.unity3d.com/2021.2/Documentation/Manual/roslyn-analyzers.html

Also, record, init will not compile as-is due to lack of IsExternalInitAttribute.

This is an unfortunate issue because Unity does not yet support .NET 5 class libraries. You can add a definition of IsExternalInitAttribute to work around this issue though.

Also, the csproj generated by Unity is still in legacy format, is the IDE support ok?

I'm not certain about this - but yes, Unity's generated .csproj files are still in the older format.

[neuecc]

@joshpeterson
Thanks, I've reported to forum.
It's probably not working.
https://forum.unity.com/threads/unity-future-net-development-status.1092205/page-11#post-7715218

[V000id]

I just like to say, I as probably many others look forward a lot to see BenchmarkDotNet working with Unity.
(I didn't found another way to upvote this much appreciated ticket)

[Blackclaws]

So I've been looking for a way to run a set of standardized benchmarks in Unity that use the Graphics engine as well. BenchmarkDotNet seemed like an obvious candidate, however it seems that it currently doesn't support Unity officially at least.

One way I can think off is that you build a ready to benchmark executable with Unity and then just use BenchmarkDotNet to run that executable, passing various parameters. However this currently doesn't really instrument very well. You can do this by running executables from within the code being benchmarked but that measures all kind of overhead from the operating system as well. It would be nice if you could add instrumentation to the Unity code and let that communicate with the external benchmarking process.

Especially since you might want to benchmark your code on different hardware platforms as well this could take the form of a server-client architecture where Unity acts as the client to a benchmarking process that runs on the machine collecting data but the app under test would run on a device connected via the local network.

[Blackclaws]

Having thought about this some more I don't see why BenchmarkDotNet wouldn't be able to run from within Unity if it has the required extension points for custom builders etc.

Some target platforms require manual intervention anyway such as iOS/Android where you deploy to a remote device. In those cases automation might be hard and BenchmarkDotNet can't run completely autonomous. In those cases it would act as the coordinator which it could do just as well from within Unity, waiting at certain points for the interaction with the built projects to be ready.

Does BenchmarkDotNet have extension points for adding runtimes/build scripts/communication with created projects or is it extremely opinionated in that regard?

[adamsitnik]

Does BenchmarkDotNet have extension points for adding runtimes/build scripts/communication

We provide such extension points and contributions that add support for new runtimes are welcomed.

In #1483 you can see what it took to implement a WASM toolchain. Toolchain consists of generator (generates C# code and a project file), builder (a thing that knows how to build given project) and an executor (a thing that runs the produced exe).

[xoofx]

@tgjones developed recently some specialized toolchain+runtime for running BenchmarkDotNet with Unity Mono, IL2CPP and even Burst. He had only one small change via #2285 to make it possible. We cannot make it OSS but for anyone that would like to develop an OSS version, it's doable to make it working (and Tim might be able to give some high level pointers here if there are questions)

[adamsitnik]

developed recently some specialized toolchain+runtime for running BenchmarkDotNet with Unity Mono, IL2CPP and even Burst

Impressive!

We cannot make it OSS

@xoofx is there any chance you could release it as a NuGet package?