Launch benchmarks on multiple targets from a single separate source set

[alexismanin]

Is it possible to create a single benchmark class in a dedicated source set, and then define multiple benchmark targets (one for each kotlin multiplatform compilation target in the project) to launch the same benchmark ?

I am currently trying to achieve that in this little Github project (currently on a dedicated refactor/kotlinx-benchmark branch).

So far, I succeeded to launch my benchmark from a separate source set on JVM target only. All other targets launches, but no benchmark is executed for them.

I have not found any example that do exactly that, so I am wondering if it is currently achievable. If not, what are the alternatives ? Would it work if I move my benchmarks in "commonMain" source set ?

For reference, here is what I've tried:

I've set up my build file by mixing information from doc: separate benchmark source set and this issue about simplifying separate source-set.

Therefore, I've:

1. Applied kotlin multiplatform and kotlinx benchmark plugins
2. registered kotlin multiplatform targets (jvm, linuxX64, js, wasmJs)
3. applied Kotlin default hierarchy template,
4. Defined a common main source set for my code that is not benchmark code
5. Defined a benchmark source set that contains my Benchmark code
6. Modified all defined targets to depend on the benchmark code (create a dedicated KotlinCompilation depending on benchmark source set)
7. Added benchmark runtime dependency on common main implementation
8. Defined benchmark targets : jvmBenchmark, linuxX64Benchmark, jsBenchmark, wasmJsBenchmark

Regards,

[fzhinkin]

@alexismanin kotlinx-benchmark can require a class holding benchmarks to be annotated with @State (which is quite inconvenient, #29). And I don't think kotlinx-benchmark supports separate state object.

The change like this makes benchmarks running:

@Warmup(iterations = 1, time = 1)
 @Measurement(iterations = 1, time = 1)
+@State(Scope.Benchmark)
 open class StringArrayToList {
+    @Param("32")
+    var arraySize: Int? = null
 
-    @State(Scope.Benchmark)
-    open class ArrayInput {
-
-        @Param("32")
-        var arraySize: Int? = null
-        // For now, do not make it varying, because array to list should only copy references
-        val txtSize: Int = 32
-        lateinit var array: Array<String>
+    // For now, do not make it varying, because array to list should only copy references
+    val txtSize: Int = 32
+    lateinit var array: Array<String>
 
-        @Setup
-        fun init() {
-            val arraySize = checkNotNull(arraySize) { "arraySize parameter not set"}
-            array = Array(arraySize) { Random.nextBytes(txtSize).joinToString() }
-        }
+    @Setup
+    fun init() {
+        val arraySize = checkNotNull(arraySize) { "arraySize parameter not set" }
+        array = Array(arraySize) { Random.nextBytes(txtSize).joinToString() }
     }
 
     @Benchmark
-    open fun usingStd(state: ArrayInput) = state.array.toList()
+    open fun usingStd() = array.toList()
 
     @Benchmark
-    open fun usingListOf(state: ArrayInput) = state.array.toListUsingListOf()
+    open fun usingListOf() = array.toListUsingListOf()
 
 
     @Benchmark
-    open fun usingCopyOfAsList(state: ArrayInput) = state.array.toListUsingAsList()
+    open fun usingCopyOfAsList() = array.toListUsingAsList()
 }

[alexismanin]

Thank you @fzhinkin . Your solution works perfectly. I've completely missed this requirement.

If there's no plan to remove the requirement to annotate benchmark classes with @State in short term, maybe we could make it clearer for users, by :

• Issuing a compiler warning when compiling a benchmark whose class do not hold a state annotation (maybe that's difficult from a technical point, I don't know)
• Emphasize the requirement in the README, either by making related text bold, or, even better, by putting it in a dedicated admonition (i.e. "IMPORTANT" block).

I can provide a PR for the second point (make it clearer that @State is required in documentation), if there's any interest about that.

P.S : The "Contribution guidelines" link on the bottom of the project README is broken. It links to a "CONTRIBUTING.md" file at the root of the project, and there is none currently.