libfuzzer-coverage analyzes unrelated build plugin classes

[ben-manes]

The Caffeine fuzz test failed because the jacoco code coverage tool is used to analyze all *.class files regardless of their relevance to the fuzz execution. As the build tool compiles its plugins into class files, those are being picked up for analysis despite being unrelated to the project's own source code. The project uses Java 11, the build tool uses Java 21, and oss-fuzz infrastructure uses Java 17 for running its own tooling. This results in an unsupported class file version, even though each of these should be fully independent.

A best practice is to have Gradle builds organized as custom plugins that configure external plugins, such as testing or static analysis configuration. This way the project build files are short and declarative. This is known as pre-compiled script plugins where the plugins are their own project being built, the jar is placed onto the build tool's classpath, and then available when running the main project's build. This separation allows the build tool to use a different version of Java, which it can self-install, and it prefers the latest for the best developer experience. As a result, the generated class files are a mix of Java versions.

The oss-fuzz execution fails because it tries to parse the class file

./gradle/plugins/build/classes/kotlin/main/Spotbugs_caffeine_gradle$5.class

which is in Java 21 and completely unrelated to the library's code, e.g.

./caffeine/build/classes/java/main/com/github/benmanes/caffeine/cache/LoadingCache.class

The libfuzzer-coverage tool is run by the infrastructure, independent to the configuration provided in projects/caffeine. This means there is no way for a maintainer to resolve this problem directly. Instead, there is an implicit dependency that the covered projects must limit themselves to only use your Java version in their build infrastructure.

Would it be reasonable for the analyzer to treat this as a warning instead of error, skipping over the class files it cannot parse?

logs

Step #5: Running CaffeineSpecFuzzer
Step #5: [INFO] Loading execution data file /workspace/out/libfuzzer-coverage-x86_64/dumps/CaffeineSpecFuzzer.exec.
Step #5: [INFO] Analyzing 65 classes.
Step #5: [INFO] Loading execution data file /workspace/out/libfuzzer-coverage-x86_64/dumps/CaffeineSpecFuzzer.exec.
Step #5: [INFO] Writing execution data to /workspace/out/libfuzzer-coverage-x86_64/dumps/jacoco.merged.exec.
Step #5: [INFO] Loading execution data file /workspace/out/libfuzzer-coverage-x86_64/dumps/jacoco.merged.exec.
Step #5: Exception in thread "main" java.io.IOException: Error while analyzing /workspace/out/libfuzzer-coverage-x86_64/dumps/classes/Spotbugs_caffeine_gradle$5.class.
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzerError(Analyzer.java:162)
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:134)
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:157)
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeAll(Analyzer.java:193)
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeAll(Analyzer.java:226)
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeAll(Analyzer.java:221)
Step #5: 	at org.jacoco.cli.internal.commands.Report.analyze(Report.java:110)
Step #5: 	at org.jacoco.cli.internal.commands.Report.execute(Report.java:84)
Step #5: 	at org.jacoco.cli.internal.Main.execute(Main.java:90)
Step #5: 	at org.jacoco.cli.internal.Main.main(Main.java:105)
Step #5: Caused by: java.lang.IllegalArgumentException: Unsupported class file major version 65
Step #5: 	at org.jacoco.cli.internal.asm.ClassReader.<init>(ClassReader.java:196)
Step #5: 	at org.jacoco.cli.internal.asm.ClassReader.<init>(ClassReader.java:177)
Step #5: 	at org.jacoco.cli.internal.asm.ClassReader.<init>(ClassReader.java:163)
Step #5: 	at org.jacoco.cli.internal.core.internal.instr.InstrSupport.classReaderFor(InstrSupport.java:280)
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:107)
Step #5: 	at org.jacoco.cli.internal.core.analysis.Analyzer.analyzeClass(Analyzer.java:132)
Step #5: 	... 8 more