Class versions V1_5 or less must use F_NEW frames

[stanosamek]

Hi,

I would like to ask. During the startup of application, Otel agent is logging exception below. Is this correct behaviour or where can be a problem? As you can see we are using Spring Boot, Netty with Java 11.0.11. When I'm starting application without Otel agent, application is starting without this error.

Thank you

2021-12-27 10:21:40.303 app INFO [] [] [] [] [main] istrationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.security.config.annotation.method.configuration.ReactiveMethodSecurityConfiguration' of type [org.springframework.security.config.annotation.method.configuration.ReactiveMethodSecurityConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) java.lang.IllegalArgumentException: Class versions V1_5 or less must use F_NEW frames. at reactor.tools.shaded.net.bytebuddy.jar.asm.MethodWriter.visitFrame(MethodWriter.java:779) at reactor.tools.shaded.net.bytebuddy.jar.asm.MethodVisitor.visitFrame(MethodVisitor.java:300) at reactor.tools.shaded.net.bytebuddy.jar.asm.ClassReader.readCode(ClassReader.java:2075) at reactor.tools.shaded.net.bytebuddy.jar.asm.ClassReader.readMethod(ClassReader.java:1514) at reactor.tools.shaded.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:744) at reactor.tools.shaded.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:424) at reactor.tools.agent.ReactorDebugAgent$1.transform(ReactorDebugAgent.java:104) at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246) at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188) at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at org.springframework.boot.web.embedded.netty.NettyWebServer.<init>(NettyWebServer.java:87) at org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactory.createNettyWebServer(NettyReactiveWebServerFactory.java:80) at org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactory.getWebServer(NettyReactiveWebServerFactory.java:72) at org.springframework.boot.web.reactive.context.WebServerManager.<init>(WebServerManager.java:49) at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.createWebServer(ReactiveWebServerApplicationContext.java:93) at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.onRefresh(ReactiveWebServerApplicationContext.java:78) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:577) at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:63) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:771) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:763) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:438) at org.springframework.boot.SpringApplication.run(SpringApplication.java:339) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318)

[iNikem]

It seems that exception is thrown by Reactor Debug Agent. Can you try to disable it temporarily and run your application again? This way we can make sure that the combination of this agent and OpenTelemetry ones causes the problem.

[stanosamek]

Hi @iNikem,

I didn't find a way to temporarily disable Reactor debug agent. This agent is automatically initialized if reactor-tools is in application classpath. So, I'll have to wait to colleague for editing application.

[laurit]

Firstly Bean 'org.springframework.security.config.annotation.method.configuration.ReactiveMethodSecurityConfiguration' of type [org.springframework.security.config.annotation.method.configuration.ReactiveMethodSecurityConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) and IllegalArgumentException are really two separate messages. Which of these are you concerned about? Assuming it is the IllegalArgumentException it is not printed by otel agent but reactor debug agent see https://github.com/reactor/reactor-tools/blob/1e7d455aa0f038b53c3a02f143aba3c6c778da58/reactor-tools/src/main/java/reactor/tools/agent/ReactorDebugAgent.java#L156 Although you can see that reactor debug agent rethrows this exception it will not propagate into application code because sun.instrument.TransformerManager#transform swallows all exceptions that happen during transformation. Due to this even when this stack trace is logged it should not break your application.
It is unlikely that we'll be able to make any progress with this without your help. I'd start by confirming that you really don't have the same stack trace when running without otel agent. If you are completely sure that this happens only when otel agent is present, attach debugger and figure out the class for which the transformation fails. You could also write the class bytes to disk and compare them with the bytes of the same class that is included inside your application to see whether the class has been transformed.