Seeking advice for handling java.lang.VerifyError with java.util.concurrent (a.k.a executor) instrumentation

[chris-pinola-rf]

Hi folks,

I'm fairly new to the JVM ecosystem, but I'm attempting to instrument a variety of Kotlin and Java applications with the OpenTelemetry agent.

The applications that I'm instrumenting also use the Rollout.io/CloudBees Feature Management (docs, Maven package) SDK. The Rollout SDK uses java.util.concurrent to perform asynchronous network I/O. Therefore the OpenTelemetry agent's executor module is injecting some bytecode into those library functions to facilitate context propagation.

The problem is that when I enable the OpenTelemetry agent on these applications, the Rollout SDK fails to initialize and throws a java.lang.VerifyError. From what I can tell, for security purposes, the JVM attempts to verify library bytecode before executing it. Since the agent has injected bytecode into some of the java.util.concurrent classes, that library bytecode no longer matches the JVM's "known good" version for said classes and throws this error at runtime.

I've identified two possible workarounds:

1. Disable the java.util.concurrent instrumentation by setting the OTEL_INSTRUMENTATION_EXECUTOR_ENABLED environment variable to false.
2. Disable the JVM's "verifier" by setting the JDK_JAVA_OPTIONS environment variable to -noverify.

Both workarounds, however, have significant downsides. We have some code that also uses java.util.concurrent and would benefit from the executor instrumentation module. Also, my gut tells me that disabling verification is going to open the door to some potentially dangerous/difficult to debug issues.

So, I'm looking for advice on what to do. Are there other alternatives that I haven't discovered? Have I completely misdiagnosed the situation? I would greatly appreciate any suggestions. Thanks!

[trask]

hey @chris-pinola-rf! it seems like a bug in the javaagent (the goal is definitely for this kind of thing not to happen 😅), is there any more detail that is printed out with the VerifyError?

[mateuszrzeszutek]

Hey @chris-pinola-rf
Asking just to make sure: which version of the javaagent are you using?

[chris-pinola-rf]

This exception was observed from an application that was running version 1.6.2 of the javaagent, but I can reproduce this exact error on 1.7.0 too.

[mateuszrzeszutek]

Looks like a fresh bug then. Can you create a bug issue for this? Please include the JVM version and vendor, javaagent version, rollout library version, etc.
It'd be very helpful if you could provide a link to a repro scenario; a simple app that causes this issue.
Thanks!

[chris-pinola-rf]

@mateuszrzeszutek Okay, I'll try to have that to you later today. 👍

[chris-pinola-rf]

I've filed a bug issue #4511 which also links to the repro repo. Please let me know if I can provide any more info.