Fix #3 - Live Lock. Custom scopes that depend on custom scopes can sometimes create a 'Rescursive Update' problem depending on thread ordering. Fix using a CompletableFuture and atomic map operations.

[exabrial]

Fix #3 - Live Lock. Custom scopes that depend on custom scopes can sometimes create a 'Rescursive Update' problem depending on thread ordering. Fix using a CompletableFuture and atomic map operations.

This is a bit more of an elegant fix than whats suggested in #3

[dblevins]

@exabrial Hey Agent Fisher, can you create one basic test in a separate PR that replicates the issue? (i.e. it fails and shows the problem)

Here are some examples of how I typically write tests for multi-threaded issues.

• https://github.com/apache/tomee/blob/main/container/openejb-core/src/test/java/org/apache/openejb/util/CachedSupplierTest.java#L561-L591

I'd love to see that test to better understand the issue and help ensure the fix is solid.

[exabrial]

Hello David! It appears to be related to: having a bean of one custom scope, containing an injection a bean of the same custom scope, and @PostConstruct methods are being invoked on one or both. We can't reliably recreate the problem which is unfortunate.... it only occurs when you're really busy, production is down, and the boss is wondering what is taking so long to get a release out the door. The blisteringly annoying "Recursive Update" Exception destroys any evidence, and stopping anything with a debugger forces ordering and the problem goes away.

The "Recursive Update" Exception is notated here: https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/ConcurrentHashMap.html#computeIfAbsent(K,java.util.function.Function). The notable line is: The mapping function must not modify this map during computation

So my understanding of the problem is the construction of the scope invokes computeIfAbsent(), which will begin constructing dependencies, which then also attempt to modify the map (while computeIfAbsent() is being invoked), causing the "Recursive Update" Exception.

Given that shaky understanding, I designed this fix around putIfAbsent(...) which is an atomic operation. This sidesteps the problem because constructing a scope may modify the map, and this reserves the place in the map first, then constructs the scope afterwards using CompleteableFuture, so no inflight putIfAbsent(...) operations will complain if the scope construction modifies the map.

Hoping that explanation makes sense and my understanding of the problem is correct. I'll try to create something that reliably reproduces the problem.

[exabrial]

Man, I cannot get this to crash outside of proprietary code. I've tried two custom scopes intermixing calls between in postConstructs.

[exabrial]

Ah ok, I figured it out. I can reproduce it reliably now. The good news is, it's not "random"... the "randomness" is likely due to the order in which tests are firing.

So you have to have a circular injection (I know, yuck. I didn't realize we had one. But technically it is allowed by the CDI Spec). Our circular dependency is that the EntityManager needs a datasource, but the datasource manager needs the name of the entity manager. The datasource manager starts an in-memory db with the entity manager name as part of the url so it can be found later by the entity manager when it creates its connection pool.

Anyway I have a small reproducer with a proprietary custom scope. Trying to see if I can reproduce it using one of the custom scopes in this project.

[exabrial]

I stand corrected, while I can now reproduce the issue, I can't reproduce it without the test target eventually calling a stack overflow. Our production code doesn't cause this of course, but here we are :)