Fix for the "compiler mirror not found" error when using dynamic execution + improvements made along the way #91
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jjudd
force-pushed
the
jjudd-scala-compiler-mirror-error-fix
branch
from
7054423 to
3abe071
Compare
This was referenced Apr 18, 2025
jnowjack-lucidchart
approved these changes
Apr 18, 2025
src/main/scala/higherkindness/rules_scala/workers/zinc/compile/ZincRunner.scala
Show resolved
Hide resolved
src/main/scala/higherkindness/rules_scala/common/worker/WorkerMain.scala
Show resolved
Hide resolved
jadenPete
reviewed
Apr 18, 2025
src/main/scala/higherkindness/rules_scala/common/worker/WorkerMain.scala
Outdated
Show resolved
Hide resolved
src/main/scala/higherkindness/rules_scala/common/worker/WorkerMain.scala
Outdated
Show resolved
Hide resolved
src/main/scala/higherkindness/rules_scala/common/worker/CancellableTask.scala
Show resolved
Hide resolved
| @@ -76,6 +76,10 @@ object ZincRunnerWorkerConfig { | |||
| */ | |||
| object ZincRunner extends WorkerMain[ZincRunnerWorkerConfig] { | |||
|
|
|||
| // Interrupting concurrent Zinc/Scala compilation with a shared ScalaInstance (and thus shared classloaders) | |||
There was a problem hiding this comment.
It may be worth elaborating here on your hypothesis for why these errors are happening.
src/main/scala/higherkindness/rules_scala/workers/common/AnnexScalaInstance.scala
Show resolved
Hide resolved
src/main/scala/higherkindness/rules_scala/workers/zinc/compile/ZincRunner.scala
Show resolved
Hide resolved
jadenPete
approved these changes
Apr 18, 2025
We were copying a file used by multiple threads directly to its destination. Problem is that copying is not an atomic action, so we could end up in states where the file wasn't correct when it was used. This should avoid that issue by first copying the file to a temp file and then using an atomic move to move the file to the destination used by other threads.
jjudd
force-pushed
the
jjudd-scala-compiler-mirror-error-fix
branch
2 times, most recently
from
3f63f80 to
815b291
Compare
…s with Thread.interrupt Prior to this change there was a bug when ScalaCompile actions were used with dynamic execution. The bug would cause builds to fail with the error message "compiler mirror not found". It is my understanding this message typically indicates a problem loading the Scala Library jar during compilation. The error would always occur in the local multiplex worker's logs and not the remote logs. This commit fixes this error by not using Thread.interrupt when cancelling the future running the ScalaCompile action. My hypothesis on why this error occurs is as follows: imagine one of the classloaders in the shared ScalaInstance is in use on Thread A. That thread gets cancelled and interrupted via Thread.interrupt. This causes some kind of persistent error inside either the classloader or another part of Zinc or the Scala compiler. We correctly ignore the error on Thread A because the request it was handling was cancelled. Another thread is working a non-cancelled request, it uses that same ScalaInstance, hit that persistent error, and fails. By not using Thread.interrupt we don't trigger the persistent error and thus avoid the bug.
jjudd
force-pushed
the
jjudd-scala-compiler-mirror-error-fix
branch
from
815b291 to
9534999
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The primary problem this PR fixes is described below. I made a few other improvements along the way and included them on this PR as well.
Fix race condition in ScalaCompile by selectively interrupting futures with Thread.interrupt
Prior to this change there was a bug when ScalaCompile actions were used with dynamic execution. The bug would cause builds to fail with the error message "compiler mirror not found". It is my understanding this message typically indicates a problem loading the Scala Library jar during compilation.
The error would always occur in the local multiplex worker's logs and not the remote logs.
This commit fixes this error by not using Thread.interrupt when cancelling the future running the ScalaCompile action.
My hypothesis on why this error occurs is as follows: imagine one of the classloaders in the shared ScalaInstance is in use on Thread A. That thread gets cancelled and interrupted via Thread.interrupt. This causes some kind of persistent error inside either the classloader or another part of Zinc or the Scala compiler. We correctly ignore the error on Thread A because the request it was handling was cancelled. Another thread is working a non-cancelled request, it uses that same ScalaInstance, hit that persistent error, and fails.
By not using Thread.interrupt we don't trigger the persistent error and thus avoid the bug.