Unexpected zinc cache invalidation after sbt remote cache hit

[bjaglin]

steps

Unfortunately, I haven't been able to extract a repro for this, as this issue is transient (but consistent for a given remote cache artifact). I decided to open a ticket to start adding information along my investigation to see if I can find a pattern.

Context

• Zulu OpenJDK 11.0.8 on Linux
• scala 2.13.4 and 2.13.5
• sbt 1.4.6 and 1.4.8
• remote caching activated with an adhoc remoteCacheId
  • remote cache artifact producer & consumer is the same machine (CI Jenkins slave running sbt in Docker)
  • problem is repeatable on another machine consuming the artifact

problem

Zinc runs incrementally after a remote cache artifact hit on IntegrationTest, detecting a change in an untouched marker trait trait ConnectTransactor defined in Compile (on which Zinc had a cache hit after pulling the remote cache).

expectation

Zinc cache hit - "No changes"

notes

Here are incOptions.value.withApiDebug(true) logs:

[debug] [zinc] IncrementalCompile -----------
[debug] IncrementalCompile.incrementalCompile
[debug] previous = Stamps for: 181 products, 50 sources, 27 libraries
[debug] current source = Set(${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/CreateManagedBuyerSeatForEntityHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/UpdateBuyerSeatFeeHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/addUrlBasedConversion/AddUrlBasedConversionTestRunner.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/ensureVisitsConversionExistsForBuyerPixelAndDomain/EnsureVisitsConversionExistsForBuyerPixelAndDomainSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/removeUrlBasedConversion/RemoveUrlBasedConversionSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/CreateIoBuyerSeatForEntityHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/TransitionIoBuyerSeatToManagedHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/CreateAdvertiserForPublisherHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/ActivateFeatureForSeatHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/TransitionSelfServedBuyerSeatToManagedHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/scalatest/Fixtures.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/removeUrlBasedConversion/RemoveUrlBasedConversionTestRunner.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/CreateSelfServedBuyerSeatForEntityHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/CreateAgencyForPublisherHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/DeleteAdvertiserHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/models/BuyerSeat.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/listeners/CleanOutdatedConversionsStatisticsListenerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/repositories/BuyerSeatRepositorySpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/OneAdvertiserAppPerCucumberScenario.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/DeleteAgencyHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/RenameAgencyHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/addEventConversionToAdvertiser/AddEventConversionToAdvertiserSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/MarkClientMsaFilledForSeatHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/EnableBuyerSeatHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/scalatest/OneAdvertiserAppPerSuite.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/MarkClientBillingDetailsFilledForSeatHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/setWebsitesDomainNamesForPixelsAnonymously/SetWebsitesDomainNamesForPixelsAnonymouslySpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/DisableBuyerSeatHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/TransitionIoBuyerSeatToSelfServedHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/repositories/StandardEventRepositorySpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/ensureVisitsConversionExistsForBuyerPixelAndDomain/EnsureVisitsConversionExistsForBuyerPixelAndDomainTestRunner.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/RenameAdvertiserHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/SharePixelWithBuyerHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/scalatest/AdvertiserSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/MarkTeadsFinanceApprovalFilledForSeatHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/repositories/AdvertiserRepositoryFixtures.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/WriteThroughDatabaseCache.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/scalatest/AdvertiserFixtures.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/models/Advertiser.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/TransitionManagedBuyerSeatToSelfServedHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/StepCommon.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/addEventConversionToAdvertiser/AddEventConversionToAdvertiserTestRunner.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/controllers/handlers/DeactivateFeatureForSeatHandlerSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/scalatest/PixelServiceSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/scalatest/ConversionServiceSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/models/Agency.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/setWebsitesDomainNamesForPixelsAnonymously/TestRunner.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/repositories/ConversionRepositorySpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/cucumber/addUrlBasedConversion/AddUrlBasedConversionSpec.scala, ${BASE}/domains/advertiser/advertiser-commands/src/it/scala/tv/teads/domains/advertiser/commands/services/repositories/PixelRepositorySpec.scala)
[debug] [diff] Detected a change in a public API (tv.teads.domains.advertiser.commands.ConnectTransactor):
[debug] [diff] abstract trait tv.teads.domains.advertiser.commands.ConnectTransactor extends java.lang.this#Object with scala.this#Any {
[debug] [diff] 
[debug] [diff] 
[debug] [diff] 
[debug] [diff] }
[debug] [diff] object tv.teads.domains.advertiser.commands.ConnectTransactor extends  {
[debug] [diff] 
[debug] [diff] 
[debug] [diff] 
[debug] [diff] }
[debug] [diff] Detected a change in a public API (tv.teads.domains.advertiser.commands.AdvertiserCommandsDependencies):
[debug] [diff] abstract trait tv.teads.domains.advertiser.commands.AdvertiserCommandsDependencies extends tv.teads.domains.commons.this#CommonDependencies with java.lang.this#Object with scala.this#Any {
[debug] [diff] 
[debug] [diff] 
[debug] [diff] 
[debug] [diff] }
[debug] [diff] object tv.teads.domains.advertiser.commands.AdvertiserCommandsDependencies extends  {
[debug] [diff] 
[debug] [diff] 
[debug] [diff] 
[debug] [diff] }
[debug] > initialChanges = InitialChanges(Changes(added = Set(), removed = Set(), changed = Set(), unmodified = ...),Set(),Set(),API Changes: Set(NamesChange(tv.teads.domains.advertiser.commands.ConnectTransactor,ModifiedNames(changes = UsedName(finalize,[Default]))), NamesChange(tv.teads.domains.advertiser.commands.AdvertiserCommandsDependencies,ModifiedNames(changes = UsedName(finalize,[Default])))))
[debug] The tv.teads.domains.advertiser.commands.ConnectTransactor has the following regular definitions changed:
[debug] 	UsedName(finalize,[Default]).
...

I have tried to diff the text format representation of inc_compile.bin for Compile & IntegrationTest before and after the recompilation, but I haven't found anything interesting (the diff is hard to read as order changes). I am not including it here as it's massive, but happy to provide extracts.

[bjaglin]

I have compared the relevant part of api_companions.bin (both taken from the remote cache artifacts) between Compile and IntegrationTest, and it confirms that the stamping differs.

I still don't understand why though. Could it be related to https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/java/lang/Object.java#L557 ? A classpath change between 1614849670371 & 1614849586071 that would either expose the annotation or not? The same JDK was used for both compilations. The fact that the Compile timestamp comes ater the IntegrationTest one is suspicious.

-  internal {
+  external {
     key: "tv.teads.domains.advertiser.commands.ConnectTransactor"
     value {
-      compilationTimestamp: 1614849670371
+      compilationTimestamp: 1614849586071
       name: "tv.teads.domains.advertiser.commands.ConnectTransactor"
       api {
         classApi {
           name: "tv.teads.domains.advertiser.commands.ConnectTransactor"
           access {
             public {
             }
           }
           modifiers {
             flags: 1
           }
           selfType {
             emptyType {
             }
           }
           structure {
             parents {
               projection {
                 id: "Object"
                 prefix {
                   singleton {
                     path {
                       components {
                         id {
                           id: "java"
                         }
                       }
                       components {
                         id {
                           id: "lang"
                         }
                       }
                       components {
                         this {
                         }
                       }
                     }
                   }
                 }
               }
             }
             parents {
               projection {
                 id: "Any"
                 prefix {
                   singleton {
                     path {
                       components {
                         id {
                           id: "scala"
                         }
                       }
                       components {
                         this {
                         }
                       }
                     }
                   }
                 }
               }
             }
           }
           savedAnnotations: "scala.native"
           savedAnnotations: "jdk.internal.HotSpotIntrinsicCandidate"
           topLevel: true
         }
         objectApi {
           name: "tv.teads.domains.advertiser.commands.ConnectTransactor"
           access {
             public {
             }
           }
           modifiers {
           }
           definitionType: MODULE
           selfType {
             emptyType {
             }
           }
           structure {
           }
           topLevel: true
         }
       }
-      apiHash: 176768709
+      apiHash: -488308903
       nameHashes {
         name: "notify"
         hash: 1514289040
       }
       nameHashes {
         name: "wait"
         hash: 1963066972
       }
       nameHashes {
         name: "equals"
         hash: 1626438048
       }
       nameHashes {
         name: "asInstanceOf"
         hash: -2008440760
       }
       nameHashes {
         name: "synchronized"
         hash: 1581358093
       }
       nameHashes {
         name: "notifyAll"
         hash: 287659096
       }
       nameHashes {
         name: "isInstanceOf"
         hash: -42141924
       }
       nameHashes {
         name: "ConnectTransactor"
         hash: 1351354997
       }
       nameHashes {
         name: "=="
         hash: 1290070777
       }
       nameHashes {
         name: "clone"
         hash: -1872329449
       }
       nameHashes {
         name: "toString"
         hash: -799817619
       }
       nameHashes {
         name: "!="
         hash: -1478124343
       }
       nameHashes {
         name: "getClass"
         hash: 1196203508
       }
       nameHashes {
         name: "ne"
         hash: -1383471943
       }
       nameHashes {
         name: "eq"
         hash: -491545741
       }
       nameHashes {
         name: "##"
         hash: 1393487132
       }
       nameHashes {
         name: "finalize"
-        hash: -245298853
+        hash: 1383774555
       }
       nameHashes {
         name: "hashCode"
         hash: -2114247454
       }
-      extraHash: 176768709
+      extraHash: -488308903
     }
   }