Possible deadlock introduced in `anvil >= 1.1.0`

[MartinquaXD]

Component

Anvil

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

No response

What version of Foundryup are you on?

No response

What command(s) is the bug in?

No response

Operating System

None

Describe the bug

We are using anvil for e2e tests. Most of the tests use an empty testnet (i.e. default anvil) but some rely on a forked network (i.e. --fork-url ). Our forked e2e tests started failing with v1.1.0 but worked fine with v1.0.0. The tests using empty testnets don't seem to be affected by the switch to the new version.

Unfortunately the errors always happen in CI but so far never when I ran the tests locally which makes debugging this more difficult. From what I can tell it seems like anvil can get stuck (perhaps a deadlock?) after receiving a evm_mine request (presumably only for forked setups).
This also looks like a race condition. When I ran anvil with RUST_LOG=debug which introduces some overhead due to excessive logging the issue happened less often.
Please see this PR for some more context of the investigation.

Given that this is probably a PITA to debug because reproducing this locally wasn't possible so far I could use our CI to do a git bisect to hopefully find the culprit. Please let me know if this is necessary since this will be quite time consuming.

[mattsse]

hmm,

could you perhaps try to repro this with dc69097

and if unable to repro maybe with f474801

which could be the culprit

[grandizzy]

@MartinquaXD the releases to pin in CI for commits above are nightly-54af8693b8fcc7a3f65136c1188e89661155955d if this works OK please pin to nightly-23191fbeccfcf901f7c28590cb962d9693373c21 and see if reproducible. thank you!

[MartinquaXD]

nightly-54af8693b8fcc7a3f65136c1188e89661155955d and nightly-23191fbeccfcf901f7c28590cb962d9693373c21 both did not reproduce the error.

[grandizzy]

@MartinquaXD thank you, could you also please check nightly-f0e24fb6cfc9018d986d3136ab00aa5077994994. thank you!

[MartinquaXD]

nightly-f0e24fb6cfc9018d986d3136ab00aa5077994994 reproduced the issue, although it's not the culprit since the release right before (nightly-bfb1cb6b118b7e07a1de0fdea64a6cf44eb3a174) already had the problem.

[grandizzy]

I see, would you be able to bisect / try different nightly versions to figure out which one is the culprit or can you share a way I can conduct such? thank you!