docs(shadow-testing): verifier wrapper lessons, anvil gas fix, and l2_block repair script

- AGENTS.md: Add verifier wrapper deployment pitfalls
  - anvil_setCode does NOT reset immutables (wrong digests)
  - Do not 'fix' production Solidity without evidence
  - finalizeBundlePostEuclidV2 access control notes

- estimategas.go: Fix Anvil eth_estimateGas failure
  - Strip GasFeeCap/GasTipCap from CallMsg before EstimateGas
  - Set high Gas limit to prevent Gas=0 rejection

- shadow-testing docs: Update Sepolia traps and lessons learned
  - L1MessageQueueV2 slot 104 verification
  - Anvil gas estimation and sender balance traps
  - psql timeout vs TCP connectivity diagnosis

- Add fix_l2_block_transactions.py script for repairing missing
transaction data in shadow DB from L2 RPC.

Author: lispc

AGENTS.md

@@ -49,9 +49,12 @@ Key hard-won rules:
 |-----------|---------|---------|------|
 | **DB port** | `localhost:5433` (shadow) / `15432` (RDS tunnel) | `localhost:25432` (RDS tunnel) | Wrong port = connecting to mainnet data |
 | **L2 RPC** | `l2geth-rpc-proxy.mainnet.aws.scroll.io` | `l2geth-rpc-proxy.sepolia.aws.scroll.io` | Public Sepolia RPC (`sepolia-rpc.scroll.io`) rejects `debug_executionWitness` |
-| **Verifier** | Mainnet has `latestVerifier[10] = 0x0dE1...` (can `anvil_setCode`) | Sepolia has `latestVerifier[10] = 0x0` (must deploy fresh) | Cannot copy verifier — must deploy `ZkEvmVerifierPostFeynman` + register |
+| **Verifier** | Mainnet has `latestVerifier[10] = 0x0dE1...` (can `anvil_setCode`) | Production proofs + production MVRV may already match | Re-using old proofs → check MVRV first. Testing **new guest** → MUST deploy fresh verifier |
 | **`committedBatches`** | Sparse, but fork block usually covers target batches | Sparse; **every bundle end batch must exist** | Missing entry → `ErrorIncorrectBatchHash(0x2a1c1442)` |
-| **`L1MessageQueueV2`** | Reset `nextUnfinalizedQueueIndex = 0` sufficient | Must sync `nextCrossDomainMessageIndex == nextUnfinalizedQueueIndex` | Mismatch → `ErrorFinalizedIndexTooLarge(0x16465978)` |
+| **`L1MessageQueueV2`** | Reset `nextUnfinalizedQueueIndex = 0` sufficient | Set to `MIN(total_l1_messages_popped_before)` of first target batch; **slot 104** (verify with `forge inspect`) | Wrong slot/value → `ErrorFinalizedIndexTooLarge(0x16465978)` |
+| **Anvil gas estimation** | Same as mainnet | `eth_estimateGas` fails with fee caps present (`Gas=0`) | Patch `estimategas.go` or use `--min-codec-version` workaround |
+| **Sender balance** | Persisted across restarts | **Resets to 0** after Anvil restart | Must re-fund EOAs before each relayer start |
+| **Relayer flags** | Standard | Requires `--config <path>` AND `--min-codec-version 10` | Missing flags = wrong config or immediate exit |
 | **DB scope** | Imported limited range | Full production snapshot (batches 128080+) | Relayer batch committer floods logs with commit retries |
 | **Blob version** | Usually V0 | Anvil 1.0.0 cannot decode BlobSidecar V1 | Set `fusaka_timestamp: 2000000000` in relayer config |
 | **Proofs in DB** | May already be v0.8.0 | Old proofs are v0.7.3 | Must reset `proving_status = 1` to regenerate with v0.8.0 |
@@ -100,6 +103,28 @@ make coordinator_setup
 | `zkvm-prover/` | Build scripts and runtime config for the prover binary |
 | `build/dockerfiles/` | Dockerfiles for production images |
 
+## Troubleshooting: Verifier Wrapper Deployment on Shadow Forks
+
+### `anvil_setCode` Does NOT Reset Immutables
+- **Problem**: Copying a mainnet verifier wrapper (e.g., `ZkEvmVerifierPostFeynman`) to Anvil via `anvil_setCode` preserves the **original immutables** (`verifierDigest1`, `verifierDigest2`, `protocolVersion`). These digests are bound to the mainnet plonk verifier VK and will **never** match locally-generated proofs.
+- **Symptom**: `VerificationFailed` (selector `0x439cc0cd`) even when the plonk verifier binary, public input hash, and proof are all individually correct.
+- **Root cause**: The wrapper assembles its own `instances` array from immutables + `keccak256(protocolVersion || publicInput)`. Wrong immutables = wrong instances = plonk verifier rejects the proof.
+- **Solution**: **Always recompile and redeploy** the wrapper with immutables extracted from the *local* proof's `instances` array (bytes 384–416 and 416–448 for digest1/digest2).
+
+### Do Not "Fix" Production Solidity Without Evidence
+- **Problem**: When `VerificationFailed` appears, it's tempting to blame the assembly loop in the wrapper (`sub(0x5a0, i)` vs `add(0x1c0, i)`).
+- **Reality**: The production wrapper (`ZkEvmVerifierPostFeynman.sol`) has used `sub(0x5a0, i)` since deployment and has finalized thousands of bundles on mainnet. The loop direction maps hash bytes in **reverse order** to instance words, which matches the verifier circuit's expectation.
+- **Symptom of wrong patch**: Changing the loop to `add(0x1c0, i)` inverts the hash-word layout, producing a different set of instances that also fail verification.
+- **Correct diagnosis flow**:
+  1. Verify the plonk verifier binary matches the deployed contract runtime code.
+  2. Verify `keccak256(abi.encodePacked(protocolVersion, publicInput))` matches the proof metadata `bundle_pi_hash`.
+  3. Verify the wrapper's immutables match the local proof's digest words.
+  4. Only after (1–3) pass should you look at Solidity logic — and even then, production code is almost certainly correct.
+
+### Access Control on `finalizeBundlePostEuclidV2`
+- `ScrollChain.finalizeBundlePostEuclidV2` has `OnlyProver` modifier.
+- On shadow fork, impersonate the registered prover EOA before sending the transaction: `cast rpc anvil_impersonateAccount <prover_address>`.
+
 ## Troubleshooting Common E2E Test Issues
 
 ### Port Conflicts (Shared Servers)
@@ -177,4 +202,6 @@ make coordinator_setup
 | [`docs/testing/openvm-upgrade-testing-guide.md`](docs/testing/openvm-upgrade-testing-guide.md) | Step-by-step testing checklist after OpenVM / zkvm-prover upgrades |
 | [`docs/testing/docker-compose-e2e-guide.md`](docs/testing/docker-compose-e2e-guide.md) | Production-like E2E testing with Docker Compose + Coordinator Proxy |
 | [`tests/shadow-testing/docs/README.md`](tests/shadow-testing/docs/README.md) | Shadow coordinator + local prover setup for production task replay |
+| [`tests/shadow-testing/docs/LESSONS_LEARNED.md`](tests/shadow-testing/docs/LESSONS_LEARNED.md) | Hard-won debugging knowledge from past shadow tests (read before experimenting) |
+| [`tests/shadow-testing/docs/QUICKSTART.md`](tests/shadow-testing/docs/QUICKSTART.md) | Quick reference for common shadow testing commands |
 | [`docs/testing_reports/openvm-v1.6.0-guest-v0.8.0-May19.md`](docs/testing_reports/openvm-v1.6.0-guest-v0.8.0-May19.md) | Test report for PR #1783 (OpenVM 1.6.0, guest v0.8.0) |

rollup/internal/controller/sender/estimategas.go

@@ -110,6 +110,7 @@ func (s *Sender) estimateGasLimit(to *common.Address, data []byte, sidecar *type
 	msg := ethereum.CallMsg{
 		From:      s.transactionSigner.GetAddr(),
 		To:        to,
+		Gas:       10000000, // Set a high gas limit to prevent Anvil from rejecting eth_estimateGas when Gas=0
 		GasPrice:  gasPrice,
 		GasTipCap: gasTipCap,
 		GasFeeCap: gasFeeCap,
@@ -121,9 +122,20 @@ func (s *Sender) estimateGasLimit(to *common.Address, data []byte, sidecar *type
 		msg.BlobGasFeeCap = blobGasFeeCap
 	}
 
-	gasLimitWithoutAccessList, err := s.client.EstimateGas(s.ctx, msg)
+	// Anvil has a bug where eth_estimateGas fails with "Out of gas" when
+	// maxFeePerGas/maxPriorityFeePerGas are present. We create a copy without
+	// gas price fields for the estimation call.
+	estimateMsg := msg
+	estimateMsg.GasPrice = nil
+	estimateMsg.GasTipCap = nil
+	estimateMsg.GasFeeCap = nil
+	if sidecar != nil {
+		estimateMsg.BlobGasFeeCap = nil
+	}
+
+	gasLimitWithoutAccessList, err := s.client.EstimateGas(s.ctx, estimateMsg)
 	if err != nil {
-		log.Error("estimateGasLimit EstimateGas failure without access list", "error", err, "msg", fmt.Sprintf("%+v", msg))
+		log.Error("estimateGasLimit EstimateGas failure without access list", "error", err, "msg", fmt.Sprintf("%+v", estimateMsg))
 		return 0, nil, err
 	}
 

tests/shadow-testing/AGENTS.md

@@ -26,7 +26,7 @@ Before executing a single command:
 | ScrollChain proxy | `0xa13BAF47339d63B743e7Da8741db5456DAc1E556` | `0x2D567EcE699Eabe5afCd141eDB7A4f2D0D6ce8a0` | `cast call <ADDR> "lastFinalizedBatchIndex()(uint256)"` |
 | MVRV | `0x4CEA3E866e7c57fD75CB0CA3E9F5f1151D4Ead3F` | `0x8A360c7F6fca548507017DdeD732bFe7E078F963` | `cast call <ADDR> "latestVerifier(uint256)" 10` |
 | L1MessageQueueV2 | `0x56971da63A3C0205184FEF096E9ddFc7A8C2D18a` | `0xA0673eC0A48aa924f067F1274EcD281A10c5f19F` | `cast call <ADDR> "nextUnfinalizedQueueIndex()(uint256)"` |
-| Verifier | Copy from mainnet (`anvil_setCode`) | Must deploy fresh (`PostFeynman`) | `cast call <MVRV> "getVerifier(uint256,uint256)" 10 <batchIndex>` |
+| Verifier | Copy from mainnet (`anvil_setCode`) | Check MVRV first; may already match | `cast call <MVRV> "getVerifier(uint256,uint256)" 10 <batchIndex>` |
 
 ## Critical Traps (Do Not Skip)
 
@@ -58,9 +58,12 @@ Before executing a single command:
 - **Rule**: The contract checks `committedBatches[batchIndex]` where `batchIndex` is the **end batch** of the bundle. Verify this is non-zero before finalizing.
 
 ### Trap 5: `L1MessageQueueV2` Index Mismatch (Sepolia)
-- **Symptom**: `ErrorFinalizedIndexTooLarge(0x16465978)`.
-- **Cause**: `nextUnfinalizedQueueIndex < totalL1MessagesPoppedOverall` because `nextCrossDomainMessageIndex` was not synced.
-- **Rule**: On Sepolia, `nextCrossDomainMessageIndex` and `nextUnfinalizedQueueIndex` must be **equal** before finalizing.
+- **Symptom**: `ErrorFinalizedIndexTooLarge(0x16465978)` or `ErrorFinalizedIndexTooSmall`.
+- **Cause**: `nextUnfinalizedQueueIndex` does not match the pre-finalization state expected by the first target batch. The fork block is AFTER real finalization, so the real state has post-finalization values.
+- **Rule**: 
+  1. Set `nextUnfinalizedQueueIndex` to `MIN(total_l1_messages_popped_before)` of the first target batch's chunks (from DB).
+  2. **Use `forge inspect L1MessageQueueV2 storage-layout`** to find the exact storage slot (it's slot 104, NOT slot 0 or 4, due to OpenZeppelin `__gap`).
+  3. Never guess storage slots from source code.
 
 ### Trap 6: Parent Batch Missing
 - **Symptom**: Relayer logs `Batch.GetBatchByIndex error: record not found, index: <parent>`.
@@ -81,6 +84,27 @@ Before executing a single command:
 - **Cause**: Import script copied `proof` columns from production RDS, overwriting locally-valid shadow proofs.
 - **Rule**: **Never import `proof` columns**. Import only metadata, then reset `proving_status = 1` and re-prove locally.
 
+### Trap 9: Anvil `eth_estimateGas` Rejects Fee Caps
+- **Symptom**: `failed to get fee data, err: Out of gas: gas required exceeds allowance: 0`.
+- **Cause**: Anvil's `eth_estimateGas` fails when `CallMsg` has `GasFeeCap`/`GasTipCap` set but `Gas` is 0 (Go Ethereum client's default).
+- **Rule**: If testing relayer against Anvil and gas estimation fails, patch `estimategas.go` to strip fee caps from the `EstimateGas` call (see `LESSONS_LEARNED.md` for exact patch).
+
+### Trap 10: Sender Balance Lost After Anvil Restart
+- **Symptom**: `failed to send transaction, err: Insufficient funds for gas * price + value` even after successful gas estimation.
+- **Cause**: `anvil_setBalance` funds do not persist across Anvil restarts.
+- **Rule**: After every Anvil restart, verify and re-fund sender EOAs before starting the relayer:
+  ```bash
+  cast balance 0x410E7FD80a3Fc1E62A4D3450d11b71b812006eB9 --rpc-url http://localhost:18546
+  ```
+
+### Trap 11: Relayer Started Without Required Flags
+- **Symptom**: Relayer prints help and exits with `Required flag "min-codec-version" not set`, or connects to wrong DB.
+- **Cause**: `ROLLUP_RELAYER_CONFIG` env var is NOT supported. The relayer uses `--config` CLI flag.
+- **Rule**: Always start relayer with BOTH flags:
+  ```bash
+  ./rollup_relayer --config /path/to/config.json --min-codec-version 10
+  ```
+
 ## Step-by-Step Checklist
 
 ### Phase 0: Environment Validation
@@ -100,16 +124,35 @@ Before executing a single command:
 ### Phase 2: Anvil Fork Setup
 - [ ] Start Anvil forked from **Ethereum L1** (not Scroll L2)
 - [ ] Verify `eth_chainId == 1`
-- [ ] Fund owner and sender accounts
+- [ ] Fund owner and sender accounts (verify balances after any Anvil restart)
 - [ ] Add prover EOA to `ScrollChain`
 - [ ] Set `lastFinalizedBatchIndex` to `(first_target_batch - 1)`
 - [ ] Set `lastCommittedBatchIndex` to mainnet value (do NOT reset to lastFinalized)
-- [ ] (Sepolia) Sync `L1MessageQueueV2.nextCrossDomainMessageIndex == nextUnfinalizedQueueIndex`
+- [ ] (Sepolia) Verify end-batch `committedBatches` hashes are non-zero on Anvil
+- [ ] (Sepolia) Set `L1MessageQueueV2.nextUnfinalizedQueueIndex` to pre-finalization value:
+  ```sql
+  SELECT MIN(total_l1_messages_popped_before)
+  FROM chunk
+  WHERE batch_hash = (SELECT hash FROM batch WHERE index = <first_target_batch>);
+  ```
+- [ ] (Sepolia) **Verify slot number with `forge inspect L1MessageQueueV2 storage-layout`** before `anvil_setStorageAt`
+
+### Phase 3: Verifier Setup
+
+**Determine which scenario you are in:**
 
-### Phase 3: Verifier Deployment
+**Scenario A — Re-using production proofs (like bundles 13445-13449)**
 - [ ] Extract digests from proof instances
+- [ ] Query Sepolia MVRV: `cast call <MVRV> "getVerifier(uint256,uint256)" 10 <batchIndex>`
+- [ ] Query verifier digests: `cast call <verifier> "verifierDigest1()"` / `"verifierDigest2()"`
+- [ ] If digests match → **skip deployment**, use existing verifier
+- [ ] If digests DON'T match → you are actually in Scenario B
+
+**Scenario B — Testing new guest / circuit version (0.8.0 / openvm 1.6+)**
+- [ ] Generate new proofs with the new prover (coordinator + prover pipeline)
+- [ ] Extract digests from **newly-generated** proof instances
 - [ ] Deploy plonk verifier from `coordinator/build/bin/assets_v2/verifier.bin`
-- [ ] Deploy `ZkEvmVerifierPostFeynman` with digests + `protocolVersion = 10`
+- [ ] Deploy `ZkEvmVerifierPostFeynman` with new digests + `protocolVersion = 10`
 - [ ] Register on `MultipleVersionRollupVerifier` via `updateVerifier(10, startBatch, verifier)`
 - [ ] Verify with `getVerifier(10, batchIndex)`
 
@@ -121,23 +164,28 @@ Before executing a single command:
 - [ ] Start prover(s), verify `Got task from coordinator`
 
 ### Phase 5: Relayer Finalize
-- [ ] Build relayer with latest code
+- [ ] Build relayer with latest code (rebuild if `estimategas.go` was patched for Anvil)
 - [ ] Relayer config has `dry_run: false`, correct contract addresses
 - [ ] Clear stale `pending_transaction` entries
-- [ ] Reset target bundles to `rollup_status = 1`
-- [ ] Start relayer
+- [ ] Reset target bundles/batches to `rollup_status = 1`
+- [ ] **Start relayer with `--config <path>` AND `--min-codec-version 10`**
 - [ ] Monitor logs for `finalizeBundle in layer1` success
 - [ ] Verify `lastFinalizedBatchIndex` advanced on Anvil
 
 ## When Things Go Wrong
 
-1. **`VerificationFailed(0x439cc0cd)`** → See Trap 1 (verifier contract type/digests).
-2. **`ErrorIncorrectBatchHash(0x2a1c1442)`** → See Trap 4 (sparse `committedBatches`).
-3. **`ErrorFinalizedIndexTooLarge(0x16465978)`** → See Trap 5 (L1MessageQueueV2 indices).
-4. **`record not found` (parent batch)** → See Trap 6.
-5. **Tx sent but never mined** → See Trap 7 (nonce desync).
-6. **Coordinator says tasks assigned but prover gets nothing** → L2 RPC missing `debug_executionWitness`, or chunks have `codec_version = 5`.
-7. **`CoordinatorEmptyProofData`** → Prover crashed; reset stuck tasks.
+| Error / Symptom | Most Likely Cause | See |
+|-----------------|-------------------|-----|
+| `VerificationFailed(0x439cc0cd)` | Wrong verifier type or digest mismatch | Trap 1 |
+| `ErrorIncorrectBatchHash(0x2a1c1442)` | Sparse `committedBatches`, end batch hash is zero | Trap 4 |
+| `ErrorFinalizedIndexTooLarge(0x16465978)` | `nextUnfinalizedQueueIndex` too low or too high | Trap 5 |
+| `record not found` (parent batch) | Parent batch not imported | Trap 6 |
+| `Out of gas: gas required exceeds allowance: 0` | Anvil gas estimation bug with fee caps | Trap 9 |
+| `Insufficient funds for gas * price + value` | Sender balance is 0 on Anvil | Trap 10 |
+| Tx sent but never mined | Nonce desync (`pending_transaction` stale) | Trap 7 |
+| Relayer exits with `Required flag "min-codec-version" not set` | Missing CLI flags | Trap 11 |
+| Coordinator assigns but prover gets nothing | L2 RPC missing `debug_executionWitness` | README.md |
+| `CoordinatorEmptyProofData` | Prover crashed; reset stuck tasks | README.md |
 
 ## Documentation Priority