Bug: Chain fork on 2026-01-04 due to nondeterministic AT ordering (created_when ties)

[QuickMythril]

Summary

A chain fork occurred on 2026-01-05 where peers diverged at the same height. Investigation shows the root cause is nondeterministic ordering of ATs when multiple ATs share the same created_when timestamp. That ordering affects AT state hashing, so different nodes can compute different AT state hashes for the same block, causing AT_STATES_MISMATCH and a persistent fork.

Impact

• Nodes on different forks rejected each other’s blocks with AT_STATES_MISMATCH.
• Some nodes could not reorg to the higher-weight chain because validation failed on AT state hash mismatch.
• Restarting a node can change DB row order and thus AT execution order, leading to “random” fork selection.

Fork Details

• Fork height: 2424270 (last common height 2424269)
• Two competing blocks at 2424270 (different minter/timestamp/transactionsSignature)
• AT ordering tie observed for two ATs with identical created_when
  • ATs: AS9tDBxMXawVPM1gH4wcdMP3TC9PsTjj7i and ASE6ZEuU3YaMqxqKA3tvwh6pa7ZdyyfHWh
  • Same creation timestamp: 1767469125122
• Error observed: AT_STATES_MISMATCH (not a timestamp validation issue)

Root Cause (Code-Level)

• getAllExecutableATs() (AT execution) orders by created_when only.
• getBlockATStatesAtHeight() (AT state retrieval) orders by created_when only.
• When created_when ties occur, DB row order is undefined.
• AT transactions preserve existing order (AT vs AT comparator returns 0), so the tie order leaks into AT state hashing.
• AT state hash depends on AT order → different order → different hash → consensus failure.

Proposed Fix (Consensus Change)

Introduce deterministic ordering at/after a new activation height:

• Order by created_when, then AT_address (lexicographic).
• Apply to:
  • executable AT list
  • per-block AT state ordering
• This is consensus-critical; must be activated at an agreed height.

Implementation

Branch: bugfix/at-ordering

Includes:

• New feature trigger: deterministicAtOrderingHeight (placeholder height)
• Deterministic ordering at/after trigger:
  • ORDER BY created_when, AT_address
• Test: AtRepositoryTests#testDeterministicAtOrderingAfterTrigger
• Updated blockchain config + test chain configs to include the new trigger

Not included:

• Any fork-rescue or reorg tooling
• Additional investigation documents (omitted from this branch)

Notes for Deployment

• Requires a coordinated activation height on mainnet.
• This prevents repeat forks after activation but does not automatically repair existing divergence.
• A manual reorg/rescue is still required if the network is already split.

References

• Commit: fb570d1 (Add deterministic AT ordering trigger)