Nimbus Bug Report: SSZ Deserialization and Database Corruption Issues in v25.5.0

[andib007]

Nimbus Bug Report: SSZ Deserialization and Database Corruption Issues in v25.5.0
Summary
When running trusted node sync with Nimbus v25.5.0 on the Ethereum mainnet, the process fails with the error SSZ BeaconState: object dynamic portion starts at invalid offset. Additionally, after a seemingly successful sync, starting the beacon node via systemctl results in a fatal error: Could not load head state, database corrupt?. Downgrading to v25.4.1 resolves both issues, indicating a bug in v25.5.0.
Environment

Nimbus Version: v25.5.0 (failing), v25.4.1 (working)
Operating System: Linux (specific distribution anonymized)
Hardware: ARM-based single-board computer, 4GB RAM, SSD storage
Network: Ethereum mainnet
Checkpoint Providers Tested:
https://mainnet.checkpointz.sigp-dev.net
https://checkpointz.pietjepuk.net

Execution Client: Geth (version anonymized, confirmed synced in working case)

Steps to Reproduce

Run trusted node sync with v25.5.0:/usr/bin/nimbus_beacon_node trustedNodeSync --network=mainnet --trusted-node-url=https://mainnet.checkpointz.sigp-dev.net --data-dir=/home/XXXX/YYYY/ethereum/consensus/nimbus_database --log-level=DEBUG --backfill=false --reindex=false --state-id=finalized

Observe the sync fails with:ERR 2025-05-22 11:22:39.909+02:00 Unable to download checkpoint state error="SSZ BeaconState: object dynamic portion starts at invalid offset" syncTarget=finalized restUrl=https://mainnet.checkpointz.sigp-dev.net stateId=finalized

Run the same command without sudo -u XXXX:nimbus_beacon_node trustedNodeSync --network=mainnet --trusted-node-url=https://mainnet.checkpointz.sigp-dev.net --data-dir=/home/XXXX/YYYY/ethereum/consensus/nimbus_database --log-level=DEBUG --backfill=false --reindex=false --state-id=finalized

Sync succeeds, logging:NTC 2025-05-22 11:27:01.920+02:00 Database initialized from genesis blockRoot=4d611d5b93fdab69013a7f0a2f961caca0c853f87cfe9595fe50038163079360 stateRoot=7e76880eb67bbdc86250aa578958e9d0675e64e714337855204fb5abaaf82c2b stateSlot=0
NTC 2025-05-22 11:31:00.763+02:00 Checkpoint written to database blockRoot=87bda670a0290cadadfeea3fa852b82fcfb6811c363ceddcf354a6d13be27ccc stateRoot=49a293c4ccf22e1124cbda5dbce09b207c09fef9213daa0ff28b60dc9ddd82da stateSlot=11756736

Note: The stateSlot=0 log is incorrect; the correct slot (11756736) is logged later.
Start the Nimbus service via systemctl:sudo systemctl start nimbus_beacon_node

Observe the service fails with:FAT 2025-05-22 11:44:38.725+02:00 Could not load head state, database corrupt? head=87bda670:11756736 tail=87bda670:11756736

Downgrade to v25.4.1 and repeat steps 1-5. The sync and service startup succeed without errors.

Expected Behavior

Trusted node sync with v25.5.0 should successfully download and deserialize the finalized state from the checkpoint provider.
The database should initialize with the correct slot (e.g., ~11756736) and load without corruption errors when starting the beacon node via systemctl.

Actual Behavior

In v25.5.0, trusted node sync fails with SSZ BeaconState: object dynamic portion starts at invalid offset when run with sudo -u XXXX.
Even when sync succeeds without sudo, the database logs stateSlot=0 initially (likely a logging bug), and the systemctl service fails with Could not load head state, database corrupt?.
Downgrading to v25.4.1 resolves both issues, with successful sync and service startup.

Additional Observations

The Geth execution client logged a warning:WARN [05-22|11:44:43.018] Post-merge network, but no beacon client seen. Please launch one to follow the chain!

This was resolved by ensuring Geth and Nimbus use the same JWT secret and Engine API port (8551).
The stateSlot=0 log during successful sync in v25.5.0 is misleading, as the correct slot (11756736) is logged later, suggesting a logging or initialization bug.
The issue persists across multiple checkpoint providers, indicating a client-side problem in v25.5.0.

Logs
Relevant logs from v25.5.0:
NTC 2025-05-22 11:25:36.116+02:00 Starting trusted node sync databaseDir=/home/XXXX/YYYY/ethereum/consensus/nimbus_database/db backfill=false reindex=false syncTarget=finalized restUrl=https://mainnet.checkpointz.sigp-dev.net
NTC 2025-05-22 11:25:36.144+02:00 Downloading checkpoint state syncTarget=finalized restUrl=https://mainnet.checkpointz.sigp-dev.net stateId=finalized
DBG 2025-05-22 11:25:36.148+02:00 Sending REST request to remote server remote=mainnet.checkpointz.sigp-dev.net:443 request=/eth/v2/debug/beacon/states/finalized http_method=GET
DBG 2025-05-22 11:25:37.584+02:00 Got REST response headers from remote server remote=mainnet.checkpointz.sigp-dev.net:443 request=/eth/v2/debug/beacon/states/finalized status=200 http_method=GET
DBG 2025-05-22 11:26:38.720+02:00 Received REST response body from remote server remote=mainnet.checkpointz.sigp-dev.net:443 request=/eth/v2/debug/beacon/states/finalized contentType=application/octet-stream size=271079848
NTC 2025-05-22 11:27:01.920+02:00 Database initialized from genesis blockRoot=4d611d5b93fdab69013a7f0a2f961caca0c853f87cfe9595fe50038163079360 stateRoot=7e76880eb67bbdc86250aa578958e9d0675e64e714337855204fb5abaaf82c2b stateSlot=0
NTC 2025-05-22 11:31:00.763+02:00 Checkpoint written to database blockRoot=87bda670a0290cadadfeea3fa852b82fcfb6811c363ceddcf354a6d13be27ccc stateRoot=49a293c4ccf22e1124cbda5dbce09b207c09fef9213daa0ff28b60dc9ddd82da stateSlot=11756736
INF 2025-05-22 11:31:09.463+02:00 Loading finalized blocks finHigh=none() finalizedHead=87bda670:11756736
INF 2025-05-22 11:31:10.267+02:00 Block DAG initialized head=87bda670:11756736 finalizedHead=87bda670:11756736 tail=87bda670:11756736
FAT 2025-05-22 11:44:38.725+02:00 Could not load head state, database corrupt? head=87bda670:11756736 tail=87bda670:11756736

Suggested Fix

Investigate SSZ deserialization in v25.5.0 for compatibility with mainnet checkpoint providers.
Fix the database initialization to prevent corruption when loading via systemctl.
Correct the logging of stateSlot=0 during trusted node sync to reflect the actual finalized slot.

Workaround

Downgrade to Nimbus v25.4.1:sudo apt install nimbus=25.4.1

Ensure the systemctl service runs as the same user (XXXX) as the sync command:[Service]
User=XXXX
Group=XXXX
ExecStart=/usr/bin/nimbus_beacon_node --network=mainnet --data-dir=/home/XXXX/YYYY/ethereum/consensus/nimbus_database --log-level=INFO --tcp-port=9000 --udp-port=9000 --jwt-secret=/path/to/jwtsecret

Verify Geth sync and Engine API configuration.

Additional Notes

The issue was reproducible on an ARM-based single-board computer with SSD storage, suggesting possible architecture-specific bugs in v25.5.0.
Community assistance was sought via EthStaker Discord, but no direct fix was found until downgrading.

References

Nimbus Guide: Trusted Node Sync
Nimbus GitHub Releases
EthStaker Discord

[etan-status]

Works on macOS ARM with v25.5.0-d2f233-stateofus.

 % build/nimbus_beacon_node trustedNodeSync --network=mainnet --trusted-node-url=https://mainnet.checkpointz.sigp-dev.net --data-dir=$HOME/Downloads/ethereum/consensus/nimbus_database --log-level=DEBUG --backfill=false --reindex=false --state-id=finalized
NTC 2025-05-22 14:11:33.520+02:00 Starting trusted node sync                 databaseDir=/Users/etan/Downloads/ethereum/consensus/nimbus_database/db backfill=false reindex=false syncTarget=finalized restUrl=https://mainnet.checkpointz.sigp-dev.net
NTC 2025-05-22 14:11:33.564+02:00 Downloading checkpoint state               syncTarget=finalized restUrl=https://mainnet.checkpointz.sigp-dev.net stateId=finalized
DBG 2025-05-22 14:11:33.565+02:00 Sending REST request to remote server      remote=mainnet.checkpointz.sigp-dev.net:443 request=/eth/v2/debug/beacon/states/finalized http_method=GET
DBG 2025-05-22 14:11:35.237+02:00 Got REST response headers from remote server remote=mainnet.checkpointz.sigp-dev.net:443 request=/eth/v2/debug/beacon/states/finalized status=200 http_method=GET
DBG 2025-05-22 14:12:05.364+02:00 Received REST response body from remote server remote=mainnet.checkpointz.sigp-dev.net:443 request=/eth/v2/debug/beacon/states/finalized contentType=application/octet-stream size=271016456
NTC 2025-05-22 14:12:06.711+02:00 Database initialized from genesis          blockRoot=4d611d5b93fdab69013a7f0a2f961caca0c853f87cfe9595fe50038163079360 stateRoot=7e76880eb67bbdc86250aa578958e9d0675e64e714337855204fb5abaaf82c2b stateSlot=0
NTC 2025-05-22 14:12:23.493+02:00 Checkpoint written to database             blockRoot=a7c01df1ccde9cc72c521ee67e9ce5f2d924abc5c1c11762ff249fe8981c131e stateRoot=91894b385b3b8ba71013c54927358cbe4c8179827c13a4d0b446e0fc3e9bb119 stateSlot=11757568
INF 2025-05-22 14:12:24.431+02:00 Loading finalized blocks                   finHigh=none() finalizedHead=a7c01df1:11757568
INF 2025-05-22 14:12:24.606+02:00 Block DAG initialized                      head=a7c01df1:11757568 finalizedHead=a7c01df1:11757568 tail=a7c01df1:11757568 backfill="(11757569, \"a7c01df1\")" loadDur=385us summariesDur=838ms950us417ns finalizedDur=273ms953us83ns frontfillDur=96us125ns keysDur=70us584ns
NTC 2025-05-22 14:12:24.606+02:00 Database initialized, historical blocks will be backfilled when starting the node missingSlots=1056769 backfill="(slot: 11757569, parent_root: a7c01df1ccde9cc72c521ee67e9ce5f2d924abc5c1c11762ff249fe8981c131e)" horizon=10700800 syncTarget=finalized restUrl=https://mainnet.checkpointz.sigp-dev.net
NTC 2025-05-22 14:12:24.606+02:00 Done, your beacon node is ready to serve you! Don't forget to check that you're on the canonical chain by comparing the checkpoint root with other online sources. See https://nimbus.guide/trusted-node-sync.html for more information. checkpoint=a7c01df1:11757568 syncTarget=finalized restUrl=https://mainnet.checkpointz.sigp-dev.net

[etan-status]

The only change between v25.4.1 and v25.5.0 is MEV related:

https://github.com/status-im/nimbus-eth2/compare/v25.4.1..v25.5.0

So, I doubt that this is a version specific issue.

[etan-status]

Also works with the exact same slot that you fetched (using different trusted server, because the one that you listed already deleted that checkpoint).

% build/nimbus_beacon_node trustedNodeSync --network=mainnet --trusted-node-url=http://unstable.mainnet.beacon-api.nimbus.team --data-dir=$HOME/Downloads/ethereum/consensus/nimbus_database --log-level=DEBUG --backfill=false --reindex=false --state-id=11756736
NTC 2025-05-22 14:18:32.819+02:00 Starting trusted node sync                 databaseDir=/Users/etan/Downloads/ethereum/consensus/nimbus_database/db backfill=false reindex=false syncTarget=11756736 restUrl=http://unstable.mainnet.beacon-api.nimbus.team
NTC 2025-05-22 14:18:32.821+02:00 Downloading checkpoint state               syncTarget=11756736 restUrl=http://unstable.mainnet.beacon-api.nimbus.team stateId=11756736
DBG 2025-05-22 14:18:32.821+02:00 Sending REST request to remote server      remote=unstable.mainnet.beacon-api.nimbus.team:80 request=/eth/v2/debug/beacon/states/11756736 http_method=GET
DBG 2025-05-22 14:18:45.010+02:00 Got REST response headers from remote server remote=unstable.mainnet.beacon-api.nimbus.team:80 request=/eth/v2/debug/beacon/states/11756736 status=200 http_method=GET
DBG 2025-05-22 14:18:54.765+02:00 Received REST response body from remote server remote=unstable.mainnet.beacon-api.nimbus.team:80 request=/eth/v2/debug/beacon/states/11756736 contentType=application/octet-stream size=271079848
NTC 2025-05-22 14:18:56.123+02:00 Database initialized from genesis          blockRoot=4d611d5b93fdab69013a7f0a2f961caca0c853f87cfe9595fe50038163079360 stateRoot=7e76880eb67bbdc86250aa578958e9d0675e64e714337855204fb5abaaf82c2b stateSlot=0
NTC 2025-05-22 14:19:12.836+02:00 Checkpoint written to database             blockRoot=87bda670a0290cadadfeea3fa852b82fcfb6811c363ceddcf354a6d13be27ccc stateRoot=49a293c4ccf22e1124cbda5dbce09b207c09fef9213daa0ff28b60dc9ddd82da stateSlot=11756736
INF 2025-05-22 14:19:13.787+02:00 Loading finalized blocks                   finHigh=none() finalizedHead=87bda670:11756736
INF 2025-05-22 14:19:13.961+02:00 Block DAG initialized                      head=87bda670:11756736 finalizedHead=87bda670:11756736 tail=87bda670:11756736 backfill="(11756737, \"87bda670\")" loadDur=282us833ns summariesDur=842ms599us125ns finalizedDur=281ms647us667ns frontfillDur=108us250ns keysDur=28us208ns
NTC 2025-05-22 14:19:13.961+02:00 Database initialized, historical blocks will be backfilled when starting the node missingSlots=1056769 backfill="(slot: 11756737, parent_root: 87bda670a0290cadadfeea3fa852b82fcfb6811c363ceddcf354a6d13be27ccc)" horizon=10699968 syncTarget=11756736 restUrl=http://unstable.mainnet.beacon-api.nimbus.team
NTC 2025-05-22 14:19:13.961+02:00 Done, your beacon node is ready to serve you! Don't forget to check that you're on the canonical chain by comparing the checkpoint root with other online sources. See https://nimbus.guide/trusted-node-sync.html for more information. checkpoint=87bda670:11756736 syncTarget=11756736 restUrl=http://unstable.mainnet.beacon-api.nimbus.team

[etan-status]

For reference, my hardware, working well here.

% sysctl hw.model
hw.model: Mac16,7
 
% sw_vers
ProductName:		macOS
ProductVersion:		15.5
BuildVersion:		24F74

So, at the very least, the "architecture-specific bugs" don't seem ARM related.

[etan-status]

As 25.5.0 does not have significant changes compared to 25.4.1, and 25.4.1 worked well for you, would you mind please retrying the process, so that intermittent network issues with that specific checkpoint provider can be ruled out?

Otherwise, please add more detailed info about your specific hardware.

Please further try with a different checkpoint provider. This issue is unlikely to be 25.5.0 vs 25.4.1 specific.

[andib007]

The issue arrived on an arm64 on a raspberry Pi4. The checkpoint syn with the trusted node sync command went through but the nimbus node could not process the fetched checkpoint. It raised the error that the database is corrupted. After downgrading to 25.4.1 the sync process started flawlessly.

[etan-status]

I don't think the node sync command "went through", as your logs lack this message:

NTC 2025-05-22 14:19:13.961+02:00 Done, your beacon node is ready to serve you! Don't forget to check that you're on the canonical chain by comparing the checkpoint root with other online sources. See https://nimbus.guide/trusted-node-sync.html for more information. checkpoint=87bda670:11756736 syncTarget=11756736 restUrl=http://unstable.mainnet.beacon-api.nimbus.team

When this message is missing, the sync was not completed successfully in your case, and the database may have been left in an inconsistent state, which explains why subsequently trying to load it won't work.

I've also verified that I can launch the BN from the DB

% build/nimbus_beacon_node --network=mainnet --data-dir=$HOME/Downloads/ethereum/consensus/nimbus_database --log-level=DEBUG --el=http://127.0.0.1:8551 --jwt-secret=/Users/etan/Library/Ethereum/geth/jwtsecret 
INF 2025-05-22 14:31:35.383+02:00 Launching beacon node                      topics="beacnde" version=v25.5.0-d2f233-stateofus bls_backend=BLST const_preset=mainnet cmdParams=...
INF 2025-05-22 14:31:36.448+02:00 Threadpool started                         topics="beacnde" numThreads=14
INF 2025-05-22 14:31:38.840+02:00 Loading block DAG from database            topics="beacnde" path=/Users/etan/Downloads/ethereum/consensus/nimbus_database/db
INF 2025-05-22 14:31:40.043+02:00 Block DAG initialized                      head=87bda670:11756736 finalizedHead=87bda670:11756736 tail=87bda670:11756736 backfill="(11756737, \"87bda670\")" loadDur=1ms155us750ns summariesDur=842ms634us finalizedDur=358ms991us584ns frontfillDur=92us791ns keysDur=32us834ns
DBG 2025-05-22 14:31:40.043+02:00 Pruning history                            topics="chaindag" horizon=10699968 blockHorizon=10698945 stateHorizon=10699776 lastHorizon=10699968 lastBlockHorizon=10698944 tail=87bda670:11756736 head=87bda670:11756736
DBG 2025-05-22 14:31:40.044+02:00 Backfill database has been loaded          topics="beacnde" path=/Users/etan/Downloads/ethereum/consensus/nimbus_database/db head=[none] tail=[none]
INF 2025-05-22 14:31:40.044+02:00 Backfill database initialized              topics="beacnde" path=/Users/etan/Downloads/ethereum/consensus/nimbus_database/db head=[none] tail=[none]
DBG 2025-05-22 14:31:40.044+02:00 Initializing ELManager                     topics="beacnde" depositContractBlockNumber=11052984 depositContractBlockHash=44bca881

Was the --data-dir that you use completely empty before you started the trustedNodeSync command?

[etan-status]

And, also please confirm that the systemctl start nimbus_beacon_node was not also running at the same time as the trustedNodeSync; even if you did not start it manually, it may still have been run automatically, possibly interfering with the sync, as both the trustedNodeSync command and the main startup command operate on the same --data-dir.

In any case, 25.5.0 and 25.4.1 are almost identical. Nothing changed between those two versions that would explain suddenly having SSZ processing issues. The full difference can be seen here (yes, it's really just that tiny change):

• https://github.com/status-im/nimbus-eth2/compare/v25.4.1..v25.5.0

If you could reproduce it one more time, it may be worth investigating further. But as you got it working in v25.4.1, and it works well on my system as well, there is not enough info to systematically analyze this.

[tersec]

The only change between v25.4.1 and v25.5.0 is MEV related:

https://github.com/status-im/nimbus-eth2/compare/v25.4.1..v25.5.0

So, I doubt that this is a version specific issue.

Also the BN is functionally identical (it's possible there's some slight codegen differences due to changing function names in some modules, but the BN does not call those functions), even for MEV. The MEV fix is 100% for the VC, not the BN.