Skip to content

Some Cardano DB snapshots require full chain replay #2474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 tasks done
jpraynaud opened this issue May 13, 2025 · 0 comments · Fixed by #2484
Closed
2 tasks done

Some Cardano DB snapshots require full chain replay #2474

jpraynaud opened this issue May 13, 2025 · 0 comments · Fixed by #2484
Assignees
@dlachaume
Labels
bug ⚠️ Something isn't working

Comments

@jpraynaud
Copy link
Member

jpraynaud commented May 13, 2025 •
edited by dlachaume
Loading

Why

Some Cardano DB snapshots provided by Mithril require a full chain replay when restarting the Cardano node. This is likely due to an invalid ledger state snapshot being embedded in the snapshot. This has been reported by SPOs on Discord.

The error message is:

Invalid snapshot DiskSnapshot {dsNumber = 79987069, dsSuffix = Nothing}InitFailureRead (ReadSnapshotFailed (ReadFailed (DeserialiseFailure 443263608 "end of input"))) This is most likely an expected change in the serialization format, which currently requires a chain replay

It has been reproduced locally with the snapshot c27d3a7cfa5836cbb393e9c75626f7b467ac2fe871a65140742cbcf1bcbd9064 of the pre-release-preview network:

[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:18.83 UTC] Started opening Chain DB
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:18.83 UTC] Started opening Immutable DB
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:19.41 UTC] Validating chunk no. 18515 out of 18515. Progress: 99.99%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:19.72 UTC] Validated chunk no. 18515 out of 18515. Progress: 100.00%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:19.72 UTC] Found a valid last location at chunk 18515 with tip 4717551f930b8eddbad8c68fb3edf47e8be95613d9f2fb87695d7eb394705c8e@79987069.
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:19.72 UTC] Opened imm db with immutable tip at 4717551f930b8eddbad8c68fb3edf47e8be95613d9f2fb87695d7eb394705c8e at slot 79987069 and chunk 18515
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:19.72 UTC] Started opening Volatile DB
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:19.72 UTC] Opened vol db with max slot number NoMaxSlotNo
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:19.72 UTC] Started opening Ledger DB
[e3b3f767:cardano.node.ChainDB:Error:5] [2025-05-13 09:00:24.92 UTC] Invalid snapshot DiskSnapshot {dsNumber = 79987069, dsSuffix = Nothing}InitFailureRead (ReadSnapshotFailed (ReadFailed (DeserialiseFailure 443263608 "end of input"))) This is most likely an expected change in the serialization format, which currently requires a chain replay
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:24.92 UTC] Replaying ledger from genesis
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:24.93 UTC] Replayed block: slot 0 out of 79987069. Progress: 0.00%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:24.95 UTC] Replayed block: slot 4300 out of 79987069. Progress: 0.01%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:24.97 UTC] Replayed block: slot 8620 out of 79987069. Progress: 0.01%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.00 UTC] Replayed block: slot 12940 out of 79987069. Progress: 0.02%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.02 UTC] Replayed block: slot 17260 out of 79987069. Progress: 0.02%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.05 UTC] Replayed block: slot 21580 out of 79987069. Progress: 0.03%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.07 UTC] Replayed block: slot 25900 out of 79987069. Progress: 0.03%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.10 UTC] Replayed block: slot 30220 out of 79987069. Progress: 0.04%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.12 UTC] Replayed block: slot 34540 out of 79987069. Progress: 0.04%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.15 UTC] Replayed block: slot 38860 out of 79987069. Progress: 0.05%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.17 UTC] Replayed block: slot 43180 out of 79987069. Progress: 0.05%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.20 UTC] Replayed block: slot 47500 out of 79987069. Progress: 0.06%
[e3b3f767:cardano.node.ChainDB:Info:5] [2025-05-13 09:00:25.22 UTC] Replayed block: slot 51820 out of 79987069. Progress: 0.06%

What

Investigate and fix the source of the problem with the ledger state snapshot.

How

  • Investigate the source of the problem (in particular try to reproduce on preprod and mainnet)
  • Create a fix for the invalid ledger state snapshot creation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dlachaume dlachaume

Labels
bug ⚠️ Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix: include the two latest ledger state snapshots in the ancillary archive input-output-hk/mithril
2 participants
@jpraynaud @dlachaume