Allow constructing `Wollet` from materialized state, skipping `Update` replay

[ledegenbtc]

Hi everyone,

I run a production hot wallet on top of lwk_wollet that has accumulated 30k+ transactions over its lifetime. As the update log and the in-memory caches grew, the process began to OOM-crash on boot: rehydration via Wollet::new allocates the full historical state (transactions, derivation maps, heights, timestamps) into resident memory before the wallet is usable, and on our hardware that final state crosses the available RAM budget.

We already maintain an external indexer that holds the canonical transaction history and can serve any subset of it on demand, so most of what Wollet rebuilds at boot is data we do not need to keep in process to operate. What we are missing from LWK is a way to construct a Wollet directly from a minimal, application-supplied state — bypassing the Update-replay path that forces full history into memory.

This issue describes that gap and proposes a narrow addition (Wollet::from_state) to close it, while being explicit about which existing LWK mechanisms are not sufficient and why.

Summary

lwk_wollet already exposes a public Persister trait (persister.rs:35) and a Wollet::new(network, Arc<dyn DynStore>, descriptor) constructor (wollet.rs:448) that accepts any custom backend, so applications can already plug in their own storage (database, key-value store, etc.). That part is solved.

The remaining gap is that rehydration always goes through linear replay of every persisted Update in Wollet::restore_updates (update.rs:378):

pub(crate) fn restore_updates(&mut self) -> Result<(), Error> {
    for i in 0.. {
        if let Some(update) = self.updates_persister.get_update(i)? {
            self.apply_update_inner(update)?;
        } else {
            let mut next_update_index = self.updates_persister.next_update_index()?;
            *next_update_index = self.updates_persister.merge_updates(i)?;
            break;
        }
    }
    Ok(())
}

I would like a constructor that takes already-materialized state and skips replay entirely, e.g. Wollet::from_state(network, descriptor, WolletFullState).

Motivation

In production we operate large, long-lived wallets whose persisted update log grows to hundreds of MB and whose fully-rehydrated Wollet reaches multi-GB resident memory. Boot time is dominated by restore_updates, and steady-state memory is dominated by the in-memory caches that replay populates. We have an external indexer that already holds the canonical transaction history and can produce a minimal materialized state on demand; we need a way to feed that state into Wollet without round-tripping through the Update log.

Why existing mechanisms do not solve this

LWK already has compaction-flavored machinery, and it is worth being explicit about why none of it closes this gap:

1. only_tip coalescing (update.rs:145-159). When a new Update only moves the tip and the previous one did too, the persister overwrites the previous entry instead of appending. The Persister trait doc explicitly asks implementors to behave this way.
2. Update::prune (update.rs:231) and the per-batch prune in update.rs:36. Strips transactions/scripts not relevant to the wallet from a single Update before persisting.
3. merge_threshold (wollet.rs:113, implemented in update.rs:91). When more than threshold updates are persisted, they are folded into a single consolidated Update at index 0 via Update::merge (update.rs:314). With with_persisted_txs (wollet.rs:103) this is forced to Some(1), so at most one Update is ever on disk.

These three mechanisms address disk footprint and boot time. They do not address resident memory, which is the motivation here.

A consolidated snapshot-style update — even a hypothetical perfect one — produces the same in-memory Wollet state as replaying the original sequence. After construction, cache.all_txs, the script/derivation maps, the height map, and the unspent set hold exactly the same data regardless of whether they were built from 10,000 small Updates or one merged one. The cost is in the final state, not in the replay path. Replacing many small persisted entries with one large entry is a change of intermediate representation, not of steady-state memory.

Concretely, in our deployment:

Mechanism	Disk	Boot time	Resident RAM
Status quo	~800 MB	~30 s	~7 GB
Hypothetical aggressive merge	~50 MB	~5 s	~7 GB
from_state with semantic prune	application-controlled	~0 s	application-controlled

What would actually reduce resident memory

Semantic pruning, not compaction. A from_state path lets the application supply only the data it needs to operate going forward — for example:

• Only transactions producing UTXOs that are still unspent.
• Only the derivation range from last_unused onward.
• Only the current tip, not historical tips/timestamps.

Trade-offs the application opts into when using this path:

• wallet.transactions() no longer returns full history.
• Reorgs deeper than the snapshot horizon require a rescan.
• Audit/tax features built on top of the wallet must source history elsewhere.

These are acceptable for our use case (transaction history lives in an external indexer); they would not be acceptable as defaults. Hence the request is an additional constructor, not a behavior change to existing ones.

Proposed shape

impl Wollet {
    /// Construct a `Wollet` from already-materialized state, skipping the
    /// `Update`-replay path used by `Wollet::new`.
    ///
    /// The caller is responsible for the consistency of `state`. Fields the
    /// caller omits will be reflected as-is in the resulting wallet — e.g.
    /// transactions not included will not appear in `wallet.transactions()`,
    /// and reorgs deeper than the snapshot horizon will require a rescan.
    pub fn from_state(
        network: Network,
        descriptor: WolletDescriptor,
        state: WolletFullState,
    ) -> Result<Self, Error>;
}

Where WolletFullState is a public, serializable struct exposing the fields needed to populate Cache directly: tip, scripts/paths maps, heights, unspent set, last_unused counters, and whatever subset of transactions the caller chooses to include.

The existing private WolletConciseState (wollet.rs:261) is close to what is needed for the read/scan side, but a write-side equivalent would need the transaction set too. Happy to iterate on the exact shape.

Scope and non-goals

• Pure addition; no change to existing constructors, persister semantics, or default behavior.
• Does not require touching merge_threshold, only_tip coalescing, or Update::prune.
• Not proposing to remove or weaken history retention by default — applications that need full history continue to use Wollet::new.
• Migration path for existing deployments is the application's responsibility: build a WolletFullState once from a full-replayed Wollet, persist it externally, then use from_state thereafter.

Alternatives considered

• Tuning merge_threshold — addressed above; reduces disk and boot, not RAM.
• Custom Persister that fabricates a single synthetic Update — possible today, but Update is not a stable serialization target for materialized state, and the replay still goes through apply_update_inner, paying the same allocation cost. A direct Cache injection is cleaner and explicit about the contract.
• Lazy/streamed transaction loading inside Cache — larger refactor, changes semantics of existing accessors, and not needed if the application can supply a pruned state up front.

Happy to send a PR if the direction is acceptable.

[ledegenbtc]

Sharing context from a related but different wallet — similar external symptoms, different point in the problem space. Posting in case it helps others triaging, and to refine where this issue's from_state ask remains necessary.

Different wallet, ~10k txs, exhibiting excessive memory growth that would lead to recurring OOM kills (RSS climbing into multi-GB territory under steady operation). After instrumenting, the root cause was heap fragmentation in glibc's ptmalloc2, not the size of the materialized Wollet state. Two factors combined:

1. The periodic scan loop in our application calls full_scan and rebuilds derived state every ~30 s, producing a bursty allocation pattern — large transient Vec<WalletTx>, UtxoData clones, network buffers — allocated and dropped within the same cycle.
2. ptmalloc2 is conservative about returning freed pages to the kernel and creates per-thread arenas that fragment under this pattern. RSS climbs to a plateau driven by peak transient use plus accumulated fragmentation, with no monotonic increase in logical memory.

Swapping the global allocator to jemalloc with background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000 brought steady-state RSS from 5-6 GB down to ~1-2 GB. jemalloc.stats.allocated confirmed the logical footprint of Wollet for ~10k txs is in that 1-2 GB range — the previous excess was fragmentation overhead, not accumulation.

This likely matters for the 30k+ case described in this issue as well. I have not yet measured allocated on that wallet, but the same fragmentation dynamics from restore_updates plus the steady-state scan loop are present there. The ~7 GB RSS in the issue body is plausibly part fragmentation, part logical state — the split is empirical and worth measuring before assuming either is the whole story.

That said, the architectural argument for from_state does not depend on the split. Wollet's materialized state scales linearly with transaction history. For any sufficiently large wallet, no allocator can keep allocated under a fixed RAM budget — only application-level pruning can. The 30k wallet may be close to the threshold today; whatever portion of the 7 GB is genuine logical state will grow monotonically as history accumulates. The fix this issue proposes addresses that fundamental scaling, which an allocator change does not.

Triage for anyone hitting OOM on lwk_wollet:

1. Switch the global allocator to jemalloc with the config above and instrument both resident (from /proc/self/status) and allocated (via tikv-jemalloc-ctl::stats::allocated).
2. If resident >> allocated and allocated stays well under the RAM budget: fragmentation was the problem. Done.
3. If allocated itself approaches the RAM budget — or scales linearly with Wollet's tx count — then from_state is the structural fix. Allocator changes cannot help further.

For our deployment, the plan is to apply (1) on the 30k wallet and measure where allocated lands. If the headroom turns out to be sufficient, from_state becomes future-proofing rather than urgent — but still necessary, since transaction history does not shrink.

[LeoComandini]

Hello @ledegenbtc

Thanks for the issue

I run a production hot wallet on top of lwk_wollet that has accumulated 30k+ transactions over its lifetime.

That's nice to hear!

We've been working on improving things for your use case over these months.
We're still improving on it, so feedback from users who manage high-volume wallets is very welcome.

We already maintain an external indexer that holds the canonical transaction history and can serve any subset of it on demand

Is that similar to Electrum, Esplora or Waterfalls?
Or is it something else?

If you can and want share more details we might be able to help you better

[LeoComandini]

These three mechanisms [only_tip, prune, merge_threshold] address disk footprint and boot time. They do not address resident memory

That's correct.

To reduce RAM usage, you need to use the txs_store (transactions key-value store).
We implemented it recently, so you might need to use master.

Example usage:
https://github.com/Blockstream/lwk/blob/master/lwk_wollet/tests/txs_store.rs#L111-L199

The idea behind it is fairly simple: Liquid transactions are huge, so we allow callers to configure LWK to store them in a way that does not bloat their RAM: WolletBuilder::with_txs_store().

The transaction store can be whatever fits your needs, for instance it can be a local db (eg SqliteStore) storing the txs on disk, or it can be an interface that fetches them from a remote indexer, doing network calls every time you (really) need a tx.
This choice depends on the user setup, and users can experiment with those.

Note that storing the txs on disk or fetching them every time might have performance implications, since it's slower than fetching them from memory.
For instance Wollet::transactions() (which returns all txs) might become significantly slower.
To mitigate this we introduced Wollet::txs(), which deals more efficiently with the txs store.

[LeoComandini]

FWIW we also added a doc section for high-volume wallets, with various suggestions.
https://blockstream.github.io/lwk/book/highvolume.html