Skip to content

Commit 27a770b

Browse files
committed
Merge bitcoin#28280: Don't empty dbcache on prune flushes: >30% faster IBD
589db87 validation: don't erase coins cache on prune flushes (Andrew Toth) 0e89187 Add linked-list test to CCoinsViewCache::SanityCheck (Pieter Wuille) 05cf4e1 coins: move Sync logic to CoinsViewCacheCursor (Andrew Toth) 7825b8b coins: pass linked list of flagged entries to BatchWrite (Andrew Toth) a14edad test: add cache entry linked list tests (Andrew Toth) 24ce37c coins: track flagged cache entries in linked list (Andrew Toth) 58b7ed1 coins: call ClearFlags in CCoinsCacheEntry destructor (Andrew Toth) 8bd3959 refactor: require self and sentinel parameters for AddFlags (Andrew Toth) 75f36d2 refactor: add CoinsCachePair alias (Andrew Toth) f08faea refactor: move flags to private uint8_t and rename to m_flags (Andrew Toth) 4e4fb4c refactor: disallow setting flags in CCoinsCacheEntry constructors (Andrew Toth) 8737c0c refactor: encapsulate flags setting with AddFlags and ClearFlags (Andrew Toth) 9715d3b refactor: encapsulate flags get access for all other checks (Andrew Toth) df34a94 refactor: encapsulate flags access for dirty and fresh checks (Andrew Toth) Pull request description: Since bitcoin#17487 we no longer need to clear the coins cache when syncing to disk. A warm coins cache significantly speeds up block connection, and only needs to be fully flushed when nearing the `dbcache` limit. For frequent pruning flushes there's no need to empty the cache and kill connect block speed. However, simply using `Sync` in place of `Flush` actually slows down a pruned full IBD with a high `dbcache` value. This is because as the cache grows, sync takes longer since every coin in the cache is scanned to check if it's dirty. For frequent prune flushes and a large cache this constant scanning starts to really slow IBD down, and just emptying the cache on every prune becomes faster. To fix this, we can add two pointers to each cache entry and construct a doubly linked list of dirty entries. We can then only iterate through all dirty entries on each `Sync`, and simply clear the pointers after. With this approach a full IBD with `dbcache=16384` and `prune=550` was 32% faster than master. For default `dbcache=450` speedup was ~9%. All benchmarks were run with `stopatheight=800000`. | | prune | dbcache | time | max RSS | speedup | |-----------:|----------:|------------:|--------:|-------------:|--------------:| | master | 550 | 16384 | 8:52:57 | 2,417,464k | - | | branch | 550 | 16384 | 6:01:00 | 16,216,736k | 32% | | branch | 550 | 450 | 8:05:08 | 2,818,072k | 8.8% | | master | 10000 | 5000 | 8:19:59 | 2,962,752k | - | | branch | 10000 | 5000| 5:56:39 | 6,179,764k | 28.8% | | master | 0 | 16384 | 4:51:53 | 14,726,408k | - | | branch | 0 | 16384 | 4:43:11 | 16,526,348k | 2.7% | | master | 0 | 450 | 7:08:07 | 3,005,892k | - | | branch | 0 | 450 | 6:57:24 | 3,013,556k |2.6%| While the 2 pointers add memory to each cache entry, it did not slow down IBD. For non-pruned IBD results were similar for this branch and master. When I performed the initial IBD, the full UTXO set could be held in memory when using the max `dbcache` value. For non-pruned IBD with max `dbcache` to tip ended up using 12% more memory, but it was also 2.7% faster somehow. For smaller `dbcache` values the `dbcache` limit is respected so does not consume more memory, and the potentially more frequent flushes were not significant enough to cause any slowdown. For reviewers, the commits in order do the following: First 4 commits encapsulate all accesses to `flags` on cache entries, and then the 5th makes `flags` private. Commits `refactor: add CoinsCachePair alias` to `coins: call ClearFlags in CCoinsCacheEntry destructor` create the linked list head nodes and cache entry self references and pass them into `AddFlags`. Commit `coins: track flagged cache entries in linked list` actually adds the entries into a linked list when they are flagged DIRTY or FRESH and removes them from the linked list when they are destroyed or the flags are cleared manually. However, the linked list is not yet used anywhere. Commit `test: add cache entry linked list tests` adds unit tests for the linked list. Commit `coins: pass linked list of flagged entries to BatchWrite` uses the linked list to iterate through DIRTY entries instead of using the entire coins cache. Commit `validation: don't erase coins cache on prune flushes` uses `Sync` instead of `Flush` for pruning flushes, so the cache is no longer cleared. Inspired by [this comment](bitcoin#15265 (comment)). Fixes bitcoin#11315. ACKs for top commit: paplorinc: ACK 589db87 sipa: reACK 589db87 achow101: ACK 589db87 mzumsande: re-ACK 589db87 Tree-SHA512: 23b2bc01c83edacb5b39aa60bb0b766de9a74ce17f0c59bf13b97b4328a7b758ad9aff6581c3ca88e2973f7658380651530d497444f48d6e22ea0bfc51cc921d
2 parents 0f68a05 + 589db87 commit 27a770b

10 files changed

+463
-84
lines changed

src/Makefile.test.include

+1
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ BITCOIN_TESTS =\
8585
test/checkqueue_tests.cpp \
8686
test/cluster_linearize_tests.cpp \
8787
test/coins_tests.cpp \
88+
test/coinscachepair_tests.cpp \
8889
test/coinstatsindex_tests.cpp \
8990
test/common_url_tests.cpp \
9091
test/compilerbug_tests.cpp \

src/coins.cpp

+61-49
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
bool CCoinsView::GetCoin(const COutPoint &outpoint, Coin &coin) const { return false; }
1313
uint256 CCoinsView::GetBestBlock() const { return uint256(); }
1414
std::vector<uint256> CCoinsView::GetHeadBlocks() const { return std::vector<uint256>(); }
15-
bool CCoinsView::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlock, bool erase) { return false; }
15+
bool CCoinsView::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &hashBlock) { return false; }
1616
std::unique_ptr<CCoinsViewCursor> CCoinsView::Cursor() const { return nullptr; }
1717

1818
bool CCoinsView::HaveCoin(const COutPoint &outpoint) const
@@ -27,14 +27,16 @@ bool CCoinsViewBacked::HaveCoin(const COutPoint &outpoint) const { return base->
2727
uint256 CCoinsViewBacked::GetBestBlock() const { return base->GetBestBlock(); }
2828
std::vector<uint256> CCoinsViewBacked::GetHeadBlocks() const { return base->GetHeadBlocks(); }
2929
void CCoinsViewBacked::SetBackend(CCoinsView &viewIn) { base = &viewIn; }
30-
bool CCoinsViewBacked::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlock, bool erase) { return base->BatchWrite(mapCoins, hashBlock, erase); }
30+
bool CCoinsViewBacked::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &hashBlock) { return base->BatchWrite(cursor, hashBlock); }
3131
std::unique_ptr<CCoinsViewCursor> CCoinsViewBacked::Cursor() const { return base->Cursor(); }
3232
size_t CCoinsViewBacked::EstimateSize() const { return base->EstimateSize(); }
3333

3434
CCoinsViewCache::CCoinsViewCache(CCoinsView* baseIn, bool deterministic) :
3535
CCoinsViewBacked(baseIn), m_deterministic(deterministic),
3636
cacheCoins(0, SaltedOutpointHasher(/*deterministic=*/deterministic), CCoinsMap::key_equal{}, &m_cache_coins_memory_resource)
37-
{}
37+
{
38+
m_sentinel.second.SelfRef(m_sentinel);
39+
}
3840

3941
size_t CCoinsViewCache::DynamicMemoryUsage() const {
4042
return memusage::DynamicUsage(cacheCoins) + cachedCoinsUsage;
@@ -51,7 +53,7 @@ CCoinsMap::iterator CCoinsViewCache::FetchCoin(const COutPoint &outpoint) const
5153
if (ret->second.coin.IsSpent()) {
5254
// The parent only has an empty entry for this outpoint; we can consider our
5355
// version as fresh.
54-
ret->second.flags = CCoinsCacheEntry::FRESH;
56+
ret->second.AddFlags(CCoinsCacheEntry::FRESH, *ret, m_sentinel);
5557
}
5658
cachedCoinsUsage += ret->second.coin.DynamicMemoryUsage();
5759
return ret;
@@ -93,10 +95,10 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
9395
//
9496
// If the coin doesn't exist in the current cache, or is spent but not
9597
// DIRTY, then it can be marked FRESH.
96-
fresh = !(it->second.flags & CCoinsCacheEntry::DIRTY);
98+
fresh = !it->second.IsDirty();
9799
}
98100
it->second.coin = std::move(coin);
99-
it->second.flags |= CCoinsCacheEntry::DIRTY | (fresh ? CCoinsCacheEntry::FRESH : 0);
101+
it->second.AddFlags(CCoinsCacheEntry::DIRTY | (fresh ? CCoinsCacheEntry::FRESH : 0), *it, m_sentinel);
100102
cachedCoinsUsage += it->second.coin.DynamicMemoryUsage();
101103
TRACE5(utxocache, add,
102104
outpoint.hash.data(),
@@ -108,10 +110,13 @@ void CCoinsViewCache::AddCoin(const COutPoint &outpoint, Coin&& coin, bool possi
108110

109111
void CCoinsViewCache::EmplaceCoinInternalDANGER(COutPoint&& outpoint, Coin&& coin) {
110112
cachedCoinsUsage += coin.DynamicMemoryUsage();
111-
cacheCoins.emplace(
113+
auto [it, inserted] = cacheCoins.emplace(
112114
std::piecewise_construct,
113115
std::forward_as_tuple(std::move(outpoint)),
114-
std::forward_as_tuple(std::move(coin), CCoinsCacheEntry::DIRTY));
116+
std::forward_as_tuple(std::move(coin)));
117+
if (inserted) {
118+
it->second.AddFlags(CCoinsCacheEntry::DIRTY, *it, m_sentinel);
119+
}
115120
}
116121

117122
void AddCoins(CCoinsViewCache& cache, const CTransaction &tx, int nHeight, bool check_for_overwrite) {
@@ -138,10 +143,10 @@ bool CCoinsViewCache::SpendCoin(const COutPoint &outpoint, Coin* moveout) {
138143
if (moveout) {
139144
*moveout = std::move(it->second.coin);
140145
}
141-
if (it->second.flags & CCoinsCacheEntry::FRESH) {
146+
if (it->second.IsFresh()) {
142147
cacheCoins.erase(it);
143148
} else {
144-
it->second.flags |= CCoinsCacheEntry::DIRTY;
149+
it->second.AddFlags(CCoinsCacheEntry::DIRTY, *it, m_sentinel);
145150
it->second.coin.Clear();
146151
}
147152
return true;
@@ -178,67 +183,64 @@ void CCoinsViewCache::SetBestBlock(const uint256 &hashBlockIn) {
178183
hashBlock = hashBlockIn;
179184
}
180185

181-
bool CCoinsViewCache::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlockIn, bool erase) {
182-
for (CCoinsMap::iterator it = mapCoins.begin();
183-
it != mapCoins.end();
184-
it = erase ? mapCoins.erase(it) : std::next(it)) {
186+
bool CCoinsViewCache::BatchWrite(CoinsViewCacheCursor& cursor, const uint256 &hashBlockIn) {
187+
for (auto it{cursor.Begin()}; it != cursor.End(); it = cursor.NextAndMaybeErase(*it)) {
185188
// Ignore non-dirty entries (optimization).
186-
if (!(it->second.flags & CCoinsCacheEntry::DIRTY)) {
189+
if (!it->second.IsDirty()) {
187190
continue;
188191
}
189192
CCoinsMap::iterator itUs = cacheCoins.find(it->first);
190193
if (itUs == cacheCoins.end()) {
191194
// The parent cache does not have an entry, while the child cache does.
192195
// We can ignore it if it's both spent and FRESH in the child
193-
if (!(it->second.flags & CCoinsCacheEntry::FRESH && it->second.coin.IsSpent())) {
196+
if (!(it->second.IsFresh() && it->second.coin.IsSpent())) {
194197
// Create the coin in the parent cache, move the data up
195198
// and mark it as dirty.
196-
CCoinsCacheEntry& entry = cacheCoins[it->first];
197-
if (erase) {
198-
// The `move` call here is purely an optimization; we rely on the
199-
// `mapCoins.erase` call in the `for` expression to actually remove
200-
// the entry from the child map.
199+
itUs = cacheCoins.try_emplace(it->first).first;
200+
CCoinsCacheEntry& entry{itUs->second};
201+
if (cursor.WillErase(*it)) {
202+
// Since this entry will be erased,
203+
// we can move the coin into us instead of copying it
201204
entry.coin = std::move(it->second.coin);
202205
} else {
203206
entry.coin = it->second.coin;
204207
}
205208
cachedCoinsUsage += entry.coin.DynamicMemoryUsage();
206-
entry.flags = CCoinsCacheEntry::DIRTY;
209+
entry.AddFlags(CCoinsCacheEntry::DIRTY, *itUs, m_sentinel);
207210
// We can mark it FRESH in the parent if it was FRESH in the child
208211
// Otherwise it might have just been flushed from the parent's cache
209212
// and already exist in the grandparent
210-
if (it->second.flags & CCoinsCacheEntry::FRESH) {
211-
entry.flags |= CCoinsCacheEntry::FRESH;
213+
if (it->second.IsFresh()) {
214+
entry.AddFlags(CCoinsCacheEntry::FRESH, *itUs, m_sentinel);
212215
}
213216
}
214217
} else {
215218
// Found the entry in the parent cache
216-
if ((it->second.flags & CCoinsCacheEntry::FRESH) && !itUs->second.coin.IsSpent()) {
219+
if (it->second.IsFresh() && !itUs->second.coin.IsSpent()) {
217220
// The coin was marked FRESH in the child cache, but the coin
218221
// exists in the parent cache. If this ever happens, it means
219222
// the FRESH flag was misapplied and there is a logic error in
220223
// the calling code.
221224
throw std::logic_error("FRESH flag misapplied to coin that exists in parent cache");
222225
}
223226

224-
if ((itUs->second.flags & CCoinsCacheEntry::FRESH) && it->second.coin.IsSpent()) {
227+
if (itUs->second.IsFresh() && it->second.coin.IsSpent()) {
225228
// The grandparent cache does not have an entry, and the coin
226229
// has been spent. We can just delete it from the parent cache.
227230
cachedCoinsUsage -= itUs->second.coin.DynamicMemoryUsage();
228231
cacheCoins.erase(itUs);
229232
} else {
230233
// A normal modification.
231234
cachedCoinsUsage -= itUs->second.coin.DynamicMemoryUsage();
232-
if (erase) {
233-
// The `move` call here is purely an optimization; we rely on the
234-
// `mapCoins.erase` call in the `for` expression to actually remove
235-
// the entry from the child map.
235+
if (cursor.WillErase(*it)) {
236+
// Since this entry will be erased,
237+
// we can move the coin into us instead of copying it
236238
itUs->second.coin = std::move(it->second.coin);
237239
} else {
238240
itUs->second.coin = it->second.coin;
239241
}
240242
cachedCoinsUsage += itUs->second.coin.DynamicMemoryUsage();
241-
itUs->second.flags |= CCoinsCacheEntry::DIRTY;
243+
itUs->second.AddFlags(CCoinsCacheEntry::DIRTY, *itUs, m_sentinel);
242244
// NOTE: It isn't safe to mark the coin as FRESH in the parent
243245
// cache. If it already existed and was spent in the parent
244246
// cache then marking it FRESH would prevent that spentness
@@ -251,12 +253,10 @@ bool CCoinsViewCache::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlockIn
251253
}
252254

253255
bool CCoinsViewCache::Flush() {
254-
bool fOk = base->BatchWrite(cacheCoins, hashBlock, /*erase=*/true);
256+
auto cursor{CoinsViewCacheCursor(cachedCoinsUsage, m_sentinel, cacheCoins, /*will_erase=*/true)};
257+
bool fOk = base->BatchWrite(cursor, hashBlock);
255258
if (fOk) {
256-
if (!cacheCoins.empty()) {
257-
/* BatchWrite must erase all cacheCoins elements when erase=true. */
258-
throw std::logic_error("Not all cached coins were erased");
259-
}
259+
cacheCoins.clear();
260260
ReallocateCache();
261261
}
262262
cachedCoinsUsage = 0;
@@ -265,16 +265,12 @@ bool CCoinsViewCache::Flush() {
265265

266266
bool CCoinsViewCache::Sync()
267267
{
268-
bool fOk = base->BatchWrite(cacheCoins, hashBlock, /*erase=*/false);
269-
// Instead of clearing `cacheCoins` as we would in Flush(), just clear the
270-
// FRESH/DIRTY flags of any coin that isn't spent.
271-
for (auto it = cacheCoins.begin(); it != cacheCoins.end(); ) {
272-
if (it->second.coin.IsSpent()) {
273-
cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
274-
it = cacheCoins.erase(it);
275-
} else {
276-
it->second.flags = 0;
277-
++it;
268+
auto cursor{CoinsViewCacheCursor(cachedCoinsUsage, m_sentinel, cacheCoins, /*will_erase=*/false)};
269+
bool fOk = base->BatchWrite(cursor, hashBlock);
270+
if (fOk) {
271+
if (m_sentinel.second.Next() != &m_sentinel) {
272+
/* BatchWrite must clear flags of all entries */
273+
throw std::logic_error("Not all unspent flagged entries were cleared");
278274
}
279275
}
280276
return fOk;
@@ -283,7 +279,7 @@ bool CCoinsViewCache::Sync()
283279
void CCoinsViewCache::Uncache(const COutPoint& hash)
284280
{
285281
CCoinsMap::iterator it = cacheCoins.find(hash);
286-
if (it != cacheCoins.end() && it->second.flags == 0) {
282+
if (it != cacheCoins.end() && !it->second.IsDirty() && !it->second.IsFresh()) {
287283
cachedCoinsUsage -= it->second.coin.DynamicMemoryUsage();
288284
TRACE5(utxocache, uncache,
289285
hash.hash.data(),
@@ -324,17 +320,33 @@ void CCoinsViewCache::ReallocateCache()
324320
void CCoinsViewCache::SanityCheck() const
325321
{
326322
size_t recomputed_usage = 0;
323+
size_t count_flagged = 0;
327324
for (const auto& [_, entry] : cacheCoins) {
328325
unsigned attr = 0;
329-
if (entry.flags & CCoinsCacheEntry::DIRTY) attr |= 1;
330-
if (entry.flags & CCoinsCacheEntry::FRESH) attr |= 2;
326+
if (entry.IsDirty()) attr |= 1;
327+
if (entry.IsFresh()) attr |= 2;
331328
if (entry.coin.IsSpent()) attr |= 4;
332329
// Only 5 combinations are possible.
333330
assert(attr != 2 && attr != 4 && attr != 7);
334331

335332
// Recompute cachedCoinsUsage.
336333
recomputed_usage += entry.coin.DynamicMemoryUsage();
334+
335+
// Count the number of entries we expect in the linked list.
336+
if (entry.IsDirty() || entry.IsFresh()) ++count_flagged;
337+
}
338+
// Iterate over the linked list of flagged entries.
339+
size_t count_linked = 0;
340+
for (auto it = m_sentinel.second.Next(); it != &m_sentinel; it = it->second.Next()) {
341+
// Verify linked list integrity.
342+
assert(it->second.Next()->second.Prev() == it);
343+
assert(it->second.Prev()->second.Next() == it);
344+
// Verify they are actually flagged.
345+
assert(it->second.IsDirty() || it->second.IsFresh());
346+
// Count the number of entries actually in the list.
347+
++count_linked;
337348
}
349+
assert(count_linked == count_flagged);
338350
assert(recomputed_usage == cachedCoinsUsage);
339351
}
340352

0 commit comments

Comments
 (0)