Navigation performance: library walk, shelf paint, settings heap peak

CHANGELOG.md

@@ -12,8 +12,13 @@
 - **Indexing a chapter is substantially faster, on every book and every build.** Saving a laid-out page to the SD card used to hand the card one tiny piece at a time — a separate write for each coordinate, each word, each style flag — which for a typical page meant around two thousand separate writes and took about half a second. Almost all of that was the cost of talking to the card, not of the data itself: a page is only a few kilobytes. Pages are now assembled in a small block of memory and handed over in a handful of writes instead. The saved file is byte-for-byte identical, so existing cached chapters stay valid and nothing needs re-indexing.
 - **Chinese, Japanese and Korean line breaking now follows the upstream CrossPoint implementation.** Ported by hand from upstream PR #2288, which shipped a more complete version of work this fork had been doing in parallel. Two visible improvements: an opening bracket or quote (「, （, 《) can no longer be stranded alone at the end of a line — previously only *closing* punctuation was protected from starting one — and the rules now cover Hangul as well as Chinese and Japanese. Internally it separates "no line break may fall here" from "no space belongs here", which this fork had conflated into a single flag. Because line breaking changed, cached layouts for CJK books are re-indexed once on first open.
 - **Saving and loading settings uses far less memory.** The settings list was being copied in full every time settings were written or read — dozens of allocations before a single value was stored. Both paths now read the list in place. This removes the most likely cause of the out-of-memory restart that could happen when holding the power button, and makes a low-memory save fail cleanly instead of writing a truncated settings file.
+- **The restarts CrumBLE does for itself are about a second and a half shorter.** Some navigation — opening Settings, leaving a book for the home screen — quietly restarts the device to tidy up memory, and that restart is the pause you feel. Part of it was spent re-reading every book file on the card to look for new ones, even though the device had just read them moments earlier and nothing could have changed in between. That scan is now skipped when the device restarted itself and the card genuinely cannot have changed. Anything that can add or remove books — sending books over Wi-Fi from a browser or from Calibre, deleting a book, swapping the card, or a power cycle — still triggers a full scan, so a newly added book always shows up.
+- **Book covers on the home shelf are saved ready-to-draw, so the shelf paints without decoding them.** Each cover was decoded from its image file every time the shelf was painted from cold, through a memory cache that refuses to allocate when memory is short — which is exactly the state startup leaves behind, so on a cold start the covers took the slowest possible route. The shelf now keeps a ready-to-draw copy of each cover next to the image it came from, the same trick the carousel already uses for its side covers. A cover is prepared once and read straight to the screen after that, needing no decoding and no spare memory, so the shelf no longer gets slower just because memory is tight. The first paint after upgrading prepares the four visible covers as it draws them, so it is marginally slower; every paint after that is faster.
+- **The home screen appears faster on the first paint after starting up.** Deciding whether a book's cover thumbnail had already been made asked the SD card up to four separate questions per cover, when one answers it for a cover that already exists — and card lookups are the slow part. It now asks the one question first and only falls back to the rest when a thumbnail actually needs making. Later paints were already skipping this work; this is the paint you wait on.
+- **About 13 KB of memory is free again on every screen.** The list of settings is built once and kept for as long as the device is on, and it was being held in a block sized for 128 entries when there are 69 — the difference sat unused and unavailable for the whole session. Opening Settings also no longer briefly needs a second copy of that list at the moment it is assembled. Free memory is what decides whether navigation restarts the device to recover, so this makes those restarts less likely everywhere, not just in Settings.
 
 ### Fixed
+- **The card was being scanned for books twice at once on startup, by two parts of the firmware at the same time.** The home screen reads the book list from two places — one that decides what the buttons should do, and one that draws the shelf — and those run independently of each other. Each checked "has the card been scanned yet?", both got told no, and both started a scan, because the answer is only recorded when a scan *finishes* and a scan takes over a second. Besides doing the work twice, the two scans were rebuilding the same list at the same time, one freeing memory the other was still reading — a long-standing cause of unexplained restarts around the home screen. Scans are now serialised: the second asks, waits for the first to finish, and finds the list already built. Startup does one scan instead of two.
 - **Cached chapter layouts are rebuilt once after upgrading, on the Chinese-fallback build only.** Page positions are computed when a chapter is first indexed and stored in its cache, so a chapter indexed by older firmware kept its old layout no matter what later versions fixed -- a chapter you had already opened would load instantly and still show the justification overflow above. The cache-compatibility window was also wider than intended, so version bumps alone never invalidated anything; both bounds now move together. Only the Chinese-fallback build rebuilds: the layout bug could only affect text with Chinese line-break opportunities, so English books laid out identically before and after the fix and their caches are kept. Books read on the main build are untouched and re-index nothing.
 - **Justified Chinese text no longer runs off the right edge of the screen.** Chinese sets solid, with no space between characters, so the line breaker fills each line right up to the margin on the basis that those boundaries are exactly zero wide. Justification then treated every one of them as a gap to stretch — and with no slack left to give, the extra width pushed the end of each line past the edge of the panel, further with every character. A run of Chinese now justifies as the solid block it already is, since every character shares one width; English words on a mixed line still space out normally. This diverges deliberately from upstream, which stretches those boundaries.
 - **Chinese characters no longer render as a wall of empty boxes when an SD-card font is selected.** With a Latin-only SD font as the reading font, every ideograph came back as that font's "missing character" box, because the lookup consulted the SD card first and its failure path substitutes the replacement glyph — a technically valid answer that ended the search before the built-in CJK font was ever reached. The flash CJK font is now consulted first, which also removes a pointless SD-card read per character. This affects the `tiny-bitter-chinese-fallback` build only.

HANDOFF-PERF-REMAINING.md

@@ -0,0 +1,118 @@
+# Handoff: remaining navigation-performance work
+
+_Continues the perf pass on `feat/dict-casper`. Items 1-5 are done and committed
+there; what follows is measured, not guessed._
+
+## Why this exists
+
+Users report CrumBLE navigation and button response feel slow. Instrumentation
+added in `c7cb7113` (`PERF` log tag) turned that into numbers on a real X4.
+
+Headline: a navigation "loading moment" is a **deliberate reboot**. When heap is
+fragmented, entering Settings or exiting to Home silent-restarts to defragment,
+so the pause the user feels is a full boot.
+
+Measured on the user's X4 (43 books, 35 sleep images):
+
+```
+boot->ready 6887 ms (silentRestart=1 target=0 lean=0)   <- Home-target restart
+boot->ready 3300 ms (silentRestart=0 ...)               <- cold boot
+```
+
+Breakdown of the 6.9 s, from log timestamps:
+
+| Stage | Cost |
+|---|---|
+| init -> CollectionsStore loaded | ~0.75 s |
+| e-ink refresh | 1.71 s |
+| `/sleep` bake scan (35 images) | ~1.8 s -> **FIXED** in `5f467975` |
+| library SD walk (43 files) | ~1.5 s -> **ITEM A below** |
+| shelf render (`shelf=1277`) | 1.28 s -> **ITEM B below** |
+| final panel | 0.57 s |
+
+Other measurements worth keeping:
+- `onEnter Home` 1197-1744 ms, `RecentBooksGrid` 1392 ms, `BookStats` 1943 ms,
+  `Sleep` 2718 ms.
+- `rebuildSettingsLists cost: free 73452->54256 (-19196)` — the Settings entry
+  peak, i.e. what trips the restart gate in the first place.
+- Steady-state panel time is ~570 ms per fast refresh, ~1800 ms full. That is
+  e-ink physics, not our code — do not chase it.
+
+## Already done (do not redo)
+
+| Commit | Change |
+|---|---|
+| `c7cb7113` | BT auto-reconnect backoff (was freezing the loop 2-3 s per button press when a bonded remote is off) + `PERF` instrumentation |
+| `9ca01478` | Settings added to `isLibraryLightBoot` — Settings restarts skip the library load. Also made Rebuild Author Keys call the idempotent `begin()` first, or it would clear an empty index and save it over the user's real one |
+| `5f467975` | Sleep-bake scan verdict cache (the ~1.8 s win) |
+| `13cd9a96` | Lost-setting fix: lend the framebuffer to the settings write on silent restart instead of dropping the change |
+
+A separate session is already rewriting the settings JSON write to stream
+(removing the 20 KB peak). Do not duplicate that.
+
+---
+
+## ITEM A — library SD walk on every restart (~1.5 s)
+
+`LibraryIndex::ensureWalked()` (`src/LibraryIndex.cpp:240`) is gated by
+`walkPerformed`, a **plain RAM bool**. It resets on every reboot, so each silent
+restart re-walks the card even though the walk it just did is still valid.
+Triggered indirectly via `CollectionsStore.cpp:320`.
+
+Intended fix: persist "already walked" across a *silent restart only* (RTC-backed,
+same pattern as the other silent-restart state in `SilentRestart.h`), honoured
+only when `isContinuingFromSilentReboot()` is true. A cold boot or power-cycle
+must always walk.
+
+**The danger — audit before writing code.** The walk is how new books are
+discovered. File Transfer adds books AND is itself a silent-restart target, so a
+naive flag makes newly uploaded books invisible until a power cycle. That is a
+much worse bug than 1.5 s of boot. Enumerate every path that can change the book
+set (FT upload/delete, web file manager, deferred delete in `main.cpp`, SD swap)
+and clear the flag on each. If that set cannot be enumerated confidently, prefer
+a cheap directory signature over a flag.
+
+## ITEM B — first shelf paint is 3-4x slower than later ones
+
+`shelf=1277` on the restart boot versus `shelf=286..443` on subsequent renders
+(see `RPROF slow-render stages` lines). The first paint does cover work the later
+ones skip. This is where Duet's "input-first shelves" idea (task #171) applies:
+paint placeholders first, hydrate covers at yield points so navigation stays
+responsive. Compare against our existing v316 shelf-covers fast-path gate and the
+v209 `CoverTiles` pre-baked tiles before adding anything new.
+
+## ITEM C — shrink the Settings entry peak (fires fewer restarts)
+
+`rebuildSettingsLists` costs ~19 KB, and `SettingsActivity::onEnter`'s pre-flight
+(`src/activities/settings/SettingsActivity.cpp:486`) restarts when free < 45 KB or
+maxAlloc < 30 KB. Main already halved this once by moving rows out of
+`allSettings` instead of copying (see the v4.7.3 comment at
+`SettingsActivity.cpp:171`). Remaining ideas: build only the submenu being shown
+rather than the whole tree; avoid materialising the flat list twice. Every KB
+here removes restarts entirely, which beats making restarts faster.
+
+## ITEM D — refresh-mode audit (HOLD)
+
+Was item 5 of the original plan. **Do not start** while the
+`fix/display-grayscale-baseline` session is live — it is fixing a grayscale
+baseline regression on the same panel path, and two threads changing refresh
+behaviour at once makes both un-diagnosable. Revisit after that lands.
+
+---
+
+## Bugs the same log surfaced (separate from perf)
+
+1. `[BMC] getSpineCumulativeSize index 8507 out of range` (twice) — bogus spine
+   index while opening the Recent Books grid.
+2. `[RBGA] OOM: grid snapshot (48000 bytes)` plus `NoIndex gen failed` /
+   `Heavy gen also failed` cover generation at ~54 KB free — the grid asks for a
+   full 48 KB framebuffer snapshot it cannot get.
+3. Device rebooted at checkpoint `cps:serialize` — same crash class the 4.6.1
+   floor was raised for; still reachable. The streaming-JSON work may remove it.
+
+## Ground rules
+
+- Build `pio run -e tiny-bitter`; stay under 6,553,600 bytes (SD-flash promise).
+- Copy builds to `~/Downloads/crumble-<ver>-<tag>.bin`.
+- Re-measure with the `PERF` lines after each change — that is the point.
+- No emojis, no AI attribution in code/commits/PRs.

lib/Epub/Epub.cpp

@@ -248,6 +248,11 @@ Epub::Epub(std::string filepath, const std::string& cacheDir) : filepath(std::mo
   migrateLegacyCachePath(cacheDir);
 }
 
+std::string Epub::thumbBmpPathForDimensions(const std::string& filepath, const std::string& cacheDir, int width,
+                                            int height) {
+  return getThumbBmpPathForDimensions(cachePathForFilePath(filepath, cacheDir), width, height);
+}
+
 std::string Epub::cachePathForFilePath(const std::string& filepath, const std::string& cacheDir) {
   // Keep on-disk EPUB cache keys stable across standard library/toolchain changes.
   return cacheDir + "/epub_" + std::to_string(ZipFile::fnvHash64(filepath.c_str(), filepath.size()));

lib/Epub/Epub.h

@@ -52,6 +52,15 @@ class Epub {
   explicit Epub(std::string filepath, const std::string& cacheDir);
   ~Epub() = default;
   static std::string cachePathForFilePath(const std::string& filepath, const std::string& cacheDir);
+  // v4.7.5: SD-free "where would this book's WxH thumbnail live" derivation.
+  // Equivalent to Epub(filepath, cacheDir).getThumbBmpPath() resolved for the
+  // given dimensions, but touches no SD: the constructor stats the card for
+  // the legacy-cache migration, and callers that only want to probe for an
+  // already-generated thumbnail should not pay for that. Kept here rather
+  // than assembled at the call site so the on-disk thumbnail layout stays
+  // owned by this class.
+  static std::string thumbBmpPathForDimensions(const std::string& filepath, const std::string& cacheDir, int width,
+                                               int height);
   std::string& getBasePath() { return contentBasePath; }
   bool load(bool buildIfMissing = true, bool skipLoadingCss = false);
   bool clearCache() const;

lib/I18n/translations/english.yaml

@@ -682,6 +682,8 @@ STR_BOOK_UNFINISHED: "Marked as Unfinished"
 STR_BOOK_SETTINGS: "Book Settings"
 STR_OPTIMIZE_CHAPTER_INDEXING: "Optimize Chapter Indexing"
 STR_SLEEP_CYCLE_SKIP_GRAYSCALE: "1-pass B/W screensaver cycling"
+STR_SLEEP_CYCLE_DOUBLE_TAP_BACK: "Double-tap for previous screensaver"
+STR_SLEEP_CYCLE_DAILY_MODE: "Change screensaver once a day"
 STR_LOOKUP: "Lookup"
 STR_LOOKED_UP_WORDS: "Looked Up Words"
 STR_NO_DEFINITION_FOUND: "No definition found"

lib/hal/BluetoothHIDManager.cpp

@@ -336,6 +336,9 @@ bool BluetoothHIDManager::enable() {
   }
 
   _enabled = true;
+  // Deliberate enable (BT menu entry, quick-connect) is an explicit "try now":
+  // never make the user wait out a backoff earned by earlier failures.
+  resetAutoReconnectBackoff();
   // v18.9.9.48: successful init clears the skip-teardown flag so callers
   // don't keep silent-restarting after we've re-entered a clean state.
   _nimbleStateSkippedTeardown = false;
@@ -2200,6 +2203,11 @@ void BluetoothHIDManager::updateActivity() {
   }
 }
 
+void BluetoothHIDManager::resetAutoReconnectBackoff() {
+  _reconnectFailures = 0;
+  _nextReconnectAllowedMs = 0;
+}
+
 void BluetoothHIDManager::checkAutoReconnect(bool userInputDetected) {
   if (!_enabled) {
     return;
@@ -2245,6 +2253,16 @@ void BluetoothHIDManager::checkAutoReconnect(bool userInputDetected) {
     LOG_DBG("BT", "AutoReconnect skipped: cooldown active (%lu ms)", now - lastReconnectAttempt);
     return;
   }
+
+  // 4.7.4: failure backoff. Each attempt below blocks the loop for 2-3 s when
+  // the remote isn't reachable, which reads as "the buttons are laggy". After
+  // a failure, hold off progressively (10s, 20s, 40s, ... capped at 5 min)
+  // instead of re-freezing on the next press.
+  if (_nextReconnectAllowedMs != 0 && now < _nextReconnectAllowedMs) {
+    LOG_DBG("BT", "AutoReconnect skipped: backoff for %lu more ms after %u failures",
+            _nextReconnectAllowedMs - now, static_cast<unsigned>(_reconnectFailures));
+    return;
+  }
   lastReconnectAttempt = now;
 
   if (_bondedDeviceAddress.empty()) {
@@ -2257,8 +2275,16 @@ void BluetoothHIDManager::checkAutoReconnect(bool userInputDetected) {
 
   if (connectToDevice(_bondedDeviceAddress)) {
     LOG_INF("BT", "Reconnected to bonded device %s", _bondedDeviceAddress.c_str());
+    resetAutoReconnectBackoff();
   } else {
-    LOG_ERR("BT", "Reconnect to bonded device %s failed: %s", _bondedDeviceAddress.c_str(), lastError.c_str());
+    if (_reconnectFailures < 5) _reconnectFailures++;
+    // 10s, 20s, 40s, 80s, 160s -- then hold at 300s.
+    unsigned long holdMs = 10000UL << (_reconnectFailures - 1);
+    if (holdMs > 300000UL) holdMs = 300000UL;
+    _nextReconnectAllowedMs = now + holdMs;
+    LOG_ERR("BT", "Reconnect to bonded device %s failed: %s (backoff %lu ms, failures=%u)",
+            _bondedDeviceAddress.c_str(), lastError.c_str(), holdMs,
+            static_cast<unsigned>(_reconnectFailures));
   }
 }
 

lib/hal/BluetoothHIDManager.h

@@ -143,6 +143,10 @@ class BluetoothHIDManager {
   void setBondedDevice(const std::string& address, const std::string& name = "");
   void updateActivity();  // Call periodically to check inactivity timeout
   void checkAutoReconnect(bool userInputDetected = false);  // Reconnect bonded device when disconnected
+  // Clear the auto-reconnect failure backoff. Called on a successful connect
+  // and whenever the user deliberately acts on BT (enable, scan, pair), so an
+  // explicit "connect now" is never held off by earlier failures.
+  void resetAutoReconnectBackoff();
 
   // Check if BLE has had activity recently (within last 4 minutes)
   // Used by power manager to prevent sleep during BLE use
@@ -253,6 +257,14 @@ class BluetoothHIDManager {
     std::function<void(uint8_t)> _buttonActivityNotifier;
   bool _debugCaptureEnabled = false;
   std::string _bondedDeviceAddress;
+  // 4.7.4: auto-reconnect backoff. connectToDevice() BLOCKS the main loop for
+  // 2-3 s when the bonded remote is off or out of range, and the reconnect is
+  // driven by local button presses -- so a user who owns a page-turner but
+  // isn't using it right now paid that freeze on button after button. Count
+  // consecutive failures and push the next attempt out exponentially; a
+  // successful connect (or a deliberate BT enable) clears it.
+  uint8_t _reconnectFailures = 0;
+  unsigned long _nextReconnectAllowedMs = 0;
   std::string _bondedDeviceName;
 
   // Inactivity timeout (milliseconds)