feat(mbtiles): add a new cache schema#3009
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new non-standard “cache” MBTiles schema intended for persistent on-disk tile caching with expires/etag metadata, while still exposing a spec-compatible tiles view so the resulting .mbtiles remains readable by standard MBTiles readers.
Changes:
- Introduces the cache schema DDL (
tile_cache+cache_data+tilesview) and Rust read/write APIs (get_cached,set_cached,purge_expired), including xxh3-based blob de-duplication. - Adds a new
MbtilesPool::open_cacheentry point for writable cache usage (WAL + busy timeout) and tests for persistence and purge behavior. - Updates schema-init logic to correctly append
STRICTalongside other SQLite table options (e.g.WITHOUT ROWID) and expands documentation to describe the new schema.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| mbtiles/src/schemas/mod.rs | Registers and re-exports the new cache schema helpers. |
| mbtiles/src/schemas/cache.rs | Adds cache-schema detection + creation helpers (but currently has an unused import warning). |
| mbtiles/src/queries.rs | Fixes STRICT table-option injection when other options exist. |
| mbtiles/src/pool.rs | Adds open_cache plus pooled cache read/write/purge APIs and a new roundtrip test. |
| mbtiles/src/lib.rs | Exposes new cache types (CacheEntryMeta, CachedTile) from the crate root. |
| mbtiles/src/errors.rs | Adds CacheKeyExhausted error for collision probe exhaustion. |
| mbtiles/src/cache.rs | Implements cache schema creation + cached tile read/write/purge logic and unit tests. |
| mbtiles/sql/init-cache.sql | Defines the new cache schema tables and spec-compatible tiles view. |
| mbtiles/.sqlx/query-3bbfb455…json | Adds sqlx offline metadata for the cache schema detection query. |
| justfile | Sorts generated sqlx JSON deterministically (jq) in prepare-sqlite. |
| docs/content/mbtiles-schema.md | Documents the new cache schema and includes its SQL definition. |
Files not reviewed (1)
- mbtiles/.sqlx/query-3bbfb4551cbb745ab79195a1a201176be1172dfe72cc60f1bae5958aa07e0cfb.json: Generated file
Comment on lines
+324
to
+336
| pub async fn open_cache<P: AsRef<Path>>(filepath: P) -> MbtResult<Self> { | ||
| let mbtiles = Mbtiles::new(filepath)?; | ||
| let opt = SqliteConnectOptions::new() | ||
| .filename(mbtiles.filepath()) | ||
| .create_if_missing(true) | ||
| .journal_mode(SqliteJournalMode::Wal) | ||
| .busy_timeout(Duration::from_secs(5)); | ||
| let pool = SqlitePool::connect_with(opt).await?; | ||
| let mut conn = pool.acquire().await?; | ||
| mbtiles.create_cache_schema(&mut *conn, false).await?; | ||
| drop(conn); | ||
| Ok(Self { mbtiles, pool }) | ||
| } |
Performance Comparison
|
CommanderStorm
approved these changes
Jul 14, 2026
CommanderStorm
left a comment
CommanderStorm
left a comment
Member
There was a problem hiding this comment.
LGTM, with one minor nit
nyurik
force-pushed
the
cache
branch
2 times, most recently
from
July 15, 2026 14:37
725a985 to
80ad6bd
Compare
nyurik
changed the title
nyurik
changed the title
nyurik
force-pushed
the
cache
branch
2 times, most recently
from
July 16, 2026 01:06
abe6f3d to
8f64ff3
Compare
CommanderStorm
marked this pull request as draft
July 16, 2026 12:29
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3009 +/- ##
==========================================
+ Coverage 86.24% 86.33% +0.09%
==========================================
Files 120 123 +3
Lines 17892 18465 +573
Branches 17892 18465 +573
==========================================
+ Hits 15431 15942 +511
- Misses 1745 1778 +33
- Partials 716 745 +29 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ge additions Address PR feedback and close the API gaps a web-fetching consumer needs: - New `MbtilesCache` type: pooled writable cache handle (WAL, busy timeout, create-if-missing, incremental auto_vacuum for new files). `MbtilesPool` reverts to its read-only main-branch surface. - `MbtilesCache::open` refuses files with a non-cache schema (NotACacheFile) instead of silently creating cache tables inside an existing tileset. - `update_cached_meta`: bump expires/etag without rewriting the blob (HTTP 304 revalidation path). - `purge_cache_to_size`: evict soonest-expiring entries until the file's live size fits a byte budget. - Purge functions now run `PRAGMA incremental_vacuum` so cache files shrink on disk when auto_vacuum is enabled. - Document the empty-blob negative-caching convention; note that a set_cached overwrite may orphan a blob until the next purge. - Tests: overwrite-orphan GC, metadata-only revalidation, size-bounded purge eviction order, negative-cache roundtrip, schema guard, auto_vacuum mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds `MbtType::Cache` (and `MbtTypeCli::Cache`), fulfilling the promise that cache files can be examined/copied/diffed like any other mbtiles: - detect_type recognizes the cache schema (checked last, so no behavior change for existing files); uniqueness constraint satisfied by the tile_cache WITHOUT-ROWID primary key. - New `xxh3_64_int` SQLite scalar UDF (bit-identical to the runtime content_key) powers bulk SQL paths. - copy: any schema -> cache (blobs keyed by content, NULL expires/etag), cache -> any schema (via the tiles view, expires/etag dropped), and cache -> cache verbatim (expires/etag and probed keys preserved, zoom/ bbox filters and on-duplicate modes honored). Bulk copies verify no xxh3-64 collision occurred and fail with CacheCopyCollision otherwise. - diff/apply-patch/bindiff/transcode into or onto a cache file are rejected with clear errors (deletion markers cannot be represented); cache files still work as the compared-against/patch-source side. - validate: FK integrity (tile_cache -> cache_data) plus content-key check with the linear-probe tolerance window; orphaned blobs are legal. - summary/meta-*/martin serving work unchanged via the tiles view. - insert_tiles supports Cache, so `martin-cp --mbtiles-type cache` can prepopulate cache files. - New `mbtiles cache-purge <file> [--max-size <MB>]` subcommand. - Docs: mbtiles-schema.md gains a supported-operations matrix; cache module rustdoc documents bulk-copy vs runtime collision semantics. - Tests: schema detection, convert round-trips (flat/hash/norm <-> cache), cache->cache metadata preservation, diff-input support, rejection errors, bulk-insert dedup, validation negatives, CLI golden files. BREAKING CHANGE: `MbtType` and `MbtTypeCli` gain a `Cache` variant; exhaustive matches on these enums must add an arm (mbtiles 0.19.0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirror the flat vs normalized split of the regular schemas:
- `CacheSchema::Flat`: a single tile_cache table with the blob inline -
simple and fast when few tiles share content.
- `CacheSchema::Normalized` (previously the only layout): tile_cache
pointing into the xxh3-64-keyed, de-duplicated cache_data blob table -
the recommended default for web caches.
Both keep the tile_cache name for the coordinate/metadata table
(detection tells them apart by the sixth column: tile_data vs tile_id)
and both expose the spec-compatible tiles view.
- `MbtType::Cache { schema: CacheSchema }`, mirroring Normalized.
- CLI values `cache-flat` and `cache-normalized`; `cache` stays as an
alias for cache-normalized.
- Runtime API takes the schema like insert_tiles takes mbt_type;
`MbtilesCache` detects and stores the layout (`open` creates
normalized; `open_with_schema` chooses).
- Copies preserve expires/etag in any cache-to-cache direction,
including across layouts, via a shared canonical select; only
normalized needs the blob-insert + collision-check path.
- Flat purges skip the orphan-blob GC; per-tile validation of flat
caches has nothing to hash-check, like flat.
- Cache detection and reads use runtime SQL (the two layouts' tile_cache
columns cannot coexist in the sqlx-prepare database), dropping two
.sqlx entries.
- Tests parametrized over both layouts (unit, pool, copy round-trips,
2x2 cache-to-cache matrix, diff inputs); CLI golden tests cover both;
normalized golden outputs are byte-identical to before.
BREAKING CHANGE: `MbtType::Cache` gains a payload, `MbtTypeCli::Cache`
is replaced by `CacheFlat`/`CacheNormalized`, and the cache read/write
APIs take a `CacheSchema` parameter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both cache schemas gain a `fetched INTEGER` column (Unix-epoch seconds, right before `expires`) recording when the tile was downloaded/added/ last refreshed - the input for HTTP Age computation and staleness heuristics. - CacheEntryMeta/CachedTile gain a `fetched` field (before `expires`); CacheEntryMeta::new is now (fetched, expires, etag). - update_cached_meta bumps it on 304 revalidation along with expires/etag. - Cache-to-cache copies (any layout direction) preserve it; bulk copies from non-cache sources leave it NULL rather than stamping copy time, keeping identical copy runs byte-identical (existing CLI golden outputs did not change). - Schema detection now expects the six shared columns + one layout column. BREAKING CHANGE: cache schema DDL and `CacheEntryMeta::new` signature changed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the normalized (de-duplicated) cache variant and everything that
existed only to support it, leaving one straightforward layout:
tile_cache(zoom_level, tile_column, tile_row,
fetched, expires, etag, tile_data)
plus the spec-compatible tiles view.
Removed: the CacheSchema enum and every schema-dispatching code path,
the cache_data blob table, the xxh3_64_int SQLite function, content-key
linear probing and its CacheKeyExhausted/CacheCopyCollision errors, the
copier's blob-insert + collision-verification passes, orphan-blob GC in
the purge functions, and the cache-normalized per-tile validation
(cache now skips hash checks exactly like flat).
MbtType::Cache is a unit variant again, the CLI value is just `cache`,
set_cached is a plain upsert, and cache-to-cache copies are a single
INSERT..SELECT that preserves fetched/expires/etag.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
nyurik
marked this pull request as ready for review
July 19, 2026 05:13
CommanderStorm
approved these changes
Jul 19, 2026
CommanderStorm
left a comment
CommanderStorm
left a comment
Member
There was a problem hiding this comment.
this is not going to get better and if there is somethign wrong, we can also fix it in main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
adds a new caching schema to the .mbtiles files, which allows various tools to store web-retrieved tiles in a standard .mbtiles file with the expiration and etag. This way cache file can be examined / prepopulated / diffed / copied / ... and many other options as any other .mbtiles file.