feat(mbtiles)!: add a fetched timestamp to both cache layouts

Both cache schemas gain a `fetched INTEGER` column (Unix-epoch seconds,
right before `expires`) recording when the tile was downloaded/added/
last refreshed - the input for HTTP Age computation and staleness
heuristics.

- CacheEntryMeta/CachedTile gain a `fetched` field (before `expires`);
  CacheEntryMeta::new is now (fetched, expires, etag).
- update_cached_meta bumps it on 304 revalidation along with
  expires/etag.
- Cache-to-cache copies (any layout direction) preserve it; bulk copies
  from non-cache sources leave it NULL rather than stamping copy time,
  keeping identical copy runs byte-identical (existing CLI golden
  outputs did not change).
- Schema detection now expects the six shared columns + one layout
  column.

BREAKING CHANGE: cache schema DDL and `CacheEntryMeta::new` signature
changed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: nyurik

docs/content/mbtiles-schema.md

@@ -68,9 +68,9 @@ In our next semver major, we plan to switch this default and produce `tiles_shal
 
 ## cache
 
-The `cache` schemas store extra cache metadata (`expires` and `etag`) alongside each tile, so a file can serve as a persistent web-tile cache.
+The `cache` schemas store extra cache metadata alongside each tile - `fetched` (when the tile was downloaded/added/last refreshed), `expires`, and `etag` - so a file can serve as a persistent web-tile cache.
 Two layouts exist, mirroring the `flat` vs `normalized` split of the regular schemas.
-Both center on a `tile_cache` table holding the tile Z,X,Y coordinates and the cache metadata, and both create a spec-compatible `tiles` view so the file can still be read by any standard MBTiles reader (the `expires`/`etag` columns are simply invisible to it).
+Both center on a `tile_cache` table holding the tile Z,X,Y coordinates and the cache metadata, and both create a spec-compatible `tiles` view so the file can still be read by any standard MBTiles reader (the extra columns are simply invisible to it).
 
 ### cache-flat
 
@@ -95,8 +95,8 @@ This is the recommended default for web-tile caches, where identical (e.g. empty
 The `mbtiles` tool treats both cache layouts as first-class schemas, with a few deliberate restrictions:
 
 * `summary`, `validate`, `meta-*`, and serving the file with `martin` all work.
-* `copy` **from** a cache file to any schema works (reading via the `tiles` view); the per-tile `expires`/`etag` values are dropped, since standard schemas cannot store them.
-* `copy` **into** a cache file works from any schema (including `martin-cp --mbtiles-type cache-flat|cache-normalized`); the copied entries get `NULL` `expires`/`etag` (never expire). Copies between cache files - including across the two layouts - preserve `expires`/`etag`.
+* `copy` **from** a cache file to any schema works (reading via the `tiles` view); the per-tile `fetched`/`expires`/`etag` values are dropped, since standard schemas cannot store them.
+* `copy` **into** a cache file works from any schema (including `martin-cp --mbtiles-type cache-flat|cache-normalized`); the copied entries get `NULL` `fetched`/`expires`/`etag` (unknown fetch time, never expire; identical copy runs stay byte-identical). Copies between cache files - including across the two layouts - preserve all cache metadata.
 * `diff`, `apply-patch`, and bin-diff **into or onto** a cache file are rejected: the `NOT NULL` blob storage joined through the `tiles` view cannot represent the `NULL` "deleted tile" markers a diff needs. A cache file *can* be the compared-against or patch-source side (it is read through the view).
 * `cache-purge <file> [--max-size <MB>]` removes expired entries (and optionally evicts soonest-expiring entries until the file fits the size budget), then reclaims free pages via `PRAGMA incremental_vacuum`.
 

mbtiles/sql/init-cache-flat.sql

@@ -2,6 +2,7 @@ CREATE TABLE tile_cache (
     zoom_level INTEGER NOT NULL,
     tile_column INTEGER NOT NULL,
     tile_row INTEGER NOT NULL,
+    fetched INTEGER,
     expires INTEGER,
     etag TEXT,
     tile_data BLOB NOT NULL,

mbtiles/sql/init-cache-normalized.sql

@@ -7,6 +7,7 @@ CREATE TABLE tile_cache (
     zoom_level INTEGER NOT NULL,
     tile_column INTEGER NOT NULL,
     tile_row INTEGER NOT NULL,
+    fetched INTEGER,
     expires INTEGER,
     etag TEXT,
     tile_id INTEGER NOT NULL,

mbtiles/src/cache.rs

@@ -2,21 +2,21 @@
 //!
 //! This is a **non-standard** schema (not part of the `MBTiles` specification) built on
 //! top of the same `SQLite` file format. It stores tiles together with cache metadata
-//! (`expires` and `etag`) and de-duplicates identical tile blobs, and it is intended to
+//! (`fetched`, `expires`, and `etag`) and de-duplicates identical tile blobs, and it is intended to
 //! be embedded by other systems that need a simple on-disk tile cache.
 //!
 //! # Schemas
 //!
 //! Two layouts are supported, chosen via [`CacheSchema`]. Both center on a `tile_cache`
-//! table storing tile coordinates with `expires`/`etag` metadata, and both expose a
+//! table storing tile coordinates with `fetched`/`expires`/`etag` metadata, and both expose a
 //! spec-compatible `tiles` view so the file can still be opened by any standard
-//! `MBTiles` reader (the `expires`/`etag` columns are simply invisible to it).
+//! `MBTiles` reader (the extra cache columns are simply invisible to it).
 //!
-//! - [`CacheSchema::Flat`]: `tile_cache(zoom_level, tile_column, tile_row, expires,
-//!   etag, tile_data)` - the blob is stored inline. Simple and fast, best when few
-//!   tiles share content.
+//! - [`CacheSchema::Flat`]: `tile_cache(zoom_level, tile_column, tile_row, fetched,
+//!   expires, etag, tile_data)` - the blob is stored inline. Simple and fast, best when
+//!   few tiles share content.
 //! - [`CacheSchema::Normalized`]: `tile_cache(zoom_level, tile_column, tile_row,
-//!   expires, etag, tile_id)` (`WITHOUT ROWID`) plus `cache_data(tile_id INTEGER
+//!   fetched, expires, etag, tile_id)` (`WITHOUT ROWID`) plus `cache_data(tile_id INTEGER
 //!   PRIMARY KEY, tile_data BLOB)`. `tile_id` is the
 //!   [xxh3-64](https://github.com/Cyan4973/xxHash) hash of `tile_data`, stored as an
 //!   `INTEGER PRIMARY KEY` so it aliases the rowid (single B-tree, no secondary index).
@@ -65,6 +65,9 @@ pub(crate) const MAX_KEY_PROBES: u32 = 1024;
 pub struct CachedTile {
     /// The tile blob.
     pub data: Vec<u8>,
+    /// Unix-epoch (seconds) time the tile was downloaded/added/last refreshed, or `None`
+    /// if unknown (e.g. the entry was bulk-imported with `mbtiles copy`).
+    pub fetched: Option<i64>,
     /// Unix-epoch (seconds) expiration time, or `None` if the entry never expires.
     ///
     /// The value is returned exactly as stored; the cache does **not** filter out expired
@@ -78,20 +81,26 @@ pub struct CachedTile {
 /// Cache metadata attached to a tile when writing it with [`Mbtiles::set_cached`].
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
 pub struct CacheEntryMeta<'a> {
+    /// Unix-epoch (seconds) time the tile was downloaded/added/last refreshed, or `None`
+    /// if unknown.
+    pub fetched: Option<i64>,
     /// Unix-epoch (seconds) expiration time, or `None` for an entry that never expires.
     pub expires: Option<i64>,
     /// Upstream validator (e.g. an HTTP `ETag`), or `None`.
     pub etag: Option<&'a str>,
 }
 
 impl<'a> CacheEntryMeta<'a> {
-    /// Create cache metadata with both an `expires` (Unix-epoch seconds) and an `etag`.
+    /// Create cache metadata with a `fetched` and an `expires` time (both Unix-epoch
+    /// seconds) and an `etag`.
     ///
-    /// For entries missing one or both, construct the struct directly (its fields are
-    /// public) or use [`CacheEntryMeta::default`] for "never expires, no etag".
+    /// For entries missing some of these, construct the struct directly (its fields are
+    /// public) or use [`CacheEntryMeta::default`] for "unknown fetch time, never expires,
+    /// no etag".
     #[must_use]
-    pub fn new(expires: i64, etag: &'a str) -> Self {
+    pub fn new(fetched: i64, expires: i64, etag: &'a str) -> Self {
         Self {
+            fetched: Some(fetched),
             expires: Some(expires),
             etag: Some(etag),
         }
@@ -145,7 +154,7 @@ impl Mbtiles {
 
     /// Look up a cached tile by its XYZ coordinates.
     ///
-    /// Returns the tile together with its `expires`/`etag` metadata, or `None` if there is
+    /// Returns the tile together with its `fetched`/`expires`/`etag` metadata, or `None` if there is
     /// no entry at the given coordinates. Expired entries are still returned (with their
     /// stored `expires`) so the caller can decide whether to serve stale, revalidate via
     /// `etag`, or refetch.
@@ -162,12 +171,12 @@ impl Mbtiles {
     {
         let sql = match schema {
             CacheSchema::Flat => {
-                "SELECT tile_data, expires, etag
+                "SELECT tile_data, fetched, expires, etag
                  FROM tile_cache
                  WHERE zoom_level = ?1 AND tile_column = ?2 AND tile_row = ?3"
             }
             CacheSchema::Normalized => {
-                "SELECT d.tile_data, c.expires, c.etag
+                "SELECT d.tile_data, c.fetched, c.expires, c.etag
                  FROM tile_cache c
                  JOIN cache_data d ON d.tile_id = c.tile_id
                  WHERE c.zoom_level = ?1 AND c.tile_column = ?2 AND c.tile_row = ?3"
@@ -182,12 +191,13 @@ impl Mbtiles {
 
         Ok(row.map(|row| CachedTile {
             data: row.get(0),
-            expires: row.get(1),
-            etag: row.get(2),
+            fetched: row.get(1),
+            expires: row.get(2),
+            etag: row.get(3),
         }))
     }
 
-    /// Insert or replace a cached tile, with its [`CacheEntryMeta`] (`expires`/`etag`).
+    /// Insert or replace a cached tile, with its [`CacheEntryMeta`] (`fetched`/`expires`/`etag`).
     ///
     /// With [`CacheSchema::Flat`], this is a plain upsert with the blob stored inline.
     ///
@@ -213,12 +223,13 @@ impl Mbtiles {
             // Inline blob: a plain upsert, no de-duplication or key probing involved.
             query(
                 "INSERT OR REPLACE INTO tile_cache
-                     (zoom_level, tile_column, tile_row, expires, etag, tile_data)
-                 VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
+                     (zoom_level, tile_column, tile_row, fetched, expires, etag, tile_data)
+                 VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
             )
             .bind(z)
             .bind(x)
             .bind(invert_y_value(z, y))
+            .bind(meta.fetched)
             .bind(meta.expires)
             .bind(meta.etag)
             .bind(data)
@@ -273,16 +284,18 @@ impl Mbtiles {
         };
 
         query(
-            "INSERT INTO tile_cache (zoom_level, tile_column, tile_row, expires, etag, tile_id)
-             VALUES (?1, ?2, ?3, ?4, ?5, ?6)
+            "INSERT INTO tile_cache (zoom_level, tile_column, tile_row, fetched, expires, etag, tile_id)
+             VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
              ON CONFLICT(zoom_level, tile_column, tile_row)
-             DO UPDATE SET expires = excluded.expires,
+             DO UPDATE SET fetched = excluded.fetched,
+                           expires = excluded.expires,
                            etag = excluded.etag,
                            tile_id = excluded.tile_id",
         )
         .bind(z)
         .bind(x)
         .bind(invert_y_value(z, y))
+        .bind(meta.fetched)
         .bind(meta.expires)
         .bind(meta.etag)
         .bind(resolved)
@@ -296,8 +309,8 @@ impl Mbtiles {
         Ok(())
     }
 
-    /// Update only the `expires`/`etag` metadata of an existing cache entry, without
-    /// touching the tile blob.
+    /// Update only the `fetched`/`expires`/`etag` metadata of an existing cache entry,
+    /// without touching the tile blob.
     ///
     /// This is the revalidation path: after a conditional refetch (e.g. HTTP
     /// `If-None-Match` answered with `304 Not Modified`), the cached bytes are still
@@ -318,12 +331,13 @@ impl Mbtiles {
         for<'e> &'e mut T: SqliteExecutor<'e>,
     {
         let updated = query(
-            "UPDATE tile_cache SET expires = ?4, etag = ?5
+            "UPDATE tile_cache SET fetched = ?4, expires = ?5, etag = ?6
              WHERE zoom_level = ?1 AND tile_column = ?2 AND tile_row = ?3",
         )
         .bind(z)
         .bind(x)
         .bind(invert_y_value(z, y))
+        .bind(meta.fetched)
         .bind(meta.expires)
         .bind(meta.etag)
         .execute(conn)
@@ -494,7 +508,7 @@ mod tests {
             1,
             2,
             b"hello",
-            CacheEntryMeta::new(100, "etag-1"),
+            CacheEntryMeta::new(42, 100, "etag-1"),
         )
         .await
         .unwrap();
@@ -504,6 +518,7 @@ mod tests {
             .unwrap()
             .unwrap();
         assert_eq!(got.data, b"hello");
+        assert_eq!(got.fetched, Some(42));
         assert_eq!(got.expires, Some(100));
         assert_eq!(got.etag.as_deref(), Some("etag-1"));
 
@@ -525,6 +540,7 @@ mod tests {
             .unwrap()
             .unwrap();
         assert_eq!(got.data, b"world");
+        assert_eq!(got.fetched, None);
         assert_eq!(got.expires, None);
         assert_eq!(got.etag, None);
     }
@@ -679,14 +695,14 @@ mod tests {
         let (mbt, mut conn) = cache(schema).await;
         let stale = CacheEntryMeta {
             expires: Some(50),
-            etag: None,
+            ..Default::default()
         };
         mbt.set_cached(&mut conn, schema, 0, 0, 0, b"stale", stale)
             .await
             .unwrap();
         let fresh = CacheEntryMeta {
             expires: Some(200),
-            etag: None,
+            ..Default::default()
         };
         mbt.set_cached(&mut conn, schema, 1, 0, 0, b"fresh", fresh)
             .await
@@ -786,21 +802,21 @@ mod tests {
             1,
             2,
             b"payload",
-            CacheEntryMeta::new(100, "etag-1"),
+            CacheEntryMeta::new(42, 100, "etag-1"),
         )
         .await
         .unwrap();
 
         // No entry at these coordinates - the caller must fall back to set_cached.
-        let missing = CacheEntryMeta::new(1, "x");
+        let missing = CacheEntryMeta::new(1, 1, "x");
         assert!(
             !mbt.update_cached_meta(&mut conn, 3, 0, 0, missing)
                 .await
                 .unwrap()
         );
 
         // Revalidation bumps the metadata in place; the blob stays untouched.
-        let bumped = CacheEntryMeta::new(500, "etag-2");
+        let bumped = CacheEntryMeta::new(20, 500, "etag-2");
         assert!(
             mbt.update_cached_meta(&mut conn, 3, 1, 2, bumped)
                 .await
@@ -812,6 +828,7 @@ mod tests {
             .unwrap()
             .unwrap();
         assert_eq!(got.data, b"payload");
+        assert_eq!(got.fetched, Some(20));
         assert_eq!(got.expires, Some(500));
         assert_eq!(got.etag.as_deref(), Some("etag-2"));
         assert_eq!(blob_count(&mut conn, schema).await, 1);
@@ -836,7 +853,7 @@ mod tests {
             1,
             1,
             b"",
-            CacheEntryMeta::new(60, "miss-etag"),
+            CacheEntryMeta::new(5, 60, "miss-etag"),
         )
         .await
         .unwrap();
@@ -846,6 +863,7 @@ mod tests {
             .unwrap()
             .unwrap();
         assert!(got.data.is_empty());
+        assert_eq!(got.fetched, Some(5));
         assert_eq!(got.expires, Some(60));
         assert_eq!(got.etag.as_deref(), Some("miss-etag"));
 
@@ -905,7 +923,7 @@ mod tests {
             let meta = if i < 80 {
                 CacheEntryMeta {
                     expires: Some(i64::from(i)),
-                    etag: None,
+                    ..Default::default()
                 }
             } else {
                 CacheEntryMeta::default()

mbtiles/src/cache_pool.rs

@@ -31,16 +31,16 @@ use crate::{CacheEntryMeta, CacheSchema, CachedTile, MbtError, Mbtiles, Metadata
 /// # async fn example() -> Result<(), Box<dyn std::error::Error>> {
 /// let cache = MbtilesCache::open("cache.mbtiles").await?;
 ///
-/// // Store a fetched tile with its freshness metadata.
-/// cache.set_cached(3, 1, 2, b"tile-bytes", CacheEntryMeta::new(1700000000, "etag-1")).await?;
+/// // Store a downloaded tile with its fetch time and freshness metadata.
+/// cache.set_cached(3, 1, 2, b"tile-bytes", CacheEntryMeta::new(1700000000, 1700003600, "etag-1")).await?;
 ///
 /// // Later: read it back, expired entries included (freshness is the caller's call).
 /// if let Some(tile) = cache.get_cached(3, 1, 2).await? {
 ///     println!("{} bytes, expires {:?}", tile.data.len(), tile.expires);
 /// }
 ///
 /// // After an HTTP 304 revalidation, bump the metadata without rewriting the blob.
-/// cache.update_cached_meta(3, 1, 2, CacheEntryMeta::new(1700003600, "etag-1")).await?;
+/// cache.update_cached_meta(3, 1, 2, CacheEntryMeta::new(1700003600, 1700007200, "etag-1")).await?;
 /// # Ok(())
 /// # }
 /// ```
@@ -121,7 +121,7 @@ impl MbtilesCache {
         self.schema
     }
 
-    /// Look up a cached tile and its `expires`/`etag` metadata.
+    /// Look up a cached tile and its `fetched`/`expires`/`etag` metadata.
     ///
     /// See [`Mbtiles::get_cached`] for the semantics (expired entries are still returned).
     #[hotpath::measure]
@@ -132,7 +132,7 @@ impl MbtilesCache {
             .await
     }
 
-    /// Insert or replace a cached tile with its [`CacheEntryMeta`] (`expires`/`etag`).
+    /// Insert or replace a cached tile with its [`CacheEntryMeta`] (`fetched`/`expires`/`etag`).
     ///
     /// See [`Mbtiles::set_cached`] for de-duplication and collision behavior.
     #[hotpath::measure]
@@ -150,7 +150,7 @@ impl MbtilesCache {
             .await
     }
 
-    /// Update only the `expires`/`etag` metadata of an existing entry (revalidation).
+    /// Update only the `fetched`/`expires`/`etag` metadata of an existing entry (revalidation).
     ///
     /// Returns `false` if there is no entry at the given coordinates.
     /// See [`Mbtiles::update_cached_meta`].
@@ -228,7 +228,7 @@ mod tests {
             let cache = MbtilesCache::open_with_schema(&path, schema).await.unwrap();
             assert_eq!(cache.schema(), schema);
             cache
-                .set_cached(2, 1, 1, b"tile-a", CacheEntryMeta::new(50, "v1"))
+                .set_cached(2, 1, 1, b"tile-a", CacheEntryMeta::new(40, 50, "v1"))
                 .await
                 .unwrap();
             cache
@@ -243,18 +243,20 @@ mod tests {
         assert_eq!(cache.schema(), schema);
         let a = cache.get_cached(2, 1, 1).await.unwrap().unwrap();
         assert_eq!(a.data, b"tile-a");
+        assert_eq!(a.fetched, Some(40));
         assert_eq!(a.expires, Some(50));
         assert_eq!(a.etag.as_deref(), Some("v1"));
         assert_eq!(cache.get_tile(2, 1, 2).await.unwrap().unwrap(), b"tile-b");
 
         // Revalidate the expiring entry without rewriting its blob.
         assert!(
             cache
-                .update_cached_meta(2, 1, 1, CacheEntryMeta::new(75, "v1"))
+                .update_cached_meta(2, 1, 1, CacheEntryMeta::new(60, 75, "v1"))
                 .await
                 .unwrap()
         );
         let a = cache.get_cached(2, 1, 1).await.unwrap().unwrap();
+        assert_eq!(a.fetched, Some(60));
         assert_eq!(a.expires, Some(75));
 
         // Purge the expired entry; the permanent one survives.