v5, createPersister and IndexedDB: absolutely amazing!

[frederikhors]

I'm using the latest v5 version: AMAZING!!!!!!! Thanks!

I tried the experimental_createPersister and it works amazingly too!

I'm using it like this, what do you think?

import {
  experimental_createPersister,
  type AsyncStorage,
} from "@tanstack/query-persist-client-core";
import { get, set, del, createStore, type UseStore } from "idb-keyval";

function newIdbStorage(idbStore: UseStore): AsyncStorage {
  return {
    getItem: async (key) => await get(key, idbStore),
    setItem: async (key, value) => await set(key, value, idbStore),
    removeItem: async (key) => await del(key, idbStore),
  };
}

export const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      gcTime: 1000 * 30, // 30 seconds
      persister: !browser
        ? undefined
        : experimental_createPersister({
            storage: newIdbStorage(createStore("db_name", "store_name")),
            maxAge: 1000 * 60 * 60 * 12, // 12 hours
          }),
    },
  },
});

[TkDodo]

looks lovely

[TheTimeWalker]

Hey! Thank you so much for the snippet. Works pretty great!

One suggestion: The advantage of IndexedDB is that you can safely put objects into the storage without having to rely on serializing/deserializing the content like e.g. with LocalStorage. I have noticed that right now it would by default still stringify the content and then save it to IndexedDB. My suggestion would be to additionally add this into your persister:

experimental_createPersister({
  storage: newIdbStorage(createStore("db_name", "store_name")),
  maxAge: 1000 * 60 * 60 * 12, // 12 hours,
  // Just pass the data
  serialize: (persistedQuery) => persistedQuery,
  deserialize: (cached) => cached,
})

The only problem is that TypeScript will complain that this serialize/deserialize function is not assignable, because it expects a string as a return type and cached type.

[TkDodo]

one problem is that we default to JSON.stringify and JSON.parse, so if you pass one of the methods but not the other, typescript complains. If you can figure it out, I'll be happy to merge it :)

[TheTimeWalker]

I think that shouldn't be a problem 👍🏻
Since if you change the serializer to not do stringify, you'll have to change the deserializer anyway because it would try to parse a string by default. 😛

[TkDodo]

yes, but we need to get this right on type level. This shouldn't err at runtime

[frederikhors]

@TkDodo @TheTimeWalker What do you think if I open an RFC in the issues to discuss it better?

[TkDodo]

Yes please 👍

[TheTimeWalker]

With the newest version (v5.10), you can now add a type to the persister, so now we can combine my serializer solution with your IndexedDB solution :)

import {
  experimental_createPersister,
  type AsyncStorage,
  type PersistedQuery
} from "@tanstack/query-persist-client-core";
import { get, set, del, createStore, type UseStore } from "idb-keyval";

function newIdbStorage(idbStore: UseStore): AsyncStorage<PersistedQuery> {
  return {
    getItem: async (key) => await get(key, idbStore),
    setItem: async (key, value) => await set(key, value, idbStore),
    removeItem: async (key) => await del(key, idbStore),
  };
}

export const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      gcTime: 1000 * 30, // 30 seconds
      persister: !browser
        ? undefined
        : experimental_createPersister<PersistedQuery>({
            storage: newIdbStorage(createStore("db_name", "store_name")),
            maxAge: 1000 * 60 * 60 * 12, // 12 hours,
            serialize: (persistedQuery) => persistedQuery,
            deserialize: (cached) => cached,
          }),
    },
  },
});

Under AsyncStorage and createrPersister, just add <PersistedQuery>

[frederikhors]

Amazing! Amazing you, tanstack's team, all the open source world!

[tam-eoh]

When I use persistQueryClient, my data immediately appears upon reloading. However, when I switch to experimental_createPersister, my data is not there when reload. Could this mean I've set up something incorrectly?

[TkDodo]

That is the intended behavior because the data is only loaded when a useQuery call needs it. That is more memory efficient because not everything is loaded upfront. If you want to load everything upfront, the PersitQueryClient is what you want. Both approaches have their pros and cons.

[hverlin]

Re-reading this question, seems similar to my question: #6213 (comment)

Is the solution to use both persister together? Or use setQueryData for critical queries if we want to avoid any loading states?

[joelbrewer]

I must be doing something wrong, but I'm getting the following error when using experimental_createPersister :

Here's my code:

...
import { QueryClient } from '@tanstack/react-query';
import { PersistQueryClientProvider } from '@tanstack/react-query-persist-client'
import { experimental_createPersister } from "@tanstack/query-persist-client-core";
import { get, set, del, createStore } from "idb-keyval";
...

function newIdbStorage(idbStore) {
  return {
    getItem: async (key) => await get(key, idbStore),
    setItem: async (key, value) => await set(key, value, idbStore),
    removeItem: async (key) => await del(key, idbStore),
  };
}

export const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      gcTime: 1000 * 30, // 30 seconds
    },
  },
});

const Root = () => {
  const persister = experimental_createPersister({
    storage: newIdbStorage(createStore("my_db", "my_store")),
    maxAge: 1000 * 60 * 60 * 12, // 12 hours
  }); 
 
...

  return (
    <UserProvider>
      <PersistQueryClientProvider
        client={queryClient}
        persistOptions={{ persister }}
      >
        <RouterProvider router={router}>
          <ReactQueryDevtools initialIsOpen={false} />
        </RouterProvider>
      </PersistQueryClientProvider>
    </UserProvider>
  );
}

export default Root;

[TkDodo]

experimental_createPersister is not something you can pass to PersistQueryClientProvider

[hverlin]

@frederikhors @TkDodo @TheTimeWalker Thank you for the suggested solutions here!

I have an additional question regarding this new experimental_createPersister API, do you restore the cache immediately when you load the app?

import { createStore, del, get, set, UseStore, keys, getMany } from 'idb-keyval';

export async function restoreQueryClientCache() {
  if (!idbStore) {
    return;
  }

  const cachedKeys = await keys(idbStore);
  const cachedValues = await getMany<PersistedQuery>(cachedKeys, idbStore);
  for (const queryData of cachedValues) {
    if (queryData?.buster !== UI_VERSION) {
      continue;
    }

    if (!queryData?.state?.data) {
      continue;
    }

    queryClient.setQueryData(queryData.queryKey, queryData.state.data, {
      updatedAt: queryData.state.dataUpdatedAt,
    });
  }
}

restoreQueryClientCache().catch(console.error);

root.render(...);

I am migrating from the solution which was suggested here.

import { get, set, del } from 'idb-keyval'
import { PersistedClient, Persister } from '@tanstack/react-query-persist-client'

export function createIDBPersister(idbValidKey: IDBValidKey = 'reactQuery') {
  return {
    persistClient: async (client: PersistedClient) => { await set(idbValidKey, client) },
    restoreClient: async () => { return await get<PersistedClient>(idbValidKey) },
    removeClient: async () => { await del(idbValidKey) },
  } as Persister
}

With the previous solution, the query would initially resolve with the data from the cache (which was quite nice as if one reloads the page, they immediately see the previous data - no loading state). After migrating to the new experimental_createPersister, the query will initially have no data until it reads from index db, leading to some flickering.

Is a workaround to run queryClient.setQueryData for all the queries stored in the cache before calling React render, as otherwise deserialization is lazy and will lead to flickering?

Wondering if there would be an option to make the query client immediately restore some of the queries?

[frederikhors]

@hverlin just to be clearer, are you talking about the loading that may occur between when the application starts and when the data is fetched from the cache?

[hverlin]

yes. Since fetching the data from the cache is async, if you don't use setQueryData you get some very short loading state

[AnandChowdhary]

Here is my solution, in case it helps someone. It doesn't write to IndexedDB immediately after each request, instead doing it when there's idle time and no more than once every 10 seconds. It also doesn't wait more than a minute and guarantees writing to cache before unload and when changing tabs, so I feel like it's the best balance.

import {
  experimental_createPersister,
  type AsyncStorage,
  type PersistedQuery,
} from "@tanstack/query-persist-client-core";
import { QueryClient } from "@tanstack/react-query";
import { createStore, del, get, set, type UseStore } from "idb-keyval";

/**
 * Creates an IndexedDB storage for React Query persistence
 *
 * This storage:
 * - Groups writes together to improve performance
 * - Uses browser idle time when available
 * - Handles different queue sizes with different strategies
 * - Ensures data is saved before page unload or tab switch
 *
 * @param idbStore - The IndexedDB store from idb-keyval
 * @returns Storage compatible with TanStack Query persister
 */
function newIdbStorage(idbStore: UseStore): AsyncStorage<PersistedQuery> {
  let pendingWrites = new Map<string, PersistedQuery>();
  let isProcessing = false;
  let idleCallbackId: number | null = null;
  let maxDelayTimer: number | null = null;
  let lastProcessTime = 0;

  const MAX_DELAY = 60000; // 1 minute max before forced processing
  const CRITICAL_THRESHOLD = 100; // Number of writes that triggers immediate processing
  const THROTTLE_DELAY = 10000; // 10 seconds between processing

  /**
   * Process all pending writes to IndexedDB
   *
   * Steps:
   * 1. Gather all pending writes
   * 2. Clear scheduled timers
   * 3. Write items to IndexedDB in parallel
   * 4. Re-queue any failed writes
   */
  const processPendingWrites = async () => {
    if (pendingWrites.size === 0 || isProcessing) return;

    const now = Date.now();
    if (now - lastProcessTime < THROTTLE_DELAY) {
      // Too soon to process again, reschedule
      scheduleWrites();
      return;
    }

    console.debug("Processing pending writes");

    try {
      isProcessing = true;
      lastProcessTime = now;
      const writes = Array.from(pendingWrites.entries());
      pendingWrites.clear();

      if (idleCallbackId) {
        if ("requestIdleCallback" in window) window.cancelIdleCallback(idleCallbackId);
        idleCallbackId = null;
      }

      if (maxDelayTimer) {
        clearTimeout(maxDelayTimer);
        maxDelayTimer = null;
      }

      await Promise.all(
        writes.map(async ([key, value]) => {
          try {
            await set(key, value, idbStore);
          } catch (error) {
            console.error(`Failed to persist query for key ${key}:`, error);
            // Re-add failed writes for retry
            pendingWrites.set(key, value);
          }
        }),
      );
    } finally {
      isProcessing = false;
      if (pendingWrites.size > 0) scheduleWrites();
    }
  };

  /**
   * Schedule processing of pending writes
   *
   * Uses two ways:
   * 1. requestIdleCallback for when browser is not busy
   * 2. setTimeout as a backup to ensure it happens anyway
   */
  const scheduleWrites = () => {
    // Skip if already scheduled
    if (idleCallbackId || maxDelayTimer) return;

    const timeUntilNextProcess = Math.max(0, THROTTLE_DELAY - (Date.now() - lastProcessTime));

    if ("requestIdleCallback" in window) {
      idleCallbackId = window.requestIdleCallback(() => processPendingWrites(), {
        timeout: MAX_DELAY, // Ensure callback runs within this time even if browser is busy
      }) as unknown as number;
    }

    // Ensure we don't process more often than every 10 seconds
    maxDelayTimer = window.setTimeout(processPendingWrites, Math.max(timeUntilNextProcess, MAX_DELAY));
  };

  // Set up event listeners for persistence guarantees
  if (typeof window !== "undefined") {
    // Process writes before page unload
    window.addEventListener(
      "beforeunload",
      (event) => {
        if (pendingWrites.size > 0) {
          // May trigger browser confirmation in some browsers
          event.preventDefault();
          // Force processing regardless of throttle on page unload
          lastProcessTime = 0;
          processPendingWrites();
        }
      },
      { capture: true }, // Run before other handlers
    );

    // Process writes when tab becomes hidden
    // Ensures data is saved when switching tabs or minimizing
    window.addEventListener("visibilitychange", () => {
      if (document.visibilityState === "hidden" && pendingWrites.size > 0) {
        // Force processing regardless of throttle when tab becomes hidden
        lastProcessTime = 0;
        processPendingWrites();
      }
    });
  }

  return {
    getItem: async (key) => {
      if (pendingWrites.has(key)) return pendingWrites.get(key);
      return await get(key, idbStore);
    },

    /**
     * Set an item in storage
     *
     * Handles storage differently based on queue size:
     * 1. Small queue: Wait for idle time or up to 60s
     * 2. Large queue (≥100 items): Save immediately
     */
    setItem: async (key, value) => {
      pendingWrites.set(key, value);

      if (pendingWrites.size >= CRITICAL_THRESHOLD) {
        // Process immediately when queue gets too large
        // May briefly slow UI but prevents memory problems
        const now = Date.now();
        if (now - lastProcessTime >= THROTTLE_DELAY) {
          await processPendingWrites();
        } else {
          scheduleWrites();
        }
      } else scheduleWrites();
    },

    removeItem: async (key) => {
      pendingWrites.delete(key);
      await del(key, idbStore);
    },
  };
}

export const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      gcTime: 1000 * 30, // After 30 seconds read from persistence
      persister:
        typeof window === "undefined"
          ? undefined
          : experimental_createPersister<PersistedQuery>({
              storage: newIdbStorage(createStore("db_name", "store_name")),
              maxAge: Infinity, // Keep persisted queries forever
              serialize: (persistedQuery) => persistedQuery,
              deserialize: (cached) => cached,
            }),
    },
  },
});

[frederikhors]

I think this is what experimental_createPersister does already, nope?

[spookyuser]

I think it might be a good idea to include this in experimental_createQueryPersister

function newIdbStorage(idbStore: UseStore): AsyncStorage {
  return {
    getItem: async (key) => await get(key, idbStore),
    setItem: async (key, value) => await set(key, value, idbStore),
    removeItem: async (key) => await del(key, idbStore),
  };
}

It feels like it's basically recommended everywhere experimental_createQueryPersister is and if you define it slightly differently to this, typescript goes crazy

[luca992]

Fyi, for others, using idb-keyval isn't the best, onversionchange isn't handled at all by idb-keyval. If you have more than one tab open and delete the database in one tab, the delete will wait for all other tabs to close that have opened an IDB connection with idb-keyval before continuing. Making all react query request freeze until those tabs close.

My current solution:

import {
  type AsyncStorage,
  experimental_createQueryPersister,
  type PersistedQuery,
} from "@tanstack/query-persist-client-core";

const DB_NAME = "db-name";
const STORE_NAME = "store";
const VERSION = 1;

let dbPromise: Promise<IDBDatabase> | null = null;

function getDb(): Promise<IDBDatabase> {
  if (dbPromise) return dbPromise;

  dbPromise = new Promise((resolve, reject) => {
    const req = indexedDB.open(DB_NAME, VERSION);

    req.onupgradeneeded = () => {
      const upgradeDb = req.result;
      if (!upgradeDb.objectStoreNames.contains(STORE_NAME)) {
        upgradeDb.createObjectStore(STORE_NAME);
      }
    };

    req.onsuccess = () => {
      const conn = req.result;
      conn.onversionchange = () => {
        console.trace("[IDB] version change – closing connection");
        conn.close();
        // clear the cached promise so the next getDb call re-opens a connection
        dbPromise = null;
      };
      resolve(conn);
    };

    req.onblocked = () => {
      console.warn("[IDB] open blocked: another connection is holding the DB");
    };

    req.onerror = () => {
      const err = req.error;
      reject(
        err instanceof Error
          ? err
          : new Error(`IndexedDB open failed: ${String(err)}`),
      );
      dbPromise = null;
    };
  });

  return dbPromise;
}

const storage: AsyncStorage<PersistedQuery> = {
  async getItem(key: string) {
    const db = await getDb();
    return new Promise<PersistedQuery | undefined>((res, rej) => {
      const r = db
        .transaction(STORE_NAME, "readonly")
        .objectStore(STORE_NAME)
        .get(key);
      r.onsuccess = () => res(r.result as PersistedQuery);
      r.onerror = () => rej(r.error ?? new Error("getItem failed"));
    });
  },

  async setItem(key: string, value: PersistedQuery) {
    return getDb()
      .then((db) => {
        if (!db) return;
        return new Promise<void>((resolve, reject) => {
          const req = db
            .transaction(STORE_NAME, "readwrite")
            .objectStore(STORE_NAME)
            .put(value, key);
          req.onsuccess = () => resolve();
          req.onerror = () => reject(req.error ?? new Error("setItem failed"));
        });
      })
      .catch((err) => {
        // experimental_createQueryPersister expects no error to be thrown
        console.trace("[IDB] skipping setItem:", err);
      });
  },
  
  async removeItem(key: string) {
    return getDb()
      .then((db) => {
        if (!db) return;
        return new Promise<void>((resolve, reject) => {
          const req = db
            .transaction(STORE_NAME, "readwrite")
            .objectStore(STORE_NAME)
            .delete(key);
          req.onsuccess = () => resolve();
          req.onerror = () =>
            reject(req.error ?? new Error("removeItem failed"));
        });
      })
      .catch((err) => {
        // experimental_createQueryPersister expects no error to be thrown
        console.trace(
          "[IDB] skipping removeItem:",
          err,
        );
      });
  },
};

export const persister = experimental_createQueryPersister<PersistedQuery>({
  storage,
  maxAge: 1000 * 60 * 60 * 24 * 30, // 30 days
  serialize: (persistedQuery) => persistedQuery,
  deserialize: (cached) => cached,
});

[luca992]

Just saying its probably safe to throw an error there if you want instead of an empty array on error

[frederikhors]

Do you mean this?

async entries(): Promise<[string, PersistedQuery][]> {
    try {
        const db = await getDb();
        const transaction = db.transaction(STORE_NAME, 'readonly');
        const store = transaction.objectStore(STORE_NAME);
        const keysRequest = store.getAllKeys();
        const valuesRequest = store.getAll();

        const [keys, values] = await Promise.all([
            promisifyRequest(keysRequest),
            promisifyRequest(valuesRequest)
        ]);

        return keys.map((key, index) => [String(key), values[index]]);
    } catch (error) {
        console.error('failed entries:', error);

        throw error;
    }
}

[luca992]

Just

async entries(): Promise<[string, PersistedQuery][]> {
        const db = await getDb();
        const transaction = db.transaction(STORE_NAME, 'readonly');
        const store = transaction.objectStore(STORE_NAME);
        const keysRequest = store.getAllKeys();
        const valuesRequest = store.getAll();

        const [keys, values] = await Promise.all([
            promisifyRequest(keysRequest),
            promisifyRequest(valuesRequest)
        ]);

        return keys.map((key, index) => [String(key), values[index]]);
}

[luca992]

But also, It'd be safer to do with a single cursor, something like:

async entries(): Promise<[string, PersistedQuery][]> {
  const db = await getDb();
  const tx = db.transaction(STORE_NAME, 'readonly');
  const store = tx.objectStore(STORE_NAME);

  const out: [string, PersistedQuery][] = [];

  await new Promise<void>((resolve, reject) => {
    tx.oncomplete = () => resolve();
    tx.onerror = () => reject(tx.error);

    const req = store.openCursor();
    req.onerror = () => reject(req.error);
    req.onsuccess = () => {
      const cursor = req.result as IDBCursorWithValue | null;
      if (!cursor) return; // done
      out.push([String(cursor.key), cursor.value]);
      cursor.continue();
    };
  });

  return out;
}

[frederikhors]

Works amazingly well! Thanks!

[jakearchibald]

Fyi, for others, using idb-keyval isn't the best, onversionchange isn't handled at all by idb-keyval.

idb-keyval only ever opens a single version of the database, so it never has a version change.

If you have more than one tab open and delete the database in one tab

Yeah, idb-keyval assumes it is in control of the database, and it doesn't offer the ability to delete the database. If you want lower-level control, use https://www.npmjs.com/package/idb.

[frederikhors]

@luca992 do you know how to arrange a minimal reproduction of the issue?

[luca992]

@jakearchibald "made up my mind, regardless of evidence", idk what high horse you think you're on but this is not a discussion on your library. If I wanted to discuss issues with your library I would have submitted an issue in your repo about the fact that idb-keyval does not handle recovering from onversionchange when the db it uses is deleted, which is a bug in your implementation that could 100% be handled.

Plus, 1.2kb is not nothing, and I had no plans to use it anywhere else in my project. There's no need for a dependency to make a experimental_createPersister instance backed by idb.

I'm sure idb-keyval has it's uses but my point is it's not a great choice for this discussion which is creating a experimental_createPersister because of this edge case. I spent days figuring out why all react-query operations were stalling, and figured out it was due to idb-keyval holding open the db after deleting the dbs in another tab. I decided to share because I thought it would prevent other from having to deal with the pain of debugging this edge case.

@frederikhors It's quite easy to reproduce. You just delete all indexDb databases in one tab, while idb-keyval holds it open in another. All further idb-keyval requests will stall until the tab holding the database open with idb-keyval is closed

[jakearchibald]

https://random-stuff.jakearchibald.com/idb-keyval/clear/ - here's a basic example. Data can be set & cleared with a single command, even if there are multiple tabs.

[luca992]

I am well aware clear would work, that's not what I said

[jakearchibald]

I'm replying to #6213 (reply in thread)