Performance expectations for large libraries (~110k items) and dashboard loading

[9acca9]

What are you trying to do?

I am trying to use the main dashboard to manage a large library of approximately 110,000 books. I expect the main page and the library counters (total books, series, authors) to load quickly or asynchronously after the initial scan is already finished, allowing for a smooth navigation experience.

What's happening instead?

Every time I access the main page, the UI takes about 30 seconds to load and often feels "stuck" or unresponsive until the book and series counts are displayed.

Since this happens even when no active scan is running, I am not sure if this is the expected behavior for a library of this size or if there is a configuration issue on my end. It seems like the application might be performing a live database count/query on every page load instead of retrieving cached values or metadata. Is there any way to optimize this or is it a known limitation for collections of this scale?

Your setup

grimmory: 2.3.0
installation: docker
s.o: lxc/debian

Anything else that might help?

No response

Before submitting

• I've searched existing discussions and I can't find anything that looks like my problem
• I've reviewed this report and have completed it as fully as I can

[obviouslyallie]

We don't exactly have data or test models on libraries that large. Contributors have already identified issues with hibernation and pagination in general which is affecting loading and I'm sure its worse on larger libraries. Some of those items are being addressed for next version and likely more updates in those areas to come.
Large library testing will probably still be a problem for awhile.

[9acca9]

I see. I'm available to run some tests with my collection if that helps.
thanks!

[chemicoholic21]

Yeah so your hunch is probably right, those counters are most likely just running a full COUNT query every single time you hit the page. At 110k books that's just going to be slow, there's no way around it until someone caches those values properly.
Not much you can do on your end config wise honestly. You could poke around and make sure your container has decent memory allocated and that WAL mode is on for the database, but realistically even with that you're probably shaving seconds off not fixing it entirely. It's just an architectural thing that needs to be solved in the code.
The offer to help test is actually really useful though. Big libraries like yours are hard to replicate for testing so having someone willing to run stuff against a real 110k collection is not something that comes up often. If you wanted to dig into it you could try pulling the actual query times directly from the DB while a page load is happening, that would at least tell everyone whether the bottleneck is the query itself or something else going on in the stack. That kind of info would probably be more useful than anything else right now while the pagination and loading fixes are still being worked on.

[9acca9]

Excellent. When i run the
docker logs <container_name> --follow

i dont get nothing when grimmory is loading (a boring video):

Grabacion.de.pantalla.desde.2026-04-18.18-00-30.mp4

[chemicoholic21]

Ok that's actually really telling. If the container logs are completely silent during those 30 seconds, the backend probably isn't even the bottleneck the hang might be happening on the frontend before it even fires the requests, or there's a request sitting in a queue that the app isn't logging.

Best thing to do now is open your browser dev tools during a load. In Chrome/Firefox just hit F12, go to the Network tab, clear it, then navigate to the dashboard and watch what happens.

You're looking for:

• Any request that shows a long "waiting" time (the TTFB column, or hover over the bar)
• Requests that are stacked up and starting one after another instead of in parallel
• Anything that shows as "pending" for a long time

If you can screenshot or screen-record the Network tab during a full load and post it here, that will tell us a lot more than the logs did. Specifically whether the 30s is one slow request or a chain of many.

[9acca9]

mmmm, so books.
Maybe the "problem" is of what it seems read all the books at the beginning? (i dont know, just read the word "books" there, lol, that is 82,79MB) or maybe the lazy-img.....

video:

Grabacion.de.pantalla.desde.2026-04-18.19-51-26.mp4

picture:

Thanks for your help!

[chemicoholic21]

@9acca9 look at the books request, 82MB in one shot, that's your 17 seconds right there. the app is just dumping your entire library on every load with zero pagination. the lazy thumbnails are just piling on after that.
nothing you can do on your end, this needs to be fixed in the app. but this screenshot is perfect evidence, worth posting as a separate bug report and tagging @obviouslyallie since they mentioned pagination is already being worked on.

[obviouslyallie]

Thank ya'll for going through this. Super helpful. Concrete evidence makes this so much easier.
Post 3.0 release I think getting some more technical eyes on this and scope out an issue that could help with this.

I think on that release there have been some PRs that have already been merged vs 2.3, but I think this could likely use more work.

[9acca9]

@chemicoholic21 Excellent. Thanks for guiding me!
@obviouslyallie If you need me to do any tests in the future, just tell me.
By the way, should I create an issue? Or is that unnecessary now that you've seen the post?
Regards

[balazs-szucs]

Hi,

apologies for the late response, yes, from 3.0.0 onward we will have server-side-pagination that should solve this issue. There are parts of the codebase that still does not scale well, e.g., readingSessions/thumbnails and such, however most of the bigger complaints should be solved, shortly.

[9acca9]

mmm, it seems that version 3.0.0 is ever worse for me:

2026-04-22T23:29:02.377-03:00 INFO 1 --- [booklore-api] [cat-handler-111] org.hibernate.SQL_SLOW : Slow query took 1632 milliseconds [select se1_0.id,(SELECT COUNT(*) FROM book_shelf_mapping bsm WHERE bsm.shelf_id = se1_0.id),se1_0.icon,se1_0.icon_type,se1_0.is_public,se1_0.name,se1_0.sort,se1_0.user_id from shelf se1_0 where se1_0.user_id=? or se1_0.is_public=true]

2026-04-22T23:30:03.027-03:00  INFO 1 --- [booklore-api] [cat-handler-123] org.hibernate.SQL_SLOW                   : Slow query took 1365 milliseconds [select be1_0.id,be1_0.added_on,be1_0.audiobook_cover_hash,be1_0.book_cover_hash,bf2_0.book_id,bf2_0.id,bf2_0.added_on,bf2_0.alt_format_current_hash,bf2_0.archive_type,bf2_0.bitrate,bf2_0.book_type,bf2_0.channels,bf2_0.chapter_count,bf2_0.chapters_json,bf2_0.codec,bf2_0.current_hash,bf2_0.description,bf2_0.duration_seconds,bf2_0.file_name,bf2_0.file_size_kb,bf2_0.file_sub_path,bf2_0.is_folder_based,bf2_0.initial_hash,bf2_0.is_book,bf2_0.is_fixed_layout,bf2_0.sample_rate,be1_0.deleted,be1_0.deleted_at,be1_0.is_physical,be1_0.library_id,l1_0.id,l1_0.allowed_formats,l1_0.file_naming_pattern,l1_0.format_priority,l1_0.icon,l1_0.icon_type,l1_0.metadata_source,l1_0.name,l1_0.organization_mode,l1_0.sort,l1_0.watch,lp1_0.id,lp1_0.library_id,lp1_0.path,m1_0.book_id,m1_0.abridged,m1_0.abridged_locked,m1_0.age_rating,m1_0.age_rating_locked,m1_0.amazon_rating,m1_0.amazon_rating_locked,m1_0.amazon_review_count,m1_0.amazon_review_count_locked,m1_0.asin,m1_0.asin_locked,m1_0.audible_id,m1_0.audible_id_locked,m1_0.audible_rating,m1_0.audible_rating_locked,m1_0.audible_review_count,m1_0.audible_review_count_locked,m1_0.audiobook_cover_locked,m1_0.audiobook_cover_updated_on,m1_0.authors_locked,m1_0.categories_locked,m1_0.comicvine_id,m1_0.comicvine_id_locked,m1_0.content_rating,m1_0.content_rating_locked,m1_0.cover_locked,m1_0.cover_updated_on,m1_0.description,m1_0.description_locked,m1_0.embedding_updated_at,m1_0.goodreads_id,m1_0.goodreads_id_locked,m1_0.goodreads_rating,m1_0.goodreads_rating_locked,m1_0.goodreads_review_count,m1_0.goodreads_review_count_locked,m1_0.google_id,m1_0.google_id_locked,m1_0.hardcover_book_id,m1_0.hardcover_book_id_locked,m1_0.hardcover_id,m1_0.hardcover_id_locked,m1_0.hardcover_rating,m1_0.hardcover_rating_locked,m1_0.hardcover_review_count,m1_0.hardcover_review_count_locked,m1_0.isbn_10,m1_0.isbn_10_locked,m1_0.isbn_13,m1_0.isbn_13_locked,m1_0.language,m1_0.language_locked,m1_0.lubimyczytac_id,m1_0.lubimyczytac_id_locked,m1_0.lubimyczytac_rating,m1_0.lubimyczytac_rating_locked,m1_0.moods_locked,m1_0.narrator,m1_0.narrator_locked,m1_0.page_count,m1_0.page_count_locked,m1_0.published_date,m1_0.published_date_locked,m1_0.publisher,m1_0.publisher_locked,m1_0.ranobedb_id,m1_0.ranobedb_id_locked,m1_0.ranobedb_rating,m1_0.ranobedb_rating_locked,m1_0.rating,m1_0.review_count,m1_0.reviews_locked,m1_0.series_name,m1_0.series_name_locked,m1_0.series_number,m1_0.series_number_locked,m1_0.series_total,m1_0.series_total_locked,m1_0.subtitle,m1_0.subtitle_locked,m1_0.tags_locked,m1_0.title,m1_0.title_locked,be1_0.metadata_for_write_updated_at,be1_0.metadata_match_score,be1_0.metadata_updated_at,be1_0.scanned_on from book be1_0 join book_file bf1_0 on be1_0.id=bf1_0.book_id left join book_file bf2_0 on be1_0.id=bf2_0.book_id join library l1_0 on l1_0.id=be1_0.library_id left join library_path lp1_0 on lp1_0.id=be1_0.library_path_id left join book_metadata m1_0 on be1_0.id=m1_0.book_id where bf1_0.current_hash=? and bf1_0.is_book=true and (be1_0.deleted is null or be1_0.deleted=false) order by bf2_0.id]
2026-04-22T23:30:03.064-03:00  INFO 1 --- [booklore-api] [cat-handler-123] o.b.service.koreader.KoreaderService     : getProgress: fetched progress='/body/DocFragment[18]/body/p[9]/text().181' percentage=0.33 for userId=2 bookHash=27ab4b529d7dd063b712b9ed725bc3b4

1:29 min to load the books:

And a big log with some errors:
https://pastebin.com/9bNvmcU9

Do you want i open issue?

Thanks