Compare online library XML and CMS one

[benoit74]

Before moving to production the CMS, we need to ensure library.xml produced by the CMS is equivalent to the one produced by the library generator.

[benoit74]

For the record, this is what I run on the shuttle to load books to the CMS

cd /app/maint-scripts/
pip install "libzim==3.7.0" requests
export QUARANTINE_WAREHOUSE_ID="1e9783eb-296e-4cb0-853a-5355a92b2323"
export STAGING_WAREHOUSE_ID="1e9783eb-296e-4cb0-853a-5355a92b2323"
export QUARANTINE_BASE_PATH="quarantine"
export STAGING_BASE_PATH="staging"
python create_collection_titles.py
python populate_cms_database.py > logs.txt 2>&1
python compare_catalogs.py

To clean afterwards, I run these DB statements:

DELETE FROM book_location;
DELETE FROM book;
DELETE FROM collection_title;
DELETE FROM title;

Before production, we will need to run (to delete fake records created by create_collection_titles)

DELETE FROM collection_title WHERE title_id IN (SELECT id FROM title WHERE name LIKE 'foo\_%');
DELETE FROM title WHERE name LIKE 'foo\_%';

[benoit74]

We know have only differences in terms of which books are present. Data of books is then strictly identical (beside a +1 in some books size - in kiB - because we've decide to round up the size where library generator script seems to have simply make a round).

Books only in CMS (not in current library.xml)

• https://cms.openzim.org/book/07d8974c-5dcd-e90b-5bd3-c5f16519561a
• https://cms.openzim.org/book/14fdd465-f5f8-622c-6938-bda4a3829ea0
• https://cms.openzim.org/book/2b291282-8230-6515-7be6-b778962d6c5c
• https://cms.openzim.org/book/2d26b30a-6cf2-6fa2-361b-0c2dfe3b7528
• https://cms.openzim.org/book/2fa5d719-ccc3-381d-9b09-00a3f701f479
• https://cms.openzim.org/book/5fe7276f-04a8-be6c-ed0e-55e2bdb1dc3a
• https://cms.openzim.org/book/62edec2b-91cb-cc81-8733-bda7da4f2638
• https://cms.openzim.org/book/6b180680-022b-28e2-1ba5-80631ac69b8c
• https://cms.openzim.org/book/6d69b939-32a1-21d6-3276-dc2ba84ea157
• https://cms.openzim.org/book/6dee01df-d475-6d26-3fd8-90e737b713c8
• https://cms.openzim.org/book/75b27b51-f779-8728-43bf-1f4300d9f7d4
• https://cms.openzim.org/book/8f090b50-eda0-ba61-305d-26f50987ff7c
• https://cms.openzim.org/book/9b4bc967-1063-0ad2-f446-10721f452930
• https://cms.openzim.org/book/b4cca23e-bd83-8cb2-10e0-9000a1aa7a88
• https://cms.openzim.org/book/d2fe3e62-a39d-401d-3ce9-275c29cbb74e

I've identified following issues:

• Delete Wikisummaries zim files zim-requests#429
• Empty flavour should be considered as null in DB  #182
• Delete (very) old ZIMs from download with bad metadata zim-requests#1653
• Scraper should not finish ZIM when bad exception arises devdocs#54

Regarding devdocs, I've rebuild the ZIM and issue will settle without needing scraper to be released (even if it would be way better). But it proves that we need #183 to also refuse these ZIMs in the normal publishing from Zimfarm notification.

Books only in current library.xml (not in CMS):

• 0548eb6b-92f2-3b17-d964-81200c6dd68b / phet/phet_mo_all_2025-03.zim
• 14d10547-5a2a-4cd1-0668-bf4f8c6efe19 / devdocs/devdocs_en_latex_2026-02.zim
• 967e7254-3e06-702d-ef61-714c638028f2 / phet/phet_ef_all_2025-03.zim
• d8c1a965-0081-2d42-d826-26c9fa586161 / devdocs/devdocs_en_vuex_2026-02.zim

I've identified following issues:

• phet_mo_all ZIMs have a bad phets_ro_all name zim-requests#1654 / mo language is wrongly interpreted as ro/ron phet#307
• same devdocs issue than above
• phet_ef_all ZIM has a bad ZIM Name zim-requests#1655 / Correctly use ISO639 codes in ZIM Name and ZIM filename, and align both phet#308

[benoit74]

We now have only 5 differences left, all covered in #200 "hopefully".