Changelog

Unreleased:

1.14.2:
* FIX: Ignore pages without a single revision / revid (@benoit74 #2091)
* CHANGED: Set default log level to `log` to sufficiently verbose logs by default (@benoit74 #2121)
* FIX: Custom home page thumbnails caption are getting overflowed (@benoit74 #1972)
* CHANGED: Allow to process wikis with mwWikiPath=/ (@benoit74 #2163)
* FIX: Some images of custom home page thumbnails are missing (@benoit74 #2158)
* FIX: Enhance robustness of coordinates API detection (@benoit74 #2167)

1.14.1:
* FIX: Ensure AWS SDK has access to object size when issuing an upload (@benoit74 #2117)
* FIX: Change log level of S3 missing keys message (@benoit74 #2144)
* FIX: Logic to set .webp path prefix on reencoded images is skewed (@benoit74 #2140)
* FIX: S3 cached images are missing (@benoit74 #2136)
* FIX: Do not rely on URL filename extension to detect images (@benoit74 #2088)
* FIX: S3 cached image are never used (@benoit74 #2138)
* FIX: RequestTimeTooSkewed error while downloading from cache (@benoit74 #2137)
* FIX: Axios timeouts are not properly handled (@benoit74 #2139)
* FIX: Many tasks are hanging forever (@benoit74 #2141)
* CHANGED: Pre-install all Node.JS dependencies in Docker build (@benoit74 #2148)

1.14.0:
* FIX: Remove S3 upload concurrency to avoid 'RequestTimeTooSkewed' errors (@benoit74 #2118)
* FIX: Strip HTML tags from <title> HTML nodes (@pavel-karatsiuba #1801)
* FIX: Remove inline JS to comply with restrictive CSPs (@audiodude #2111)
* FIX: Properly download maps by fixing MWoffliner user-agent (@audiodude #2068 #2062)
* FIX: Clearly configure Node.js supported versions (@kelson42 #2065)
* FIX: Properly await async polyfill function to avoid race condition (@audiodude #2055)
* FIX: Always use fallback language strings if needed (@audiodude #2050)
* FIX: Improve API end-point - response mime-type - detection (@audiodude #2041)
* FIX: Measure ZIM Metadata length by grapheme (@kelson42 #2023)
* FIX: Multiple imporvement around the usage() (@kelson42 #1996)
* FIX: Error 'stack: "Error: Input file is missing' @VadimKKovalenkoSNF #1905)
* UPDATE: Use MediaWiki 'action' API version 2 (@VadimKovalenkoSNF #1877)
* NEW: Introduce '--forceRender' paramter (@VadimKovalenko #1901)
* NEW: Introduce hardcoded blacklist of namespaces (@kelson42 #1919)
* NEW: '--insecure' option to disable client certificate verification (@kelson42 #2010)
* NEW: Add support of Wikimedia `mobile-html` API end-point (@audiodude #1903 #2101 #2043 #2113, @kelson42 #2012, @VadimKovalenkoSNF #1947)
* NEW: Add email notification to `main` based CI (@audiodude #2059)
* NEW: Proper modular API end-point renderer (@VadimKovalenkoSNF #1839 #1886 #1897 #1898 #1929 #1933 #1939 )
* DEL: Support of `mobile-sections` end-point (@VadimKovalenkoSNF #1902)

1.13.0:
* FIX: Many akwards scenarios around Redis (@uriesk #1756)
* FIX: Wrongly downloaded hidden media files (@pavel-karatsiuba #1780 #849)
* FIX: Incompatibility scenario in hacking the start module (@pavel-karatsiuba #1783)
* FIX: Bad ordering in Javascript module loading at reading time (@pavel-karatsiuba #1766)
* FIX: Unpredictive behaviour depending the order of --format (@pavel-karatsiuba #1826)
* FIX: Broken auto-redirect in a few Wiktionaries (@pavel-karatsiuba #1811)
* FIX: Mosaic selection welcome page with irrelevant pictures (@pavel-karatsiuba #1794)
* FIX: Broken logo downloading because of protocol mismatch (@pavel-karatsiuba #1800)
* UPDATE: Improved container image documentation (@kelson42 #1827)
* UPDATE: Container image using latest versions of Node.js & Redis (@kelson42 #1814)
* UPDATE: Container images only pushed to ghcr.io (@rgaudin #1812)
* UPDATE: Better Axios error reporting (@FledgeXu #1795)
* NEW: Better checking of command line arguments (@pavel-karatsiuba #1823 #1806)
* NEW: Introduce option --customZimLanguage (@kelson42 #1841)
* NEW: Synchronization of UI translations with Translatewiki.net (@kelson42 #116)
* NEW: Threshold specification to the --verbose option (@pavel-karatsiuba #1785)

1.12.1:
* FIX: Bad behaviour because of wrong mime-type in S3 (@kelson42 #1772)
* FIX: Unefficient CI steps (@kelson42 1770)

1.12.0:
* NEW: Add --articleListToIgnore option (@uriesk #1706)
* NEW: Add --longDescription option (@pavel-karatsiuba #1755)
* NEW: Remove support of LZMA (@kelson42 #1643)
* NEW: Drop support of Node.js 14 (@kelson42 #1671)
* NEW: Revamping of interactions with Redis (@uriesk #1719 #1720)
* NEW: Use Jest in place of Tape for automated tests (@pavel-karatsiuba #1716)
* NEW: Introduce/migrate to usage of ESM (@pavel-karatsiuba #1688)
* NEW: Use Eslint instead Tslint (@pavel-karatsiuba #1738 #1747)
* FIX: Simplisitc HTTP retry strategy for downloading files (@uriesk #1751)
* FIX: Broken support of OGA audio files (@pavel-karatsiuba#1763)
* FIX: Problematic inline Javascript to comply with CSP (@pavel-karatsiuba #1752)
* FIX: Partly broken WEBP related mime-type mgmt (@uriesk 1723)
* FIX: Broken show/hide toggle in Wikipedia infoboxes (@zhuowei #1665)
* FIX: Handle properly Fandom images src URL (@uriesk #1712)
* FIX: Upgrade most of the dependences (@uriesk @pavel-karatsiuba #1715)

1.11.12:
* FIX: ZIM illustration now always in PNG format
* FIX: Somehow cropped ZIM illustration
* NEW: Add support for mw:File typeof
* NEW: Use keep-alive TCP flag
* UPDATE: Redis 8 & Node 18 in Docker image

1.11.11:
* FIX: Workaround wrong Wep mime-type returns sporadically by Mediawiki
* UPDATE: Better speed computation

1.11.10:
* FIX: Videos in with too low resolution
* UPDATE: Allow to have JSON error 'missingarticle' in module dependence retrieval
* UPDATE: Stop allow Mediawiki instance with misconfigured SSL

1.11.9:
* FIX: Thumbnail URL retrieval with new edge cases
* FIX: Crash in case of badly formated URL in HTML
* UPDATE: Better video picker - respect the HTML element size
* UPDATE: Reduce number of parallel requests by 30%
* UPDATE: HTTP request timeout increased to 120 seconds

1.11.8
* FIX: Redirect retrieval if lots of them
* FIX: Webp polyfill concurency problems

1.11.7
* NEWS: Better ingest Windows encoded article lists
* UPDATE: Use libzim 2.4.4

1.11.6
* FIX: Fix local Parsoid upstream URL config (regression)

1.11.5
* FIX: Better handle of score pictures
* FIX: Desktop API detection (broken in case of redirected welcome page)
* FIX: Handling of articles deleted during the scrape
* FIX: Mediawiki detection crashing if very old Mediawiki
* FIX: Crash if article name is 'constructor'

1.11.4
* NEW: Check properly for unsupported Mediawiki versions
* FIX: Remove surnumerous underscores in M/Flavour
* FIX: Various bugs around ResourceLoader
* FIX: Edge-case bugs around API detection
* FIX: Buggy links for article starting with a /
* UPDATE: Stops when an article details retrieval fails

1.11.3
* FIX: Partly broken Webp image conversion
* FIX: HTTP error status code because wrong 'accept' HTTP request header
* FIX: Bad exit code in some error cases
* UPDATE: Remove a few additional Parsoid images attributes
* NEW: Allow S3 unencrypted connections

1.11.2
* FIX: Local Parsoid for HTTP only instances
* FIX: Buggy sub-page URL rewritting
* FIX: Buggy empty paragraph removal
* FIX: Private Mediawiki login/scraping
* FIX: VisualEditor buggy detection
* FIX: WikiHiero pictures scraping
* NEW: Better --customMainPage mgmt
* NEW: Simplify CSS/JS/Media paths a bit
* NEW: Always show first single section
* NEW: Use latest libzim

1.11.1
* FIX: Buggy API end-points detection
* FIX: Weak image mime-type assignement
* FIX: Partly missing revision ids
* UPDATE: Article details retrieval speed-up
* UPDATE: Use Node.js 14 in Docker

1.11.0
* FIX: multiple bugs around redirects
* FIX: multiple regression if local Parsoid is used
* NEW: image lazy loading
* NEW: option to convert png/jpg images to WebP

1.10.12:
* NEW: Add ability to make ZIM files using Zstandard compression

1.10.11:
* UPDATE: Use node-libzim 2.4.2
* FIX: first section scraping
* FIX: custom plugin for Wiktionary
* FIX: ZIM filename date
* FIX: broken scraping with local Parsoid

1.10.10:
* UPDATE: Small memory usage improvement
* FIX: Regression by article retrieval (from namespace)

1.10.9:
* FIX: Support of video subtitles
* FIX: Run generating ZIM files with different dates in filename
* UPDATE: Better default standard Redis client conf
* UPDATE: Use node-libzim version 2.2.4
* UPDATE: Optimisation of the S3 workflow
* UPDATE: Various revamping to simplify code base
* DEL: Custom download cache

1.10.8:
* UPDATE: Use node-libzim version 2.2.3

1.10.7:
* FIX: Improving parsoid/source select logic
* FIX: Better errors and timeouts handling
* FIX: AWS S3 build errors and logging
* FIX: title on main pages and footer links
* FIX: scraping articles with question marks
* UPDATE: Add support for node v12+
* UPDATE: Migrate to node-libzim v2.2.2
* UPDATE: Migrate to Redis 6 and
* UPDATE: Migrate to sharp
* UPDATE: Remove odd metadata and article dependencies requests
* UPDATE: Add support for parametrized translatable strings + add Spanish translation
* UPDATE: Simplify automated tests call

1.10.6:
* FIX: Hotfix for TypeScript build config

1.10.5:
* FIX: Fix docker build

1.10.4:
* FIX: Optimizations/fixes for better scaling (#1025)
* FIX: Refactoring / making the code clean

1.10.3:
* FIX: Better logging while fetching the articles

1.10.2:
* FIX: Fix broken npm package

1.10.1:
* NEW: Support "http" prefixed URLs for --optimisationCacheUrl

1.10.0:
* NEW: Support of S3 --optimisationCacheUrl

1.9.13:
* NEW: Docker image has an embedded Redis daemon
* DEL: --keepHtml and --format=nozim
* UPDATE: Rename --useCache in --useDownloadCache
* UPDATE: Rename --cacheDirectory in --downloadCacheDirectory

1.9.12:
* UPDATE: Release documentation

1.9.11:
* FIX: broken '_details' tag values (#980)
* FIX: non-empty sections removal (#986)
* NEW: --osTmpDir optional argument (#966)
* NEW: --flavour for specifying pre/post processing scripts

1.9.10:
* NEW: nothing

1.9.9:
* FIX: lib imports

1.9.8:
* UPDATE: update libzim
* UPDATE: update zimcheck
* UPDATE: KaiOS Rendering Hack (#931)
* UPDATE: Use execa instead of execPromise (#948)
* UPDATE: Add canonical meta
* UPDATE: Keep inline mathml HTML (#164)
* UPDATE: Replace Kartographer maplink numbers (#871)
* FIX: Wikivoyage DE rendering (#873)
* FIX: Remove duplicate article redirects (#889)
* FIX: No-JS mobile rendering (#962)

1.9.7:
* NEW: Add 'Flavour' ZIM metadata
* NEW: revamped tag system

1.9.6:
* FEATURE: Add format aliasing option
* UPDATE: Remove skipCacheCleaning parameter
* FIX: Welcome page detection & rendering

1.9.5:
* UPDATE: Update node-libzim
* UPDATE: Cache logic (only use FS)
* UPDATE: Remove JSON request cache
* UPDATE: Specify tile layout for screens ~8096px
* FIX: Welcome page '/' rewriting
* FIX: Empty paragraph removal

1.9.4:
* UPDATE: README improvements (#880)
* UPDATE: Smarter and more efficient article fetching
* UPDATE: remove hard-coded font-sizes for ehadings (#825)
* UPDATE: write categories to 'U' namespace
* UPDATE: redis handling improvements (#448)
* UPDATE: improve code quality (#687)
* UPDATE: improve testing
* FIX: implement scraping of <audio> elements
* FIX: Thumbnail image fetching on nopic (#867)

1.9.3:
* FIX: coordinate retrieval continue bug
* FEATURE: add throttled keepalive log (#791)
* FEATURE: add scraper metadata to ZIM file (#787)
* UPDATE: only collect relevant redirects (#790)
* UPDATE: disable quirks mode for article html (#717)
* UPDATE: improve no-image articlelist home page (#759)
* UPDATE: subsections open by default (#786)

1.9.2:
* FIX: favicon writing logic
* FIX: special uris no longer re-written (#771)
* FIX: prefer remote Parsoid for fallbacks (#767)
* FIX: redirect-writing issues (#769)
* FIX: --help option (#768)
* UPDATE: improve external link handling
* UPDATE: stop retrying 404s (#600)

1.9.1:
* UPDATE: Update node-libzim (and libzim) to improve memory usage (#706)
* UPDATE: Implement basic category scraping (#15)
* UPDATE: Increase default scrape "speed" (#744)
* UPDATE: Report on failed article downloads ()
* UPDATE: Retry all failed media downloads at the end of scrape (#723)
* UPDATE: Re-introduce --useCache option (#700)
* UPDATE: Reduce system dependencies (#681)
* UPDATE: Use native <details> element for collapsible sections in ZIM files (#677)
* FIX: Update article title logic (#605)
* FIX: Logging format improvements (#662)
* FIX: Redis clearing improved (#732, #731, #448)
* FIX: Text only article list main page background colour fixed (#728)
* FIX: Resource relative url improvement (#726)

1.8.6:
* UPDATE: Parallel processing improvements
* UPDATE: Increased Parsoid and MCS limits #710
* UPDATE: Switched to native `<details>` element for collapsible sections #677
* FIX: Re-introduced automatic ZIM tagging #709

1.8.5:
* FIX: Thumbnail nopic quality improvements #693
* FIX: Add error handling for "URI malformed" error #697
* FIX: Add checks to stop "normalized of undefined" errors #698
* FIX: Clear file store between dumps #699
* UPDATE: Re-implement useCache for downloader #700

1.8.4:
* FIX: Article title logic #605
* FIX: Log formatting issue #662
* FIX: Clear Redis when exiting #448
* UPDATE: Moved "convert" utility to a node library #681

1.8.3:
* FIX: Parsoid wt2html limit increased #664
* FIX: Parsoid request timeout increased # 660
* FIX: Logging bug #662
* FIX: Remote article list fixed #661

1.8.2:
* UPDATE: Improved dump performance
* FIX: Reduce memory usage
* FIX: #654 Duplicate jsConfigVars.js
* FIX: #645 Welcome page mirroring
* FIX: #643 --customMainPage

1.8.1:
* FIX: ZIM filename language code #624

1.8.0:
* UPDATE: Removed 'tmp' files and directory #448 #575
* UPDATE: Write ZIM files directly (Using Libzim) #184
* UPDATE: Removed --deflateTmpHTML and --tmpDirectory arguments #575 #576
* UPDATE: Implemented better request backoff #496
* UPDATE: Change file names/paths #278
* UPDATE: Removed --writeHtmlRedirects argument #506
* UPDATE: Removed --localMCS option (automatically detect) #490
* UPDATE: Updated documentation #423
* FIX: Other stability, logging and error handling fixes

1.7.1:
* FIX: Broken Docker image #534

1.7.0:
* UPDATE: Deprecated desktop scraping (not --mobileLayout)
* UPDATE: Improve the article list welcome page #449
* UPDATE: Add string translation mechanism #16
* UPDATE: Fetch hundreds of redirects at a time instead of one #325
* UPDATE: Integrated MCS alongside Parsoid #388 #347 #404
* UPDATE: Introduce automated testing #146
* UPDATE: Support remote articleList and favicon #457
* UPDATE: Added --useCache option #263

1.6.0:
* FIX: Template referencing #421
* FIX: Generated files are more no-js friendly #324
* FIX: Added top level section when it doesn't exist in API Response #180
* FIX: Fixed --addNamespace #338
* FIX: CSS Image issue #418
* UPDATE: Implemented PDF downloads #338
* UPDATE: Added mwmatrixoffliner tests
* UPDATE: Added git revision to version log #426
* UPDATE: Normalised Resources and Templates
* UPDATE: Strip HTML from title tags #159
* UPDATE: Improved Error Handling
* UPDATE: Converted to TypeScript #386

1.5.0:
* FIX: Favicon related crashes #347 #354
* FIX: Improved error logging when invalid logo information is returned from the server #363
* UPDATE: Updated parsoid version to latest security patch (not latest Parsoid)

1.4.1:
* FIX: Fix support of nodejs versions with V8 core #281
* FIX: Missing external link icon with desktop layout #279
* FIX: Sporadic wrong redirects #258
* FIX: Remove empty tags #259
* FIX: Remove sections sur-numerous braces #205