Sitemap Extractor: optimize download & parsing (streaming XML, gzip, keep-alive, concurrent child fetch, incremental skip, dedupe)

[nicklamonov]

Summary

Independent of the overlap change (#339), the sitemap download/parse path itself can be made substantially faster and lighter for large sitemap trees.

Context

Same 500-domain test as #339 (250 most-popular + 250 random domains, Apr–Jun 2026). Large sites carry sitemap indexes referencing hundreds of child sitemaps, each up to 50k URLs. 448,562 pages were enumerated across the test; the largest sites timed out at 1h primarily in the sitemap phase.

Proposed optimizations

• Streaming XML parsing (SAX, e.g. sax / saxes) instead of buffering + DOM-parsing each child sitemap — lower memory and faster time-to-first-URL on 50k-entry files.
• Guarantee gzip transfer (Accept-Encoding: gzip) with transparent decompression — sitemaps are frequently served as .xml.gz; large savings on the wire.
• HTTP keep-alive / connection reuse (shared agent) across the many child-sitemap requests to the same host.
• Concurrent child-sitemap fetching with a bounded pool (also enables the overlap in Sitemap Extractor: overlap sitemap download with crawling (enqueue URLs as they are parsed, don't block on full enumeration) #339).
• Dedupe child-sitemap URLs — indexes commonly list duplicates.
• Incremental skip via sitemap-index <lastmod>: skip child sitemaps unchanged since the previous run using If-Modified-Since / If-None-Match (conditional GET). Big win for re-crawls, and foundational for a future "crawl only new content" feature.
• Bound per-host concurrency to avoid anti-bot triggers.

Expected impact

Faster sitemap enumeration, lower memory footprint, fewer timeouts on large sites, and materially cheaper re-crawls.

---

🤖 Filed from a data-driven analysis with Claude Code, based on a 500-domain Sitemap Extractor test (Apr–Jun 2026).