[CI] External link checking improvements for 24H2-25H1

[chalin]

Successor to #2429. Looking forward, the plan is to:

• Resolve Don't update old blogs, and possibly warn readers #3493 - show an "obsolete" banner for old blog pages, in particular indicating that links might no longer be valid
• Adjust our external link checking to ignore obsolete blog pages
  • Blog section for 2025 #5874 goes the route of ignoring internal and external links in obsolete blog pages
  • Maybe we still want to do internal link checking for the obsolete blog pages too? I don't see an way to do this efficiently, so maybe the answer is "no" for this.
• Set up a cron job to refresh the refcache's oldest X entries every Y days, and report an issue and/or submit a PR with the cache updates - CI: refcache update policy and cron-job proposal #2554
• [infra] Support checking of crates.io URLs #6175

Here are items carried over from #2429:

• Clean up links that shouldn't be external:
  • Are Docsy-generated breadcrumb links still external? Resolved via Change breadcrumb and print links to RelPermalink google/docsy#1699
• Check more GitHub links
  • [CI] Check links with references into GitHub #5951
• Normalize links to avoid checking a(n external) link (to the same destination) more than once, and creating duplicate entries in the ref cache:
  • external links: consistently end with a slash or not (for domain-only URLs).
  • external links: use apex domain (w/ or w/o www), as appropriate

Related improvements:

• [CI] Fix link checking for all builds #5551

Related issues & blockers:

• Query parameter gets overly escaped when saved to the refcache wjdp/htmltest#239