feat(url): verify TLS with the OS and read embedded data from JavaScript shells

.github/distribution.json

@@ -52,26 +52,15 @@
     "linux": "Archive SHA-256 checksums; no detached publisher signature"
   },
   "release_features": [
-    "Add --render-js auto, the recommended mode for unknown pages: acquire statically and run JavaScript only on a JavaScript shell page; bare --render-js still means always. Pages that need browser APIs outside the bounded engine, such as crates.io, refuse at javascript_required.",
-    "Refuse captcha, bot challenge, login and JavaScript shell pages with a plain reason at stage challenge or javascript_required, instead of near-empty successes. Challenges are named, such as cloudflare, datadome or qrator, never solved.",
-    "Report http_status, redirect_count, challenge, charset, charset_source and render_js in URL receipts; brewdoc.receipt/2 changes only by adding keys.",
-    "Add --no-links for HTML: keep link text, omit destinations and disclose it in the receipt.",
-    "Decode mislabelled HTML charsets and undeclared Latin or Cyrillic HTML by fallback instead of refusing, disclosed as charset_fallback in local receipts.",
-    "Write links inside code and around images as standard Markdown links instead of [Link target] records; local HTML Markdown changes for these links."
+    "Convert the embedded data of a JavaScript shell page instead of refusing it: title, description, JSON-LD text and noscript text from the static HTML, when the body text reaches 200 characters. The capture is partial, snapshot kind embedded_data, with one javascript_required error. This reads pages such as an Airbnb listing, a SoundCloud track or a Bluesky profile; crates.io still refuses.",
+    "Name AWS WAF challenge pages as challenge aws-waf, such as IMDb or Binance pages answered with HTTP 202, instead of refusing them as javascript_required."
   ],
   "release_fixes": [
-    "Keep the HTTP status and the redirect reason in URL refusals instead of a generic stage message.",
-    "Fall back to the acquired HTML when the JavaScript heap runs out, the DOM snapshot check fails or a page runs more than 200 timer callbacks, instead of refusing.",
-    "Start no script or module fetch after the JavaScript soft window, so script-heavy pages stop sooner.",
-    "Run scripts served or typed with any WHATWG JavaScript MIME type, such as application/x-javascript.",
-    "Let a non-HTTP(S) sub-request fail alone and ignore a non-HTTP(S) base URL, instead of refusing the JavaScript capture.",
-    "Convert pages that create upper-case STYLE, TITLE or TEXTAREA elements during JavaScript capture.",
-    "Offer ALPN http/1.1 over HTTPS, so WordPress.com sites that answered 403 to brewdoc, such as acloserlisten.com, now load."
+    "Verify --url TLS with the operating system trust store through truststore 0.10.5, so pages whose root is only in that store, such as GlobalSign Root R46 on macOS, load instead of refusing at tls. The macOS and Windows verifiers may fetch missing intermediates themselves."
   ],
-  "release_pr": 67,
+  "release_pr": 70,
   "release_examples": [
-    "brewdoc --url \"https://hn.algolia.com/?q=rust\" --render-js auto --out search.md",
-    "brewdoc page.html --no-links --out page.md"
+    "brewdoc --url https://soundcloud.com/forss/flickermood --out track.md"
   ],
   "releases": {
     "0.2.0": {

FORMATS.md

@@ -1,8 +1,9 @@
 # Formats
 
 Supported extensions cover the profiles below, not every possible file. PDF needs a text layer;
-unreadable, malformed and unsupported input is refused by name. Base dependencies are pdfplumber
-and python-calamine; static HTML adds its conditional native parser. Other routes use those or stdlib.
+unreadable, malformed and unsupported input is refused by name. Base dependencies are pdfplumber,
+python-calamine and `truststore==0.10.5` for `--url` TLS; static HTML adds its conditional native
+parser. Other routes use those or stdlib.
 
 ## Supported
 
@@ -357,15 +358,18 @@ credentials or a non-default port refuses. The default clean body retains source
 Local files and static capture do not execute scripts.
 
 Only public destinations/default ports 80/443 pass. All DNS answers, peers and redirects are
-validated. TLS verifies certificates/hostnames against the Python runtime's default trust store,
-with no bundled CA list: a store that lacks the issuing root refuses at `tls`. HTTPS offers ALPN
-`http/1.1`, the protocol actually spoken; since 0.7.0 this lets WordPress.com edges that answered
-403 to a ClientHello without ALPN serve the page. GET excludes ambient credentials/cookies/proxy
-environment. At most five redirects; downgrades, unsupported schemes/private/special-use addresses
-refuse. Location values have tabs and line breaks removed and spaces, controls and other
-whitespace percent-encoded as UTF-8; raw non-UTF-8 bytes become `%XX`; a lone `%` or `\` refuses.
-The main request sends `Accept: text/html,application/xhtml+xml,*/*;q=0.8` and every request
-sends `Accept-Encoding: gzip`; identity and bounded gzip responses are accepted.
+validated. Since 0.7.1, TLS verifies certificates/hostnames with the operating system verifier
+through truststore 0.10.5: Security framework on macOS, CryptoAPI on Windows, OpenSSL default
+paths on Linux. There is no bundled CA list; a root only in the OS store, such as GlobalSign Root
+R46 on macOS, now passes, and a store that lacks the issuing root refuses at `tls`. The macOS and
+Windows verifiers may fetch missing intermediates themselves, outside brewdoc's address checks.
+HTTPS offers ALPN `http/1.1`, the protocol actually spoken; since 0.7.0 this lets WordPress.com
+edges that answered 403 to a ClientHello without ALPN serve the page. GET excludes ambient
+credentials/cookies/proxy environment. At most five redirects; downgrades, unsupported
+schemes/private/special-use addresses refuse. Location values have tabs and line breaks removed and
+spaces, controls and other whitespace percent-encoded as UTF-8; raw non-UTF-8 bytes become `%XX`; a
+lone `%` or `\` refuses. The main request sends `Accept: text/html,application/xhtml+xml,*/*;q=0.8`
+and every request sends `Accept-Encoding: gzip`; identity and bounded gzip responses are accepted.
 
 No request is re-sent. A host's validated addresses are tried in alternating families, starting
 with the first answer's, at most three attempts. Only a TCP connect failure moves to the next
@@ -399,31 +403,49 @@ fully. Runtimes without the HTML parser still acquire documents; an HTML respons
 Since 0.7.0, decoded URL HTML is checked before conversion. Captcha, bot challenge and login
 pages refuse at stage `challenge` with
 `URL captcha or bot challenge refused: <name> (HTTP <status>)` or `URL login page refused`;
-receipt `challenge` names cloudflare, ddos-guard, yandex-smartcaptcha, hcaptcha, recaptcha,
-datadome, qrator, generic or login. Evidence that decides alone: a `cf-mitigated: challenge`
-header, or a Cloudflare challenge script with a title holding "Just a moment" or "Attention
-required"; a final path `/showcaptcha` or a form posting to `/checkcaptcha`
-(yandex-smartcaptcha); a password field in a form with the exact title Login, Log in or Sign in,
-in any case. DataDome and QRATOR count only on a non-2xx answer: an `x-datadome` header or a
+receipt `challenge` names cloudflare, ddos-guard, aws-waf, yandex-smartcaptcha, hcaptcha,
+recaptcha, datadome, qrator, generic or login. Evidence that decides alone: a
+`cf-mitigated: challenge` header, or a Cloudflare challenge script with a title holding "Just a
+moment" or "Attention required"; a final path `/showcaptcha` or a form posting to `/checkcaptcha`
+(yandex-smartcaptcha); a password field in a form with the exact title Login, Log in or Sign in, in
+any case. DataDome and QRATOR count only on a non-2xx answer: an `x-datadome` header or a
 captcha-delivery.com reference (datadome); HTTP 439 or a `Server: QRATOR` header (qrator). Weak
-evidence counts only when the visible body text, without script, style, noscript and template,
-is under 1,000 characters: `Server: ddos-guard`, `/.well-known/ddos-guard/`, hCaptcha, reCAPTCHA
-or Turnstile hosts, and phrases such as "checking your browser", "please enable cookies" or
-"Проверяем браузер" (generic).
+evidence counts only when the visible body text, without script, style, noscript and template, is
+under 1,000 characters: `Server: ddos-guard`, `/.well-known/ddos-guard/`, `token.awswaf.com` or
+`gokuProps` (aws-waf, since 0.7.1; such as IMDb or Binance pages answered with HTTP 202), hCaptcha,
+reCAPTCHA or Turnstile hosts, and phrases such as "checking your browser", "please enable cookies"
+or "Проверяем браузер" (generic).
 
 A page with that little text and at least one script is a JavaScript shell when it holds an
 enable-JavaScript message, or an empty `#root`, `#app`, `#__next` or `#__nuxt` element and under
-200 visible characters. Shell evidence beats the hCaptcha, reCAPTCHA and Turnstile hosts: such a
-page is a shell, not a challenge; other weak evidence still names a challenge. Static acquisition
-refuses it at `javascript_required` with `URL page needs JavaScript; retry with --render-js auto`.
-After JS capture the same check runs on the captured DOM, and a remaining shell refuses with
-`URL page content was not produced by the bounded JavaScript engine`. A non-2xx HTML answer with
+200 visible characters. Shell evidence beats the AWS WAF SDK host `token.awswaf.com` and the
+hCaptcha, reCAPTCHA and Turnstile hosts: such a page is a shell, not a challenge; other weak
+evidence, including `gokuProps`, still names a challenge. Static acquisition refuses it at
+`javascript_required` with `URL page needs JavaScript; retry with --render-js auto`. After JS
+capture the same check runs on the captured DOM, and a remaining shell refuses with
+`URL page content was not produced by the bounded JavaScript engine`.
+
+Since 0.7.1, a page that is still a shell, statically or after JS capture, first tries its static
+HTML's embedded data. The title is `head title`, else `og:title`; the description is the
+`description` meta, else `og:description`. Body text comes in document order from JSON-LD and
+noscript. JSON-LD objects with `@type` give string headline, name, description, articleBody,
+text and reviewBody fields, HTML removed. Objects whose `@type`, a string or a list, names WebSite,
+Organization, BreadcrumbList, ListItem or SearchAction are skipped with their children; an invalid
+or too deeply nested JSON-LD block is ignored whole. Noscript gives the text of innermost h1-h4, p,
+li, dt, dd and blockquote blocks; a block holding "enable JavaScript", "JavaScript enabled",
+"включите JavaScript", "отключен JavaScript" or "JavaScript is disabled", in any case, drops.
+Repeated texts and copies of the title or description drop. With at least 200 body characters, the
+snapshot has the title as heading, then the description and body texts as paragraphs. The capture
+is partial, snapshot kind `embedded_data`, with one `javascript_required` error carrying the final
+URL; the exit code is 0. Otherwise the refusal above stands. In the 2026-10 pilot this read an
+Airbnb listing, a SoundCloud track and a Bluesky profile (static mode); crates.io and a dvnovosti
+story still refuse. Embedded data is a summary, not the rendered page. A non-2xx HTML answer with
 recognized challenge evidence refuses at `challenge`; otherwise, including a shell or an
-undecodable error page, it keeps `URL HTTP status refused: <status>` at `status`. The non-2xx
-check needs the HTML parser; outside CPython 3.12-3.14 every such answer refuses at `status`.
-0.6.0 recognized only titled Cloudflare and login pages; other captcha and shell pages answered
-with 200 converted as near-empty successes. These are fixed markers, not a general bot detector;
-brewdoc never solves a challenge.
+undecodable error page, it keeps `URL HTTP status refused: <status>` at `status`. The non-2xx check
+needs the HTML parser; outside CPython 3.12-3.14 every such answer refuses at `status`. 0.6.0
+recognized only titled Cloudflare and login pages; other captcha and shell pages answered with 200
+converted as near-empty successes. These are fixed markers, not a general bot detector; brewdoc
+never solves a challenge.
 
 --timeout is a positive finite waiting budget within clock range, default 30 seconds. One budget
 covers prerequisites/DNS/connections/TLS/redirects/body/scripts/resources/JS waiting. Ready content
@@ -535,15 +557,16 @@ URL receipts use brewdoc.receipt/2 with acquisition: redacted requested/final/ba
 snapshot identities, mode, capture status, effective timeout_seconds and explicit omissions.
 JS snapshot kind is javascript_dom, initial_html_timeout_fallback, initial_html_heap_fallback,
 initial_html_dom_fallback or initial_html_timer_fallback; the fallbacks are original HTML, capture
-status partial, and add their own name as an error category.
+status partial, and add their own name as an error category. A static or JS capture of a shell can
+instead have kind embedded_data, described above.
 
 Since 0.7.0, acquisition also always has these keys:
 
 | Key | Value |
 |---|---|
 | `http_status` | Final main-document HTTP status, including 200; on a refusal the last received status; null without a response |
 | `redirect_count` | Main-document redirects followed, 0-5 |
-| `challenge` | cloudflare, ddos-guard, yandex-smartcaptcha, hcaptcha, recaptcha, datadome, qrator, generic or login on a `challenge` refusal; else null |
+| `challenge` | cloudflare, ddos-guard, aws-waf, yandex-smartcaptcha, hcaptcha, recaptcha, datadome, qrator, generic or login on a `challenge` refusal; else null |
 | `charset`, `charset_source` | HTML decoding charset (WHATWG name) and its source: bom, header, meta, default or fallback; null for documents and before decoding |
 | `render_js` | `auto` or `always` as requested; null for static |