feat(url): read JavaScript pages and refuse challenges honestly

.github/distribution.json

@@ -52,27 +52,26 @@
     "linux": "Archive SHA-256 checksums; no detached publisher signature"
   },
   "release_features": [
-    "Write clean Markdown by default: minimal escaping, images as ![alt](src), no form-control or list-item records and no page landmark chrome.",
-    "Add opt-in --body-only output.",
-    "Accept PDF, DOCX, XLSX and other supported documents through --url.",
-    "Decode legacy HTML charsets such as koi8-r and windows-1251, and request gzip responses.",
-    "Finish JavaScript capture when the page becomes quiet."
+    "Add --render-js auto, the recommended mode for unknown pages: acquire statically and run JavaScript only on a JavaScript shell page; bare --render-js still means always. Pages that need browser APIs outside the bounded engine, such as crates.io, refuse at javascript_required.",
+    "Refuse captcha, bot challenge, login and JavaScript shell pages with a plain reason at stage challenge or javascript_required, instead of near-empty successes. Challenges are named, such as cloudflare, datadome or qrator, never solved.",
+    "Report http_status, redirect_count, challenge, charset, charset_source and render_js in URL receipts; brewdoc.receipt/2 changes only by adding keys.",
+    "Add --no-links for HTML: keep link text, omit destinations and disclose it in the receipt.",
+    "Decode mislabelled HTML charsets and undeclared Latin or Cyrillic HTML by fallback instead of refusing, disclosed as charset_fallback in local receipts.",
+    "Write links inside code and around images as standard Markdown links instead of [Link target] records; local HTML Markdown changes for these links."
   ],
   "release_fixes": [
-    "Fix PDF word spacing in tightly kerned text, such as LaTeX output.",
-    "Keep rotated watermark text out of PDF prose.",
-    "Decode entities in HTML titles.",
-    "Fall back to the full page when --article finds no article, instead of refusing.",
-    "Follow URL redirects whose location contains raw spaces.",
-    "Report the real refusal stage and resource_url.",
-    "Try the next address before any request bytes are sent.",
-    "Keep a JavaScript capture when one sub-request fails."
+    "Keep the HTTP status and the redirect reason in URL refusals instead of a generic stage message.",
+    "Fall back to the acquired HTML when the JavaScript heap runs out, the DOM snapshot check fails or a page runs more than 200 timer callbacks, instead of refusing.",
+    "Start no script or module fetch after the JavaScript soft window, so script-heavy pages stop sooner.",
+    "Run scripts served or typed with any WHATWG JavaScript MIME type, such as application/x-javascript.",
+    "Let a non-HTTP(S) sub-request fail alone and ignore a non-HTTP(S) base URL, instead of refusing the JavaScript capture.",
+    "Convert pages that create upper-case STYLE, TITLE or TEXTAREA elements during JavaScript capture.",
+    "Offer ALPN http/1.1 over HTTPS, so WordPress.com sites that answered 403 to brewdoc, such as acloserlisten.com, now load."
   ],
-  "release_pr": 65,
+  "release_pr": 67,
   "release_examples": [
-    "brewdoc --body-only paper.pdf --out paper.md",
-    "brewdoc --url https://arxiv.org/pdf/1706.03762 --out attention.md",
-    "brewdoc koi8r-page.html --out page.md"
+    "brewdoc --url \"https://hn.algolia.com/?q=rust\" --render-js auto --out search.md",
+    "brewdoc page.html --no-links --out page.md"
   ],
   "releases": {
     "0.2.0": {

INSTALL.md

@@ -1,14 +1,15 @@
 # Installation
 
-These examples select 0.6.0. Use a channel listed as available in
-[that release](https://github.com/kochetkov-ma/brewdoc/releases/tag/v0.6.0).
+These examples select 0.7.0. Use a channel listed as available in
+[that release](https://github.com/kochetkov-ma/brewdoc/releases/tag/v0.7.0).
 X.Y.Z marks an exact published update version to replace.
 After installation, local conversion and self-check need no network or runtime downloads.
 
 Article mode is introduced in 0.4.0. Use local `brewdoc page.html --article`, or combine
 `--article` with `--url` and optional `--render-js`. See the [article contract](FORMATS.md#article-selection).
 Version 0.6.0 adds `--body-only` and document URLs, for example
 `brewdoc --url https://arxiv.org/pdf/1706.03762 --body-only --out attention.md`.
+Version 0.7.0 adds `--render-js auto` and HTML `--no-links`; see the [URL policy](FORMATS.md#url-acquisition).
 
 ## Python prerequisites
 
@@ -31,7 +32,7 @@ Linux/macOS:
 
 ```sh
 python3.12 -m venv .venv
-.venv/bin/python -m pip install "brewdoc==0.6.0"
+.venv/bin/python -m pip install "brewdoc==0.7.0"
 .venv/bin/python -c "from importlib.metadata import version; print(version('brewdoc'))"
 .venv/bin/brewdoc --self-check
 .venv/bin/brewdoc file.pdf --out file.md
@@ -41,7 +42,7 @@ Windows PowerShell:
 
 ```powershell
 py -3.12 -m venv .venv
-.\.venv\Scripts\python.exe -m pip install "brewdoc==0.6.0"
+.\.venv\Scripts\python.exe -m pip install "brewdoc==0.7.0"
 .\.venv\Scripts\python.exe -c "from importlib.metadata import version; print(version('brewdoc'))"
 .\.venv\Scripts\brewdoc.exe --self-check
 .\.venv\Scripts\brewdoc.exe file.pdf --out file.md
@@ -52,16 +53,16 @@ Update with the same environment's Python and an exact published version:
 Remove with `.venv/bin/python -m pip uninstall brewdoc` on Unix or
 `.\.venv\Scripts\python.exe -m pip uninstall brewdoc` in PowerShell. Dependencies
 remain; remove the dedicated environment when no longer needed. Keep inputs/output outside it.
-For a container recipe using installed Python: `RUN pip install brewdoc==0.6.0`.
+For a container recipe using installed Python: `RUN pip install brewdoc==0.7.0`.
 
 ### uv
 
 Requires installed [uv](https://docs.astral.sh/uv/getting-started/installation/);
 uv can obtain Python when needed. Temporary invocations:
 
 ```sh
-uvx --python 3.12 --from "brewdoc==0.6.0" brewdoc --self-check
-uvx --python 3.12 --from "brewdoc==0.6.0" brewdoc file.pdf --out file.md
+uvx --python 3.12 --from "brewdoc==0.7.0" brewdoc --self-check
+uvx --python 3.12 --from "brewdoc==0.7.0" brewdoc file.pdf --out file.md
 ```
 
 uvx uses a cached environment, creates no persistent PATH command and needs no
@@ -71,7 +72,7 @@ uninstall. Choose another exact version on the next invocation to update.
 For a persistent tool:
 
 ```sh
-uv tool install --python 3.12 "brewdoc==0.6.0"
+uv tool install --python 3.12 "brewdoc==0.7.0"
 uv tool update-shell
 ```
 
@@ -85,7 +86,7 @@ Replace X.Y.Z first. Remove with `uv tool uninstall brewdoc`.
 Requires installed [pipx](https://pipx.pypa.io/stable/how-to/install-pipx.html) and Python 3.12.
 
 ```sh
-pipx install --python 3.12 "brewdoc==0.6.0"
+pipx install --python 3.12 "brewdoc==0.7.0"
 pipx ensurepath
 ```
 
@@ -108,18 +109,18 @@ establish older OS support. Linux musl/ARM64 and macOS Intel are outside this se
 Linux x64/glibc:
 
 ```sh
-curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.6.0/brewdoc-0.6.0-linux-x64.tar.gz"
-curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.6.0/SHA256SUMS"
-sha256sum brewdoc-0.6.0-linux-x64.tar.gz
-awk '$2 == "brewdoc-0.6.0-linux-x64.tar.gz" { print }' SHA256SUMS
+curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.7.0/brewdoc-0.7.0-linux-x64.tar.gz"
+curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.7.0/SHA256SUMS"
+sha256sum brewdoc-0.7.0-linux-x64.tar.gz
+awk '$2 == "brewdoc-0.7.0-linux-x64.tar.gz" { print }' SHA256SUMS
 ```
 
 Continue only when the two SHA-256 values match:
 
 ```sh
-mkdir brewdoc-0.6.0-linux-x64
-tar -xzf brewdoc-0.6.0-linux-x64.tar.gz -C brewdoc-0.6.0-linux-x64
-cd brewdoc-0.6.0-linux-x64
+mkdir brewdoc-0.7.0-linux-x64
+tar -xzf brewdoc-0.7.0-linux-x64.tar.gz -C brewdoc-0.7.0-linux-x64
+cd brewdoc-0.7.0-linux-x64
 cat brewdoc/version.json
 ./brewdoc/brewdoc --self-check
 ./brewdoc/brewdoc /absolute/path/file.pdf --out file.md
@@ -129,18 +130,18 @@ export PATH="$PWD/brewdoc:$PATH"
 macOS ARM64:
 
 ```sh
-curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.6.0/brewdoc-0.6.0-macos-arm64.tar.gz"
-curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.6.0/SHA256SUMS"
-shasum -a 256 brewdoc-0.6.0-macos-arm64.tar.gz
-awk '$2 == "brewdoc-0.6.0-macos-arm64.tar.gz" { print }' SHA256SUMS
+curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.7.0/brewdoc-0.7.0-macos-arm64.tar.gz"
+curl -fLO "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.7.0/SHA256SUMS"
+shasum -a 256 brewdoc-0.7.0-macos-arm64.tar.gz
+awk '$2 == "brewdoc-0.7.0-macos-arm64.tar.gz" { print }' SHA256SUMS
 ```
 
 Continue only when the two SHA-256 values match:
 
 ```sh
-mkdir brewdoc-0.6.0-macos-arm64
-tar -xzf brewdoc-0.6.0-macos-arm64.tar.gz -C brewdoc-0.6.0-macos-arm64
-cd brewdoc-0.6.0-macos-arm64
+mkdir brewdoc-0.7.0-macos-arm64
+tar -xzf brewdoc-0.7.0-macos-arm64.tar.gz -C brewdoc-0.7.0-macos-arm64
+cd brewdoc-0.7.0-macos-arm64
 cat brewdoc/version.json
 ./brewdoc/brewdoc --self-check
 ./brewdoc/brewdoc /absolute/path/file.pdf --out file.md
@@ -150,17 +151,17 @@ export PATH="$PWD/brewdoc:$PATH"
 Windows PowerShell:
 
 ```powershell
-Invoke-WebRequest "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.6.0/brewdoc-0.6.0-windows-x64.zip" -OutFile "brewdoc-0.6.0-windows-x64.zip"
-Invoke-WebRequest "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.6.0/SHA256SUMS" -OutFile SHA256SUMS
-(Get-FileHash "brewdoc-0.6.0-windows-x64.zip" -Algorithm SHA256).Hash.ToLower()
-Select-String -Path SHA256SUMS -SimpleMatch "  brewdoc-0.6.0-windows-x64.zip"
+Invoke-WebRequest "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.7.0/brewdoc-0.7.0-windows-x64.zip" -OutFile "brewdoc-0.7.0-windows-x64.zip"
+Invoke-WebRequest "https://github.com/kochetkov-ma/brewdoc/releases/download/v0.7.0/SHA256SUMS" -OutFile SHA256SUMS
+(Get-FileHash "brewdoc-0.7.0-windows-x64.zip" -Algorithm SHA256).Hash.ToLower()
+Select-String -Path SHA256SUMS -SimpleMatch "  brewdoc-0.7.0-windows-x64.zip"
 ```
 
 Continue only when the SHA-256 values match:
 
 ```powershell
-Expand-Archive "brewdoc-0.6.0-windows-x64.zip" -DestinationPath "brewdoc-0.6.0-windows-x64"
-Set-Location "brewdoc-0.6.0-windows-x64"
+Expand-Archive "brewdoc-0.7.0-windows-x64.zip" -DestinationPath "brewdoc-0.7.0-windows-x64"
+Set-Location "brewdoc-0.7.0-windows-x64"
 Get-Content .\brewdoc\version.json
 .\brewdoc\brewdoc.exe --self-check
 .\brewdoc\brewdoc.exe "C:\path\file.pdf" --out file.md
@@ -204,16 +205,20 @@ the managed keg. Other installations on PATH can still provide a `brewdoc` comma
 
 ## Optional JavaScript capture
 
-Python installs use `brewdoc[render-js]==0.6.0` instead of the base package in the
+Python installs use `brewdoc[render-js]==0.7.0` instead of the base package in the
 chosen install command. It includes QuickJS-ng 0.17.0.1 and bundled LinkeDOM 0.18.13.
 JS needs CPython 3.12-3.14 on macOS ARM64 or 64-bit glibc Linux x86_64/aarch64.
 Windows, macOS Intel, musl and 32-bit systems refuse before requests.
 Standalone bundles include admitted JS components on Linux x64 and macOS ARM64.
 
 ```sh
 brewdoc --url https://example.com --render-js --timeout 30 --out page.md
+brewdoc --url https://example.com --render-js auto --out page.md
 ```
 
+Bare `--render-js` means `always`; `auto` runs JS only when the static page is a JavaScript shell.
+Prefer `auto` for unknown pages: `always` runs JS on every page and can hit an engine
+quota on a page that static mode reads.
 No Node/browser/driver/engine downloads occur during conversion. Explicit --url
 fetches the requested page or document and page resources under the [URL policy](FORMATS.md#url-acquisition).
 From source, use `uv sync --locked --extra render-js` and
@@ -224,12 +229,12 @@ From source, use `uv sync --locked --extra render-js` and
 The npm channel targets Linux x64/glibc and macOS ARM64 only. Windows uses the
 GitHub native archive or Python installation above; there is no Windows npm package.
 
-Use npm/npx only when listed as available in [the release](https://github.com/kochetkov-ma/brewdoc/releases/tag/v0.6.0).
+Use npm/npx only when listed as available in [the release](https://github.com/kochetkov-ma/brewdoc/releases/tag/v0.7.0).
 Tested launcher tooling is Node 24.21.0/npm 12.2.0.
 Keep optional platform packages enabled. Missing packages/unsupported targets refuse.
 
 ```sh
-npm install --global brewdoc@0.6.0
+npm install --global brewdoc@0.7.0
 npm list --global brewdoc --depth=0
 brewdoc --self-check
 brewdoc file.pdf --out file.md
@@ -240,8 +245,8 @@ for another exact published version; uninstall with `npm uninstall --global brew
 Temporary cached invocation:
 
 ```sh
-npx --yes --package brewdoc@0.6.0 brewdoc --self-check
-npx --yes --package brewdoc@0.6.0 brewdoc file.pdf --out file.md
+npx --yes --package brewdoc@0.7.0 brewdoc --self-check
+npx --yes --package brewdoc@0.7.0 brewdoc file.pdf --out file.md
 ```
 
 npx creates no persistent global tool and needs no tool uninstall. Choose another