feat: add web_search and web_fetch tools

.gitignore

@@ -30,3 +30,4 @@ pnpm-debug.log*
 .tmp/
 tmp/
 *.tmp
+PULL_REQUEST.md

README.md

@@ -19,6 +19,19 @@ This is not a standalone browser testing framework and does not host a daemon. I
 
 中文文档: [README.zh-CN.md](./README.zh-CN.md)
 
+## Tools
+
+### `browser_execute`
+Run arbitrary JavaScript in a visible Chrome browser via CDP. Supports console capture, screenshot collection, and workspace imports.
+
+### `web_search`
+Search Google using a visible Chrome browser via CDP. Extracts up to 20 visible result links along with a text snapshot of the search results page as structured markdown. Handles Google consent dialogs automatically.
+
+### `web_fetch`
+Fetch and extract page content from any URL using a visible Chrome browser. Dynamically scrolls to trigger lazy loading, then extracts structured markdown from semantic HTML elements (headings, paragraphs, lists, code blocks, blockquotes). Truncates content at 900KB.
+
+All three tools share the same Chrome profile for session continuity and can use `profileDir` to reuse an existing Chrome session.
+
 ## Quick Start
 
 ### 1. Install the extension
@@ -45,7 +58,10 @@ Pi will connect to an authorized Chromium browser, drive the page, inspect the r
 
 ## What it gives Pi
 
-- `browser_execute`: Pi-callable tool name.
+- `browser_execute`: Pi-callable tool for running arbitrary JavaScript in the browser.
+- `web_search`: Pi-callable tool for Google search via a visible Chrome browser.
+- `web_fetch`: Pi-callable tool for fetching and extracting page content as structured markdown.
+- `session`: persistent CDP session; multiple calls in the same Pi session reuse browser state.
 - `session`: persistent CDP session; multiple calls in the same Pi session reuse browser state.
 - `console`: captures `log`, `error`, `warn`, `info`, and `debug` output and streams it back in the tool result.
 - Screenshot collection: successful `Page.captureScreenshot` calls are automatically converted into Pi image content.
@@ -81,6 +97,28 @@ Do not use this for:
 - Pure unit testing; Playwright or Vitest is more direct.
 - Untrusted pages or untrusted CDP endpoints. CDP can control the connected browser, so only connect to browsers you authorize.
 
+## Auto-launching Chrome
+
+When you pass a `profileDir` to `browser_execute`, the extension can automatically launch Chrome if no browser with remote debugging is already connected.
+
+The tool accepts a `profileDir` parameter:
+
+```javascript
+await session.connect({
+  profileDir: '/home/user/.ds4/browser',
+  launchBrowser: true,
+});
+```
+
+The extension will:
+1. Create the profile directory if it doesn't exist
+2. Launch Chrome with `--remote-debugging-port=0 --user-data-dir=<profileDir> --no-first-run --no-default-browser-check`
+3. Wait for Chrome to write `DevToolsActivePort` and connect
+
+If the directory already has a Chrome instance running with remote debugging, it reuses that instance.
+
+To customize the Chrome executable path, set `BROWSER_PATH` or `CHROME_BIN` environment variable.
+
 ## Configuration
 
 Environment variables:
@@ -103,7 +141,7 @@ npm run typecheck
 npm test
 ```
 
-Current tests cover session reuse/isolation, workspace imports, console streaming, timeout handling, screenshot collection, CDP target filtering, active `sessionId` routing, and Pi image content conversion.
+Current tests cover session reuse/isolation, workspace imports, console streaming, timeout handling, screenshot collection, CDP target filtering, active `sessionId` routing, Pi image content conversion, web search and fetch logic, Google consent handling, dynamic scrolling, and Pi extension adapter integration.
 
 ## Acknowledgements
 

README.zh-CN.md

@@ -43,9 +43,24 @@ pi install .
 
 Pi 会连接已授权的 Chromium 浏览器，打开页面、读取结果，并把截图附在回复里。
 
+## 工具
+
+### `browser_execute`
+通过 CDP 在可见的 Chrome 浏览器中执行任意 JavaScript。支持 console 捕获、截图收集和 workspace 导入。
+
+### `web_search`
+通过可见的 Chrome 浏览器搜索 Google。提取最多 20 条可见结果链接及搜索结果页面的文本摘要，以结构化 markdown 返回。自动处理 Google 同意弹窗。
+
+### `web_fetch`
+通过可见的 Chrome 浏览器抓取任意 URL 的页面内容。自动滚动触发懒加载，然后从语义化 HTML 元素（标题、段落、列表、代码块、引用块）中提取结构化 markdown。900KB 截断。
+
+三个工具共享同一个 Chrome 配置用于 session 连续性，都可以使用 `profileDir` 复用已有的 Chrome session。
+
 ## 给 Pi 提供什么
 
 - `browser_execute`：Pi 可调用的工具名。
+- `web_search`：通过可见 Chrome 浏览器搜索 Google 的工具。
+- `web_fetch`：抓取页面内容并提取结构化 markdown 的工具。
 - `session`：持久 CDP Session，同一个 Pi session 内多次调用会复用状态。
 - `console`：捕获 `log/error/warn/info/debug`，作为工具输出流式返回。
 - 截图收集：成功的 `Page.captureScreenshot` 会自动转成 Pi image content。
@@ -103,7 +118,7 @@ npm run typecheck
 npm test
 ```
 
-当前测试覆盖包括：session 复用/隔离、workspace import、console streaming、timeout、screenshot 收集、CDP target 过滤、active sessionId 路由、Pi image content 转换。
+当前测试覆盖包括：session 复用/隔离、workspace import、console streaming、timeout、screenshot 收集、CDP target 过滤、active sessionId 路由、Pi image content 转换、web search 和 fetch 逻辑、Google 同意弹窗处理、动态滚动、Pi extension adapter 集成。
 
 ## 致谢
 

extensions/browser-execute-web.ts

@@ -0,0 +1,188 @@
+import path from "node:path";
+import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
+import type { Component } from "@earendil-works/pi-tui";
+import { Type } from "typebox";
+import { executeBrowserCode, type BrowserExecuteParameters } from "../src/browser-execute.js";
+import { webSearch, webFetch, type WebSearchOptions, type WebFetchOptions } from "../src/web-fetch.js";
+
+// ============================================================================
+// Web Search Tool
+// ============================================================================
+
+const WebSearchParams = Type.Object({
+  query: Type.String({
+    description: "The search query to look up on Google. Examples: 'typescript async await', 'latest react patterns 2024', 'pi coding agent documentation'.",
+  }),
+  profileDir: Type.Optional(
+    Type.String({
+      description: "Chrome user-data directory. If provided, connects to (or launches) Chrome with --user-data-dir=profileDir. E.g. /home/user/.ds4/browser",
+    }),
+  ),
+});
+
+// ============================================================================
+// Web Fetch Tool
+// ============================================================================
+
+const WebFetchParams = Type.Object({
+  url: Type.String({
+    description: "The full URL to fetch and extract. Examples: 'https://example.com', 'https://github.com/citrolabs/pi-browser-cdp-extension'.",
+  }),
+  profileDir: Type.Optional(
+    Type.String({
+      description: "Chrome user-data directory. If provided, connects to (or launches) Chrome with --user-data-dir=profileDir. E.g. /home/user/.ds4/browser",
+    }),
+  ),
+});
+
+// ============================================================================
+// Extension: Web Search + Web Fetch
+// ============================================================================
+
+function workspaceDirOf(cwd: string): string {
+  return path.join(cwd, ".pi", "browser-execute-workspace");
+}
+
+export default function browserExecuteWebExtension(pi: ExtensionAPI) {
+  // --- web_search tool ---
+  pi.registerTool({
+    name: "web_search",
+    label: "Web Search",
+    description: `Search Google and return structured results as markdown. Always prefer this tool over curl for web searches.
+
+This tool queries Google Search and returns up to 20 visible result links with their text and URLs, plus a text snapshot of the results page.
+
+Use this tool instead of curl when:
+- The user asks to search the web or find information
+- You need to see Google search results in a structured format
+- You want to discover URLs before reading them with web_fetch
+
+Returns structured markdown with links and a text snapshot of results.`,
+    promptSnippet: "Search Google and return structured results as markdown. Use instead of curl for web searches.",
+    promptGuidelines: [
+      "Use web_search (NOT curl) whenever the user asks to search the web or find information online.",
+      "Pass a descriptive, specific query for best results.",
+      "Use web_fetch to read the full content of URLs discovered through search.",
+      "web_search and web_fetch share the same browser session for continuity.",
+    ],
+    parameters: WebSearchParams,
+    renderCall: (args: { query: string }, ..._rest: any[]) => {
+      // eslint-disable-next-line @typescript-eslint/no-require-imports
+      const { Text } = require("@earendil-works/pi-tui");
+      const text = new Text(0, 0);
+      const q = args.query?.trim() || "...";
+      text.setText(`Web search: ${q}`);
+      return text;
+    },
+    async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
+      const options: WebSearchOptions = {
+        query: params.query,
+        profileDir: params.profileDir,
+        onProgress: (status) => {
+          _onUpdate?.({
+            content: [{ type: "text" as const, text: status }],
+            details: { status },
+          });
+        },
+      };
+
+      const result = await webSearch(options);
+
+      if ("error" in result) {
+        return {
+          content: [
+            { type: "text" as const, text: `Error: web_search failed:\n${result.message}` },
+          ],
+          details: {
+            query: options.query,
+            error: result.message,
+          },
+        };
+      }
+
+      return {
+        content: [
+          { type: "text" as const, text: result.markdown },
+        ],
+        details: {
+          query: options.query,
+          searchUrl: result.searchUrl,
+          linkCount: result.linkCount,
+        },
+      };
+    },
+  });
+
+  // --- web_fetch tool ---
+  pi.registerTool({
+    name: "web_fetch",
+    label: "Web Fetch",
+    description: `Fetch a URL and extract its content as structured markdown. Always prefer this tool over curl for fetching web pages.
+
+This tool fetches any URL and returns structured markdown including the page title, content from semantic HTML elements (headings, paragraphs, lists, code, blockquotes), and visible links.
+
+Use this tool instead of curl when:
+- The user asks to read, fetch, or get the content of a webpage
+- You need structured markdown output instead of raw HTML
+- The page uses JavaScript rendering or lazy loading
+- You want to avoid parsing HTML yourself
+
+Returns clean structured markdown, not raw HTML or curl output.`,
+    promptSnippet: "Fetch any URL and extract content as structured markdown. Use instead of curl for web pages.",
+    promptGuidelines: [
+      "Use web_fetch (NOT curl) whenever the user asks to read, fetch, or get the content of a webpage.",
+      "Pass the complete URL including the protocol (https://).",
+      "For dynamic pages with lazy loading, this tool automatically scrolls to load more content.",
+      "web_fetch and web_search share the same browser session for continuity.",
+    ],
+    parameters: WebFetchParams,
+    renderCall: (args: { url: string }, ..._rest: any[]) => {
+      // eslint-disable-next-line @typescript-eslint/no-require-imports
+      const { Text } = require("@earendil-works/pi-tui");
+      const text = new Text(0, 0);
+      const u = args.url?.trim() || "...";
+      text.setText(`Web fetch: ${u}`);
+      return text;
+    },
+    async execute(_toolCallId, params, _signal, _onUpdate, ctx) {
+      const options: WebFetchOptions = {
+        url: params.url,
+        profileDir: params.profileDir,
+        onProgress: (status) => {
+          _onUpdate?.({
+            content: [{ type: "text" as const, text: status }],
+            details: { status },
+          });
+        },
+      };
+
+      const result = await webFetch(options);
+
+      if ("error" in result) {
+        return {
+          content: [
+            { type: "text" as const, text: `Error: web_fetch failed:\n${result.message}` },
+          ],
+          details: {
+            url: options.url,
+            error: result.message,
+          },
+        };
+      }
+
+      return {
+        content: [
+          { type: "text" as const, text: result.markdown },
+        ],
+        details: {
+          url: options.url,
+          finalUrl: result.finalUrl,
+          title: result.title,
+          linkCount: result.linkCount,
+          lineCount: result.lineCount,
+          scrolled: result.scrolled,
+        },
+      };
+    },
+  });
+}