Skip to content

Commit 8c0972c

Browse files
Richard Wheelerclaude
authored andcommitted
feat(codegen): mobile capture (Layer 3), verified live; flip codegen to shipped (v5.1.0)
Add the native-mobile codegen capture layer and wire it into the CLI, then mark the codegen recorder shipped across the docs. mobile/codegen_capture.v + _test.v: - Pure, offline-tested core: quote-aware accessibility-tree XML parser (parse_nodes), smallest-area hit-test (hit_test_index), getevent stream parser (GetEventParser), coordinate scaling (scale_point), wm-size / getevent -lp parsers, and per-platform selector synthesis matching the mobile/selectors.v priority (test-id -> resource-id/label -> text -> role -> positional xpath). - Live MobileRecorder (record_tap_at / record_assert_at / record_text / flush_pending_edit / emit) reusing page_source / tap_at / element_attribute, plus adb helpers (screen_size, touch_axis_max, start_touch_stream). tools/codegen.v: - Wire `android` (passive getevent loop, Enter to stop) and `ios` (assisted tap/text/assert/done REPL), replacing the prior "not available" stub. Verified: 24 offline tests; generated mobile program type-checks; and a live round-trip on the Android emulator (Pixel AVD, API 34; UiAutomator2 v10.2.1) re-resolved 5/5 synthesized selectors against the running tree. Docs: flip codegen Planned->shipped in COMPARISON.md, COMPARISON_WITH_PLAYWRIGHT.md, docs/preview/comparison-table.html, and DocsPage.jsx; add CHANGELOG 5.1.0; update README; bump v.mod to 5.1.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 73eb440 commit 8c0972c

11 files changed

Lines changed: 1322 additions & 48 deletions

File tree

CHANGELOG.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,43 @@
11
# WebDriver V Library - Changelog
22

3+
## [5.1.0] - 2026-06-03 - Codegen / session recorder (web + mobile)
4+
5+
A **codegen recorder** that turns a live session into runnable vebidor
6+
source. Both front-ends feed one capture-agnostic action model and the same
7+
emitters, so the output goes through the semantic `get_by_*` engines a human
8+
would hand-write — refactor-resistant locators, not brittle coordinates.
9+
10+
- **Web capture** (over WebDriver-BiDi) was already shipped and verified live
11+
on Edge (capture → emit → compile → replay round-trip).
12+
- **Mobile capture (Layer 3)** is new this release and **verified live on the
13+
Android emulator** (Pixel AVD, API 34; UiAutomator2 v10.2.1): an on-device
14+
round-trip re-resolved 5/5 synthesized selectors against the live tree.
15+
16+
### ✨ Added
17+
18+
- **Mobile capture** ([`mobile/codegen_capture.v`](mobile/codegen_capture.v)) —
19+
`MobileRecorder` (`record_tap_at` / `record_assert_at` / `record_text` /
20+
`flush_pending_edit` / `emit`). Android is passive: `start_touch_stream()`
21+
spawns `adb shell getevent -lt`, `GetEventParser` decodes finger-ups, and
22+
`scale_point` maps raw touch coords → pixels using `screen_size()` and
23+
`touch_axis_max()`. On each tap, `page_source()` is parsed (`parse_nodes`),
24+
the smallest containing leaf is hit-tested (`hit_test_index`), and
25+
`synth_locator` builds a `LocatorSpec` via the `mobile/selectors.v` priority
26+
(test-id → resource-id/label → text → role → positional xpath). iOS uses an
27+
assisted REPL (no passive touch stream on XCUITest).
28+
- **CLI** ([`tools/codegen.v`](tools/codegen.v)) — `android` (passive,
29+
`getevent`-driven) and `ios` (`tap` / `text` / `assert` / `done` REPL)
30+
subcommands, replacing the prior "not available" stub.
31+
- Offline test suite ([`mobile/codegen_capture_test.v`](mobile/codegen_capture_test.v))
32+
covering the parsers, coordinate scaling, hit-test, and per-platform
33+
synthesis branches.
34+
35+
### 📝 Notes
36+
37+
- iOS synthesis is offline-tested but not yet exercised on a physical device.
38+
- Real video capture and a binary trace viewer remain the only Playwright-only
39+
conveniences; codegen is no longer a gap.
40+
341
## [5.0.0] - 2026-06-01 - Native mobile (iOS + Android)
442

543
A new **`vebidor.mobile`** module: native automation of real apps on iOS

CODEGEN_HANDOFF.md

Lines changed: 26 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
11
# Codegen / session recorder — handoff
22

3-
Status as of this commit. Web codegen is built and **verified live on Edge**;
4-
mobile capture is designed but not yet built (needs an Android emulator).
3+
Status as of this commit. Web codegen is built and **verified live on Edge**.
4+
Mobile capture (Layer 3) is now built and **verified live on the Android
5+
emulator** — 24 offline tests plus an on-device round-trip (5/5 synthesized
6+
selectors re-resolved against the live UiAutomator2 tree). iOS uses the same
7+
core via an assisted REPL (no passive touch stream); its synthesis is
8+
offline-tested but not yet exercised on a physical device.
59

610
## What's done (this commit)
711

@@ -25,19 +29,26 @@ button/textbox/checkbox/link/Enter flow re-ran headless with exit 0).
2529

2630
## What's left
2731

28-
### Mobile capture (Layer 3) — needs adb + a running Android emulator
29-
Design (from the approved plan):
30-
- **Android passive tap-capture (primary):** stream `adb shell getevent -lt`,
31-
parse `ABS_MT_POSITION_X/Y` + `BTN_TOUCH`, scale touch-device coords → screen
32-
using ranges from `getevent -p`. On touch-up: snapshot `page_source()` (XML a11y
33-
tree w/ bounds), hit-test the topmost leaf containing the point, synthesize a
34-
`LocatorSpec` via the `mobile/selectors.v` priority, record a tap. Text entry:
35-
after a tap on an `EditText`, poll the node's `text` attr → emit a `fill`.
36-
- **Cross-platform assisted mode (iOS's only path):** operator points at a target
37-
in a REPL; recorder hit-tests current `page_source()`, performs via existing
38-
`tap`/`fill`/gestures, records. **iOS has no passive touch stream** → assisted only.
39-
- New file: `mobile/codegen_capture.v` (+ `_test.v`); wire `android`/`ios` into
40-
`tools/codegen.v`. Verify on the emulator (the verified-live platform).
32+
### Mobile capture (Layer 3) — DONE (verified live on the Android emulator)
33+
Built in [`mobile/codegen_capture.v`](mobile/codegen_capture.v) (+ offline tests
34+
in [`mobile/codegen_capture_test.v`](mobile/codegen_capture_test.v)); `android`
35+
and `ios` are wired into [`tools/codegen.v`](tools/codegen.v).
36+
- **Android passive tap-capture (primary):** `MobileSession.start_touch_stream()`
37+
spawns `adb shell getevent -lt`; `GetEventParser` parses `ABS_MT_POSITION_X/Y`
38+
+ `ABS_MT_TRACKING_ID ffffffff` / `BTN_TOUCH UP`; `scale_point` maps raw coords
39+
→ screen px using `screen_size()` (`wm size`) and `touch_axis_max()`
40+
(`getevent -lp`). On finger-up: `record_tap_at` snapshots `page_source()`,
41+
`parse_nodes` + `hit_test_index` find the smallest containing leaf, and
42+
`synth_locator` builds a `LocatorSpec` via the `mobile/selectors.v` priority.
43+
Text entry: a tap on an `EditText` defers to `flush_pending_edit`, which
44+
re-reads the field's `text` and emits a `fill`.
45+
- **iOS assisted REPL (no passive touch stream):** `tap x y` / `text <s>` /
46+
`assert x y` / `done`; each hit-tests the current `page_source()`, performs via
47+
`tap_at` / `MobileLocator.fill`, and records. Synthesis is offline-tested;
48+
not yet run on a physical device.
49+
- **Verified:** `v test mobile/codegen_capture_test.v` (24 cases) + a live
50+
on-emulator round-trip — 5/5 synthesized selectors re-resolved against the
51+
running UiAutomator2 tree, and a generated program type-checks.
4152

4253
### Docs (Layer 5) — not started
4354
Flip codegen "Planned" → shipped (web) across `COMPARISON.md`,

COMPARISON.md

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,8 @@ per-tool docs:
3636
| **Cross-browser incl. Safari** | ✅ (W3C) || ⚠️ bundled engines | n/a |
3737
| **Native iOS/Android** | ✅ no Node hop ||||
3838
| **Hybrid-app webviews** | ❌ not yet | n/a | n/a | ✅ mature |
39-
| **Trace viewer / video / codegen** | ⚠️ JSON tracer | ⚠️ basic | ✅ full | ⚠️ basic |
39+
| **Trace viewer / video** | ⚠️ JSON tracer | ⚠️ basic | ✅ full | ⚠️ basic |
40+
| **Codegen / session recorder** | ✅ web + mobile ||||
4041
| **Ecosystem / cloud grids** | early | ✅ huge | ✅ growing | ✅ huge |
4142

4243
Legend: ✅ first-class · ⚠️ partial / via add-on · ❌ not available
@@ -56,10 +57,11 @@ capable.
5657
Vebidor delivers Playwright's developer experience (lazy locators, semantic
5758
selectors, retrying assertions, one-call `launch()`, route/fulfill mocking) on
5859
a **W3C-standards transport** rather than Chromium-only CDP — so it reaches any
59-
conformant driver, Safari included. Playwright still wins on polished tooling:
60-
binary trace viewer and video capture (conveniences, not capability gaps) — plus
61-
codegen, which is a genuine authoring-time gap on vebidor's roadmap (a standards-based
62-
recorder would emit semantic locators from the accessibility tree).
60+
conformant driver, Safari included. Vebidor now ships a **codegen / session
61+
recorder** too — web (over BiDi) and native mobile (over the a11y tree) — that
62+
emits semantic `get_by_*` locators rather than raw coordinates. Playwright still
63+
wins on polished tooling: the binary trace viewer and video capture
64+
(conveniences, not capability gaps).
6365

6466
### vs Appium — *same backends, no Node hop*
6567
`vebidor.mobile` talks to the **exact same on-device servers** Appium uses
@@ -124,7 +126,7 @@ and the only V-native option at all.
124126

125127
## Where the others still win
126128

127-
- **Playwright** — binary trace viewer, video capture, codegen recorder.
129+
- **Playwright** — binary trace viewer, video capture.
128130
- **Appium** — hybrid-app webview contexts, mature cloud-grid + plugin
129131
ecosystem, years of edge-case hardening.
130132
- **Selenium** — 15 years of community, Selenium Grid, clients in every

COMPARISON_WITH_PLAYWRIGHT.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,6 @@ surface and BiDi for the event-driven features. Playwright achieves the same via
7575
| Feature | Why deferred |
7676
|---------|--------------|
7777
| Real video capture | No standardized BiDi screencast yet; would need periodic screenshots + ffmpeg. The screenshot/PDF primitives are the building blocks. |
78-
| Codegen / session recorder | Not built yet, but a real authoring-time gap — it bootstraps the locator-writing loop, which is the main cost of writing a suite. On the roadmap; a standards-based version would emit semantic `get_by_*` locators from the accessibility tree, not raw coordinates. |
7978

8079
---
8180

@@ -154,7 +153,8 @@ last as it's the most driver-dependent).
154153
Vebidor now offers Playwright-style ergonomics (one-call launch, lazy auto-waiting
155154
locators, semantic selectors, web-first assertions, route/fulfill mocking) **and**
156155
standards-based WebDriver-BiDi coverage that meets or exceeds Selenium's — on a native V
157-
API, with no Node runtime or bundled browser download. The remaining Playwright-only items
158-
split two ways: the binary trace viewer and video capture are non-standard conveniences, while
159-
a codegen/session recorder is a genuine authoring-time gap that's on the roadmap (it'd emit
160-
semantic locators from the accessibility tree, staying standards-based).
156+
API, with no Node runtime or bundled browser download. Vebidor also ships a
157+
**codegen / session recorder** — web (over BiDi) and native mobile (over the
158+
accessibility tree) — that emits semantic `get_by_*` locators rather than raw
159+
coordinates. The remaining Playwright-only items are non-standard conveniences:
160+
the binary trace viewer and real video capture.

README.md

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A V language implementation of the W3C WebDriver protocol for browser automation.
44

5-
**Version 5.0.0** | **Playwright-style API + WebDriver-BiDi + native iOS/Android** | **100% Selenium parity** | **Production Ready**
5+
**Version 5.1.0** | **Playwright-style API + WebDriver-BiDi + native iOS/Android + codegen** | **100% Selenium parity** | **Production Ready**
66

77
## ⚡ Modern API (Playwright-style)
88

@@ -685,6 +685,16 @@ setup.
685685

686686
See [COMPARISON_WITH_PLAYWRIGHT.md](COMPARISON_WITH_PLAYWRIGHT.md) for the full Playwright/Selenium feature mapping.
687687

688+
## ✨ What's New in v5.1.0
689+
690+
**🎬 Codegen / session recorder** — record a live session and emit runnable
691+
vebidor source via [`tools/codegen.v`](tools/codegen.v). Web capture runs over
692+
WebDriver-BiDi; **native mobile** capture is new — Android is passive (taps read
693+
from `adb getevent`, hit-tested against the live accessibility tree), iOS is an
694+
assisted REPL. Both synthesize semantic `get_by_*` locators rather than brittle
695+
coordinates. Verified live on the Android emulator (5/5 synthesized selectors
696+
re-resolved on-device). See [CODEGEN_HANDOFF.md](CODEGEN_HANDOFF.md).
697+
688698
## ✨ What's New in v5.0.0
689699

690700
**📱 Native mobile automation** — a new [`vebidor.mobile`](mobile/) module that drives
@@ -867,11 +877,11 @@ For issues, questions, or contributions:
867877

868878
**Status**: Production-ready for web automation (Playwright-style API + WebDriver-BiDi on top of 100% Selenium parity) and native mobile automation (iOS via WebDriverAgent + Android via UiAutomator2). 🎉
869879

870-
**Version**: 5.0.0 (adds `vebidor.mobile`native iOS/Android app automation; web stack unchanged: Playwright-style Locators/assertions, `launch()`, WebDriver-BiDi, mobile emulation, 4-browser support)
880+
**Version**: 5.1.0 (adds a codegen recorder for web + native mobile — emits semantic `get_by_*` locators; on top of v5.0.0's `vebidor.mobile` and the unchanged web stack: Playwright-style Locators/assertions, `launch()`, WebDriver-BiDi, mobile emulation, 4-browser support)
871881

872882
**Selenium-parity phases**: ✅ Phase 1 | ✅ Phase 2 | ✅ Phase 3 | ✅ Phase 4 | ✅ Phase 5 | ✅ Phase 6 | ✅ Phase 7 | ✅ Phase 8
873883

874884
**Playwright-parity roadmap**: ✅ Phase 0 (transport seam) | ✅ Phase 1 (locators/assertions) | ✅ Phase 2 (launch) | ✅ Phase 3 (BiDi transport) | ✅ Phase 4 (BiDi features) | ✅ Phase 5 (tooling) | ✅ BiDi gap closure vs Selenium
875885

876-
**Latest Update**: 2026-06-01 - v5.0.0 native mobile automation (`vebidor.mobile`: iOS/WebDriverAgent + Android/UiAutomator2, cross-platform selectors, gestures, app lifecycle, device state)
886+
**Latest Update**: 2026-06-03 - v5.1.0 codegen recorder for web + native mobile (a11y-tree → semantic `get_by_*` locators; mobile capture verified live on the Android emulator)
877887

docs/preview/comparison-table.html

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@
3232
<tr><td class="feature">Auto-waiting actions</td><td class="has"><span class="em"></span> Built-in</td><td class="has"><span class="em"></span></td><td class="note">visible + enabled + scrolled-into-view</td></tr>
3333
<tr><td class="feature">Network interception</td><td class="has"><span class="em"></span> via BiDi</td><td class="has"><span class="em"></span></td><td class="note"><code>route</code> / <code>fulfill</code> / <code>abort</code></td></tr>
3434
<tr><td class="feature">Real touch-event dispatch</td><td style="color:var(--warning-fg)"><span class="em">⚠️</span> Protocol limit</td><td class="has"><span class="em"></span></td><td class="note">BiDi doesn't expose CDP touch</td></tr>
35-
<tr><td class="feature">Codegen recorder</td><td style="color:var(--fg-3)">— Planned</td><td class="has"><span class="em"></span></td><td class="note">Real authoring gap; on roadmap (a11y-tree → semantic locators)</td></tr>
35+
<tr><td class="feature">Codegen recorder</td><td class="has"><span class="em"></span> Web + mobile</td><td class="has"><span class="em"></span></td><td class="note">emits semantic <code>get_by_*</code> locators (a11y-tree → not raw coords)</td></tr>
3636
</tbody>
3737
</table></div>
3838
</body>

docs/ui_kits/docs/components/DocsPage.jsx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ function PageComparison() {
195195
["Network mocking", { chip: "BiDi", kind: "success", em: "✅" }, { chip: "Yes", kind: "success", em: "✅" }, "route / fulfill / abort / continue"],
196196
["Standards-based transport", { chip: "W3C", kind: "milestone", em: "🏆" }, { chip: "CDP", kind: "neutral", em: "" }, "talks to any conformant driver, incl. Safari"],
197197
["Real touch-event dispatch", { chip: "Protocol limit", kind: "warning", em: "⚠️" }, { chip: "Yes", kind: "success", em: "✅" }, "BiDi doesn't expose CDP touch"],
198-
["Codegen recorder", { chip: "Planned", kind: "neutral", em: "" }, { chip: "Yes", kind: "success", em: "✅" }, "real authoring gap; on roadmap (a11y-tree → semantic locators)"],
198+
["Codegen recorder", { chip: "Web + mobile", kind: "success", em: "" }, { chip: "Yes", kind: "success", em: "✅" }, "emits semantic get_by_* locators (a11y-tree → not raw coords)"],
199199
]}
200200
/>
201201

0 commit comments

Comments
 (0)