You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Summary
Brings `agents-mobile` to parity with the desktop agents app for four
composer / spawn capabilities, building on #4533 (the native mobile
slash-command composer):
- **Schema-driven new-session args** — render an agent's
`creation_schema` as native controls (feature 7)
- **Model / reasoning / speed controls at spawn** (feature 8)
- **Image attachments in the in-session chat composer** (feature 6)
- **Image attachments at spawn time** (feature 9)
**No server / runtime changes were required.** Every wire contract
already exists; this PR makes the mobile client produce the same
payloads desktop does, plus a small set of shared-UI fixes that device
testing surfaced.
## Why
#4533 reached slash-command composer parity on mobile, but the composer
was otherwise text-only. The four gaps above are all implemented on
desktop in `agents-server-ui`; this closes them on mobile.
## Key insight: 4 features → 2 capabilities, 0 server changes
1. **Features 7 + 8 are the same mechanism.** Model / reasoning / speed
are *not* special server concepts — they're ordinary `enum` properties
of the entity type's `creation_schema`, detected by field-name
convention. `creation_schema` is already synced to mobile and
`spawnEntity` already forwards `args`, so this is purely additive native
UI over data mobile already has.
2. **Features 6 + 9 share one path.** The shared send action already
threads attachments end-to-end; the only blocker was that
`uploadMessageAttachments` used browser `File` / `FormData.set`, which
React Native lacks.
## What's included
### Capability A — schema-driven spawn args + model/reasoning/speed (7,
8)
- New mobile: `lib/spawnArgs.ts`, `lib/lastPickedModel.ts`,
`components/SchemaArgsControls.tsx`, wired into `NewSessionScreen`.
- Renders the synced `creation_schema` natively, full parity with
desktop `SchemaForm`: enum → BottomSheet picker pill (model grouped by
provider, last pick remembered in AsyncStorage), boolean → `Switch`,
string/number → text field, string-array → comma field, object → JSON
field. Field labels are humanized (`reasoningEffort` → "Reasoning
Effort") and provider names mapped (`openai` → "OpenAI"). Required
fields gate "Start session"; server still validates against the schema.
- **"Start session" is pinned to the bottom** of the new-session screen
(outside the scroll) so it stays reachable as the schema / model /
sandbox sections stack up. The bar reports its height via `onLayout` and
the scroll content is padded by that height so every section still
scrolls clear; the bar owns its safe-area bottom inset (`Screen` leaves
the bottom to screen-specific controls, matching the in-session
composer).
### Capability B — image attachments (6, 9)
- New mobile: `lib/attachments.ts` + attach button / thumbnail tray in
`NativeComposer`; `image`/`camera` icons.
- Deps: `expo-image-picker` (library + camera; **iOS uses a native
`ActionSheetIOS`**), `expo-image-manipulator` (transcode to JPEG).
- In-session reuses the shared `createSendComposerInputAction({
attachments })`. At spawn it mirrors desktop `doSpawn` (spawn without
`initialMessage`, then send `immediate` with attachments). Gated on
whether the session's model supports image input
(`schemaModelSupportsImageInput`).
- **Display is handled by the existing desktop chat-log WebView embed**
— no native timeline rendering was needed; once a mobile-sent attachment
syncs, the embedded renderer shows the thumbnail.
### Shared / cross-cutting
- `agents-server-ui/lib/schemaProperties.ts` (new): the DOM-free
schema-classification helpers (inline props, model/reasoning/speed
detection + grouping, string-array/JSON parsing, object-schema guards)
extracted from `SchemaForm`/`NewSessionView` so desktop and the native
mobile composer share one source of truth — now including the
model-provider label map + `provider:model` id parsing that were
otherwise mirrored on each platform. Behaviour-preserving for desktop.
- `agents-server-ui/lib/sendMessage.ts`: `uploadMessageAttachments`
accepts `File | NativeFileDescriptor` and branches `FormData.append`
(RN) vs `.set` (web). No wire-format change.
- Embed / timeline fixes (device testing): `WorkspaceProvider` around
the embed router so the state-inspector view's `useWorkspace` resolves;
image-preview dialog respects the composer inset; iOS timeline thumbnail
containment.
- `agents/horton.ts`: robust session-title generation for
image/attachment messages.
## Key decisions (the "why", for future sessions)
- **Extract shared schema helpers rather than duplicate.** Mobile reuses
`agents-server-ui` pure logic + `agents-runtime/client`, and builds
native UI itself (the established pattern). The desktop side of the
refactor is mostly deletion.
- **Attachment display via the existing WebView embed**, not native
rendering — the desktop chat log already renders attachments; mobile
just needs to *send* them.
- **iOS picker via `ActionSheetIOS`, not our `BottomSheet`.** Presenting
the native image picker over an RN `Modal` freezes / fails to open on
iOS; the native action sheet has no such conflict. Android keeps the
`BottomSheet` (intent-based picker, no conflict).
- **Transcode every picked image to JPEG.** iOS returns HEIC, which
Anthropic/OpenAI vision models reject (the agent run errored with
`finish_reason=error`); `expo-image-manipulator` normalizes to JPEG.
- **Title generation hardening.** The low-cost, text-only title model
went conversational on image messages it couldn't see ("I'm sorry but no
images were actually shared…") and that became the title — firmer prompt
+ a guard that rejects sentence-like responses and falls back to the
local title.
## Known limitations / follow-ups
- **Android keyboard:** the composer is `position: absolute; bottom: 0`
and relies on Android window resize to lift above the IME; the iOS-style
translate is intentionally **not** applied on Android (doing so
double-offsets — device testing showed a 2× lift, proving resize is
active). If a device ever shows the composer covered under the keyboard,
the deterministic fix is `android:windowSoftInputMode="adjustNothing"`
(config plugin) + translate-only — that needs a native rebuild.
Rationale is in the `SessionScreen` composer-transform comment.
- **Image-preview bottom gap** is a tuned constant in
`AttachmentImagePreviewDialog.module.css` (`36px` = 16 gap + 20
`CHAT_COMPOSER_OVERLAP` compensation); nudge if a device shows too much
/ too little gap.
- **Title fix** only affects newly-created chats (existing bad titles
don't change retroactively).
- Attachment policy is **image-only** (desktop parity); arbitrary
documents are out of scope.
## Testing
- **Unit:** `schemaProperties` (agents-server-ui), `spawnArgs`
(agents-mobile), `generate-title` (agents) — all green. Typecheck clean
across `agents-server-ui`, `agents-mobile`, `agents-desktop`, `agents`;
`expo-doctor` 18/18.
- **Manual (iOS + Android dev builds):** schema controls +
model/reasoning pickers, required-field gating, attach from
library/camera, spawn-with-attachments + in-session send, HEIC handling,
image-input gating, timeline display, image-preview layering, keyboard
behaviour.
## Build / deploy notes
- **Native modules added** (`expo-image-picker`,
`expo-image-manipulator`) + an `expo-image-picker` config plugin (iOS
photo/camera usage strings) → requires a **native dev build** (not Expo
Go; a Metro reload is not enough).
- **Embed / CSS changes** (`agents-server-ui`) ship in the WebView DOM
bundle → Metro reload.
- **Title generation** runs server-side in `agents` (the horton handler)
→ restart / redeploy the agents server / runner.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the mobile new-session and chat composers to parity with desktop:
8
+
9
+
-**Schema-driven spawn args + model/reasoning/speed controls.** The new-session screen now renders an agent type's `creation_schema` as native controls — enum properties become picker sheets (the model enum groups options by provider and remembers the last pick), booleans become switches, string/number become text fields, string-arrays a comma-separated field, and other objects a JSON field — so agents that need structured creation args can be configured and started from mobile (full parity with the desktop `SchemaForm`). Required fields gate the **Start session** button, which is now pinned to the bottom of the screen so it stays reachable as these extra sections grow (the scroll content is padded to clear it).
10
+
-**Image attachments.** Both the in-session and new-session composers can attach images (photo library or camera) via `expo-image-picker`, gated on whether the session's model accepts image input. At spawn the first message is sent immediately after the entity is created so the upload can target it, mirroring the desktop flow. Attachments render in the chat log through the existing embedded timeline.
11
+
12
+
The shared `agents-server-ui` send path (`uploadMessageAttachments`) accepts React Native file descriptors alongside browser `File`s, and the new-session schema-classification helpers (`inlineSchemaProperties`, model/reasoning/speed detection, model-settings grouping) move into a reusable `lib/schemaProperties` module shared by desktop and mobile. No server API changes — the title hardening below is the only server-side behavior change.
13
+
14
+
Horton's session-title generation is also hardened for attachment messages: the title model could go conversational when the first message referenced images it couldn't see (e.g. apologizing that nothing was shared), and that sentence became the title. The system prompt now instructs it to infer a title from intent and never apologize, and a guard rejects sentence-like responses and falls back to the locally-derived title.
0 commit comments