Skip to content

Commit 683cfae

Browse files
msfstefclaude
andauthored
feat(agents-mobile): schema-driven spawn args, model controls & image attachments (desktop parity) (#4553)
## Summary Brings `agents-mobile` to parity with the desktop agents app for four composer / spawn capabilities, building on #4533 (the native mobile slash-command composer): - **Schema-driven new-session args** — render an agent's `creation_schema` as native controls (feature 7) - **Model / reasoning / speed controls at spawn** (feature 8) - **Image attachments in the in-session chat composer** (feature 6) - **Image attachments at spawn time** (feature 9) **No server / runtime changes were required.** Every wire contract already exists; this PR makes the mobile client produce the same payloads desktop does, plus a small set of shared-UI fixes that device testing surfaced. ## Why #4533 reached slash-command composer parity on mobile, but the composer was otherwise text-only. The four gaps above are all implemented on desktop in `agents-server-ui`; this closes them on mobile. ## Key insight: 4 features → 2 capabilities, 0 server changes 1. **Features 7 + 8 are the same mechanism.** Model / reasoning / speed are *not* special server concepts — they're ordinary `enum` properties of the entity type's `creation_schema`, detected by field-name convention. `creation_schema` is already synced to mobile and `spawnEntity` already forwards `args`, so this is purely additive native UI over data mobile already has. 2. **Features 6 + 9 share one path.** The shared send action already threads attachments end-to-end; the only blocker was that `uploadMessageAttachments` used browser `File` / `FormData.set`, which React Native lacks. ## What's included ### Capability A — schema-driven spawn args + model/reasoning/speed (7, 8) - New mobile: `lib/spawnArgs.ts`, `lib/lastPickedModel.ts`, `components/SchemaArgsControls.tsx`, wired into `NewSessionScreen`. - Renders the synced `creation_schema` natively, full parity with desktop `SchemaForm`: enum → BottomSheet picker pill (model grouped by provider, last pick remembered in AsyncStorage), boolean → `Switch`, string/number → text field, string-array → comma field, object → JSON field. Field labels are humanized (`reasoningEffort` → "Reasoning Effort") and provider names mapped (`openai` → "OpenAI"). Required fields gate "Start session"; server still validates against the schema. - **"Start session" is pinned to the bottom** of the new-session screen (outside the scroll) so it stays reachable as the schema / model / sandbox sections stack up. The bar reports its height via `onLayout` and the scroll content is padded by that height so every section still scrolls clear; the bar owns its safe-area bottom inset (`Screen` leaves the bottom to screen-specific controls, matching the in-session composer). ### Capability B — image attachments (6, 9) - New mobile: `lib/attachments.ts` + attach button / thumbnail tray in `NativeComposer`; `image`/`camera` icons. - Deps: `expo-image-picker` (library + camera; **iOS uses a native `ActionSheetIOS`**), `expo-image-manipulator` (transcode to JPEG). - In-session reuses the shared `createSendComposerInputAction({ attachments })`. At spawn it mirrors desktop `doSpawn` (spawn without `initialMessage`, then send `immediate` with attachments). Gated on whether the session's model supports image input (`schemaModelSupportsImageInput`). - **Display is handled by the existing desktop chat-log WebView embed** — no native timeline rendering was needed; once a mobile-sent attachment syncs, the embedded renderer shows the thumbnail. ### Shared / cross-cutting - `agents-server-ui/lib/schemaProperties.ts` (new): the DOM-free schema-classification helpers (inline props, model/reasoning/speed detection + grouping, string-array/JSON parsing, object-schema guards) extracted from `SchemaForm`/`NewSessionView` so desktop and the native mobile composer share one source of truth — now including the model-provider label map + `provider:model` id parsing that were otherwise mirrored on each platform. Behaviour-preserving for desktop. - `agents-server-ui/lib/sendMessage.ts`: `uploadMessageAttachments` accepts `File | NativeFileDescriptor` and branches `FormData.append` (RN) vs `.set` (web). No wire-format change. - Embed / timeline fixes (device testing): `WorkspaceProvider` around the embed router so the state-inspector view's `useWorkspace` resolves; image-preview dialog respects the composer inset; iOS timeline thumbnail containment. - `agents/horton.ts`: robust session-title generation for image/attachment messages. ## Key decisions (the "why", for future sessions) - **Extract shared schema helpers rather than duplicate.** Mobile reuses `agents-server-ui` pure logic + `agents-runtime/client`, and builds native UI itself (the established pattern). The desktop side of the refactor is mostly deletion. - **Attachment display via the existing WebView embed**, not native rendering — the desktop chat log already renders attachments; mobile just needs to *send* them. - **iOS picker via `ActionSheetIOS`, not our `BottomSheet`.** Presenting the native image picker over an RN `Modal` freezes / fails to open on iOS; the native action sheet has no such conflict. Android keeps the `BottomSheet` (intent-based picker, no conflict). - **Transcode every picked image to JPEG.** iOS returns HEIC, which Anthropic/OpenAI vision models reject (the agent run errored with `finish_reason=error`); `expo-image-manipulator` normalizes to JPEG. - **Title generation hardening.** The low-cost, text-only title model went conversational on image messages it couldn't see ("I'm sorry but no images were actually shared…") and that became the title — firmer prompt + a guard that rejects sentence-like responses and falls back to the local title. ## Known limitations / follow-ups - **Android keyboard:** the composer is `position: absolute; bottom: 0` and relies on Android window resize to lift above the IME; the iOS-style translate is intentionally **not** applied on Android (doing so double-offsets — device testing showed a 2× lift, proving resize is active). If a device ever shows the composer covered under the keyboard, the deterministic fix is `android:windowSoftInputMode="adjustNothing"` (config plugin) + translate-only — that needs a native rebuild. Rationale is in the `SessionScreen` composer-transform comment. - **Image-preview bottom gap** is a tuned constant in `AttachmentImagePreviewDialog.module.css` (`36px` = 16 gap + 20 `CHAT_COMPOSER_OVERLAP` compensation); nudge if a device shows too much / too little gap. - **Title fix** only affects newly-created chats (existing bad titles don't change retroactively). - Attachment policy is **image-only** (desktop parity); arbitrary documents are out of scope. ## Testing - **Unit:** `schemaProperties` (agents-server-ui), `spawnArgs` (agents-mobile), `generate-title` (agents) — all green. Typecheck clean across `agents-server-ui`, `agents-mobile`, `agents-desktop`, `agents`; `expo-doctor` 18/18. - **Manual (iOS + Android dev builds):** schema controls + model/reasoning pickers, required-field gating, attach from library/camera, spawn-with-attachments + in-session send, HEIC handling, image-input gating, timeline display, image-preview layering, keyboard behaviour. ## Build / deploy notes - **Native modules added** (`expo-image-picker`, `expo-image-manipulator`) + an `expo-image-picker` config plugin (iOS photo/camera usage strings) → requires a **native dev build** (not Expo Go; a Metro reload is not enough). - **Embed / CSS changes** (`agents-server-ui`) ship in the WebView DOM bundle → Metro reload. - **Title generation** runs server-side in `agents` (the horton handler) → restart / redeploy the agents server / runner. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent b8875a2 commit 683cfae

24 files changed

Lines changed: 1923 additions & 235 deletions
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
"@electric-ax/agents-mobile": patch
3+
"@electric-ax/agents-server-ui": patch
4+
"@electric-ax/agents": patch
5+
---
6+
7+
Bring the mobile new-session and chat composers to parity with desktop:
8+
9+
- **Schema-driven spawn args + model/reasoning/speed controls.** The new-session screen now renders an agent type's `creation_schema` as native controls — enum properties become picker sheets (the model enum groups options by provider and remembers the last pick), booleans become switches, string/number become text fields, string-arrays a comma-separated field, and other objects a JSON field — so agents that need structured creation args can be configured and started from mobile (full parity with the desktop `SchemaForm`). Required fields gate the **Start session** button, which is now pinned to the bottom of the screen so it stays reachable as these extra sections grow (the scroll content is padded to clear it).
10+
- **Image attachments.** Both the in-session and new-session composers can attach images (photo library or camera) via `expo-image-picker`, gated on whether the session's model accepts image input. At spawn the first message is sent immediately after the entity is created so the upload can target it, mirroring the desktop flow. Attachments render in the chat log through the existing embedded timeline.
11+
12+
The shared `agents-server-ui` send path (`uploadMessageAttachments`) accepts React Native file descriptors alongside browser `File`s, and the new-session schema-classification helpers (`inlineSchemaProperties`, model/reasoning/speed detection, model-settings grouping) move into a reusable `lib/schemaProperties` module shared by desktop and mobile. No server API changes — the title hardening below is the only server-side behavior change.
13+
14+
Horton's session-title generation is also hardened for attachment messages: the title model could go conversational when the first message referenced images it couldn't see (e.g. apologizing that nothing was shared), and that sentence became the title. The system prompt now instructs it to infer a title from intent and never apologize, and a guard rejects sentence-like responses and falls back to the locally-derived title.

packages/agents-mobile/app.config.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,16 @@ export default ({ config }: ConfigContext): ExpoConfig =>
3838
// expo-web-browser. See `plugins/with-android-on-new-intent.js`
3939
// for the underlying Expo issue.
4040
`./plugins/with-android-on-new-intent.js`,
41+
// Image attachments in the composer. The plugin injects the iOS
42+
// photo-library / camera usage strings; a dev build (not Expo Go) is
43+
// required for the native module.
44+
[
45+
`expo-image-picker`,
46+
{
47+
photosPermission: `Allow Electric Agents to attach photos to messages.`,
48+
cameraPermission: `Allow Electric Agents to take a photo to attach to a message.`,
49+
},
50+
],
4151
],
4252
ios: {
4353
...config.ios,

packages/agents-mobile/package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@
2828
"expo": "54.0.35",
2929
"expo-build-properties": "~1.0.10",
3030
"expo-constants": "~18.0.13",
31+
"expo-image-manipulator": "~14.0.8",
32+
"expo-image-picker": "~17.0.11",
3133
"expo-linking": "~8.0.12",
3234
"expo-router": "~6.0.24",
3335
"expo-status-bar": "~3.0.9",

packages/agents-mobile/src/components/Icon.tsx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ export type IconName =
4040
| `user`
4141
| `users`
4242
| `pin`
43+
| `image`
44+
| `camera`
4345

4446
const PATHS: Record<IconName, string> = {
4547
back: `M15 18l-6-6 6-6`,
@@ -68,6 +70,9 @@ const PATHS: Record<IconName, string> = {
6870
// Lucide `pin` — matches the desktop sidebar's pin toggle glyph.
6971
pin: `M12 17v5M9 10.76a2 2 0 0 1-1.11 1.79l-1.78.9A2 2 0 0 0 5 15.24V16a1 1 0 0 0 1 1h12a1 1 0 0 0 1-1v-.76a2 2 0 0 0-1.11-1.79l-1.78-.9A2 2 0 0 1 15 10.76V7a1 1 0 0 1 1-1 2 2 0 0 0 0-4H8a2 2 0 0 0 0 4 1 1 0 0 1 1 1z`,
7072
square: `M7 7h10v10H7z`,
73+
// Lucide `image` / `camera` — image attachment affordances.
74+
image: `M5 3h14a2 2 0 0 1 2 2v14a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2ZM8.5 11a1.5 1.5 0 1 0 0-3 1.5 1.5 0 0 0 0 3ZM21 15l-5-5L5 21`,
75+
camera: `M9 4 7.5 6H5a2 2 0 0 0-2 2v10a2 2 0 0 0 2 2h14a2 2 0 0 0 2-2V8a2 2 0 0 0-2-2h-2.5L15 4ZM12 17a4 4 0 1 0 0-8 4 4 0 0 0 0 8Z`,
7176
github: `M15 22v-4a4.8 4.8 0 0 0-1-3.5c3 0 6-2 6-5.5.08-1.25-.27-2.48-1-3.5.28-1.15.28-2.35 0-3.5 0 0-1 0-3 1.5-2.64-.5-5.36-.5-8 0C6 2 5 2 5 2c-.3 1.15-.3 2.35 0 3.5A5.403 5.403 0 0 0 4 9c0 3.5 3 5.5 6 5.5-.39.49-.68 1.05-.85 1.65-.17.6-.22 1.23-.15 1.85v4M9 18c-4.51 2-5-2-7-2`,
7277
// Official Google "G" mark, rendered in a single fill colour. Google's
7378
// brand guidelines permit monochrome use in CTA contexts where the

packages/agents-mobile/src/components/NativeComposer.tsx

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
import { useCallback, useMemo, useState } from 'react'
22
import {
3+
ActionSheetIOS,
4+
Image,
5+
Keyboard,
6+
Platform,
37
Pressable,
48
ScrollView,
59
StyleSheet,
@@ -23,9 +27,12 @@ import {
2327
type ComposerInsertion,
2428
type Selection,
2529
} from '../lib/slashAutocomplete'
30+
import type { AttachmentDraft } from '../lib/attachments'
2631
import { useTokens } from '../lib/ThemeProvider'
2732
import { fontSize, lineHeight, radii, spacing } from '../lib/theme'
2833
import type { Tokens } from '../lib/theme'
34+
import { BottomSheet, BottomSheetItem } from './BottomSheet'
35+
import { Icon } from './Icon'
2936

3037
export type SlashAutocomplete = {
3138
/** Whether the suggestion menu should be shown. */
@@ -217,6 +224,128 @@ export function SlashCommandMenu({
217224
)
218225
}
219226

227+
/**
228+
* Composer affordance for adding image attachments. Opens a native action
229+
* sheet (photo library / camera); the keyboard is dismissed first so the sheet
230+
* and keyboard don't fight for the bottom of the screen.
231+
*/
232+
export function AttachButton({
233+
onAddFromLibrary,
234+
onAddFromCamera,
235+
disabled,
236+
}: {
237+
onAddFromLibrary: () => void
238+
onAddFromCamera: () => void
239+
disabled?: boolean
240+
}): React.ReactElement {
241+
const tokens = useTokens()
242+
const styles = useMemo(() => createStyles(tokens), [tokens])
243+
// Android uses the bottom-sheet menu; iOS uses a native action sheet because
244+
// presenting the image picker over an RN `Modal` breaks it on iOS.
245+
const [open, setOpen] = useState(false)
246+
const onPress = (): void => {
247+
Keyboard.dismiss()
248+
if (Platform.OS === `ios`) {
249+
ActionSheetIOS.showActionSheetWithOptions(
250+
{
251+
options: [`Cancel`, `Photo Library`, `Take Photo`],
252+
cancelButtonIndex: 0,
253+
},
254+
(index) => {
255+
if (index === 1) onAddFromLibrary()
256+
else if (index === 2) onAddFromCamera()
257+
}
258+
)
259+
return
260+
}
261+
setOpen(true)
262+
}
263+
return (
264+
<>
265+
<Pressable
266+
onPress={onPress}
267+
disabled={disabled}
268+
hitSlop={8}
269+
accessibilityRole="button"
270+
accessibilityLabel="Attach image"
271+
style={({ pressed }) => [
272+
styles.attachButton,
273+
pressed && !disabled ? styles.attachButtonPressed : null,
274+
]}
275+
>
276+
<Icon
277+
name="image"
278+
size={20}
279+
color={disabled ? tokens.text4 : tokens.text2}
280+
strokeWidth={2}
281+
/>
282+
</Pressable>
283+
<BottomSheet open={open} onClose={() => setOpen(false)} title="Add image">
284+
<BottomSheetItem
285+
label="Photo Library"
286+
icon={<Icon name="image" size={18} color={tokens.text2} />}
287+
onPress={() => {
288+
setOpen(false)
289+
onAddFromLibrary()
290+
}}
291+
/>
292+
<BottomSheetItem
293+
label="Take Photo"
294+
icon={<Icon name="camera" size={18} color={tokens.text2} />}
295+
onPress={() => {
296+
setOpen(false)
297+
onAddFromCamera()
298+
}}
299+
/>
300+
</BottomSheet>
301+
</>
302+
)
303+
}
304+
305+
/**
306+
* Horizontal strip of image-attachment thumbnails with per-item remove
307+
* buttons. Renders nothing when there are no drafts.
308+
*/
309+
export function AttachmentTray({
310+
drafts,
311+
onRemove,
312+
}: {
313+
drafts: ReadonlyArray<AttachmentDraft>
314+
onRemove: (index: number) => void
315+
}): React.ReactElement | null {
316+
const tokens = useTokens()
317+
const styles = useMemo(() => createStyles(tokens), [tokens])
318+
if (drafts.length === 0) return null
319+
return (
320+
<ScrollView
321+
horizontal
322+
showsHorizontalScrollIndicator={false}
323+
keyboardShouldPersistTaps="handled"
324+
contentContainerStyle={styles.tray}
325+
>
326+
{drafts.map((draft, index) => (
327+
<View key={`${draft.uri}:${index}`} style={styles.thumbWrap}>
328+
<Image source={{ uri: draft.uri }} style={styles.thumb} />
329+
<Pressable
330+
onPress={() => onRemove(index)}
331+
hitSlop={6}
332+
accessibilityRole="button"
333+
accessibilityLabel={`Remove ${draft.name}`}
334+
style={styles.thumbRemove}
335+
>
336+
<Icon
337+
name="close"
338+
size={12}
339+
color={tokens.textOnAccent}
340+
strokeWidth={2.6}
341+
/>
342+
</Pressable>
343+
</View>
344+
))}
345+
</ScrollView>
346+
)
347+
}
348+
220349
function createStyles(tokens: Tokens) {
221350
return StyleSheet.create({
222351
menu: {
@@ -261,5 +390,48 @@ function createStyles(tokens: Tokens) {
261390
fontSize: fontSize.xs,
262391
lineHeight: lineHeight.xs,
263392
},
393+
attachButton: {
394+
width: 34,
395+
height: 34,
396+
borderRadius: radii.pill,
397+
alignItems: `center`,
398+
justifyContent: `center`,
399+
},
400+
attachButtonPressed: {
401+
backgroundColor: tokens.bgHover,
402+
},
403+
tray: {
404+
gap: spacing.sm,
405+
// Leave room for the remove (×) buttons, which protrude 6px past the
406+
// top-right of each thumbnail (incl. the last one when scrolled to end).
407+
paddingTop: 8,
408+
paddingBottom: 2,
409+
paddingHorizontal: 8,
410+
},
411+
thumbWrap: {
412+
width: 60,
413+
height: 60,
414+
},
415+
thumb: {
416+
width: 60,
417+
height: 60,
418+
borderRadius: radii.md,
419+
borderWidth: 1,
420+
borderColor: tokens.border1,
421+
backgroundColor: tokens.bgSubtle,
422+
},
423+
thumbRemove: {
424+
position: `absolute`,
425+
top: -6,
426+
right: -6,
427+
width: 20,
428+
height: 20,
429+
borderRadius: radii.pill,
430+
alignItems: `center`,
431+
justifyContent: `center`,
432+
backgroundColor: tokens.accent9,
433+
borderWidth: 1,
434+
borderColor: tokens.surface,
435+
},
264436
})
265437
}

0 commit comments

Comments
 (0)