Explore contents_trajectory(): serialize chats to trajectory-v1 - #1075
Draft
gadenbuie wants to merge 9 commits into
Draft
Explore contents_trajectory(): serialize chats to trajectory-v1#1075gadenbuie wants to merge 9 commits into
gadenbuie wants to merge 9 commits into
Conversation
Serializes a Chat/Turn/Content into a plain R list matching the
letta-ai trajectory-v1 schema shape (meta/user/reasoning/assistant/tool
records). New generic in the contents_text() family, dispatching on
Turn/Content, with a new_S3_class("Chat") method for the entry point.
Adds test-content-trajectory.R (90 assertions) plus a locally-relaxed vendored copy of the trajectory-v1 schema (timestamp made optional) used for optional jsonvalidate-based conformance checks alongside a dependency-free structural helper.
- Recommend jsonlite::toJSON(null = "null") in docs so assistant tool-call records serialize content as JSON null instead of dropping it. - Omit SystemTurn from the Turn method rather than emitting it as an assistant record. - Track built-in search/fetch ids as FIFO queues so each result links to its own request when a turn contains multiple searches or fetches.
- Synthesize a stable id for built-in search/fetch results that have no queued matching request, mirroring the ContentToolResult fallback, instead of indexing an empty vector. - Document contents_trajectory() as accepting Chat or Turn only; a bare Content has no role context to form a valid trajectory record.
A provider that emits an empty text chunk alongside tool calls (e.g. Databricks) would otherwise produce a bogus assistant record with content = "". Filter empty ContentText up front, matching the replay/provider handling.
A tool result whose value is a Content object (e.g. a screenshot tool returning ContentImageInline) previously crashed with a JSON serialization error. Images and PDFs now inline as base64 data URIs; text unwraps directly; other content falls back to its usual trajectory placeholder.
Upstream trajectory-v1 gained an optional ok: boolean on tool records (a source-native success/failure signal) after this branch started. ContentToolResult@error is an exact match, so tool_errored(content) now populates it for regular tool results; built-in web search/fetch results have no equivalent signal and omit it, matching upstream.
Collaborator
|
If you're game, let's pair on this review! Would be helpful to have some back and forth on the shape of this. :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is an exploratory draft. The implementation is in solid shape, but there's no pressure to merge this — it's here to spark discussion about whether this fits ellmer's scope. It's inspired by letta-ai/trajectory, a schema/toolkit for normalizing raw agent transcripts (Claude Code, Codex, etc.) into a common
trajectory-v1JSON shape. ellmer already holds a fully structuredChat/Round/Turn/Contentobject graph in memory, so the parsing problem that project solves doesn't apply here — this PR asks a narrower question: does ellmer's internal model map cleanly onto that shape, and is a native R serializer worth having?contents_trajectory()is a new generic, added alongside the existingcontents_text()/contents_html()/contents_markdown()family (same S7 dispatch pattern), that converts aChat,Round, orTurninto a plain R list matchingtrajectory-v1's shape: an ordered stream ofuser/assistant/reasoning/toolrecords preceded by onemetarecord. Callers serialize to JSON themselves viajsonlite::toJSON().Key design decisions (see inline docs and commit messages for detail):
@letta-ai/trajectorypackage — this is a from-scratch reshaping of data ellmer already has, not a transcript parser.trajectory-v1role appearing directly in a turn (images, PDFs, uploaded files) degrade to a short placeholder string (e.g.[inline image]) rather than being silently dropped.Content(e.g. an image returned by a screenshot tool) is a different case — that's real tool output, not decoration — so it's inlined as a base64 data URI instead of a placeholder, and instead of crashing or requiring wire-protocol-style expansion. The same base64-inline approach could be extended to images/PDFs appearing directly in a turn too, if placeholder-only turns out to be too lossy in practice.ok: booleanfield totoolrecords after this PR was started (a source-native success/failure signal, e.g. Claude Code'sis_error). ellmer has an exact match for that signal inContentToolResult@error, sotoolrecords now carryokfor regular tool calls; built-in web search/fetch results have no equivalent signal and omit it, matching upstream's "omit when unreliable" policy.Verification
See
tests/testthat/test-content-trajectory.Rfor coverage of tool calls/results, reasoning content, mixed-content assistant turns, built-in web search/fetch tools, andRound-level conversion.