feat: record model thoughts by dannykopping · Pull Request #203 · coder/aibridge

dannykopping · 2026-03-05T15:37:10Z

Required for coder/coder#22676
Closes #168

This PR adds recording of model "thoughts" (sometimes call "reasoning"). This is only available for /v1/messages and /v1/responses.

/v1/messages

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"The"}              }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":" user is asking me to get"}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":" stock prices for Apple and Google and compare"}           }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":" their"}               }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":" performance over the last week."}        }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"\n\nLet me search"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":" for both"}      }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":" stock prices in"}    }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":" parallel"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"thinking_delta","thinking":"."}       }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"signature_delta","signature":"..."}      }

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_01KLHTi1EnM4RxxG9VfFBwDt","name":"WebSearch","input":{},"caller":{"type":"direct"}}     }

...

event: content_block_start
data: {"type":"content_block_start","index":2,"content_block":{"type":"tool_use","id":"toolu_01T9we4nEzyv7WVoiazyK8VE","name":"WebSearch","input":{},"caller":{"type":"direct"}}          }

/v1/responses

{
    "type": "response.completed",
    "response": {
        ...
        "output": [
            {
                "id": "rs_0a139fe840c9c4640169b296e3eb088195886c12786026df1a",
                "type": "reasoning",
                "encrypted_content": "...",
                "summary": []
            },
            {
                "id": "msg_0a139fe840c9c4640169b296ebad1081958b3077bf7ae243ee",
                "type": "message",
                "status": "completed",
                "content": [
                    {
                        "type": "output_text",
                        "annotations": [],
                        "logprobs": [],
                        "text": "I’m reading both paths directly so I can return their contents or tell you if either one is missing or not a regular file."
                    }
                ],
                "phase": "commentary",
                "role": "assistant"
            },
            {
                "id": "fc_0a139fe840c9c4640169b296ed6ebc8195b5cc1641dac78918",
                "type": "function_call",
                "status": "completed",
                "arguments": "{\"cmd\":\"sed -n '1,250p' /tmp/foo\",\"yield_time_ms\":1000,\"max_output_tokens\":6000,\"workdir\":\"/home/coder/coder\"}",
                "call_id": "call_JmiKwclDrwqU1w2Ojk0E9wQB",
                "name": "exec_command"
            },
            {
                "id": "fc_0a139fe840c9c4640169b296ed6ecc819586447dbe18e23fff",
                "type": "function_call",
                "status": "completed",
                "arguments": "{\"cmd\":\"sed -n '1,250p' /home/bar\",\"yield_time_ms\":1000,\"max_output_tokens\":6000,\"workdir\":\"/home/coder/coder\"}",
                "call_id": "call_juUonJYUHxtCJb6ckr9tNTv5",
                "name": "exec_command"
            }
        ],
        "parallel_tool_calls": true,
        ...
    }
}

dannykopping · 2026-03-05T15:37:28Z

feat: record model thoughts #203 👈 (View in Graphite)
feat: only disable parallel tool calls when injectable tools are present #211
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

pawbana · 2026-03-06T15:06:30Z

 	InvocationError error
 	Metadata        Metadata
 	CreatedAt       time.Time
+	ModelThoughts   []*ModelThoughtRecord


How much space could ModelThoughts take?
Maybe we should consider configuring recording things not only due to compliance reasons but also to limit space usage.

It all depends on how many thinking tokens are given to the model, and even then we're only capturing a summary, so I suspect it won't be much. In any case, having this information will help provide insight into why agents take the course of action they do, which may end up being valuable for an audit scenario.

In any case, disk is cheap.

We can later add a flag to not store these conditionally.

pawbana · 2026-03-09T12:26:39Z

Maybe I'm missing something but is there a reason thinking blocks are merged into RecordToolUsage and recorded only on tool call? (inner loop abstraction?)
I understand they are usually connected but I'd think there could be some reasoning without any tool call or provider could call tool directly.
Maybe thinking blocks should be part of RecordInterceptionEnded?

ssncferreira · 2026-03-09T18:55:19Z

+					case anthropic.ThinkingBlock:
+						thoughtRecords = append(thoughtRecords, &recorder.ModelThoughtRecord{
+							Content:   variant.Thinking,
+							CreatedAt: time.Now(),


AFAIK, with streaming, the code waits until we get a stop block and processes all thinking blocks at that point, meaning they'll all have the same CreatedAt. I assume the ordering is still preserved by their position in the slice, so this probably doesn't matter, but worth noting that CreatedAt won't reflect when each block actually arrived. Could this be an issue?

This is actually true of tool usages as well which are recorded at this stage, as well.

Postgres is microsecond-precise, so some records may be persisted with the same timestamp.

The timestamp for thoughts doesn't really matter, though; as long as it's associated to a tool call it'll be displayed correctly (i.e. before the tool call).

pawbana · 2026-03-16T09:20:33Z


-		// Handle tool calls for non-streaming.
+		// Capture any thinking blocks that were returned.
+		for _, t := range i.extractModelThoughts(resp) {


nit: in responses there is separate base.recordModelThoughts method. I like this approach as it helps to keep ProcessRequest function in blocking.go and streaming.go shorter / "higher level" making it easier to understand and reason about. Would be nice to be consistent between interceptors (either both have this base method or not).

pawbana · 2026-03-16T09:30:01Z

+				// We can't guarantee the order of model thoughts since they're recorded separately, so
+				// we have to scan all thoughts for a match.
+
+				for _, expected := range tc.expectedThoughts {


This could be part of mockRecorder, eg. something like verifyThoughts method. Would remove copy paste from TestResponsesModelThoughts. It could ignore InterceptionID and CreatedAt fields then tests could use []recorder.ModelThoughtRecord as expectedThoughts field.

I believe mockRecorder stores thoughts in slice so they should be stored in order they where added which should be deterministic? I'm ok with not checking exact order but I think comment is wrong?

nit: when comparing slices without caring about order I find sorting slices then comparing equality helps to keep code more concise.

Signed-off-by: Danny Kopping <danny@coder.com>

Depends on coder/aibridge#203 Closes coder/internal#1337 --------- Signed-off-by: Danny Kopping <danny@coder.com>

dannykopping mentioned this pull request Mar 5, 2026

feat: record and correlate tool call IDs across interceptions #188

Merged

dannykopping mentioned this pull request Mar 5, 2026

feat: add client session tracking #198

Merged

dannykopping force-pushed the dk/model-thoughts branch from 8682f8d to 56ff5d0 Compare March 6, 2026 07:28

dannykopping changed the base branch from dk/session-id-tracking to graphite-base/203 March 6, 2026 11:52

dannykopping force-pushed the graphite-base/203 branch from 1d5e6ac to e559e5e Compare March 6, 2026 11:52

dannykopping force-pushed the dk/model-thoughts branch from 56ff5d0 to 11d859c Compare March 6, 2026 11:52

graphite-app bot changed the base branch from graphite-base/203 to main March 6, 2026 11:53

dannykopping force-pushed the dk/model-thoughts branch from 11d859c to 855e3a7 Compare March 6, 2026 11:53

dannykopping commented Mar 6, 2026

View reviewed changes

Comment thread intercept/responses/base.go Outdated

dannykopping marked this pull request as ready for review March 6, 2026 14:30

dannykopping requested review from pawbana and ssncferreira March 6, 2026 14:30

dannykopping force-pushed the dk/model-thoughts branch from d09d5b0 to a0ad392 Compare March 6, 2026 15:14

pawbana reviewed Mar 9, 2026

View reviewed changes

Comment thread intercept/messages/blocking.go Outdated

ssncferreira reviewed Mar 9, 2026

View reviewed changes

dannykopping changed the base branch from main to graphite-base/203 March 11, 2026 08:27

dannykopping force-pushed the dk/model-thoughts branch from a0ad392 to c2f9c79 Compare March 11, 2026 08:27

dannykopping changed the base branch from graphite-base/203 to dk/parallel-tool-calls March 11, 2026 08:27

dannykopping mentioned this pull request Mar 11, 2026

feat: only disable parallel tool calls when injectable tools are present #211

Merged

dannykopping marked this pull request as draft March 11, 2026 08:28

dannykopping force-pushed the dk/parallel-tool-calls branch from a0e150e to b2e4f03 Compare March 11, 2026 12:15

dannykopping force-pushed the dk/model-thoughts branch 2 times, most recently from 8a13f42 to a492e77 Compare March 11, 2026 12:27

dannykopping force-pushed the dk/parallel-tool-calls branch from b2e4f03 to 732f3d2 Compare March 11, 2026 12:27

dannykopping force-pushed the dk/model-thoughts branch 3 times, most recently from 158d8b0 to 25b5478 Compare March 12, 2026 12:18

dannykopping mentioned this pull request Mar 12, 2026

feat: record model thoughts coder/coder#22676

Merged

dannykopping marked this pull request as ready for review March 12, 2026 13:44

dannykopping force-pushed the dk/model-thoughts branch from 0773ea6 to 8db8a1c Compare March 12, 2026 13:45

dannykopping changed the base branch from dk/parallel-tool-calls to graphite-base/203 March 12, 2026 13:48

dannykopping force-pushed the dk/model-thoughts branch from 8db8a1c to 7a0d206 Compare March 12, 2026 13:49

dannykopping force-pushed the graphite-base/203 branch from 3466ab6 to 4fd7ded Compare March 12, 2026 13:49

graphite-app bot changed the base branch from graphite-base/203 to main March 12, 2026 13:49

dannykopping force-pushed the dk/model-thoughts branch from 7a0d206 to 87044c1 Compare March 12, 2026 13:50

pawbana reviewed Mar 13, 2026

View reviewed changes

Comment thread fixtures/anthropic/simple.txtar

Comment thread intercept/messages/blocking.go Outdated

Comment thread intercept/messages/base.go

dannykopping force-pushed the dk/model-thoughts branch from 87044c1 to 0c56c33 Compare March 13, 2026 15:45

dannykopping requested a review from pawbana March 16, 2026 08:28

pawbana reviewed Mar 16, 2026

View reviewed changes

dannykopping added 13 commits March 16, 2026 14:25

feat: record model thoughts

4875cf5

Signed-off-by: Danny Kopping <danny@coder.com>

fix: send model thoughts with tool usage recording

f878dd6

Signed-off-by: Danny Kopping <danny@coder.com>

feat: capture responses reasoning

daaec9f

Signed-off-by: Danny Kopping <danny@coder.com>

chore: refactor tests

1c9a526

Signed-off-by: Danny Kopping <danny@coder.com>

chore: test multiple thoughts

7e4a15a

Signed-off-by: Danny Kopping <danny@coder.com>

chore: refactor tests

abfc679

Signed-off-by: Danny Kopping <danny@coder.com>

feat: capture commentary as model thoughts

464670a

Signed-off-by: Danny Kopping <danny@coder.com>

chore: cleaning up /v1/messages impl

bf7d2bc

Signed-off-by: Danny Kopping <danny@coder.com>

chore: update comments

424382d

Signed-off-by: Danny Kopping <danny@coder.com>

chore: add tests for parallel tool calls

7511cc4

Signed-off-by: Danny Kopping <danny@coder.com>

chore: capture source of thinking/reasoning

befe9d9

Signed-off-by: Danny Kopping <danny@coder.com>

chore: break association between thoughts and tools

7fbfb8e

Signed-off-by: Danny Kopping <danny@coder.com>

chore: expand trace testing

33cec2a

Signed-off-by: Danny Kopping <danny@coder.com>

dannykopping force-pushed the dk/model-thoughts branch from 0c56c33 to 33cec2a Compare March 16, 2026 14:31

chore: refactor model thought assertions into common func

9207856

Signed-off-by: Danny Kopping <danny@coder.com>

pawbana approved these changes Mar 16, 2026

View reviewed changes

dannykopping merged commit 5c071a7 into main Mar 16, 2026
5 checks passed

dannykopping added a commit to coder/coder that referenced this pull request Mar 17, 2026

feat: record model thoughts (#22676)

365de3e

Depends on coder/aibridge#203 Closes coder/internal#1337 --------- Signed-off-by: Danny Kopping <danny@coder.com>

Conversation

dannykopping commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dannykopping commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

pawbana Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

dannykopping Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pawbana commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ssncferreira Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

dannykopping Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pawbana Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

pawbana Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dannykopping commented Mar 5, 2026 •

edited

Loading

dannykopping commented Mar 5, 2026 •

edited

Loading

pawbana commented Mar 9, 2026 •

edited

Loading