Skip to content

Commit 1d85d2c

Browse files
MMeteorLclaude
andcommitted
Update benchmark prompts and README for post-rebase correctness
- Add isPrimaryKey: true to the natural PK column in each of the 4 benchmark prompts so the orchestrator gets explicit pk guidance and deduplication works correctly with the run_subagent tool's .refine() enforcement - Fix investigateCalls description: investigate_row → run_subagent - Add note that enumeration-step tokens are not captured in RunMetrics - Update custom-prompt example to show isPrimaryKey usage - Update cost table model name: Kimi K2 → DeepSeek V4 Pro Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent be57305 commit 1d85d2c

2 files changed

Lines changed: 12 additions & 7 deletions

File tree

benchmarks/README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ When a user clicks "Populate" in the app, or when the benchmark runner triggers
1414
|---|---|
1515
| `searchCalls` | Calls to `search_web` (TinyFish search API) |
1616
| `fetchCalls` | Calls to `fetch_page` (TinyFish fetch API) |
17-
| `investigateCalls` | `investigate_row` dispatches from the orchestrator |
17+
| `investigateCalls` | `run_subagent` dispatches from the orchestrator |
1818
| `rowsInserted` | Rows successfully inserted into the dataset |
1919
| `tokensInput` / `tokensOutput` | Total LLM tokens across all agents |
2020
| `orchestratorTokens*` / `investigateTokens*` | Token breakdown per agent tier |
@@ -24,6 +24,8 @@ When a user clicks "Populate" in the app, or when the benchmark runner triggers
2424

2525
Each run also records `status` (`success` / `error`), any error message, and an `isBenchmark` flag so you can filter benchmark runs from real sessions.
2626

27+
> **Note:** The workflow includes an enumeration classification step that calls an LLM directly (not through an agent) to decide whether to use a scraper or search strategy. The tokens used by that step are **not** captured in the metrics above — they're a small fixed cost per run (~100–200 input tokens, ~5 output tokens) but worth knowing if you're doing precise cost accounting.
28+
2729
---
2830

2931
## Prerequisites
@@ -225,13 +227,16 @@ Edit [`prompts.json`](./prompts.json) to add your own benchmark prompts. Each en
225227
"datasetName": "Human-readable dataset name",
226228
"description": "What the dataset is about — shown to the agent",
227229
"columns": [
228-
{ "name": "column_name", "type": "text", "description": "What this field is" }
230+
{ "name": "entity_name", "type": "text", "description": "The entity name", "isPrimaryKey": true },
231+
{ "name": "other_field", "type": "text", "description": "What this field is" }
229232
]
230233
}
231234
```
232235

233236
Column types: `text`, `number`, `boolean`, `url`, `date`.
234237

238+
Mark at least one column as `"isPrimaryKey": true` — the orchestrator uses this to tell subagents which field is the unique identifier, and the workflow uses it to reject duplicate rows automatically.
239+
235240
Then run:
236241
```bash
237242
make benchmark ARGS="--prompt my-prompt-id"
@@ -241,7 +246,7 @@ make benchmark ARGS="--prompt my-prompt-id"
241246

242247
## Cost estimation
243248

244-
Rough estimates based on Kimi K2 pricing (as of writing):
249+
Rough estimates based on DeepSeek V4 Pro pricing (as of writing):
245250

246251
| Per run (20 rows target) | Approximate cost |
247252
|---|---|

benchmarks/prompts.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
"datasetName": "YC Recent Batch Companies",
55
"description": "Companies from recent Y Combinator batches (W24, S24). Include name, website, one-line description, and batch.",
66
"columns": [
7-
{ "name": "company_name", "type": "text", "description": "Company name" },
7+
{ "name": "company_name", "type": "text", "description": "Company name", "isPrimaryKey": true },
88
{ "name": "website", "type": "url", "description": "Company website URL" },
99
{ "name": "description", "type": "text", "description": "One-line description of what the company does" },
1010
{ "name": "batch", "type": "text", "description": "YC batch, e.g. W24 or S24" }
@@ -15,7 +15,7 @@
1515
"datasetName": "B2B SaaS with Free Tiers",
1616
"description": "Popular B2B SaaS tools that offer a free tier or freemium plan. Include tool name, category, free-tier summary, and pricing page URL.",
1717
"columns": [
18-
{ "name": "tool_name", "type": "text", "description": "Name of the SaaS tool" },
18+
{ "name": "tool_name", "type": "text", "description": "Name of the SaaS tool", "isPrimaryKey": true },
1919
{ "name": "category", "type": "text", "description": "Category, e.g. CRM, Analytics, DevOps" },
2020
{ "name": "free_tier_summary", "type": "text", "description": "What the free tier includes" },
2121
{ "name": "pricing_page_url", "type": "url", "description": "URL to the pricing page" }
@@ -26,7 +26,7 @@
2626
"datasetName": "US National Parks",
2727
"description": "All US National Parks with state, year established, and official NPS page URL.",
2828
"columns": [
29-
{ "name": "park_name", "type": "text", "description": "Official park name" },
29+
{ "name": "park_name", "type": "text", "description": "Official park name", "isPrimaryKey": true },
3030
{ "name": "state", "type": "text", "description": "US state(s) the park is in" },
3131
{ "name": "established_year", "type": "number", "description": "Year the park was established" },
3232
{ "name": "official_page_url", "type": "url", "description": "URL on nps.gov" }
@@ -37,7 +37,7 @@
3737
"datasetName": "University AI Research Labs",
3838
"description": "Academic AI research labs at major universities. Include lab name, university, research focus, and lab website URL.",
3939
"columns": [
40-
{ "name": "lab_name", "type": "text", "description": "Name of the research lab" },
40+
{ "name": "lab_name", "type": "text", "description": "Name of the research lab", "isPrimaryKey": true },
4141
{ "name": "university", "type": "text", "description": "Host university" },
4242
{ "name": "research_focus", "type": "text", "description": "Primary research area, e.g. NLP, robotics, RL" },
4343
{ "name": "lab_website_url", "type": "url", "description": "Lab homepage URL" }

0 commit comments

Comments
 (0)