Skip to content

Commit 9e789ba

Browse files
authored
Add benchmark to Flash README (#376)
1 parent 3f33a53 commit 9e789ba

3 files changed

Lines changed: 31 additions & 0 deletions

File tree

pageindex/flash/README.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,37 @@ missing, non-PDF, encrypted, empty, or unreadable file.
3838

3939
Page indexes are 1-based. `nodes` nests the same shape recursively.
4040

41+
## Benchmark
42+
43+
Nine PDFs, each run end to end with tree optimization: PDF parse, layout
44+
outline, merge, LLM expand, then a summary for every node.
45+
46+
![Time against document length](assets/time_vs_pages.png)
47+
48+
![Cost against document length](assets/cost_vs_pages.png)
49+
50+
Both scale close to linearly with length, at 218 s and $0.85 per 1,000 pages.
51+
Two qualifiers. Cost follows node count a little more closely than page count,
52+
$0.0007 to $0.0016 per node, so a densely structured document costs more than
53+
its length suggests. And wall clock flattens past roughly 700 pages, where
54+
summary concurrency rather than length becomes the limit.
55+
56+
| Document | Pages | Input tokens | Output tokens |
57+
|---|---:|---:|---:|
58+
| Bitcoin whitepaper | 9 | 8,715 | 4,673 |
59+
| Attention Is All You Need | 15 | 26,805 | 10,183 |
60+
| KIMI K3 | 47 | 85,704 | 35,217 |
61+
| DeepSeek-R1 | 86 | 68,398 | 26,351 |
62+
| Situational Awareness | 165 | 115,130 | 54,347 |
63+
| Federal Reserve 2023 report | 222 | 280,975 | 136,982 |
64+
| 9/11 Commission Report | 585 | 720,624 | 200,202 |
65+
| Pattern Recognition and Machine Learning | 758 | 857,983 | 277,675 |
66+
| Machine Learning: A Probabilistic Perspective | 1,098 | 1,587,265 | 646,958 |
67+
| **Total** | **2,985** | **3,751,599** | **1,392,588** |
68+
69+
Measured with `gpt-5.6-luna` at $0.20 / $1.20 per million input / output tokens,
70+
priced cold with no prompt-cache discount.
71+
4172
## Limits
4273

4374
- Scanned PDFs without embedded text are not supported.
63.6 KB
Loading
65.4 KB
Loading

0 commit comments

Comments
 (0)