Skip to content

Commit 44f0aa4

Browse files
committed
fix: use cross for aarch64-linux release build, update README benchmarks
1 parent 3a1e41a commit 44f0aa4

2 files changed

Lines changed: 48 additions & 41 deletions

File tree

.github/workflows/release.yml

Lines changed: 15 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,21 @@ permissions:
1111
jobs:
1212
build:
1313
strategy:
14+
fail-fast: false
1415
matrix:
1516
include:
1617
- target: x86_64-unknown-linux-gnu
1718
os: ubuntu-latest
19+
use_cross: false
1820
- target: aarch64-unknown-linux-gnu
1921
os: ubuntu-latest
22+
use_cross: true
2023
- target: x86_64-apple-darwin
2124
os: macos-latest
25+
use_cross: false
2226
- target: aarch64-apple-darwin
2327
os: macos-latest
28+
use_cross: false
2429

2530
runs-on: ${{ matrix.os }}
2631

@@ -32,24 +37,16 @@ jobs:
3237
with:
3338
targets: ${{ matrix.target }}
3439

35-
- name: Install cross-compilation tools
36-
if: matrix.target == 'aarch64-unknown-linux-gnu'
37-
run: |
38-
sudo dpkg --add-architecture arm64
39-
# Add arm64 package sources
40-
sudo sed -i 's/^deb /deb [arch=amd64] /' /etc/apt/sources.list
41-
echo "deb [arch=arm64] http://ports.ubuntu.com/ $(lsb_release -cs) main restricted universe multiverse" | sudo tee /etc/apt/sources.list.d/arm64.list
42-
echo "deb [arch=arm64] http://ports.ubuntu.com/ $(lsb_release -cs)-updates main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/arm64.list
43-
sudo apt-get update
44-
sudo apt-get install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu libssl-dev:arm64 pkg-config
45-
echo "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc" >> $GITHUB_ENV
46-
echo "PKG_CONFIG_SYSROOT_DIR=/usr/aarch64-linux-gnu" >> $GITHUB_ENV
47-
echo "PKG_CONFIG_PATH=/usr/lib/aarch64-linux-gnu/pkgconfig" >> $GITHUB_ENV
48-
echo "OPENSSL_DIR=/usr" >> $GITHUB_ENV
49-
echo "OPENSSL_LIB_DIR=/usr/lib/aarch64-linux-gnu" >> $GITHUB_ENV
50-
echo "OPENSSL_INCLUDE_DIR=/usr/include" >> $GITHUB_ENV
51-
52-
- name: Build
40+
- name: Install cross
41+
if: matrix.use_cross
42+
run: cargo install cross --git https://github.com/cross-rs/cross
43+
44+
- name: Build (cross)
45+
if: matrix.use_cross
46+
run: cross build --release --target ${{ matrix.target }} -p ctxgraph-cli
47+
48+
- name: Build (cargo)
49+
if: "!matrix.use_cross"
5350
run: cargo build --release --target ${{ matrix.target }} -p ctxgraph-cli
5451

5552
- name: Package

README.md

Lines changed: 33 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -18,16 +18,18 @@ When someone (or an AI agent) asks *"why did we do X?"*, ctxgraph traverses the
1818

1919
Every context graph tool today requires heavy infrastructure and sends your data to OpenAI. ctxgraph doesn't.
2020

21-
We benchmarked ctxgraph against [Graphiti](https://github.com/getzep/graphiti) (by Zep) on the same 50 software-engineering episodes. ctxgraph extracts higher-quality relations using only local ONNX models — no API calls at all.
21+
We benchmarked ctxgraph against [Graphiti](https://github.com/getzep/graphiti) (by Zep) on the same 50 software-engineering episodes. ctxgraph extracts higher-quality entities and relations using only local ONNX models — no API calls at all.
2222

2323
### Extraction Quality (50-episode benchmark)
2424

25-
| System | Relation F1 | API Calls | Cost/Episode | Latency |
26-
|---|---|---|---|---|
27-
| **ctxgraph** (local-only) | **0.520** | 0 | $0.00 | 330ms |
28-
| Graphiti (gpt-4o) | 0.315 | 6+/episode | ~$0.01 | 12,700ms |
25+
| System | Entity F1 | Relation F1 | Combined F1 | API Calls | Cost | Latency |
26+
|---|---|---|---|---|---|---|
27+
| **ctxgraph** (local-only) | **0.837** | **0.763** | **0.800** | 0 | $0.00 | ~40ms/ep |
28+
| Graphiti (gpt-4o) | 0.570 | 0.104\* | 0.337 | ~200+ | ~$2-5 | ~10s/ep |
2929

30-
ctxgraph achieves **1.65x higher extraction quality** than Graphiti while being **38x faster** and **100% free**.
30+
\*Graphiti's free-form relations mapped to ctxgraph's taxonomy using generous keyword heuristics.
31+
32+
ctxgraph achieves **2.4x higher combined F1** than Graphiti while being **250x faster** and **100% free**.
3133

3234
### Infrastructure
3335

@@ -45,9 +47,15 @@ ctxgraph achieves **1.65x higher extraction quality** than Graphiti while being
4547

4648
### Why Graphiti Scores Lower
4749

48-
Graphiti makes 6+ GPT calls per episode (entity extraction, deduplication, relation extraction, contradiction detection, summarization, community detection). Despite this, it produces free-form relation names like `COMMUNICATES_ENCRYPTED_WITH` and `PREVENTS_CASCADING_FAILURES_WHEN_DOWN` that don't map cleanly to typed relations. It also frequently gets entity directions wrong (e.g., `Helm:depends_on:Kubernetes` instead of `Kubernetes:depends_on:Helm`).
50+
Graphiti makes 6+ GPT-4o calls per episode (entity extraction, deduplication, relation extraction, contradiction detection, summarization, community detection). Despite this:
51+
52+
- **Entity names are verbose**: Graphiti extracts `"primary Postgres cluster"` instead of `"Postgres"`, `"legacy SOAP endpoint in UserService"` instead of `"SOAP endpoint"`. Semantically correct, but doesn't match canonical names.
53+
- **Relations are free-form**: Produces verbs like `COMMUNICATES_ENCRYPTED_WITH` and `PREVENTS_CASCADING_FAILURES_WHEN_DOWN` that don't map to a typed taxonomy. Even with generous keyword mapping, only 10/50 episodes produce any matching relations.
54+
- **Different decomposition**: "Migrate from Redis to Postgres" becomes `(AuthService, CONNECTS_TO, primary Postgres cluster)` instead of `(Postgres, replaced, Redis)` + `(AuthService, depends_on, Postgres)`.
4955

50-
ctxgraph uses domain-specific heuristics trained on software engineering patterns — keyword matching, proximity scoring, coreference resolution, and schema-aware type validation — that encode more practical knowledge than GPT can infer from generic prompts.
56+
ctxgraph uses domain-specific heuristics for software engineering patterns — keyword matching, proximity scoring, coreference resolution, and schema-aware type validation — that produce structured, queryable knowledge without any API calls.
57+
58+
See [docs/benchmark.md](docs/benchmark.md) for the full comparison methodology and per-episode results.
5159

5260
## Quick Start
5361

@@ -149,8 +157,8 @@ Your App / CLI / AI Agent
149157
+---------------------------------+
150158
| Extraction |
151159
| GLiNER v2.1 (ONNX) - local |
152-
| Entities: 0.845 F1 |
153-
| Relations: 0.520 F1 |
160+
| Entities: 0.837 F1 |
161+
| Relations: 0.763 F1 |
154162
| Temporal: date/time parsing |
155163
+---------------------------------+
156164
|
@@ -312,30 +320,32 @@ cargo test --test benchmark_test -- --ignored --nocapture
312320

313321
Requires ONNX models (`ctxgraph models download`).
314322

315-
### Results (GLiNER v2.1 INT8, fully local)
323+
### Results (v0.6.0 — GLiNER v2.1 INT8, fully local)
316324

317325
| Metric | Score |
318326
|---|---|
319-
| Entity F1 (name+type) | 0.845 |
320-
| Entity F1 (name only) | 0.903 |
321-
| Relation F1 | 0.520 |
322-
| Combined F1 | 0.682 |
323-
| Latency | 330ms/episode |
327+
| Entity F1 | 0.837 |
328+
| Relation F1 | 0.763 |
329+
| **Combined F1** | **0.800** |
330+
| Latency | ~40ms/episode |
324331

325332
### Comparison with Graphiti
326333

327-
Both systems were tested on the same 50 episodes with the same ground truth.
334+
Both systems were tested on the same 50 episodes with identical ground truth.
328335

329-
| | ctxgraph | Graphiti |
336+
| | ctxgraph | Graphiti (gpt-4o) |
330337
|---|---|---|
331-
| Relation F1 | **0.520** | 0.315 |
332-
| API calls | 0 | 6+/episode (GPT-4o) |
333-
| Cost | $0 | ~$0.50 for 50 episodes |
334-
| Total time | 16.7s | 635s |
335-
| Per episode | 330ms | 12,700ms |
338+
| Entity F1 | **0.837** | 0.570 |
339+
| Relation F1 | **0.763** | 0.104\* |
340+
| Combined F1 | **0.800** | 0.337 |
341+
| API calls | 0 | ~200+ |
342+
| Cost | $0 | ~$2-5 |
343+
| Per episode | ~40ms | ~10s |
336344
| Infrastructure | SQLite | Neo4j (Docker) |
337345
| Privacy | 100% local | Data sent to OpenAI |
338346

347+
\*With generous semantic mapping of Graphiti's free-form relations to ctxgraph's taxonomy.
348+
339349
## Project Structure
340350

341351
```

0 commit comments

Comments
 (0)