Skip to content

Commit 9f0186a

Browse files
committed
refactor(bfcl): rename example folder to 10_Edge_Agentic_Example; add run_accuracy.sh
Renames examples/10_BFCLv4_Example to examples/10_Edge_Agentic_Example to align with the MLPerf edge-agentic submission category name. Adds run_accuracy.sh — a single script that reproduces both single-turn and multi-turn reference accuracy numbers end-to-end with the exact validated parameters (sampling rates, temperature=0, seed=42, max-steps-per-turn=25). Updates README to lead with the one-liner quick-start referencing the script, fixes the install instructions to point to mlcommons/endpoints (not the fork), adds --seed and --max-steps-per-turn to the Step 3 MT snippet, and corrects the internal path reference in online_agentic_coding_perf.yaml.
1 parent 09602f8 commit 9f0186a

5 files changed

Lines changed: 1901 additions & 23 deletions

File tree

examples/10_BFCLv4_Example/README.md renamed to examples/10_Edge_Agentic_Example/README.md

Lines changed: 27 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,19 @@
1-
# BFCL v4 Accuracy Benchmarking
1+
# Edge-Agentic (BFCL v4) Accuracy Benchmarking
2+
3+
## Quick start
4+
5+
To reproduce all reference accuracy numbers (~2.5 h on an edge device), set
6+
your model name and endpoint URL, then run:
7+
8+
```bash
9+
MODEL=Qwen3.6-27B-Q4_K_M ENDPOINT=http://localhost:8080 bash run_accuracy.sh
10+
```
11+
12+
`run_accuracy.sh` runs both single-turn and multi-turn phases end-to-end with
13+
the exact validated parameters. See the steps below if you prefer to run each
14+
phase individually or need to customise the configuration.
15+
16+
---
217

318
## What is this?
419

@@ -98,34 +113,21 @@ docker run --rm -it --gpus all \
98113

99114
---
100115

101-
## Step 1 — Install from the PRs
102-
103-
This feature spans two pull requests. Both must be present:
104-
105-
| PR | Branch | What it does |
106-
| --- | --- | --- |
107-
| [PR #1](https://github.com/Palanivelg/endpoints/pull/1) | `chore/relax-numpy-pin` | Relaxes the `numpy` pin so `bfcl-eval` can install alongside the project |
108-
| [PR #2](https://github.com/Palanivelg/endpoints/pull/2) | `feat/bfcl-v4-combined` | Full BFCL v4 single-turn + multi-turn accuracy integration |
109-
110-
**PR #1 is a prerequisite for PR #2.** Without it, `pip install -e ".[bfcl]"`
111-
fails with a numpy version conflict.
116+
## Step 1 — Install
112117

113118
```bash
114-
# 1. Clone the fork that contains both PRs
115-
git clone https://github.com/Palanivelg/endpoints.git
119+
# 1. Clone the repo
120+
git clone https://github.com/mlcommons/endpoints.git
116121
cd endpoints
117122

118-
# 2. Check out the BFCL v4 branch (PR #2; already includes the PR #1 change)
119-
git checkout feat/bfcl-v4-combined
120-
121-
# 3. Install the package with the [bfcl] scoring extra
123+
# 2. Install the package with the [bfcl] scoring extra
122124
pip install -e ".[bfcl]"
123125

124-
# 4. Confirm the install resolved without conflict
126+
# 3. Confirm the install resolved without conflict
125127
pip show bfcl-eval numpy | grep -E "^(Name|Version)"
126-
# Expected: bfcl-eval present, numpy >= 1.26.4 (bfcl-eval may cap it < 2)
128+
# Expected: bfcl-eval present, numpy >= 1.26.4
127129

128-
# 5. Confirm the CLI is available
130+
# 4. Confirm the CLI is available
129131
inference-endpoint --help
130132
```
131133

@@ -138,8 +140,8 @@ Single-turn covers three BFCL v4 categories: `non_live`, `live`, and
138140
The YAML config in this directory has the sampling rates pre-configured.
139141

140142
```bash
141-
# Run from the examples/10_BFCLv4_Example/ directory
142-
cd examples/10_BFCLv4_Example/
143+
# Run from the examples/10_Edge_Agentic_Example/ directory
144+
cd examples/10_Edge_Agentic_Example/
143145

144146
inference-endpoint benchmark from-config \
145147
--config offline_bfcl_v4_single_turn.yaml \
@@ -180,6 +182,8 @@ python -m inference_endpoint.evaluation.bfcl_v4_multi_turn_cli \
180182
--model Qwen3.6-27B-Q4_K_M \
181183
--sample-pct 3 \
182184
--temperature 0 \
185+
--seed 42 \
186+
--max-steps-per-turn 25 \
183187
--report-dir results/bfcl_v4_multi_turn/
184188
```
185189

0 commit comments

Comments
 (0)