feat(bfcl): add BFCL v4 edge-agentic accuracy + performance integration#346
Merged
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request adds support for Berkeley Function Calling Leaderboard (BFCL) v4 accuracy benchmarking, introducing single-turn and multi-turn evaluation pipelines, datasets, and adapters, along with sequential sample ordering for deterministic evaluation. The code review feedback identifies three key improvements: moving the tool_calls parsing loop inside the try-except block in bfcl_v4_execution.py to prevent crashes on invalid JSON, conditionally including tools and tool_choice in the request payload in bfcl_v4_multi_turn_runner.py to avoid API errors when no tools are present, and guarding the n_repeats calculation in bfcl_v4_scorer.py to ensure it does not evaluate to zero if some samples fail.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Palanivelg
changed the title
Integrate Berkeley Function Calling Leaderboard (BFCL) v4 evaluation
into the accuracy pipeline. Covers both single-turn function-calling
subsets (non_live, live, hallucination) and the agentic multi-turn
subsets (multi_turn_base + variants), validated against evalscope.
Single-turn (drop-in scorer):
- BFCLv4 predefined dataset (categories=[non_live|live|hallucination],
configurable sample_pct) plus a default preset.
- BFCLv4Scorer wires bfcl-eval's ast_checker into the standard
accuracy phase via the existing scorer registry.
- FunctionCallExtractor: normalizes native tool_calls JSON, JSON
arrays of function-call objects, and text-form function calls
into the canonical BFCL input format.
- openai_msgspec_adapter now:
* passes tool_choice through,
* emits tool_calls verbatim as output_text for scoring,
* coerces whole-number temperatures (vLLM strictness),
* makes max_completion_tokens optional and uses a permissive
ColumnFilter so messages+tools datasets pass through.
Multi-turn (agentic loop, outside the standard scorer):
- BFCLExecutionBridge: parses tool_calls, executes them locally
against bfcl-eval's class instances, and constructs the tool
response messages for the next turn.
- BFCLMultiTurnRunner: drives the per-entry agentic conversation
via httpx (bounded by max_steps_per_turn / timeout_s).
- BFCLv4MultiTurnScorer: invokes bfcl-eval's multi_turn_checker.
- bfcl_v4_multi_turn_cli: standalone CLI for the multi-turn flow.
Supporting plumbing:
- SequentialSampleOrder + `sequential=` flag on create_sample_order,
used by accuracy phases so ordering matches reference runs.
- BenchmarkConfig.Dataset.params: dict for dataset-specific kwargs
(e.g. categories, sample_pct) plumbed through DataLoaderFactory.
- ScorerMethod.BFCL_V4.
- Dataset.load() preserves user-provided ColumnFilters when the
adapter would otherwise inject a conflicting one.
- `--accuracy-only` benchmark flag: skip the performance phase
entirely (forces num_workers=1, max_connections=1 for
deterministic per-sample ordering).
Optional dep: `pip install -e ".[bfcl]"` (`bfcl-eval==2026.3.23`).
The top-level numpy pin must be relaxed to `>=1.26.4` because
bfcl-eval hard-pins `numpy==1.26.4` — shipped as a separate
prerequisite commit on the chore/relax-numpy-pin branch.
Validation (Qwen3.6-27B-Q4_K_M, temperature=0):
- Single-turn live (10%): ~82% accuracy.
- Multi-turn base (full 200): 140/200 = 70.00%, exact parity
(100% match, 0 mismatches) with evalscope on identical inputs.
Add a committed config that reproduces the validated <3h BFCL v4 accuracy
run on an embedded device (Thor).
Dataset (BFCLv4.generate):
- category_sample_pct: per-category sampling rates (e.g. non_live 20 /
live 10 / hallucination 5), resolved per subset via SUBSET_TO_CATEGORY.
- subset_floor: subsets whose TOTAL size is <= floor are taken in full,
preventing tiny subsets (live_parallel=16, live_parallel_multiple=24)
from collapsing to one or two noisy samples. Selection stays
deterministic (head(n)). Plain sample_pct behavior is unchanged.
Multi-turn CLI:
- --sample-pct: deterministic per-subset sub-sampling so a 3% (~24 entry)
multi-turn run is reproducible. Defaults to all entries.
examples/10_BFCLv4_Example/:
- offline_bfcl_v4_single_turn.yaml: single-turn (non_live + live +
hallucination) accuracy config, run with --accuracy-only.
- README.md: documents the two run paths (single-turn YAML vs multi-turn
CLI), the per-category sampling + floor, and the ~2h49m Thor budget.
The example and docstring referenced a specific device; reword to the generic "edge device" since the <3h budget applies to embedded targets broadly.
In --accuracy-only runs there is no performance phase, so ctx.rt_settings is None. _run_benchmark_async read ctx.rt_settings.max_duration_ms unconditionally, raising AttributeError at session setup. The global timeout only applies to the performance phase, so default max_duration_ms to None when rt_settings is absent.
--accuracy-only forces a single connection for deterministic sample ordering, which serializes the offline MAX_THROUGHPUT burst. For large accuracy datasets the sequential processing time exceeds the hardcoded 240 s drain cap, so the phase aborted mid-run dropping in-flight samples. Make drain_timeout a per-phase field defaulting to 240.0 (performance phases unchanged). Accuracy phases pass None to drain unbounded, since every sample must complete; a dropped transport still unblocks the wait via the _receive_responses close path. Re-check inflight after clear() to close a completion/clear race on the unbounded path.
The msgspec adapter serialized tool_calls into `output` AND kept them in the
structured `tool_calls` field. TextModelOutput.__str__ then appended the
tool_calls a second time, producing duplicated, malformed JSON
(`[{...}][{...}]`) that FunctionCallExtractor could not parse. This made every
single-turn function-calling subset score 0% (and gave hallucination subsets a
spurious 100%).
Keep `output` as the textual content only; the structured tool_calls field is
the single source serialized once by __str__. This matches the non-msgspec
OpenAI adapter and the streaming accumulator, which already did this.
Multi-turn is unaffected: it uses a separate httpx runner that reads structured
tool_calls from the raw response and never touches TextModelOutput.
…mbles
When a model emits a prose preamble alongside a tool call (e.g.
"To compute this, I'll call...\n[{...}]"), str(TextModelOutput) prefixes the
tool-call JSON with that text, so the function-call parser's json.loads fast
path fails and the sample scores 0.
Override BFCLv4Scorer.get_outputs to prefer the structured tool_calls field
when present (the function call is the answer; the prose is chatter), falling
back to the full string for plain-text responses such as hallucination
refusals. Verified deterministic across repeated fresh-server single-turn runs.
Some servers (e.g. trtllm-serve on edge devices) stall when tools are present but tool_choice is omitted, relying on a server default. Set tool_choice="auto" explicitly on each single-turn sample and pass it through the function_calling preset's ColumnFilter so it reaches the request payload. Multi-turn already sends tool_choice="auto" via its dedicated runner, so this only affects the single-turn path.
Add ModelParams.seed field and propagate it to the OpenAI wire format so runs can be made reproducible: - config/schema.py: add seed field to ModelParams - openai/types.py: add seed field to ChatCompletionRequest - openai/openai_adapter.py: include seed in metadata dict - openai/openai_msgspec_adapter.py: include seed in metadata dict and ChatCompletionRequest construction - evaluation/bfcl_v4_multi_turn_runner.py: accept seed param; inject payload["seed"] when set - evaluation/bfcl_v4_multi_turn_cli.py: expose --seed CLI arg and pass it to BFCLMultiTurnRunner - commands/benchmark/cli.py: expose --seed and --report-dir overrides on the from-config subcommand - tests: unit coverage for seed propagation in msgspec adapter, multi-turn runner, and from-config CLI
Expand examples/10_BFCLv4_Example/README.md: - Add a "Reproducing from the PRs" section explaining that PR #1 (numpy pin) is a prerequisite for PR #2 to install [bfcl] - Show how to check out and install from the branches - Document --seed flag for both single-turn (from-config) and multi-turn CLI paths - Replace placeholder accuracy numbers with confirmed Thor validation results (Qwen3.6-27B-Q4_K_M, temperature=0, 456 ST samples): non_live 86.98%, live 84.12%, hallucination 94.32%, overall 87.50% (both seed runs identical); MT base 70.00% (exact evalscope parity) - Add output file paths and a quick sanity-check script
Replace the terse reference doc with a numbered walkthrough that works for someone unfamiliar with the endpoints repo: - What is this / What is the endpoints repo - Step 0: prerequisites including a llama.cpp Docker quick-start - Step 1: install from the two PRs with conflict explanation - Step 2: run single-turn (with YAML config notes) - Step 3: run multi-turn - Step 4: verify results with one-liners - Seed reproducibility section - Reference results table (Thor, two seed runs, evalscope parity)
- Fix memory requirement: ~24 GB (not 16 GB) for the Q4 GGUF + KV cache
- Replace generic Docker quick-start with Thor-specific llama.cpp native
build instructions (Docker CUDA images don't target sm_110/aarch64-SBSA)
- Add x86_64 Docker quick-start in a collapsible details block
- Fix Step 4 result path: results.json under accuracy_scores key, not
a separate accuracy_scores.json; add report.txt note
- Add server-side determinism note (--seed 42, -np 1 on llama-server)
- Replace placeholder MT numbers with actual sampled-run Thor results:
multi_turn_base 66.67% (4/6), miss_func 33.33% (2/6),
miss_param 16.67% (1/6), long_context 66.67% (4/6),
overall 45.84% (24 entries) — identical across both seed runs
- Separate full multi_turn_base parity result (140/200 = 70.00%) into
its own subsection to avoid conflating sampled and full-set numbers
- Update wall-clock: ~82 min ST + ~64 min MT ≈ 2.4–2.5 h total
bfcl-eval's Qwen model handler transitively imports qwen_agent which requires soundfile; without it the import fails on Thor and any machine where soundfile is not already installed.
… run_accuracy.sh Renames examples/10_BFCLv4_Example to examples/10_Edge_Agentic_Example to align with the MLPerf edge-agentic submission category name. Adds run_accuracy.sh — a single script that reproduces both single-turn and multi-turn reference accuracy numbers end-to-end with the exact validated parameters (sampling rates, temperature=0, seed=42, max-steps-per-turn=25). Updates README to lead with the one-liner quick-start referencing the script, fixes the install instructions to point to mlcommons/endpoints (not the fork), adds --seed and --max-steps-per-turn to the Step 3 MT snippet, and corrects the internal path reference in online_agentic_coding_perf.yaml.
Palanivelg
force-pushed
the
feat/bfcl-v4-combined
branch
from
9f0186a to
37c74d2
Compare
- bfcl_v4_execution: move tool-call argument JSON parsing inside the try-except block so json.JSONDecodeError from malformed model output (common on small/quantized models) is caught and handled gracefully rather than crashing the evaluation run. - bfcl_v4_multi_turn_runner: only include tools/tool_choice in the request payload when the tools list is non-empty, avoiding 400 errors on endpoints that reject tool_choice without accompanying tools. - bfcl_v4_scorer: guard n_repeats with max(1, ...) so a partial run where fewer samples completed than num_samples() does not produce a zero divisor and incorrect reporting.
3 tasks
- Relax base numpy to >=1.26.4 so bfcl-eval's numpy==1.26.4 pin resolves; regenerate uv.lock. - Regenerate stale *_template_full.yaml config templates after schema change. - Fix mypy: annotate tool_calls/tool_call_ids and narrow Optional messages/tools in the multi-turn runner; mark BFCLv4Scorer.score override. - Isolate the bfcl extra via [tool.uv].conflicts and add patched filelock/virtualenv floors to the dev extra so bfcl-eval's filelock==3.20.0 pin no longer drags shared tooling deps into CVE versions (CVE-2025-68146, CVE-2026-22701, CVE-2026-22702).
The accuracy phase hardcoded drain_timeout=None, which ignored a user-configured DrainConfig.accuracy_timeout_s and failed test_configured_drain_timeouts_propagate_to_phases. accuracy_timeout_s already defaults to None (unbounded), so reading it preserves the unbounded default while honoring an explicit timeout.
10 tasks
from-config has no --model-params.name / --endpoint-config.endpoints overrides, so the script errored immediately. Render a temp YAML with MODEL/ENDPOINT substituted into the committed config (anchored on the "# set to your ..." comments) so the one-liner still works without editing the tracked file.
The combined perf+accuracy run loads a repo-root-relative performance dataset path, so it must be launched from the repo root; the README Step 5 and the config header comment previously said to cd into the example dir, which breaks dataset resolution. The inline-checker verify one-liner also read a top-level `valid` key that the scorer never writes (it emits score/turns/domains/per_turn) — derive validity from turns.missing instead.
Resolve conflicts: - config/schema.py: keep both ScorerMethod members (BFCL_V4 + LEGACY_MLPERF_DEEPSEEK_R1) - commands/benchmark/execute.py: keep BFCL score_breakdown() + entry storage, adopt main's richer completeness log line - AGENTS.md: union Key Components table (main's DeepSeek-R1 row + BFCL Compliance row) - config/templates/*_full.yaml: regenerated from schema
leopck
reviewed
Jul 2, 2026
Remove throughput/latency/runtime figures (tok/s, TTFT, TPOT, per-turn latency, ISL/OSL percentiles, wall-clock/runtime columns) that are hardware-specific and not the reference target. Retain accuracy numbers (86.23% overall, 87.96% normalized, per-category, IoU 0.6335) and the reasoning ON vs OFF comparison used to justify running with reasoning off.
Address PR mlcommons#346 review: move the function-body imports (bfcl_eval and the first-party bfcl_v4_scorer/bfcl_v4_execution/scoring modules) to the top of the file per AGENTS.md no-lazy-imports rule. The smoke script is run in an env with the `bfcl` extra installed, so a top-level import is the intended behavior.
arekay-nv
reviewed
Jul 2, 2026
arekay-nv
left a comment
arekay-nv
left a comment
Collaborator
There was a problem hiding this comment.
Review Council — Multi-AI Code Review
Reviewed by Codex + Claude | Depth: thorough. 15 verified findings posted as inline comments (summary comment follows).
Collaborator
Review Council — Multi-AI Code ReviewReviewed by: Codex + Claude | Depth: thorough Found 15 issues across 12 files (17 candidates; 2 dropped in synthesis — numpy floor-pin trade-off is already documented in-code with 🔴 Must Fix (critical/high)Issues that will cause incorrect behavior in normal usage.
🟡 Should Fix (medium)Real issues that trigger under specific conditions or flaws that will compound.
🔵 Consider (low)Valid improvements that could be follow-ups.
|
arekay-nv
reviewed
Jul 6, 2026
arekay-nv
reviewed
Jul 6, 2026
arekay-nv
reviewed
Jul 6, 2026
…sion Address PR mlcommons#346 review: _percentile("99.5") previously fell back to perc.get("99") and silently returned the P99 bucket when the fractional percentile was absent. Match on float value instead so a missing "99.5" returns None rather than P99. Add regression tests.
…single-stream, docs) - extractor: guard non-dict `function` value in _try_parse_tool_calls_json so a malformed tool_calls array falls through to None instead of raising an uncaught AttributeError that aborts scoring for the whole run. - execute: accuracy-only setup now also normalizes load_pattern.target_concurrency to 1 (the compliance single_stream gate reads it), matching the num_workers/max_connections override already baked in. - README: fix the single-turn verify snippets to read ['breakdown']['overall_accuracy'] (score is a scalar float; the dict lives under breakdown); document the single-turn vs multi-turn result-shape asymmetry; correct the seed example (from-config takes model_params.seed from YAML, not --seed). - Add regression tests for the extractor guard and the accuracy-only single-stream normalization.
This was referenced Jul 6, 2026
Open
- compliance checker: FAIL explicitly when an accuracy score is present but no metric intersects the model's golden/factor tables (was a silent PASS). - session: stop_current_phase() now sets the drain event and _drain_inflight observes the per-phase stop, so an unbounded (performance_timeout_s: null) drain can't hang past the max_duration cap on a stuck in-flight response. - results_plots: slice histogram buckets and counts to a common length so a mismatch can't raise inside matplotlib's bar(). - openai_msgspec adapter: drop the dead max_new_tokens guard (the field always has a default); always emit max_completion_tokens, matching OpenAIAdapter. - bfcl_v4 multi_turn: remove unused get_tools_for_turn and excluded_function, dedupe MULTI_TURN_SUBSETS (import from package), drop unused MULTI_TURN_CATEGORY_MAP; runner imports DEFAULT_MAX_STEPS_PER_TURN from bfcl_v4_execution instead of redefining it. - Add regression tests for each fix.
…and test items
Bugs:
- bfcl_v4 scorer: score live_relevance with relevance semantics (correct when
the model DOES call a tool) via _score_relevance, instead of routing it
through the empty-ground-truth AST path that awarded credit for NOT calling
(inverted). Corrects both the reported per-subset value and its contribution
to overall_accuracy. NOTE: this changes the reference overall_accuracy; the
frozen golden should be recomputed (normalized_single_turn_score is
unaffected — live_relevance is excluded from category aggregates).
- multi_turn runner: a structurally malformed 200 body (missing choices/message,
{"error": ...} payload) now raises MalformedResponseError and force-terminates
the entry, matching the transport-error path, instead of being scored as a
degraded-but-normal completion.
- execution bridge: cleanup() now evicts bfcl-eval's cached simulated-class
instances from the multi_turn_utils module globals (same key + sanitization
bfcl-eval uses), so re-running an entry_id in-process starts from
initial_config and a full run no longer leaks thousands of instances.
Security:
- execution bridge: validate each tool call name against the entry's declared
tools before handing calls to bfcl-eval's eval()-based executor, so an
untrusted endpoint cannot inject a name like __import__('os').system that
bypasses upstream's denylist.
Tests:
- New BFCLv4MultiTurnScorer.score() coverage (force-terminated, turn-count
mismatch, checker-exception, valid/invalid bookkeeping, unweighted-mean
aggregation, empty-results zero, string-typed subset_scores).
- New BFCLv4.generate() selection coverage (unknown-category raise, all-multi-
turn-filtered empty frame, sample-pct validation, subset_floor vs pct,
_resolve_subset_pct precedence, max_samples truncation, cached-parquet SHA
re-verification both ways).
- Regression tests for live_relevance scoring, malformed-body force-terminate,
undeclared-tool-name filtering, and instance-cache eviction.
Contributor
Author
Review status updateThanks for the thorough review. All actionable threads are now fixed and resolved (with regression tests; Group 1 — no code change, left open for reviewer sign-off (2)These were addressed in-thread and don't carry a code change, so I've left them for you to resolve if you agree:
Group 2 — large design refactors, tracked as separate issues (2)Both are meaningful refactors beyond this PR's scope and are filed as follow-ups:
Everything else — doneAll remaining review items — the correctness bugs, the eval security hardening, dead-code/dedupe cleanups, and the test-coverage gaps — are fixed and their threads resolved (tracked under #391 / #392 / #393 / #394). One heads-up on the |
viraatc
requested changes
Jul 6, 2026
viraatc
reviewed
Jul 6, 2026
This was referenced Jul 6, 2026
leopck
reviewed
Jul 6, 2026
leopck
reviewed
Jul 6, 2026
…SWE-bench attribution - publish_submission: return a non-zero exit when a requested run copies zero parseable artifacts (wrong/empty path or a failed run), instead of printing "(none!)" and exiting 0 with an incomplete submission tree. Add regression tests for the empty-run and populated-run exit codes. - ATTRIBUTION: add SWE-bench (MIT, Copyright (c) 2023 Princeton NLP) as the source of the edge-agentic performance dataset (examples/11_Edge_Agentic_Example/agentic_coding_2.5h.jsonl), which replays SWE-bench task instances (e.g. django__django-16899).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Integrates the edge-agentic example (BFCL-v4 as the accuracy set): single-turn + multi-turn pipelines, datasets, adapters, and a reproducible run script. See
examples/10_Edge_Agentic_Example/README.md.Type of change
Related issues
Self-contained: this PR includes the numpy relaxation (
numpy>=1.26.4) required bybfcl-eval(which hard-pinsnumpy==1.26.4), so it no longer depends on a separate prerequisite PR. The[bfcl]extra is isolated via[tool.uv].conflictsso its old pins don't constrain the shared tooling deps. This supersedes #345 (closed as redundant).Testing
Checklist