Add greybox support by leonbett · Pull Request #764 · fandango-fuzzer/fandango

leonbett · 2026-01-13T22:09:10Z

Depends on #783

riesentoaster · 2026-01-13T23:22:36Z

(Regarding the lock files, if you haven't seen it: check the contribution guide in the docs in the repo. It isn't published yet, so the online docs aren't up to date yet. Section 5 and the pre-commit section have the necessary information for updating the lockfiles)

github-actions · 2026-01-15T00:24:50Z

Bencher Report

Branch	greybox
Testbed	ubuntu-latest

Click to view all benchmark results

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
tests/test_benchmarks.py::test_generate_with_single_hard_constraint	📈 view plot 🚷 view threshold	1,939.76 ms (+9.17%) Baseline: 1,776.79 ms	2,106.85 ms (92.07%)
tests/test_benchmarks.py::test_generate_with_single_soft_constraint	📈 view plot 🚷 view threshold	51,926.73 ms (-0.29%) Baseline: 52,076.37 ms	81,793.96 ms (63.48%)
tests/test_benchmarks.py::test_init_fandango	📈 view plot 🚷 view threshold	154.34 ms (+2.98%) Baseline: 149.87 ms	160.43 ms (96.20%)
tests/test_benchmarks.py::test_parse_spec	📈 view plot 🚷 view threshold	154.29 ms (+1.07%) Baseline: 152.66 ms	159.95 ms (96.46%)

🐰 View full continuous benchmarking report in Bencher

riesentoaster

I love how little change is actually needed for this to work. And I love the number/depth of tests you add.

I do have some initial comments, will do a second, more thorough pass once these bigger things are cleared up/we agree not to change them.

.github/workflows/python-tests.yml

src/fandango/cli/parser.py

src/fandango/evolution/algorithm.py

src/fandango/evolution/evaluation.py

src/fandango/experimental/execution/static_analysis.py

leonbett · 2026-01-23T12:20:31Z

Should I move the leonbett/fcc repo under fandango-fuzzer ?

riesentoaster · 2026-02-25T11:06:27Z

Should I move the leonbett/fcc repo under fandango-fuzzer ?

I think that makes sense, yes.

format test . Update uv.lock type annotations fix tests install llvm debug install llvm debug . debug macos . . no fcc for windows try to improve benchmark Introduce FCC class ran uv sync --locked --all-extras no exception refactor analysis.py

src/fandango/api.py

riesentoaster · 2026-03-27T14:09:39Z

I added a bunch of other changes for broken things:

The docs now also build on macOS, including in the CI
The docs error handling is now much better, including printing of the specific error messages (imagine that!)
Minor optimizations and cleanup all around

Copilot

Pull request overview

Adds “greybox” execution-feedback support by integrating FCC-based static/dynamic analysis into Fandango constraints, plus end-to-end tests and build tooling to compile FCC and run the new integration tests.

Changes:

Introduce FCC-backed StaticAnalysis/DynamicAnalysis and expose DynamicAnalysis(...) as a constraint callable when --fcc is enabled.
Extend CLI/API plumbing to support --fcc and pass PUT/args into the evolutionary evaluation pipeline.
Add execution-feedback integration tests and C fixtures (reachability, heap allocations, execution-path length).

Reviewed changes

Copilot reviewed 82 out of 85 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/test_execution_feedback.py	New integration tests for execution feedback objectives via FCC + dynamic analysis traces
tests/resources/execution_feedback/.gitignore	Ignore generated bitcode artifacts in execution-feedback fixtures
tests/resources/execution_feedback/reachability_1/Makefile	Build fixture with `fcc` into `build/`
tests/resources/execution_feedback/reachability_1/specifications/reach_bb_1.fan	Spec minimizing distance to a target basic block
tests/resources/execution_feedback/reachability_1/specifications/reach_bb_2.fan	Alternate BB-distance spec with recursive grammar
tests/resources/execution_feedback/reachability_1/specifications/reach_function_1.fan	Spec minimizing distance to a target function
tests/resources/execution_feedback/reachability_1/src/f.h	Fixture header declaring f1–f9
tests/resources/execution_feedback/reachability_1/src/f1.c	Fixture call-chain function f1
tests/resources/execution_feedback/reachability_1/src/f2.c	Fixture call-chain function f2
tests/resources/execution_feedback/reachability_1/src/f3.c	Fixture call-chain function f3
tests/resources/execution_feedback/reachability_1/src/f4.c	Fixture call-chain function f4
tests/resources/execution_feedback/reachability_1/src/f5.c	Fixture call-chain function f5
tests/resources/execution_feedback/reachability_1/src/f6.c	Fixture call-chain function f6
tests/resources/execution_feedback/reachability_1/src/f7.c	Fixture call-chain function f7
tests/resources/execution_feedback/reachability_1/src/f8.c	Fixture call-chain function f8
tests/resources/execution_feedback/reachability_1/src/f9.c	Fixture target function f9 (“needle”)
tests/resources/execution_feedback/reachability_1/src/main.c	Fixture harness reading input from a file and invoking f1
tests/resources/execution_feedback/reachability_2/Makefile	Build fixture with `fcc` into `build/`
tests/resources/execution_feedback/reachability_2/specifications/reach_bb_1.fan	Spec minimizing distance to a target basic block
tests/resources/execution_feedback/reachability_2/specifications/reach_bb_2.fan	Alternate BB-distance spec with recursive grammar
tests/resources/execution_feedback/reachability_2/specifications/reach_function_1.fan	Spec minimizing distance to a target function
tests/resources/execution_feedback/reachability_2/src/f.h	Fixture header declaring f1–f9
tests/resources/execution_feedback/reachability_2/src/f1.c	Fixture call-chain function f1
tests/resources/execution_feedback/reachability_2/src/f2.c	Fixture call-chain function f2
tests/resources/execution_feedback/reachability_2/src/f3.c	Fixture call-chain function f3
tests/resources/execution_feedback/reachability_2/src/f4.c	Fixture call-chain function f4
tests/resources/execution_feedback/reachability_2/src/f5.c	Fixture call-chain function f5
tests/resources/execution_feedback/reachability_2/src/f6.c	Fixture call-chain function f6
tests/resources/execution_feedback/reachability_2/src/f7.c	Fixture call-chain function f7
tests/resources/execution_feedback/reachability_2/src/f8.c	Fixture call-chain function f8
tests/resources/execution_feedback/reachability_2/src/f9.c	Fixture target function f9 (“needle”)
tests/resources/execution_feedback/reachability_2/src/main.c	Fixture harness reading input from a file and invoking f1
tests/resources/execution_feedback/reachability_3/Makefile	Build fixture with `fcc` into `build/`
tests/resources/execution_feedback/reachability_3/specifications/reach_bb_1.fan	Spec minimizing distance to a target basic block
tests/resources/execution_feedback/reachability_3/specifications/reach_bb_2.fan	Alternate BB-distance spec with recursive grammar
tests/resources/execution_feedback/reachability_3/specifications/reach_function_1.fan	Spec minimizing distance to a target function
tests/resources/execution_feedback/reachability_3/src/f.h	Fixture header declaring f1–f9
tests/resources/execution_feedback/reachability_3/src/f1.c	Fixture call-chain function f1
tests/resources/execution_feedback/reachability_3/src/f2.c	Fixture call-chain function f2
tests/resources/execution_feedback/reachability_3/src/f3.c	Fixture call-chain function f3
tests/resources/execution_feedback/reachability_3/src/f4.c	Fixture call-chain function f4
tests/resources/execution_feedback/reachability_3/src/f5.c	Fixture call-chain function f5
tests/resources/execution_feedback/reachability_3/src/f6.c	Fixture call-chain function f6
tests/resources/execution_feedback/reachability_3/src/f7.c	Fixture call-chain function f7
tests/resources/execution_feedback/reachability_3/src/f8.c	Fixture call-chain function f8
tests/resources/execution_feedback/reachability_3/src/f9.c	Fixture target function f9 (“needle”)
tests/resources/execution_feedback/reachability_3/src/main.c	Fixture harness reading input from a file and invoking f1
tests/resources/execution_feedback/mem_heap_allocations_1/Makefile	Build heap-allocation fixtures with `fcc`
tests/resources/execution_feedback/mem_heap_allocations_1/specifications/maximize_malloc.fan	Spec maximizing heap-allocated bytes
tests/resources/execution_feedback/mem_heap_allocations_1/src/test_input_dependent_malloc.c	Fixture where heap allocation depends on input digits
tests/resources/execution_feedback/mem_heap_allocations_1/src/test_malloc_calloc.c	Fixture with malloc/calloc loops for heap allocation tracing
tests/resources/execution_feedback/mem_heap_allocations_2/Makefile	Build heap-allocation fixtures with `fcc`
tests/resources/execution_feedback/mem_heap_allocations_2/specifications/maximize_malloc.fan	Spec maximizing heap-allocated bytes
tests/resources/execution_feedback/mem_heap_allocations_2/src/test_input_dependent_malloc.c	Fixture where heap allocation depends on input digits
tests/resources/execution_feedback/mem_heap_allocations_2/src/test_malloc_calloc.c	Fixture with malloc/calloc loops for heap allocation tracing
tests/resources/execution_feedback/execution_path_length_1/Makefile	Build execution-path-length fixtures with `fcc`
tests/resources/execution_feedback/execution_path_length_1/specifications/maximize_execution_path_length.fan	Spec maximizing execution path length
tests/resources/execution_feedback/execution_path_length_1/src/test_input_dependent.c	Fixture with input-dependent loop count
tests/resources/execution_feedback/execution_path_length_1/src/test_simple.c	Fixture with fixed loop count
tests/resources/execution_feedback/execution_path_length_2/Makefile	Build execution-path-length fixtures with `fcc`
tests/resources/execution_feedback/execution_path_length_2/specifications/maximize_execution_path_length.fan	Spec maximizing execution path length
tests/resources/execution_feedback/execution_path_length_2/src/test_input_dependent.c	Fixture with input-dependent loop count
tests/resources/execution_feedback/execution_path_length_2/src/test_simple.c	Fixture with fixed loop count
tests/resources/execution_feedback/execution_path_length_3/Makefile	Build execution-path-length fixtures with `fcc`
tests/resources/execution_feedback/execution_path_length_3/specifications/maximize_execution_path_length.fan	Spec maximizing execution path length
tests/resources/execution_feedback/execution_path_length_3/src/test_input_dependent.c	Fixture with input-dependent loop count
tests/resources/execution_feedback/execution_path_length_3/src/test_simple.c	Fixture with fixed loop count
src/fandango/execution/trace_types.py	New type aliases for CFG/CG and distance maps
src/fandango/execution/static_analysis.py	New static analysis loader + distance computations
src/fandango/execution/dynamic_analysis.py	New dynamic trace wrapper + objective functions and PUT runner
src/fandango/execution/fcc.py	New FCC orchestrator wiring static + dynamic analysis
src/fandango/execution/init.py	Package init for execution module
src/fandango/evolution/evaluation.py	Inject `DynamicAnalysis` callable into constraints when `--fcc` is enabled
src/fandango/evolution/algorithm.py	Thread FCC settings (use_fcc/put/put_args) into evaluator
src/fandango/constraints/soft.py	Replace unbounded dict cache with LRU cache
src/fandango/cli/utils.py	Pass `use_fcc` and (conditionally) PUT/args into Fandango settings
src/fandango/cli/parser.py	Add `--fcc` CLI flag
src/fandango/cli/output.py	Skip running `test_command` output hook when `--fcc` is enabled
src/fandango/api.py	Adjust runtime-end sanitization for FCC mode in infinite runs
pyproject.toml	Add `cachetools` + typing stubs; update mypy excludes for `fcc/`
docs/Constraints.md	Update example command output (reduced generations in docs snippet)
Makefile	Install/build FCC locally in `make test-tools`; update docs error checking
.gitignore	Ignore `fcc/` clone directory
.github/workflows/build-docs.yml	Run docs build on Ubuntu + macOS; conditional sudo and artifact upload

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/fandango/experimental/execution/fcc.py

src/fandango/experimental/execution/static_analysis.py

src/fandango/cli/utils.py

src/fandango/experimental/execution/static_analysis.py

src/fandango/evolution/evaluation.py

Makefile

src/fandango/experimental/execution/static_analysis.py

src/fandango/experimental/execution/dynamic_analysis.py

tests/test_execution_feedback.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

riesentoaster · 2026-03-27T16:06:14Z

(I ignored the copilot suggestions/notes that aren't about my code.)

(I also haven't taken a close look at Leon's original work. The code generally is not up to the normal standard, but I think I'd be fine merging this mostly as-is, as fcc/coverage feedback is at experimental for now.)

riesentoaster · 2026-03-27T17:15:21Z

I have moved the FCC logic to an experimental module, as discussed in a meeting at some point. All submodules in this experimental module will emit warnings when used. These can be ignored very easily in the API. I think this is a good general way of dealing with this, I expect more experimental features as time goes on.

riesentoaster requested changes Jan 15, 2026

View reviewed changes

leonbett force-pushed the greybox branch 3 times, most recently from 6de14c4 to b82e116 Compare January 21, 2026 17:02

leonbett changed the base branch from main to stop_options January 21, 2026 17:07

leonbett requested a review from riesentoaster January 21, 2026 17:25

riesentoaster force-pushed the stop_options branch from 591f732 to d08f4e7 Compare March 26, 2026 13:38

leonbett and others added 5 commits March 26, 2026 14:50

Add stop options

c4b0903

fcc integration

60993a6

format test . Update uv.lock type annotations fix tests install llvm debug install llvm debug . debug macos . . no fcc for windows try to improve benchmark Introduce FCC class ran uv sync --locked --all-extras no exception refactor analysis.py

s/leonbett/fandango-fuzzer

6fa7ae9

Exit installation if LLVM version < 18

e670a1a

Fix paths

f28821e

riesentoaster force-pushed the greybox branch from b0ba96e to f28821e Compare March 26, 2026 15:03

riesentoaster added 6 commits March 26, 2026 16:11

remove dependency on mermaid-cli

daff778

remove broken uv dependency on linux

e0272f3

More PATH fun

fc8f67d

remove no-longer-necessary PATH fixing

0479c15

Make "make fcc" part of "make test_tools"

5d4bd9d

Exclude fcc from mypy

49b0a9c

riesentoaster reviewed Mar 26, 2026

View reviewed changes

src/fandango/api.py Outdated Show resolved Hide resolved

minor change to trigger CI

e55f9eb

riesentoaster reviewed Mar 26, 2026

View reviewed changes

src/fandango/api.py Outdated Show resolved Hide resolved

riesentoaster added 4 commits March 26, 2026 23:02

Fix black

3eeb8a8

Improve error handling for docs

2fbf0bc

Split Makefile into fcc and fcc-local targets

b5e9760

Check for use_fcc value to be present before accessing it

2781f9e

riesentoaster added 7 commits March 27, 2026 13:39

Fix access to args

de5a8c3

Use getattr

ee36776

Speed up docs

2198857

Only run grep on the outputs of the docs if there even are error files

4767e76

Also build docs on macos

7ab7462

Only run install with sudo on ubuntu in docs CI; disable fail-fast

3e3593b

Only sudo on ubuntu

5691f46

riesentoaster added 2 commits March 27, 2026 15:16

Install telnet on macos

b76dea3

Cast a wider net for error detection in book

c5f8cec

riesentoaster mentioned this pull request Mar 27, 2026

Prevent runaway processes #874

Open

riesentoaster requested a review from Copilot March 27, 2026 15:45

Copilot started reviewing on behalf of riesentoaster March 27, 2026 15:46 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

riesentoaster and others added 2 commits March 27, 2026 16:57

Improve access

29665f2

Improve check for errors in book

2570293

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Introduce an experimental module and move FCC to it

e51205d

riesentoaster added 2 commits March 27, 2026 18:43

Add custom warning type for improved ignorability

e3d2b63

Make tests more stable

876613c

riesentoaster requested a review from joszamama March 30, 2026 09:02

riesentoaster added 2 commits April 2, 2026 14:41

Only emit one warning per experimental module

2d4b7e2

Introduce --enable-experimental-module

3c3bf48

Conversation

leonbett commented Jan 13, 2026 • edited by riesentoaster Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

riesentoaster commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

riesentoaster left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leonbett commented Jan 23, 2026

Uh oh!

riesentoaster commented Feb 25, 2026

Uh oh!

Uh oh!

Uh oh!

riesentoaster commented Mar 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

riesentoaster commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

riesentoaster commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leonbett commented Jan 13, 2026 •

edited by riesentoaster

Loading

riesentoaster commented Jan 13, 2026 •

edited

Loading

github-actions bot commented Jan 15, 2026 •

edited

Loading

riesentoaster commented Mar 27, 2026 •

edited

Loading