add neo vm fuzzer #3851
Draft
add neo vm fuzzer #3851
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jim8y
added
Unit Tests
NGD Review
Jim8y
added
the
Blocker
There was a problem hiding this comment.
Pull Request Overview
This PR adds a comprehensive Neo VM Fuzzer featuring coverage-guided fuzzing, DOS vector detection, and corpus management, along with robust documentation and detailed reporting.
- Introduces core fuzzer components: configuration management, script generation and mutation, execution instrumentation, and results aggregation.
- Provides extensive documentation covering usage, DOS detection, event handling, and overall architecture.
- Adds new utilities and runners to integrate with the Neo VM and systematically test its behavior.
Reviewed Changes
Copilot reviewed 40 out of 42 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| fuzzers/Neo.VM.Fuzzer/Utils/ConfigurationManager.cs | Provides singleton configuration with load/save functionality for fuzzer settings. |
| fuzzers/Neo.VM.Fuzzer/Utils/CoverageTracker.cs | Implements code coverage tracking with frequency statistics. |
| fuzzers/Neo.VM.Fuzzer/Runners/StackItem.cs | Represents VM stack items with a custom string formatter. |
| fuzzers/Neo.VM.Fuzzer/Documentation/VMRunner.md | Documents the VMRunner class functionality and usage examples. |
| fuzzers/Neo.VM.Fuzzer/Program.cs | Main fuzzer entry point integrating script generation, mutation, execution, DOS detection, and reporting. |
| fuzzers/Neo.VM.Fuzzer/Documentation/USAGE.md | Provides detailed instructions on installing, configuring, and running the fuzzer. |
| fuzzers/Neo.VM.Fuzzer/Utils/CorpusManager.cs | Manages saving/loading of test scripts, crash reports, and DOS vectors. |
| fuzzers/Neo.VM.Fuzzer/Properties/AssemblyInfo.cs | Assembly configuration exposing internals for unit tests. |
| fuzzers/Neo.VM.Fuzzer/Documentation/DOSDetection.md | Describes the DOS detection mechanisms and configuration thresholds. |
| fuzzers/Neo.VM.Fuzzer/Runners/InstrumentedExecutionEngine.cs | Extends the execution engine with instrumentation for coverage, opcode timing, and event notifications. |
| fuzzers/Neo.VM.Fuzzer/Runners/EventArgs.cs | Defines custom event argument classes for tracking execution events. |
| fuzzers/Neo.VM.Fuzzer/Documentation/EventArgs.md | Documents the structure and usage of event argument classes. |
| fuzzers/Neo.VM.Fuzzer/Runners/VMRunner.cs | Executes scripts in the Neo VM while instrumenting them for DOS detection and coverage collection. |
| fuzzers/Neo.VM.Fuzzer/README.md | Provides an overview, usage instructions, and project structure details. |
| fuzzers/Neo.VM.Fuzzer/Utils/FuzzingResults.cs | Aggregates and reports execution metrics, including performance histograms and exception statistics. |
| fuzzers/Neo.VM.Fuzzer/Utils/DOSDetector.cs | Implements DOS vector analysis based on execution metrics, with configurable thresholds and recommendations. |
| fuzzers/Neo.VM.Fuzzer/Documentation/FUZZER_ARCHITECTURE.md | Details the overall design and integration of the fuzzer components. |
| fuzzers/Neo.VM.Fuzzer/Generators/MutationEngine.cs | Implements a suite of script mutation strategies for evolving test cases. |
Files not reviewed (2)
- fuzzers/Neo.VM.Fuzzer/Neo.VM.Fuzzer.csproj: Language not supported
- fuzzers/Neo.VM.Fuzzer/dos-test-scripts/aggressive_dos.neo: Language not supported
| } | ||
|
|
||
| // Check if it crashed | ||
| if (executionResult.Crashed) |
There was a problem hiding this comment.
Crash, timeout, and new coverage counters (crashCount, timeoutCount, newCoverageCount) are not updated within the main fuzzing loop, leading to inaccurate progress reporting. Consider incrementing these variables appropriately when a script crashes, times out, or discovers new coverage.
Copilot is powered by AI, so mistakes are possible. Review output carefully before use.
shargon
reviewed
Apr 3, 2025
|
|
||
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <TargetFramework>net7.0</TargetFramework> |
There was a problem hiding this comment.
net 9?
Suggested change
| <TargetFramework>net7.0</TargetFramework> | |
| <TargetFramework>net9.0</TargetFramework> |
shargon
reviewed
Apr 3, 2025
shargon
reviewed
Apr 3, 2025
| /// <summary> | ||
| /// Represents an item on the VM execution stack | ||
| /// </summary> | ||
| public class StackItem |
shargon
reviewed
Apr 3, 2025
| public void SaveInteresting(byte[] script) | ||
| { | ||
| // Add to corpus | ||
| _corpus.Add(script); |
There was a problem hiding this comment.
Why this is added to corpus and the others don't?
There was a problem hiding this comment.
Still working on this, marked it as draft,
cschuchardt88
approved these changes
Apr 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the Neo VM Fuzzer, a new tool designed to systematically test the Neo Virtual Machine for bugs, vulnerabilities, and potential Denial of Service (DOS) vectors. The fuzzer uses coverage-guided techniques to explore the execution paths of the Neo VM, helping to identify edge cases and potential issues that might not be discovered through traditional testing methods.
Key features of the Neo VM Fuzzer:
Coverage-guided fuzzing: Tracks code coverage to prioritize inputs that explore new execution paths
DOS detection: Identifies potential DOS vectors by analyzing execution metrics (instruction count, stack depth, execution time)
Instrumented execution: Provides detailed metrics about script execution for analysis
Corpus management: Maintains and evolves a corpus of interesting test cases
Reproducible testing: Supports deterministic fuzzing with seed values for reproducibility
Comprehensive documentation: Includes detailed documentation on usage, extension, and DOS detection capabilities
The fuzzer is implemented as a standalone project within the Neo repository, with minimal dependencies on the core Neo VM code. This ensures that it can evolve independently while still providing valuable testing capabilities for the Neo VM.
Fixes #1234 (Need for systematic VM testing)
Type of change
[ ] Optimization (the change is only an optimization)
[ ] Style (the change is only a code style for better maintenance or standard purpose)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
How Has This Been Tested?
[x] Executed the fuzzer with 10,000 iterations against the current Neo VM
[x] Verified that the fuzzer correctly identifies and saves interesting test cases
[x] Tested DOS detection with known problematic scripts
[x] Verified that the fuzzer can be run with different configurations and seed values
Test Configuration:
Command: dotnet run --project fuzzers/Neo.VM.Fuzzer -- -i 10000 --detect-dos --track-memory
Checklist:
[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules