Json fuzzer

[Jim8y]

Description

This PR introduces Neo.Json.Fuzzer, a specialized fuzzing tool designed to test the Neo.Json library for bugs, vulnerabilities, and performance issues. The fuzzer generates diverse JSON inputs, applies mutations, and analyzes the behavior of Neo.Json when processing these inputs.

Fixes #

Type of change

• Optimization (the change is only an optimization)
• Style (the change is only a code style for better maintenance or standard purpose)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Test Configuration:

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[Copilot]

Pull Request Overview

This PR introduces Neo.Json.Fuzzer – a new fuzzing tool aimed at testing the Neo.Json library for bugs, vulnerabilities, and performance issues by generating and mutating JSON inputs. Key changes include:

• Implementation of various mutation components (MutationEngine, ConcurrentAccessMutations, CharacterMutations, BooleanMutations, BaseMutationEngine) to diversify JSON mutations.
• Extensive new documentation covering generation strategies, testing approaches (JPath, Unicode, numeric precision, streaming, concurrent access, DOS detection, coverage analysis), and extension guidelines.

Reviewed Changes

Copilot reviewed 86 out of 86 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
fuzzers/Neo.Json.Fuzzer/Generators/MutationEngine.cs	Core mutation engine coordinating diverse mutation strategies and handling valid/invalid JSON inputs.
fuzzers/Neo.Json.Fuzzer/Generators/ConcurrentAccessMutations.cs	Implements concurrent access mutation strategies to simulate shared data tests under multithreading.
fuzzers/Neo.Json.Fuzzer/Generators/CharacterMutations.cs	Provides character-level mutations and includes a helper to select JSON-relevant characters.
fuzzers/Neo.Json.Fuzzer/Generators/BooleanMutations.cs	Contains strategies to modify boolean values and convert non-boolean values, ensuring diverse boolean mutations.
fuzzers/Neo.Json.Fuzzer/Generators/BaseMutationEngine.cs	Base class that establishes common limits and utility functions for mutation operations.
Documentation (various .md files)	New and updated documentation outlining comprehensive testing strategies for JSON generation, mutation, DOS detection, coverage analysis, JPath testing, concurrent access, and extension guidelines.

[Copilot]

[nitpick] The string of JSON characters appears to contain a typo ('truefalsnl'). Consider verifying and correcting this to the intended set of characters (e.g. 'truefalse null' or a specific list of valid JSON punctuation).

Suggested change

	string jsonChars = "{}[]\":,0123456789.truefalsnl ";
	string jsonChars = "{}[]\":,0123456789truefalsenull ";

[Copilot]

The catch block in SelectAndApplyMutation swallows all exceptions silently. Consider logging the exception details or handling it in a way that aids debugging while still falling back to character-level mutation.

Suggested change

	catch
	{
	catch (Exception ex)
	{
	// Log the exception details
	Debug.WriteLine($"Exception in SelectAndApplyMutation: {ex}");

[Jim8y]

Neo.Json.Fuzzer

A specialized fuzzing tool for testing the Neo.Json library's robustness, security, and performance.

Overview

Neo.Json.Fuzzer systematically tests the Neo.Json library by generating diverse JSON inputs, including edge cases and malformed structures, to identify potential vulnerabilities, crashes, and denial-of-service vectors.

This fuzzer focuses specifically on testing the built-in limits and constraints of the Neo.Json library, such as maximum nesting depth, string length handling, numeric precision, and resource utilization during parsing.

Key Features

• Comprehensive JSON test case generation
• Mutation-based fuzzing with coverage feedback
• Denial-of-Service (DOS) vector detection
• Detailed coverage analysis and reporting
• Specialized testing for integer boundaries and numeric precision
• JPath query testing with complex structures
• Unicode handling verification
• Concurrent access and streaming JSON testing

Documentation

The project follows a documentation-first approach, with comprehensive documentation for all components:

Core Components

• Main Documentation - Central entry point to all documentation
• Mutation Engine - Core component for generating and mutating JSON
• JSON Generation - Strategies for generating test cases
• JSON Runner - Execution engine for running tests
• Utils - Utility functions and helpers

Testing Strategies

• JPath Testing - Testing JPath query functionality
• Unicode Testing - Testing Unicode character handling
• Numeric Precision Testing - Testing numeric precision and integer boundaries
• Streaming Testing - Testing streaming JSON functionality
• Concurrent Access Testing - Testing concurrent access

Analysis and Results

• DOS Detection - DOS detection capabilities
• DOS Testing Results - Results of DOS testing
• Coverage Analysis - Code coverage analysis
• Coverage Mapping - Mapping coverage points to Neo.Json functionality

Extension Guide

• Extending the Fuzzer - Guide for extending the fuzzer

Getting Started

Prerequisites

• .NET SDK 6.0 or later
• Neo project source code

Building

dotnet build

Running

dotnet run -- [options]

Common options:

• --runs N: Run N fuzzing iterations (default: 0 = infinite)
• --seed N: Use specific random seed for reproducible results (default: time-based)
• --output DIR: Output directory for results (default: ./output)
• --detect-dos: Enable DOS vector detection
• --verbose: Enable verbose output
• --max-depth N: Maximum JSON nesting depth to generate (default: 10)

Specialized testing options:

• --jpath-tests: Run specialized JPath query testing
• --unicode-tests: Run specialized Unicode handling tests
• --numeric-precision-tests: Run specialized numeric precision tests
• --streaming-tests: Run specialized streaming JSON tests
• --concurrent-access-tests: Run specialized concurrent access tests
• --specialized-test-type TYPE: Run a specific type of specialized test (e.g., "integer_boundaries")
• --specialized-test-count N: Number of specialized test cases to generate (default: 100)

See dotnet run -- --help for a complete list of options.

Project Structure

• Generators/: Components for generating and mutating JSON test cases

  • BaseMutationEngine.cs: Core mutation functionality and Neo.Json-specific constants
  • MutationEngine.cs: Facade coordinating various mutation strategies
  • NumericPrecisionMutations.cs: Specialized testing for numeric precision and integer boundaries
  • StringMutations.cs: String generation and mutation
  • StructureMutations.cs: Structure manipulation (objects, arrays)

• Runners/: Components for executing tests and collecting results

  • JsonRunner.cs: Executes JSON parsing operations and analyzes results
  • DOSDetector.cs: Identifies potential denial-of-service vectors

• Utils/: Utility classes for analysis, corpus management, etc.

  • CoverageTracker.cs: Tracks code coverage during fuzzing
  • CorpusManager.cs: Manages the corpus of test cases
  • FuzzingStatistics.cs: Collects and reports statistics

• Documentation/: Comprehensive documentation following a documentation-first approach

Testing Results

Initial testing has identified several areas for improvement in Neo.Json:

1. Duplicate Property Handling: Neo.Json throws an error on duplicate property names, which differs from the JSON specification
2. Maximum Nesting Depth: Hard limit of 64 levels with limited configurability
3. Performance with Large Inputs: Significant performance degradation with deeply nested structures
4. Numeric Precision Issues: Potential issues with very large or precise numeric values
5. Non-Standard Format Rejection: Some non-standard formats (like underscores in numbers) are rejected

Contributing

Contributions are welcome! Please see EXTENDING.md for guidelines on extending the fuzzer with new capabilities.