Typeberry Testing

E2E tests for Typeberry - a JAM node implementation by Fluffy Labs.

Check out our performance statistics over time at typeberry.fluffylabs.dev.

Status

Test Category	Status	Description
Docker Works		Tests Docker image functionality and basic operations
Docker Imports		Docker image is able to import standard block dumps
Docker Conformance		Tests JAM conformance using Docker with latest conformance test suite
Docker Test Vectors		Tests W3F test vectors using Docker with latest test suite
NPM Works		Tests NPM package installation and basic functionality
NPM @next Works		Tests NPM @next package installation and basic functionality
Picofuzz Fallback		Tests fallback functionality using prepared fuzz messages
Picofuzz Safrole		Tests Safrole protocol implementation with fuzzing
Picofuzz Storage		Tests storage functionality with comprehensive fuzzing
Picofuzz Storage Light		Tests storage functionality with lightweight fuzzing
Minifuzz Burn		Burn-in testing for extended fuzzing operations
Minifuzz Faulty		Tests error handling and fault tolerance
Minifuzz Forks		Tests fork handling and process management
Minifuzz No Forks		Tests single-process operation without forking

Standalone tools

• picofuzz
• perf-graph

Running Tests

Prerequisites

• Node.js 20+
• Docker
• npm

Setup

# Clone the repository with recursive submodules
git clone --recursive https://github.com/FluffyLabs/typeberry-testing.git

# Or if already cloned, initialize submodules recursively
git submodule update --init --recursive

# Install dependencies
npm install

# Fetch the latest Typeberry Docker image
npm run fetch-typeberry

Submodules

This repository uses the following submodules:

• jam-conformance - JAM conformance traces
• picofuzz-data - Data for picofuzz execution.

Running All Tests

npm test

Running Individual Tests

# Docker functionality tests
npm exec tsx --test tests/docker-works.test.ts

# Docker conformance tests
npm exec tsx --test tests/docker-conformance.test.ts

# Docker test vectors
npm exec tsx --test tests/docker-test-vectors.test.ts

# NPM package tests
npm exec tsx --test tests/npm-works.test.ts

# NPM @next package tests
npm exec tsx --test tests/npm-next-works.test.ts

# Picofuzz tests
npm exec tsx --test tests/picofuzz/fallback.test.ts
npm exec tsx --test tests/picofuzz/safrole.test.ts
npm exec tsx --test tests/picofuzz/storage.test.ts
npm exec tsx --test tests/picofuzz/storage_light.test.ts

# Minifuzz tests
npm exec tsx --test tests/minifuzz/burn.test.ts
npm exec tsx --test tests/minifuzz/faulty.test.ts
npm exec tsx --test tests/minifuzz/forks.test.ts
npm exec tsx --test tests/minifuzz/no_forks.test.ts

Running Picofuzz

Picofuzz is a lightweight fuzzing tool that sends prepared fuzz messages using the Fuzz protocol:

# Run picofuzz directly
npm start -w @fluffylabs/picofuzz [options] <directory> <socket>

# Options:
#   -f, --flavour <spec>      JAM spec: tiny | full (default: tiny)
#   -r, --repeat  <count>     Number of repetitions (default: 1)
#   -s, --stats   <file>      Append aggregated stats to a CSV file
#   -h, --help                Show help

# Examples:
npm start -w @fluffylabs/picofuzz picofuzz-data/picofuzz-data/fallback /tmp/jam_target.sock
npm start -w @fluffylabs/picofuzz -r 10 picofuzz-data/picofuzz-data/safrole /tmp/jam_target.sock
npm start -w @fluffylabs/picofuzz -s results.csv picofuzz-data/picofuzz-data/storage /tmp/jam_target.sock

See more details about [picofuzz](./picofuzz).

# Using Docker
cd picofuzz
docker build -t picofuzz .
docker run picofuzz [options] <directory> <socket>

PR Benchmark Workflow

The PR Benchmark workflow allows you to test and benchmark PRs from the typeberry repository before merging. It runs the complete picofuzz test suite against a specific Docker image build and compares the results with baseline performance metrics.

Usage

Manual Trigger:

1. Go to Actions > PR Benchmark
2. Click "Run workflow"
3. Enter the PR number from fluffylabs/typeberry (e.g., 704)

The workflow will automatically find the latest successful build-docker workflow run for that PR.

Automated Trigger:

curl -X POST \
  -H "Authorization: token YOUR_PAT" \
  https://api.github.com/repos/FluffyLabs/typeberry-testing/dispatches \
  -d '{"event_type": "benchmark-pr", "client_payload": {"pr_number": "704"}}'

Requirements:

• TYPEBERRY_PAT secret with permissions to read artifacts and post comments to fluffylabs/typeberry

Project Structure

├── tests/
│   ├── docker-works.test.ts        # Docker image functionality tests
│   ├── docker-conformance.test.ts  # JAM conformance tests using Docker
│   ├── docker-test-vectors.test.ts # W3F test vectors using Docker
│   ├── npm-works.test.ts           # NPM package tests
│   ├── npm-next-works.test.ts      # NPM @next package tests
│   ├── picofuzz/                   # Performance testing
│   │   ├── common.ts               # Common utilities for picofuzz tests
│   │   ├── fallback.test.ts        # Fallback performance
│   │   ├── safrole.test.ts         # Safrole performance
│   │   ├── storage.test.ts         # Storage performance
│   │   └── storage_light.test.ts   # Lightweight storage performance
│   └── minifuzz/                   # Minifuzz compatibility
│       ├── burn.test.ts            # Repeated execution
│       ├── faulty.test.ts          # Fault tolerance tests
│       ├── forks.test.ts           # Fork handling tests
│       └── no_forks.test.ts        # Single-chain tests
├── picofuzz/                       # Performance testing
│   ├── index.ts                    # Main entry point
│   ├── args.ts                     # Argument parsing
│   ├── files.ts                    # File processing utilities
│   ├── socket.ts                   # Socket communication
│   ├── stats.ts                    # Statistics collection
│   └── package.json                # Package configuration

Contributing

This repository contains end-to-end tests for the Typeberry project. Each test suite runs in isolation as separate CI jobs to provide granular feedback on different aspects of the system.

License

Mozilla Public License 2.0 (MPL-2.0)