This guide will show you how to run this project locally if you want to test or contribute to it.
Most projects under the umbrella of data systems are written in Rust, so we prefer using Rust tooling and community standards. Ensure you have the following tools installed:
- Rust (version 1.84.0)
- Rust Nightly (version nightly-2025-01-24)
- NodeJS (version >=22.11.0)
- Bun (version 1.2.2)
- Make
- Pre-commit
- NATS
- Tilt
- Minikube
- Kubernetes
- Python3
- Docker
First, clone this repository:
git clone [email protected]:fuellabs/data-systems.git
cd data-systems
Now, install the necessary tools to ensure code quality and standards. Use Make to simplify this process:
make setup
After setup, you'll need to create the environment configuration. First, make sure you have an Infura API key:
- Go to Infura and create an account
- Create a new project
- Copy your project ID (API key)
Then run the environment setup command:
make create-env
The script will prompt you to enter your Infura API key and will automatically:
- Generate a new keypair for P2P communication
- Create a
.env
file from the template - Configure the environment with your Infura key and the generated keypair
You can check the ./scripts/setup.sh file to see what is being installed on your machine.
Here's an overview of the project's directory structure:
crates/
: Contains the main Rust crates for the projectdata-parser/
: Utility library for encoding/decoding datamessage-broker/
: Message broker implementationfuel-streams/
: Main fuel-streams packagecore/
: Core components for working with streamsdomains/
: Domain-specific implementationssubject/
: Macro utilities for the projectstore/
: Storage implementationstypes/
: Common types and traitsweb-utils/
: Web utilitiestest/
: Testing utilities
services/
: Contains the services for the projectpublisher/
: Publisher service implementationconsumer/
: Consumer service implementationwebserver/
: WebSocket server implementation
benches/
: Benchmarking codetests/
: Integration and end-to-end testsexamples/
: Example code and usage demonstrationscluster/
: Kubernetes cluster configuration and deployment filesscripts/
: Utility scripts for setup, deployment, and maintenancegenerate-api-keys/
: Script for generating API keyssubjects-schema/
: Script for generating subjects schema
The project uses cargo-nextest for running tests. Here are the available test commands:
# Run all tests in the project
make test
# Run tests for a specific package
make test PACKAGE=<package-name>
# Run tests in watch mode
make test-watch
# Run tests with a specific profile
make test PROFILE=<profile-name>
# Run Helm chart tests
make helm-test
Command | Description |
---|---|
make build |
Build release version |
make dev-watch |
Run in development mode with auto-reload |
Command | Description |
---|---|
make fmt |
Format all code |
make fmt-rust |
Format Rust code only |
make fmt-markdown |
Format markdown files |
make fmt-prettier |
Format other files with prettier |
Command | Description |
---|---|
make lint |
Run all linters |
make lint-rust |
Run Rust linter |
make lint-clippy |
Run clippy |
make lint-markdown |
Lint markdown files |
make lint-machete |
Check for unused dependencies |
Command | Description |
---|---|
make start-docker |
Start Docker services |
make stop-docker |
Stop Docker services |
make reset-docker |
Reset Docker services |
Command | Description |
---|---|
make setup-db |
Set up database |
make reset-db |
Reset database |
Command | Description |
---|---|
make docs |
Generate documentation |
make docs-serve |
Serve documentation locally |
Command | Description |
---|---|
make version |
Show current version |
make bump-version VERSION=X.Y.Z |
Bump version to specified number |
Command | Description |
---|---|
make audit |
Run security audit |
make audit-fix-test |
Test security fixes |
make audit-fix |
Apply security fixes |
Command | Description |
---|---|
make minikube-setup |
Setup Minikube cluster |
make minikube-start |
Start Minikube cluster |
make minikube-delete |
Delete Minikube cluster |
make cluster-up |
Start local cluster |
make cluster-down |
Stop local cluster |
make cluster-reset |
Reset local cluster |
Command | Description |
---|---|
make load-test |
Run load tests |
make bench |
Run benchmarks |
The project includes several services that can be run locally:
The publisher service is responsible for fetching and publishing blockchain data. You can run it in different modes:
# Run publisher in development mode
make run-publisher-testnet-dev # For testnet
make run-publisher-mainnet-dev # For mainnet
# Run with profiling
make run-publisher-testnet-profiling
make run-publisher-mainnet-profiling
# Run with custom parameters
make run-publisher \
NETWORK=testnet \
MODE=dev \
PORT=4000 \
TELEMETRY_PORT=9001 \
NATS_URL=localhost:4222 \
FROM_HEIGHT=0
- Use
testnet-dev
when developing features against testnet - Use
mainnet-dev
when developing features against mainnet - Use
*-profiling
modes when you need to analyze performance or debug memory issues - Custom parameters:
NETWORK
: Choose betweentestnet
ormainnet
MODE
: Choose betweendev
orprofiling
PORT
: Service port (default: 4000)TELEMETRY_PORT
: Metrics port (default: 9001)NATS_URL
: NATS server URLFROM_HEIGHT
: Starting block height
The consumer service processes the published blockchain data and maintains the database state:
# Run with default settings
make run-consumer
# Run with custom parameters
make run-consumer \
NATS_URL=localhost:4222 \
PORT=9002
- Custom parameters:
NATS_URL
: NATS server URL (default: localhost:4222)PORT
: Service port (default: 9002)
This service should be running alongside the publisher to process the data stream.
The webserver provides WebSocket endpoints for clients to subscribe to real-time blockchain data:
# Run with default settings
make run-webserver-testnet-dev
make run-webserver-mainnet-dev
# Run with custom parameters
make run-webserver \
NETWORK=testnet \
MODE=dev \
PORT=9003 \
NATS_URL=nats://localhost:4222
- Use
testnet-dev
to serve testnet data during development - Use
mainnet-dev
to serve mainnet data during development - Custom parameters:
NETWORK
: Choose betweentestnet
ormainnet
MODE
: Choose betweendev
orprofiling
PORT
: Service port (default: 9003)NATS_URL
: NATS server URL
For local development, a typical setup would be:
- Start the publisher service for your desired network
- Run the consumer service to process the data
- Start the webserver to expose the processed data via WebSocket
We enforce strict code conventions to ensure quality, sustainability, and maintainability. The following tools help automate our standards:
- Conventional Commits - Standardizes commit message formats
- Pre-commit - Runs automated checks before commits
- Commitlint - Enforces commit message standards
- Release-plz - Automates our release process
All commits must follow the Conventional Commits specification using this format:
type(scope): subject
[optional body]
[optional footer(s)]
Choose the appropriate type that best describes your changes:
feat
: New features or significant functionality additionsfix
: Bug fixes and error correctionsdocs
: Documentation changes onlyrefactor
: Code changes that neither fix bugs nor add featuresperf
: Performance improvementstest
: Adding or modifying testsbuild
: Changes affecting build system or dependenciesci
: Changes to CI configuration files and scripts
The scope field is mandatory and must be one of the following:
Core Packages:
fuel-streams
: Main fuel-streams packagefuel-streams-core
: Core components and utilitiesfuel-streams-domains
: Domain-specific implementationsfuel-streams-types
: Common types and traitsfuel-streams-subject
: Macro utilitiesfuel-streams-store
: Storage implementations
Service Packages:
sv-publisher
: Publisher servicesv-consumer
: Consumer servicesv-webserver
: WebSocket server
Support Packages:
fuel-data-parser
: Data parser utilitiesfuel-message-broker
: Message broker implementationfuel-streams-test
: Testing utilitiesfuel-web-utils
: Web utilities
Repository:
repo
: Global repository changesrelease
: Automated release pull requests
For breaking changes:
- Add a
!
after the type/scope - Include a
BREAKING CHANGE:
footer - Clearly explain the changes and migration path
Example of a breaking change commit:
feat(fuel-streams-core)!: implement new streaming protocol
[optional body explaining the changes in detail]
BREAKING CHANGE: The streaming protocol has been completely redesigned.
Users need to:
1. Update client implementations to use the new StreamingClient
2. Migrate existing stream configurations
3. Update any custom protocol handlers
We use release-plz to automate our release workflow. This tool:
- Generates changelogs based on conventional commits
- Groups changes by scope in the changelog
- Determines version bumps based on commit types
- Creates release pull requests
For this automation to work effectively:
- Always use the correct commit type and scope
- Write clear, descriptive commit messages
- Include all necessary details for breaking changes
- Ensure PR titles follow the same conventional commit format
Example of how commits affect releases:
feat(fuel-streams): add new feature // Minor version bump
fix(sv-publisher): fix bug // Patch version bump
feat(fuel-streams-core)!: breaking // Major version bump
When creating a PR:
- Use the same conventional commit format in the PR title
- Include the mandatory scope field
- Reference related issues
- Provide detailed description of changes
- Add breaking change warnings if applicable
Example PR title:
feat(fuel-streams-core)!: implement new streaming protocol
This structured approach to commits and PRs ensures:
- Clear and searchable project history
- Automated and accurate changelog generation
- Proper semantic versioning
- Easy identification of breaking changes
- Efficient code review process
The project includes support for running a local Kubernetes cluster using Minikube for development and testing. Here's a quick guide to get started:
- Setup Minikube cluster:
make minikube-setup
make minikube-start
For detailed information about the necessary tools to install, cluster configuration, deployment options, and troubleshooting, please refer to the Cluster Documentation.
If you encounter any issues while setting up or contributing to the project, here are some common problems and their solutions:
-
Pre-commit hooks failing: Ensure you've installed all the required dependencies and run
make setup
. If issues persist, try runningpre-commit run --all-files
to see detailed error messages. -
Build failures: Make sure you're using the latest stable Rust version (1.84.0) and the correct nightly version (nightly-2025-01-24). You can update Rust using
rustup update stable
andrustup update nightly-2025-01-24
. -
Test failures: If specific tests are failing, try running them in isolation to see if it's a concurrency issue. Use
RUST_BACKTRACE=1
to get more detailed error information. -
Docker issues: If you encounter Docker-related issues, try:
- Ensuring Docker daemon is running
- Running
docker system prune
to clean up unused resources - Checking Docker logs with
docker logs <container-name>
-
Database issues: If you encounter database problems:
- Ensure PostgreSQL is running with
make start-docker
- Reset the database with
make reset-db
- Check database logs with
docker logs <postgres-container-name>
- Ensure PostgreSQL is running with
If you encounter any other issues not listed here, please open an issue on the GitHub repository.
- Rust Documentation
- Fuel Labs Documentation
- NATS Documentation
- Kubernetes Documentation
- Tilt Documentation
We appreciate your contributions to the Fuel Data Systems project!