Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
35e4f93
initialize work
tylerbutler Oct 28, 2025
738e9da
updates
tylerbutler Oct 29, 2025
6706c44
feat(build-tools): add cache configuration validation
tylerbutler Oct 29, 2025
c014fab
feat(build-tools): add debug logging and build output for shared cache
tylerbutler Oct 29, 2025
89cfc9d
feat(build-tools): add cache management commands
tylerbutler Oct 29, 2025
019d3fe
updates
tylerbutler Oct 29, 2025
c1fa768
updates
tylerbutler Oct 29, 2025
e2800b5
build
tylerbutler Oct 29, 2025
40ae60a
shebang
tylerbutler Oct 29, 2025
b594ae6
Add cache support for BiomeTask, ApiExtractorTask, and other tasks
tylerbutler Oct 29, 2025
8136126
updates
tylerbutler Oct 29, 2025
7fb79d2
fix
tylerbutler Oct 29, 2025
221c3f9
fix
tylerbutler Oct 29, 2025
c215a77
updates
tylerbutler Oct 29, 2025
f7209ee
Centralize and expand global cache key components
tylerbutler Oct 29, 2025
34b7581
updates
tylerbutler Oct 29, 2025
aca477f
benchmark
tylerbutler Oct 29, 2025
dadf26e
results
tylerbutler Oct 29, 2025
7378792
updates
tylerbutler Oct 29, 2025
9ca3cf2
updates
tylerbutler Oct 29, 2025
b7d8637
Merge branch 'main' into fluid-build-cache
tylerbutler Oct 29, 2025
5169666
Add shared cache support for flub generate entrypoints task
tylerbutler Oct 29, 2025
118f4aa
updates
tylerbutler Oct 29, 2025
bc32eb6
updates
tylerbutler Oct 29, 2025
298aa43
updates
tylerbutler Oct 29, 2025
cada1e1
error handling
tylerbutler Oct 29, 2025
b5ca2c1
dirs
tylerbutler Oct 29, 2025
63ef999
ctrl-c handling
tylerbutler Oct 29, 2025
5f903cb
Update cache status symbols and add legend
tylerbutler Oct 29, 2025
f72d47f
Refactor status symbols into shared constants
tylerbutler Oct 29, 2025
c2b3663
Add time saved tracking and improve legend layout
tylerbutler Oct 29, 2025
1d0f63e
Add specific reasons for cache write failures
tylerbutler Oct 29, 2025
b8b4c47
Suppress 'no output files' warning for tasks with no expected outputs
tylerbutler Oct 29, 2025
cf2a41f
Always include donefile in remote cache outputs
tylerbutler Oct 29, 2025
f9912fb
Format time saved as hours, minutes, seconds
tylerbutler Oct 29, 2025
556b5fd
Fix cache statistics and upload skip reason display
tylerbutler Oct 30, 2025
e426f3a
Merge branch 'main' into fluid-build-cache
tylerbutler Oct 30, 2025
ca6936a
Merge remote-tracking branch 'upstream/main' into fluid-build-cache
tylerbutler Oct 30, 2025
e5518f5
Merge branch 'main' into fluid-build-cache
tylerbutler Nov 1, 2025
7fadc8c
Merge branch 'main' into fluid-build-cache
tylerbutler Nov 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .claude/settings.local.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"permissions": {
"allow": [
"Bash(find:*)",
"Bash(pnpm build:*)",
"Bash(pnpm exec mocha:*)",
"Bash(pnpm run build:test:*)",
"Bash(pnpm run tsc:*)",
"Bash(node -e:*)",
"Bash(pnpm run check:biome:*)",
"Bash(pnpm run format:biome)",
"Bash(git restore:*)",
"Bash(pnpm run format:biome:*)"
],
"deny": [],
"ask": []
}
}
228 changes: 228 additions & 0 deletions BENCHMARK-GUIDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,228 @@
# Build Cache Benchmarking Guide

This guide explains how to benchmark the Fluid Framework build cache performance using the provided scripts.

## Prerequisites

### Install hyperfine

Hyperfine is a command-line benchmarking tool that provides statistical analysis of command execution times.

**Ubuntu/Debian:**
```bash
sudo apt install hyperfine
```

**macOS:**
```bash
brew install hyperfine
```

**With Cargo (Rust):**
```bash
cargo install hyperfine
```

**From releases:**
Download from https://github.com/sharkdp/hyperfine/releases

## Quick Start

### Basic Usage

Run the benchmark on the aqueduct package (default):
```bash
./benchmark-cache.sh
```

### Custom Project

Benchmark a different package:
```bash
./benchmark-cache.sh packages/runtime/container-runtime
```

### Adjust Number of Runs

Benchmark with custom number of runs (for more statistical accuracy):
```bash
./benchmark-cache.sh packages/framework/aqueduct 10 3
# Parameters: <project-dir> <benchmark-runs> <warmup-runs>
```

## Understanding the Results

The script will generate two output files:
- `benchmark-results-<project>.md` - Markdown formatted results
- `benchmark-results-<project>.json` - JSON data for further analysis

### Example Output

```
Benchmark 1: with-cache
Time (mean ± σ): 2.234 s ± 0.125 s [User: 8.1 s, System: 1.2 s]
Range (min … max): 2.105 s … 2.458 s 10 runs

Benchmark 2: without-cache
Time (mean ± σ): 4.567 s ± 0.234 s [User: 16.3 s, System: 2.1 s]
Range (min … max): 4.289 s … 4.892 s 10 runs

Summary
with-cache ran
2.04 ± 0.15x faster than without-cache
```

## What Gets Benchmarked

The script compares:

1. **with-cache**: Build using the shared build cache
- Reuses cached build artifacts when possible
- Typical of incremental builds or builds after cache warming

2. **without-cache**: Build with cache disabled (`FLUID_BUILD_CACHE_DISABLED=1`)
- Forces complete rebuild every time
- Equivalent to clean builds

## Advanced Scenarios

### Testing Different Build Tasks

Edit the `BUILD_CMD` variable in the script to test different tasks:

```bash
# Test full build (default is compile)
BUILD_CMD="pnpm fluid-build --task build --root ${PROJECT_DIR}"

# Test just TypeScript compilation
BUILD_CMD="pnpm fluid-build --task tsc --root ${PROJECT_DIR}"

# Test with linting
BUILD_CMD="pnpm fluid-build --task lint --root ${PROJECT_DIR}"
```

### Testing Multiple Projects

Create a loop to test multiple projects:

```bash
for project in packages/framework/aqueduct packages/runtime/container-runtime packages/dds/tree; do
echo "Benchmarking $project..."
./benchmark-cache.sh "$project" 5 2
done
```

### Comparing Cold vs Warm Cache

To test cold cache (first time) vs warm cache (subsequent builds):

```bash
# First, prime the cache
pnpm fluid-build --task compile --root packages/framework/aqueduct

# Then run benchmark (cache will be warm)
./benchmark-cache.sh packages/framework/aqueduct
```

## Tips for Accurate Benchmarking

1. **Close other applications**: Minimize background processes that might affect CPU/disk usage
2. **Run multiple iterations**: Use at least 5-10 runs for statistical significance
3. **Consider warmup runs**: 2-3 warmup runs help stabilize the results
4. **Test on consistent hardware**: Same machine, power settings, etc.
5. **Check disk state**: SSD vs HDD, available space, fragmentation
6. **Monitor system load**: Ensure system isn't under heavy load during benchmarks

## Interpreting Cache Performance

Good cache performance indicators:
- **2-5x speedup** for incremental builds
- **Consistent timings** across runs (low standard deviation)
- **Minimal variance** in warm cache runs

Potential issues if:
- Cache builds are slower than non-cache builds
- High variance in cached build times
- Cache hit rate is very low

## Environment Variables

The benchmark script respects these environment variables:

- `FLUID_BUILD_CACHE_DISABLED=1` - Disables the build cache
- `FLUID_BUILD_CACHE_PATH` - Custom cache location (if supported)

## Troubleshooting

### hyperfine not found
Install hyperfine using one of the methods in Prerequisites.

### Project directory not found
Ensure you're running from the repository root and the path is correct.

### Inconsistent results
- Run more iterations
- Check for background processes
- Ensure disk has sufficient space
- Try with different projects

### Cache not being used
- Check if cache directory exists and has content
- Verify no environment variables are disabling cache
- Check build tool configuration

## Examples for Different Scenarios

### Benchmark after clean install
```bash
# Clean everything
pnpm clean
rm -rf node_modules

# Install
pnpm install

# Benchmark
./benchmark-cache.sh packages/framework/aqueduct 10 3
```

### Compare cache effectiveness across packages
```bash
#!/bin/bash
packages=(
"packages/framework/aqueduct"
"packages/runtime/container-runtime"
"packages/dds/tree"
"packages/loader/container-loader"
)

for pkg in "${packages[@]}"; do
echo "=== Benchmarking $pkg ==="
./benchmark-cache.sh "$pkg" 5 2
echo ""
done
```

## Further Analysis

The JSON output can be analyzed using tools like:
- Python with pandas/matplotlib for visualization
- R for statistical analysis
- Excel/Google Sheets for charts

Example Python analysis:
```python
import json
import matplotlib.pyplot as plt

with open('benchmark-results-aqueduct.json') as f:
data = json.load(f)

times = [result['mean'] for result in data['results']]
names = [result['command'] for result in data['results']]

plt.bar(names, times)
plt.ylabel('Time (s)')
plt.title('Build Performance Comparison')
plt.show()
```
79 changes: 79 additions & 0 deletions BENCHMARK-QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Build Cache Benchmark - Quick Start

## TL;DR

```bash
# 1. Install hyperfine
sudo apt install hyperfine # or: brew install hyperfine

# 2. Run benchmark
./benchmark-cache.sh

# 3. View results
cat benchmark-results-aqueduct.md
```

## One-Liners

```bash
# Simple benchmark (default project)
./benchmark-cache.sh

# Benchmark specific project
./benchmark-cache.sh packages/runtime/container-runtime

# More accurate (10 runs)
./benchmark-cache.sh packages/framework/aqueduct 10

# Test multiple scenarios
./benchmark-cache-advanced.sh -m incremental -r 10

# Benchmark all projects
./benchmark-cache-batch.sh 5
```

## What Gets Tested

✅ **with-cache** - Build using shared cache (fast)
✅ **without-cache** - Build without cache (slow)
✅ **Speedup ratio** - How much faster cache makes builds

## Expected Results

Good cache performance: **2-5x faster** with cache enabled

```
Summary
with-cache ran 2.04 ± 0.15x faster than without-cache
```

## All Scripts

| Script | Purpose | Usage |
|--------|---------|-------|
| `benchmark-cache.sh` | Simple benchmark | `./benchmark-cache.sh [project]` |
| `benchmark-cache-advanced.sh` | Advanced options | `./benchmark-cache-advanced.sh -h` |
| `benchmark-cache-batch.sh` | Multiple projects | `./benchmark-cache-batch.sh [runs]` |

## Common Tasks

**Test cache effectiveness:**
```bash
./benchmark-cache.sh packages/framework/aqueduct 10
```

**Compare multiple projects:**
```bash
./benchmark-cache-batch.sh
```

**Test incremental builds:**
```bash
./benchmark-cache-advanced.sh -m incremental
```

## Need Help?

- `./benchmark-cache-advanced.sh --help` - Show all options
- `BENCHMARK-README.md` - Full documentation
- `BENCHMARK-GUIDE.md` - Detailed guide and tips
Loading
Loading