Cross-platform desktop application for evaluating, comparing, and fine-tuning information retrieval reranker models — no Python required.
RerankEval runs fully on .NET/C# — download HuggingFace Hub models, score them with ONNX Runtime, compute standard IR metrics (NDCG, MRR, MAP), and compare models side-by-side on your own datasets. View quality-vs-latency scatter plots, per-query NDCG distributions, and a historical model leaderboard — all without leaving the app. An AI Agent layer (Semantic Kernel) lets you orchestrate evaluation pipelines, generate reports, and inspect metrics from natural language instructions.
Most reranker evaluation tooling is Python-only and assumes a Jupyter-notebook workflow. RerankEval offers:
- No Python. Single self-contained binary. Works offline after the first model download.
- ONNX-native inference. CPU out of the box; CUDA and CoreML with no code changes.
- Graded-relevance metrics. NDCG@K, MRR@K, MAP@K, Precision@K, Recall@K, Spearman ρ, Kendall τ, calibration curves — all computed locally.
- Persistent experiment store. Every run is saved to SQLite; nothing is lost between sessions.
- Cross-platform. One codebase runs on Windows, macOS, and Linux.
Screenshots will be added once the UI stabilises. Want to contribute one? See Contributing.
Data
- Search and download models from HuggingFace Hub with HTTP resume support
- JSONL and CSV dataset loading with built-in schema validation and relevance-distribution stats
Evaluation
- Batched ONNX inference — CPU out of the box, CUDA and CoreML with no code changes
- Full IR metric suite: NDCG@K, MRR@K, MAP@K, Precision@K, Recall@K, Spearman ρ, Kendall τ, calibration curves
- Multi-model parallel evaluation in a single run; cancelable at any time
Analysis
- Quality-vs-latency scatter plot (NDCG@10 vs P50 latency) across all models in a run
- Per-query NDCG@10 histogram — select any model row to see its score distribution
- Best-model row highlighted automatically
- Export results to CSV or JSON
Deep Analysis
- Error analysis: worst-performing queries sorted by NDCG@10, full-text search, export worst-N to CSV
- Calibration: reliability diagram (score buckets vs. actual relevance fraction), domain-shift breakdown per tag
- Rank correlation: pairwise Spearman ρ and Kendall τ matrix between all models in a run
- Latency profiling: 4-phase stacked bar chart (tokenization / tensor creation / ONNX run / postprocessing) per model
- A/B test sample-size calculator — given control/treatment NDCG@10, α, and power, outputs required queries per arm
History
- Per-dataset model leaderboard aggregated across all runs
- NDCG@10 trend chart per model over time
Fine-tuning
- Step 1: upload triplet JSONL, pairwise JSONL, or CSV training data with schema validation
- Step 2: configure learning rate, epochs, batch size, frozen layers, loss function
- Step 3: live training monitor with loss curve and streaming log
- Training loop via TorchSharp (simulation mode by default; see README footnote to enable real training)
AI Agent
- Semantic Kernel agent with 6 plugins: ModelManagement, Evaluation, Dataset, MetricsAnalysis, FineTuning, Reporting
- Streaming chat responses with a live typing bubble
- Session history sidebar — pick up any past conversation
- Auto function calling — agent invokes the right tool and shows tool calls in the action log
- Supports OpenAI, Azure OpenAI, and Ollama providers — configure in Settings
- Generates Markdown evaluation reports saved to
~/.rerank_eval/exports/
Settings
- LLM provider selector (OpenAI / Azure OpenAI / Ollama)
- Masked API key input stored in
~/.rerank_eval/credentials.json - Model ID and Azure endpoint/deployment configuration
Persistence
- Persistent SQLite experiment store — every run, dataset, result, training metric, and agent session is saved between sessions
- .NET 10 SDK (or later)
- (Optional) CUDA 12.x for GPU inference on Windows / Linux
- (Optional) Apple Silicon Mac for CoreML acceleration
git clone https://github.com/Eisenmann/rerank-eval.git
cd rerank-eval
dotnet build -c Release
dotnet run --project src/ReRankEval.App -c ReleaseOn first launch the app creates ~/.rerank_eval/ and applies the database migration automatically.
# Windows x64
dotnet publish src/ReRankEval.App -r win-x64 -c Release --self-contained -o ./publish/win-x64
# macOS Apple Silicon
dotnet publish src/ReRankEval.App -r osx-arm64 -c Release --self-contained -o ./publish/osx-arm64
# Linux x64
dotnet publish src/ReRankEval.App -r linux-x64 -c Release --self-contained -o ./publish/linux-x64
chmod +x ./publish/linux-x64/ReRankEval.AppApproximate size: ~280 MB (CPU-only). CUDA build is larger due to native libtorch.
Open the Models tab, search for a model on HuggingFace Hub (e.g. cross-encoder/ms-marco-MiniLM-L-6-v2), and click Download. config.json, tokenizer.json, and weights are fetched with HTTP range resume support.
Open the Datasets tab, click Load JSONL… or Load CSV…, pick your file. Click Validate to check for schema errors, or Load Stats to see the relevance distribution.
JSONL — one object per line:
{"query": "What is BERT?", "docs": ["BERT is a transformer...", "GPT is..."], "labels": [2, 0]}
{"query": "Vector search", "docs": ["FAISS is a library...", "BM25 is sparse..."], "labels": [2, 1]}CSV — one row per query–document pair, rows with the same query are grouped automatically:
query,document,relevance,domain_tag
"What is BERT?","BERT is a transformer...",2,nlp
"What is BERT?","GPT is a language model",0,nlp
Open Evaluation, select one or more local models, pick a dataset, set K values (1,5,10), and click Run. Results are saved to SQLite and appear in the Metrics tab immediately.
The Metrics tab shows a comparison table per evaluation run. The best-scoring model is highlighted. Click any row to see its per-query NDCG@10 histogram. Use Export CSV or Export JSON to save the table.
| Metric | Description |
|---|---|
| NDCG@K | Normalized Discounted Cumulative Gain — graded relevance |
| MRR@K | Mean Reciprocal Rank — position of first relevant doc |
| MAP@K | Mean Average Precision |
| P50 / P90 latency | Per-query inference latency percentiles |
The Quality vs. Latency scatter plot (below the table) shows every model as a point — models toward the top-left are the Pareto-optimal choices.
Open the History tab, select a dataset, and see a leaderboard of every model ever evaluated on it (averaged across runs). Click a model to view its NDCG@10 trend over time.
DCG@K = Σ (2^rel_i − 1) / log₂(i + 2) for i = 0 … K−1
NDCG@K = DCG@K / IDCG@K
A perfect ranking returns 1.0; random ranking approaches 0.
1 / rank of the first relevant document, averaged over all queries, clipped at K.
Mean of per-query average precision values computed at each relevant document position up to K.
Measure how similarly two models rank documents on the same query set. Useful for model selection without a labelled dataset.
Reliability diagram — buckets of predicted scores (x) vs. actual relevance fraction (y). A well-calibrated model lies close to the diagonal.
rerank-eval/
├── src/
│ ├── ReRankEval.Domain/ # Entities, interfaces, MetricsCalculator (pure C#)
│ ├── ReRankEval.Infrastructure/ # HF Hub client, ONNX inference, EF Core, dataset parsing
│ ├── ReRankEval.Agent/ # Semantic Kernel orchestrator + 6 KernelPlugin classes
│ │ └── Plugins/ # ModelManagement, Evaluation, Dataset, MetricsAnalysis,
│ │ # FineTuning, Reporting
│ └── ReRankEval.App/ # Avalonia startup, DI wiring, XAML views
│ └── Controls/ # MarkdownViewer (code-only UserControl)
└── tests/
├── ReRankEval.Domain.Tests/ # 21 unit tests covering all metrics
└── ReRankEval.Infrastructure.Tests/
~/.rerank_eval/
├── models/{org}/{model-name}/ # downloaded weights + ONNX
├── datasets/{id}/ # imported dataset files
├── checkpoints/ # fine-tuning checkpoints (Phase 3)
├── exports/ # CSV / JSON / Markdown reports
├── experiments.db # SQLite (EF Core)
└── logs/app_YYYYMMDD.log
| Layer | Technology |
|---|---|
| UI | Avalonia UI 12 (XAML, Fluent theme) |
| MVVM | CommunityToolkit.Mvvm 8 |
| Inference | Microsoft.ML.OnnxRuntime 1.27 |
| Storage | EF Core 10 + SQLite |
| HTTP / resilience | System.Net.Http + Polly |
| Logging | Serilog (rolling file) |
| DI / hosting | Microsoft.Extensions.Hosting |
| Analytics | EF Core LINQ queries (same interface; DuckDB swap-in ready) |
| Fine-tuning | TorchSharp (simulation mode default; add native backend to enable real training) |
| AI Agent | Microsoft.SemanticKernel 1.x (OpenAI / Azure OpenAI / Ollama) |
dotnet test tests/ReRankEval.Domain.Tests
dotnet test tests/ReRankEval.Infrastructure.Tests| Phase | Status | Highlights |
|---|---|---|
| 1 — Foundation | ✅ Complete | Domain model, ONNX inference, metrics, SQLite store, basic UI |
| 2 — Evaluation engine | ✅ Complete* | DatasetView, scatter + histogram charts, export CSV/JSON, model leaderboard, NDCG trend |
| 3 — Analysis & fine-tuning | ✅ Complete† | Error analysis (worst-query table, search, export), calibration reliability diagram, domain breakdown, rank correlation matrix, 4-phase latency profiling, A/B sample-size calculator, fine-tuning wizard (validation + hyperparameters + live training monitor) |
| 4 — AI Agent | ✅ Complete‡ | Semantic Kernel orchestrator, 6 KernelPlugins, streaming chat, session history, MarkdownViewer, Settings page, credential store |
* BEIR dataset downloader not yet implemented.
† TorchSharp fine-tuning runs in simulation mode by default. Add TorchSharp-cpu (CPU) or TorchSharp-cuda-* (GPU) NuGet packages to ReRankEval.Infrastructure and uncomment #define TORCHSHARP in TorchSharpFineTuningService.cs to enable real deep fine-tuning.
‡ AI Agent requires an LLM API key — configure via the ⚙ Settings button in the sidebar. Supports OpenAI (gpt-4o-mini recommended), Azure OpenAI, and Ollama (local). Without a key configured the agent returns a helpful prompt to visit Settings.
Contributions are welcome. Please:
- Fork the repo and create a feature branch from
main. - Add unit tests for any new domain logic (
ReRankEval.Domain.Tests). - Add integration tests for infrastructure changes (
ReRankEval.Infrastructure.Tests). - Ensure
dotnet build -c Releaseproduces zero warnings. - Open a pull request describing what changed and why.
If you have a feature idea or bug report, open an issue first so we can discuss the approach.