11# Multi-Agent Banking Annotation System
22
3+ [ ![ Lint] ( https://github.com/saviolobo/agentic-annotator/actions/workflows/lint.yml/badge.svg )] ( https://github.com/saviolobo/agentic-annotator/actions/workflows/lint.yml )
4+ [ ![ Python] ( https://img.shields.io/badge/python-3.11+-blue.svg )] ( https://www.python.org/ )
5+ [ ![ LangGraph] ( https://img.shields.io/badge/LangGraph-1.2+-orange.svg )] ( https://github.com/langchain-ai/langgraph )
6+ [ ![ FastMCP] ( https://img.shields.io/badge/FastMCP-enabled-purple.svg )] ( https://github.com/jlowin/fastmcp )
7+ [ ![ License: MIT] ( https://img.shields.io/badge/License-MIT-yellow.svg )] ( LICENSE )
8+
39A production-grade multi-agent pipeline for automated intent classification
410on the Banking77 benchmark (13,083 queries, 77 intents), inspired by
511JP Morgan's MAFA system (AAAI 2026).
612
13+ > ** Why this exists:** Manual data annotation costs enterprises ~ $0.15/item and
14+ > doesn't scale. JP Morgan Chase eliminated a 1M-utterance backlog saving 5,000+
15+ > hours/year using a multi-agent approach (MAFA, AAAI 2026). This project
16+ > independently reproduces and benchmarks a similar architecture on the public
17+ > Banking77 dataset, enabling direct comparison against their published results.
18+
19+ ---
20+
721## Results
822
923| | This project | JP Morgan MAFA (AAAI 2026) |
@@ -17,43 +31,54 @@ JP Morgan's MAFA system (AAAI 2026).
1731> Full pipeline uses 5 agents with MCP-backed guideline retrieval and
1832> Redis vector search for similar-example lookup.
1933
34+ ---
35+
2036## Architecture
2137
2238Five specialized agents replace manual annotation for high-confidence items
2339while routing ambiguous cases to a human review queue.
2440
2541![ Architecture] ( docs/architecture.png )
2642
27- ** Key pattern:** LangGraph Send API runs Primary Annotator and Validator
43+ ** Key pattern:** LangGraph Send API runs Primary Annotator Agent and Validator Agent
2844in parallel on COMPLEX queries, cutting latency on the most expensive path.
2945
3046| Agent | Model | Role |
3147| ---| ---| ---|
32- | Router | Groq llama-3.1-8b-instant | Classify query complexity |
33- | Primary Annotator | Cerebras gpt-oss-120b | First-pass label + confidence |
34- | Validator | Cerebras gpt-oss-120b | Independent second annotation |
35- | Arbitrator | Cerebras gpt-oss-120b | Resolve disagreements |
36- | Quality Controller | Groq llama-3.1-8b-instant | Consistency + drift detection |
48+ | Router Agent | Groq llama-3.1-8b-instant | Classify query complexity (SIMPLE / COMPLEX) |
49+ | Primary Annotator Agent | Cerebras gpt-oss-120b | First-pass label + confidence + reasoning |
50+ | Validator Agent | Cerebras gpt-oss-120b | Independent second annotation, blind to Primary |
51+ | Arbitrator Agent | Cerebras gpt-oss-120b | Resolves disagreements, routes low-confidence to human |
52+ | Quality Controller Agent | Groq llama-3.1-8b-instant | Consistency checks + label drift detection |
53+
54+ ---
3755
3856## MCP Servers
3957
40- Three FastMCP servers provide tools to the agents at inference time:
58+ Three [ FastMCP] ( https://github.com/jlowin/fastmcp ) servers expose tools to agents at inference time:
59+
60+ | Server | Tools |
61+ | ---| ---|
62+ | ` guidelines-mcp ` | Intent definitions + Redis vector search for similar labeled examples |
63+ | ` label-schema-mcp ` | 77 valid intent labels + taxonomy + confusion-prone pairs |
64+ | ` human-review-mcp ` | SQLite-backed queue for low-confidence items awaiting human review |
4165
42- - ** guidelines-mcp** — intent definitions + Redis vector search for similar labeled examples
43- - ** label-schema-mcp** — 77 valid intent labels + taxonomy
44- - ** human-review-mcp** — SQLite-backed queue for low-confidence items
66+ ---
4567
4668## Stack
4769
4870| Layer | Technology |
4971| ---| ---|
5072| Orchestration | LangGraph + SqliteSaver checkpointing |
5173| MCP servers | FastMCP |
52- | Vector search | Redis Stack + sentence-transformers |
74+ | Vector search | Redis Stack + sentence-transformers/all-MiniLM-L6-v2 |
5375| API | FastAPI |
5476| Frontend | Next.js 15 + shadcn/ui + Recharts |
77+ | Tracing | Langfuse (self-hosted) |
5578| Dataset | Banking77 — PolyAI (CC-BY 4.0) |
5679
80+ ---
81+
5782## Setup
5883
5984``` bash
@@ -66,6 +91,8 @@ uv run python main.py # API on :8000
6691cd frontend && npm run dev # Dashboard on :3000
6792```
6893
94+ ---
95+
6996## Eval
7097
7198``` bash
@@ -80,3 +107,16 @@ caffeinate -i uv run python -m eval.run_eval \
80107caffeinate -i uv run python -m eval.retry_errors \
81108 --input eval_results.json --sleep 60
82109```
110+
111+ ---
112+
113+ ## Reference
114+
115+ Inspired by: * MAFA: Multi-Agent Framework for Annotation* , JP Morgan Chase, AAAI 2026.
116+ Dataset: [ PolyAI/banking77] ( https://github.com/PolyAI-LDN/task-specific-datasets ) — CC-BY 4.0.
117+
118+ ---
119+
120+ ## License
121+
122+ MIT © [ Savio Lobo] ( https://github.com/saviolobo )
0 commit comments