Skip to content

Commit 1ada6f0

Browse files
savioloboclaude
andcommitted
docs: professional README with badges, motivation, and reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 3fa8454 commit 1ada6f0

1 file changed

Lines changed: 51 additions & 11 deletions

File tree

README.md

Lines changed: 51 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,23 @@
11
# Multi-Agent Banking Annotation System
22

3+
[![Lint](https://github.com/saviolobo/agentic-annotator/actions/workflows/lint.yml/badge.svg)](https://github.com/saviolobo/agentic-annotator/actions/workflows/lint.yml)
4+
[![Python](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/)
5+
[![LangGraph](https://img.shields.io/badge/LangGraph-1.2+-orange.svg)](https://github.com/langchain-ai/langgraph)
6+
[![FastMCP](https://img.shields.io/badge/FastMCP-enabled-purple.svg)](https://github.com/jlowin/fastmcp)
7+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
8+
39
A production-grade multi-agent pipeline for automated intent classification
410
on the Banking77 benchmark (13,083 queries, 77 intents), inspired by
511
JP Morgan's MAFA system (AAAI 2026).
612

13+
> **Why this exists:** Manual data annotation costs enterprises ~$0.15/item and
14+
> doesn't scale. JP Morgan Chase eliminated a 1M-utterance backlog saving 5,000+
15+
> hours/year using a multi-agent approach (MAFA, AAAI 2026). This project
16+
> independently reproduces and benchmarks a similar architecture on the public
17+
> Banking77 dataset, enabling direct comparison against their published results.
18+
19+
---
20+
721
## Results
822

923
| | This project | JP Morgan MAFA (AAAI 2026) |
@@ -17,43 +31,54 @@ JP Morgan's MAFA system (AAAI 2026).
1731
> Full pipeline uses 5 agents with MCP-backed guideline retrieval and
1832
> Redis vector search for similar-example lookup.
1933
34+
---
35+
2036
## Architecture
2137

2238
Five specialized agents replace manual annotation for high-confidence items
2339
while routing ambiguous cases to a human review queue.
2440

2541
![Architecture](docs/architecture.png)
2642

27-
**Key pattern:** LangGraph Send API runs Primary Annotator and Validator
43+
**Key pattern:** LangGraph Send API runs Primary Annotator Agent and Validator Agent
2844
in parallel on COMPLEX queries, cutting latency on the most expensive path.
2945

3046
| Agent | Model | Role |
3147
|---|---|---|
32-
| Router | Groq llama-3.1-8b-instant | Classify query complexity |
33-
| Primary Annotator | Cerebras gpt-oss-120b | First-pass label + confidence |
34-
| Validator | Cerebras gpt-oss-120b | Independent second annotation |
35-
| Arbitrator | Cerebras gpt-oss-120b | Resolve disagreements |
36-
| Quality Controller | Groq llama-3.1-8b-instant | Consistency + drift detection |
48+
| Router Agent | Groq llama-3.1-8b-instant | Classify query complexity (SIMPLE / COMPLEX) |
49+
| Primary Annotator Agent | Cerebras gpt-oss-120b | First-pass label + confidence + reasoning |
50+
| Validator Agent | Cerebras gpt-oss-120b | Independent second annotation, blind to Primary |
51+
| Arbitrator Agent | Cerebras gpt-oss-120b | Resolves disagreements, routes low-confidence to human |
52+
| Quality Controller Agent | Groq llama-3.1-8b-instant | Consistency checks + label drift detection |
53+
54+
---
3755

3856
## MCP Servers
3957

40-
Three FastMCP servers provide tools to the agents at inference time:
58+
Three [FastMCP](https://github.com/jlowin/fastmcp) servers expose tools to agents at inference time:
59+
60+
| Server | Tools |
61+
|---|---|
62+
| `guidelines-mcp` | Intent definitions + Redis vector search for similar labeled examples |
63+
| `label-schema-mcp` | 77 valid intent labels + taxonomy + confusion-prone pairs |
64+
| `human-review-mcp` | SQLite-backed queue for low-confidence items awaiting human review |
4165

42-
- **guidelines-mcp** — intent definitions + Redis vector search for similar labeled examples
43-
- **label-schema-mcp** — 77 valid intent labels + taxonomy
44-
- **human-review-mcp** — SQLite-backed queue for low-confidence items
66+
---
4567

4668
## Stack
4769

4870
| Layer | Technology |
4971
|---|---|
5072
| Orchestration | LangGraph + SqliteSaver checkpointing |
5173
| MCP servers | FastMCP |
52-
| Vector search | Redis Stack + sentence-transformers |
74+
| Vector search | Redis Stack + sentence-transformers/all-MiniLM-L6-v2 |
5375
| API | FastAPI |
5476
| Frontend | Next.js 15 + shadcn/ui + Recharts |
77+
| Tracing | Langfuse (self-hosted) |
5578
| Dataset | Banking77 — PolyAI (CC-BY 4.0) |
5679

80+
---
81+
5782
## Setup
5883

5984
```bash
@@ -66,6 +91,8 @@ uv run python main.py # API on :8000
6691
cd frontend && npm run dev # Dashboard on :3000
6792
```
6893

94+
---
95+
6996
## Eval
7097

7198
```bash
@@ -80,3 +107,16 @@ caffeinate -i uv run python -m eval.run_eval \
80107
caffeinate -i uv run python -m eval.retry_errors \
81108
--input eval_results.json --sleep 60
82109
```
110+
111+
---
112+
113+
## Reference
114+
115+
Inspired by: *MAFA: Multi-Agent Framework for Annotation*, JP Morgan Chase, AAAI 2026.
116+
Dataset: [PolyAI/banking77](https://github.com/PolyAI-LDN/task-specific-datasets) — CC-BY 4.0.
117+
118+
---
119+
120+
## License
121+
122+
MIT © [Savio Lobo](https://github.com/saviolobo)

0 commit comments

Comments
 (0)