Author: Jimmy Luo
Chat agent built on OpenAI's Agent SDK and ChatKit to assist Xfinity Home Internet customers, automating as many support tasks as possible.
- Lower Cost: Decrease customer service cost.
- Higher Sat: Increase or maintain parity of customer satisfaction.
- Informational: Questions that need answers, with routing to the appropriate support agent as-needed.
- Troubleshoot: Technical troubleshooting that can be turn-based performed with the user.
- Account: Customer support needs that require account access, e.g., billing, new/modify service, cancel, move
| Effort | Informational | Troubleshoot | Account | |
|---|---|---|---|---|
| MVP | 0.1 Eng | Full Support | ||
| M2 | 2 Eng | = | Step-by-step troubleshooting | |
| M3 | 6 Eng | = | = | Account access capability; initial limited support (e.g., first-line retention & sales) with long tail. |
Unsupported scenarios at any time are automatically routed to customer support to avoid drop-off and maintain parity.
- Chat UI client: developer-hosted frontend with embedded ChatKit UI, can be embedded on any site
- Repository (this): https://github.com/jimmy-zhening-luo/agent
- Session proxy server: developer-hosted server to negotiate session between client and agent host
- Endpoint: https://chat-server.jim.so/health
- Repository: https://github.com/jimmy-zhening-luo/agent-server
- Agent host: Agents SDK logic deployed to and hosted on OpenAI
- Ingress: User Prompt
- Input Guardrail: Jailbreak, Topicality
- Intent Classifier: route to correct agent to assist with customer support need
- Agents: task handlers for individual intent taxa
- Output Guardrail
- End: User Choice
- MVP: limited to "Done?" and "Connect to a live agent?"
- Future: next turn
- MVP: single-turn for informational question-answer
- M2+: multi-turn for multi-step tasks like troubleshooting and account assistance
OpenAI's off-the-shelf guardrail (GPT-5.4-nano) terminates chat if jailbreak or off-topic.
Intent top-level taxonomy is a 1:1 map to Customer Support Need:
- Informational
- Troubleshoot
- Account
An agent (OpenAI's GPT-5.5) classifies the support prompt into one of the three L1 intents.
Each intent has a specialized agent to handle it (1-to-1 mapping between L1 intent & agent).
Prompts that cannot be classified will fallback by offering to connect to a live agent.
Keeping intent classification at L1:
- Avoids over-structuring the intent taxonomy (allows agent to use built-in intelligence to solve a task, rather than forcibly constraining the task to some narrow intent)
- Enables hillclimbing L1 classification quality
- Enables hillclimbing support agent task fulfillment/coverage within each L1 intent taxa
Intent-to-agent map is a 1:1 map of L1 intent to corresponding agent:
- Informational
- Troubleshoot
- Account
- -> (Fallback/Escalation) Connect to agent
This agent will answer an informational question, and then ask the user whether their need has been met:
- If needs met, end chat.
- Else, fallback to "connect to live agent?" prompt.
A dual-corpus search strategy is used:
The support agent, OpenAI's GPT-5.5, retrieves information using its built-in web search tool, with instructions to source its answers from the canonical Xfinity Home Internet support sitemap node: https://www.xfinity.com/support/internet
Xfinity Home Internet support documents are already pre-indexed with high query traffic on Google Search, thus the agent is expected to achieve high accuracy and precision by issuing a targeted search query and distilling the top results.
GPT's web search tool also automatically grounds answers with the relevant support URL as citation; this structured grounding serves as a naive guardrail against hallucination.
The output of the agent is then filtered by an off-the-shelf hallucination guard agent (GPT-5.5) to guard against hallucinated informational answers. The guard agent issues a semantic search on an offline vector store to corroborate the answer generated by the support agent, dropping conflicting answers. The vector store contains query-to-document embeddings:
- Crawl the Xfinity support site documents.
- Transform documents to Markdown.
- Create a vector mapping between the question (possible prompts) and answer (site documents) spaces using OpenAI's off-the-shelf* semantic embedding generator.
*Large language models have built-in semantic understanding, which would compensate for the lack of real-world prompts to use as a prior for embedding generation.
Despite being faster and possibly higher-accuracy, semantic search is only used as a guardrail:
- Online web-search corpus uses search ranking for higher precision and recall.
- Web search citations give structured grounding for free, which would increase accuracy and precision. Semantic search on embeddings is fully unstructured, meaning structured grounding would need to be done as a separate step.
- Latency is not a critical factor here. Information lookup latency is relatively small compared to the existing informational customer support flow of finding and manually parsing information online.
In MVP, the Troubleshoot & Account agents will forward to the "connect to live agent" fallback.
Human raters will label golden eval sets for intent classifier and agents respectively. Release criteria will be gated on auto-evaluation results using GPT-5.5-Pro with extended thinking:
- Auto-evaluations will be run on m examples n times
- A random sample of disagreements between auto-evaluation runs, and a smaller random sample will be manually cross-checked and loss-bucketed by human raters for future hill-climbing of:
- Auto-evaluation prompt
- Objective criteria
- Golden set selection
- Golden set labeling
A golden eval set of minimum 60 (n^k) prompts must be completely, correctly classified 10 times out of 10 to meet release criteria (600 out of 600 correct). For each L1 intent added, the golden eval set grows exponentially.
The intent classifier golden set will include the eval examples from all agents, to enforce a given prompt is covered by the single, correct agent.
Eval set: https://github.com/jimmy-zhening-luo/agent/blob/main/eval/intent.csv
Each agent will also have a golden eval set of prompts that must meet user-needs-met release criteria. Each agent golden set is expected to grow over time (see Continuous Refinement below).
Each agent's user-needs-met will be evaluated by a human rater using the objective criteria defined for that agent.
10 canonical informational needs (pretend data that these are the top known info needs) are included in a golden eval with the following columns and to be evaluated as follows:
- The critical required information is provided for the question. The evaluation output is rated from 0 (needs unmet) to 2 (needs fully met):
| Score | Meaning | Criteria |
|---|---|---|
| 0 | Unmet | The critical information is not contained in the answer. |
| 1 | Partially Met | The critical information is partially contained in the answer, or fully contained in the answer but the answer contains conflicting information. |
| 2 | Fully Met | The critical information is fully contained in the answer, with no conflicting additional information. |
- The canonical citation page is given. Check the box if the correct page is cited.
Scoring for each evaluation output will be Needs Met (0-2) +1 if the citation page checkbox is checked.
The golden eval set of 20 evals will be run 5 times. The total score must be over 200 (out of 300 possible) to meet release criteria.
Eval set: https://github.com/jimmy-zhening-luo/agent/blob/main/eval/agents/informational.csv
- Mark for manual review a random sample of prompts that passed the input guardrail but failed to be classified & a smaller random sample of prompts that failed the input guardrail for false-positive abuse tuning.
- Reviewers will bucket losses & add intent-classifier eval-set cases for further hillclimbing. Further, input guardrail false positives will be noted but for now no hillclimbing, as we consider input guardrail false positives an upstream failure (i.e.., jailbreak identification is owned by the vendor, not by us).
- After release, a random sample of the total querystream & a higher random sample of the NOT user-needs-met querystream will be marked for manual review.
- Reviewers will bucket both losses & wins, and add agent eval-set cases for further hillclimbing in a query-weighted fashion:
- Common wins not covered in golden set evals would earn a canonical eval set example to avoid future regression
- Most common losses will also earn an eval-set slot for hillclimbing for next release.
- Some losses (but should be very few) may be a result of incorrect intent classification. These will be remitted back to intent classification for loss-bucketing and hillclimbing.
Inner loop development will use new and existing examples in single-sided evaluations with human graders to hillclimb toward a new launch candidate.
Outer loop development will use either or both:
- Single-sided evaluations to demonstrate full coverage of newly identified losses, using the same auto-evaluation described in the MVP evaluation plan.
- Double-sided evaluations to demonstrate qualitative (rated) improvements since last iteration on wins, using n runs of auto-evaluators rating the two sides respectively, with the same human cross-checking plan described in the MVP evaluation plan to hill-climb auto-evaluator quality.