[prompt-clustering] 🔬 Copilot Agent Prompt Clustering Analysis - November 27, 2025

[github-actions[bot]]

Executive Summary

Analyzed 1,290 copilot-generated pull requests using NLP clustering to identify task patterns and success factors.

Key Findings:

• Overall Success Rate: 74.7% (964/1,290 merged)
• 6 Distinct Task Clusters identified with success rates ranging from 66.7% to 77.6%
• Best Practice: Tasks with structured issue descriptions achieve 77.6% success rate
• Complexity Challenge: Large cross-cutting changes (34+ files) have only 66.7% success rate

Top Insight: ⭐ Well-structured, issue-based task descriptions lead to significantly better outcomes and require less iteration (1.0 vs 2.6 average comments).

---

📊 Cluster Overview

Cluster 1: General Maintenance (32.5%)

• Success Rate: 75.9%
• Keywords: update, add, github, file, mcp
• Characteristics: Balanced mix of documentation, features, and bug fixes
• Complexity: Medium (18.4 files, 701 additions)

Cluster 2: Issue-Based Workflows (21.7%) 🏆 HIGHEST SUCCESS

• Success Rate: 77.1%
• Keywords: issue, workflow, pkg, gh aw
• Characteristics: 97.5% documentation tasks with clear requirements
• Complexity: Low (11.5 files, 445 additions)
• Comments: Only 1.0 average (fewest)

Cluster 3: Structured Issues (15.9%) 🏆 HIGHEST SUCCESS

• Success Rate: 77.6%
• Keywords: issue, summary, title, cli
• Characteristics: Mix of documentation and bug fixes with templated descriptions
• Complexity: Medium (15.2 files, 417 additions)

Cluster 4: Agentic Workflow Development (14.3%)

• Success Rate: 73.0%
• Keywords: agentic, workflow, workflows, add
• Characteristics: Meta-work - copilot improving copilot
• Complexity: High line count (1,566 additions) in few files (10.5)

Cluster 5: Agent & MCP Configuration (7.9%)

• Success Rate: 68.6%
• Keywords: agent, copilot, docs, mcp, make
• Characteristics: Configuration and integration work
• Complexity: Medium (15.3 files, 934 additions)

Cluster 6: Safe Output Implementation (7.7%) ⚠️ MOST COMPLEX

• Success Rate: 66.7% (lowest)
• Keywords: output, safe, add, create, issue
• Characteristics: Large cross-cutting changes requiring significant iteration
• Complexity: Very High (34.6 files, 2,145 additions, 4.5 commits)
• Comments: 2.6 average (most) - indicates need for guidance

---

🎯 Actionable Recommendations

1. Adopt Structured Issue Templates 📝

Clusters 2 and 3 show that structured issue descriptions lead to 77%+ success rates. Recommended format:

[Task] Clear, actionable title

Summary: 2-3 sentences describing the goal

Context:
- Related issue: #123
- Affected files: list
- Dependencies: list

Acceptance Criteria:
- [ ] Criterion 1
- [ ] Criterion 2

2. Break Down Complex Tasks ✂️

Cluster 6 shows that tasks touching 34+ files have 66.7% success rate. Apply decomposition:

• Threshold: Max 20 files or 1,500 line additions per task
• Strategy: Split by component, layer, or feature
• Example: Instead of "Implement safe output type" → 3 tasks: define schema, implement core, add tests

3. Leverage Copilot for Documentation 📚

51.2% of tasks are documentation-related with high success rates:

• API documentation generation
• README updates
• Code comments
• Migration guides

4. Use Meta-Work Strategically 🔄

Cluster 4 shows 73% success with high-value agentic workflow improvements:

• Self-improvement loops
• Workflow optimization
• Automated maintenance

5. Provide Clear Context for Complex Work ⚠️

Safe output and agent config tasks need more guidance:

• Include code examples
• Link to similar implementations
• Specify architectural constraints
• Add explicit acceptance criteria

---

📈 Visualizations

Charts showing cluster analysis available in the workflow artifacts:

1. Cluster Sizes - Task distribution across 6 clusters
2. Success Rates by Cluster - Comparative PR merge rates
3. Clustering Optimization - Elbow and silhouette analysis
4. PCA Visualization - 2D task similarity map
5. Complexity Metrics - Files, additions, deletions by cluster

---

📋 Quick Stats

Metric	Value
Total PRs Analyzed	1,290
Merged (Success)	964 (74.7%)
Average Files Changed	16.3
Average Additions	854.2 lines
Average Commits	3.6
Average Comments	1.5

Success Rate Range: 66.7% (Safe Output) to 77.6% (Structured Issues)

Top Terms: issue, workflow, update, add, agentic, agent, copilot, github, workflows, mcp

---

🔬 Methodology

• Data: 1,290 copilot-generated PRs from githubnext/gh-aw
• NLP: TF-IDF vectorization (100 features, 1-3 word n-grams)
• Clustering: K-means with k=6 (optimized via elbow + silhouette)
• Analysis: Success rates, complexity metrics, task categorization
• Validation: Manual review of cluster coherence

---

📚 Full Report

Complete analysis with detailed findings, sample PRs, and recommendations available in workflow artifacts: clustering-report.md

Key Files:

• /tmp/gh-aw/pr-data/clustering-report.md - Full report
• /tmp/gh-aw/pr-data/cluster-analysis.json - Structured data
• /tmp/gh-aw/pr-data/processed-prompts.json - All analyzed prompts
• /tmp/gh-aw/python/charts/*.png - 5 visualizations

---

References:

• §19745795639

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot Agent Prompt Clustering Analysis.
🗺️ A newer treasure map awaits ye at Discussion #5325.
Fair winds, matey! 🏴‍☠️