[Epic][TypeSpec Authoring] Collect user feedback from early adopters and telemetry analysis

[haolingdong-msft]

Overview

Collect user feedback for the azure-typespec-author Copilot skill to measure adoption, effectiveness, and identify areas for improvement.

Feedback Channels

1. Skill Invocation Telemetry

• Number of skill invocations (total and per-user)
• Task completion and failure rates
• Task type breakdown (API Versioning, Resource Definitions, Resource Operations, Models & Types)

2. Sanitized User Prompts Telemetry

• Collect sanitized/anonymized user prompts to understand common usage patterns
• Identify most/least used prompt categories
• Detect unclear or failing prompt patterns

3. User Conversations

• Conduct feedback sessions with early adopter teams
• Gather qualitative insights on authoring experience, output quality, and pain points

4. User Quick Feedback

• Collect in-session thumbs up/down or rating feedback after task completion
• Track user satisfaction trends over time