Skip to content

Benchmarks fix reporting going over token count#14837

Open
jeo02 wants to merge 9 commits intoAzure:mainfrom
jeo02:fix-reporting
Open

Benchmarks fix reporting going over token count#14837
jeo02 wants to merge 9 commits intoAzure:mainfrom
jeo02:fix-reporting

Conversation

@jeo02
Copy link
Copy Markdown
Member

@jeo02 jeo02 commented Mar 31, 2026

  • Benchmarks reports were adding to much information into result and causing rate limiting,
  • Avoid telemetry.
  • Remove do not verify setup from tests that don't need to avoid it

@jeo02 jeo02 requested a review from a team as a code owner March 31, 2026 21:52
@github-actions github-actions bot added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Mar 31, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the azsdk-cli benchmarks harness to reduce oversized benchmark outputs (to avoid rate limiting), disable telemetry during benchmark runs, and simplify several benchmark scenario prompts/validators around setup verification.

Changes:

  • Reduce benchmark tool-call reporting payload by omitting args/results for non-azsdk_ tools and removing the recorded MCP server name field.
  • Disable telemetry during benchmark workspace execution via AZSDKTOOLS_COLLECT_TELEMETRY=false.
  • Remove “do not run azsdk_verify_setup” prompt text and the corresponding forbiddenToolNames restriction from several scenarios.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Scenarios/TypeSpec/GetModifiedTypespecProjectsScenario.cs Removes prompt instruction and validator restriction related to azsdk_verify_setup.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Scenarios/TypeSpec/CheckPublicRepoThenValidateScenario.cs Removes prompt instruction and validator restriction related to azsdk_verify_setup.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Scenarios/TypeSpec/CheckPublicRepoScenario.cs Removes prompt instruction and validator restriction related to azsdk_verify_setup.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Scenarios/ReleasePlan/LinkNamespaceApprovalIssueScenario.cs Removes prompt instruction and validator restriction related to azsdk_verify_setup.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Scenarios/ReleasePlan/CreateReleasePlanScenario.cs Removes prompt instruction and validator restriction related to azsdk_verify_setup and tightens prompt wording.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Scenarios/Pipeline/CheckSdkGenerationStatusScenario.cs Removes prompt instruction and validator restriction related to azsdk_verify_setup.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Scenarios/GitHub/GetPrLinkCurrentBranchScenario.cs Removes prompt instruction and validator restriction related to azsdk_verify_setup.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Models/ToolCallRecord.cs Removes the McpServerName field from captured tool call records.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Infrastructure/WorkspaceManager.cs Disables telemetry for benchmark runs via environment variable.
tools/azsdk-cli/Azure.Sdk.Tools.Cli.Benchmarks/Infrastructure/SessionExecutor.cs Filters captured tool args/results to reduce benchmark output size.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 8 comments.

@jeo02 jeo02 requested a review from chunyu3 April 1, 2026 01:38
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants