Skip to content

Conversation

@JennyPng
Copy link
Member

@JennyPng JennyPng commented Nov 8, 2025

Azure/azure-sdk-tools#12592

  • Adding instructions on using new Verify Setup MCP tool, which checks that requirements are installed in the environment
  • We want this check to be proactive and happen without explicit invokation. However, currently, Copilot would only remember if the tool is run in the current session, not in newer sessions, so the user would likely run verify setup in each new session due to a lack of a caching. This will be accepted for Scenario 1 and improved in future iterations

Example of implicitly running this tool before running a different requested tool
image

image

Explicit invokation
image

@JennyPng JennyPng changed the title instructions Add Env Setup to Copilot Instructions Nov 8, 2025
@JennyPng JennyPng marked this pull request as ready for review November 10, 2025 20:07
Copilot AI review requested due to automatic review settings November 10, 2025 20:07
@JennyPng JennyPng requested a review from maririos as a code owner November 10, 2025 20:07
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new prerequisite instruction to the Copilot instructions file that directs Copilot to run the azsdk_verify_setup tool before executing other MCP tools to verify the user's development environment setup.

Key changes:

  • Added a new prerequisite bullet point instructing Copilot to always run the azsdk_verify_setup tool before other tools
  • Positioned this instruction at the top of the Prerequisites section for visibility

Copy link
Member

@m-redding m-redding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lgtm

This repo contains Azure SDKs for various Azure services and tools that generate them.

## Prerequisites
- Always run the [`azsdk_verify_setup`](../eng/common/instructions/azsdk-tools/verify-setup.instructions.md) tool to check the user's development environment setup for using SDK MCP tools. DO NOT PROCEED with running other tools before completing this step. Only skip for queries that don't need tools.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you'll have to use an absolute link to pass the pipeline

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LLM seems to perform better with the relative link than if I linked to a web page, and I noticed some repos are currently using relative links 🤔 (example from rest-api-specs with check package readiness https://github.com/Azure/azure-rest-api-specs/blob/main/.github/copilot-instructions.md#release-readiness-of-sdk-and-information-about-the-release-pipeline )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohh that's interesting, but that does make sense to me since copilot likes when things are available in the same workspace. It looks like the verify links is an eng/common check. I'm not totally sure where the requirement comes from, but maybe we could revisit

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weshaggard for feedback on the link check.
I have seen in some repos the CI fails and in some it doesn't.

FYI the check is in place because our guidelines say that we should always use absolute paths

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes our link checker ensures links are absolute because most of them get published to docs sites outside of our repo.

However for instructions @jeo02 found a workaround which is to use a link like #file:../eng/common/instructions/azsdk-tools/verify-setup-instructions.md as our link checker sees that as a anchor link but it appears to still be resolved correctly by the LLM.

Copy link
Member Author

@JennyPng JennyPng Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this out, it looks like adding the #file has the same effect of using an absolute path where the LLM doesn't read it for me? (Copilot will say 1 reference used instead of 2 including the verify-setup-instructions)
image

vs.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is interesting. According to copilot chat it doesn't look at any links only open files.
image

Copy link
Member

@jeo02 jeo02 Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#file:path would only work if the file your trying to refer is in the same directory which is not the case here unfortunately. I believe in the copilot instructions in rest-api-specs we still use the relative path to refer to instructions under common/instructions.

GitHub links are hit or miss as it is up to copilot to follow it in the moment whereas relative links will add the whole file referenced to the end of the copilot instructions. (You can see this in the Copilot Logs)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok if we need to use relative links in these MD files we can either override the link checking or opt-out these instructions from link checking.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got PR Azure/azure-sdk-tools#12863 which will disable these link guidelines for instruction files. For now, we can by-pass this check and merge this PR.

@weshaggard weshaggard merged commit c169bc8 into Azure:main Nov 12, 2025
8 of 11 checks passed
@JennyPng JennyPng deleted the env-setup-instr branch November 12, 2025 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants