runTests can't find tests in a file

[pamelafox]

The runTests tool cannot find any tests in my E2E tests file (e2e.py), even though it does have many tests. It might be because I named it "e2e.py" instead of "e2e_test.py" or "test_e2e.py", the standard pytest test discovery naming. That's intentional, as I always run it separately from the fast pytest tests.

Here's the file:
https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/tests/e2e.py

GitHub Copilot logs are below.

Version: 1.104.0-insider (Universal)
Commit: e296bdfe0313d571a6f58399b22afd199f6da454
Date: 2025-09-08T05:26:36.685Z
Electron: 37.3.1
ElectronBuildId: 12259562
Chromium: 138.0.7204.235
Node.js: 22.18.0
V8: 13.8.258.31-electron.0
OS: Darwin arm64 23.6.0

Steps to Reproduce:

1. Open https://github.com/Azure-Samples/azure-search-openai-demo/
2. Use "fixer" mode in GitHub Copilot
3. Ask it to run the E2E playwright tests

Tool Call - call_IU2uqaWgR9kLS0x409vkP0n3__vscode-1757360813940

Request

id   : call_IU2uqaWgR9kLS0x409vkP0n3__vscode-1757360813940
tool : runTests
args : {
  "files": [
    "/Users/pamelafox/azure-search-openai-demo/tests/e2e.py"
  ]
}

Response

No tests found in the files. Ensure the correct absolute paths are passed to the tool.

[AnasJadoon31]

E2E Test Discovery Issue in azure-search-openai-demo

Problem Analysis

You're absolutely correct about the test discovery issue. The runTests tool is designed to work with pytest's standard test discovery patterns, which expect test files to be named with test_ prefix or _test.py suffix. Since your e2e tests are intentionally named e2e.py to keep them separate from the fast unit tests, the tool can't find them.

GitHub Copilot Logs Analysis

Version: 1.104.0-insider (Universal)
Date: 2025-09-08T05:26:36.685Z
Issue: Tool Call runTests returned "No tests found in the files"
File: /Users/pamelafox/azure-search-openai-demo/tests/e2e.py

Test File Contents

Looking at your tests/e2e.py file from the Azure-Samples/azure-search-openai-demo repository, I can see it contains comprehensive Playwright tests including:

Test Functions Identified

• test_home() - Tests the basic home page
• test_chat() - Tests the main chat functionality with streaming
• test_chat_customization() - Tests developer settings customization
• test_chat_customization_multimodal() - Tests multimodal options
• test_chat_nonstreaming() - Tests non-streaming chat mode
• test_chat_followup_streaming() - Tests follow-up questions (streaming)
• test_chat_followup_nonstreaming() - Tests follow-up questions (non-streaming)
• test_ask() - Tests the Ask functionality
• test_upload_hidden() - Tests hidden file upload scenarios
• test_upload_disabled() - Tests disabled file upload scenarios

Key Test Infrastructure

# Fixtures for test setup
@pytest.fixture(scope="session")
def free_port() -> int:
    # Returns a free port for the test server

@pytest.fixture()
def live_server_url(mock_env, mock_acs_search, free_port: int):
    # Sets up live server for testing

@pytest.fixture(params=[(480, 800), (600, 1024), (768, 1024), (992, 1024), (1024, 768)])
def sized_page(page: Page, request):
    # Tests multiple viewport sizes

Solution: Running E2E Tests Manually

To run these e2e tests, you'll need to execute them directly using pytest with the specific file path:

Prerequisites

# Ensure you have the repository cloned
git clone https://github.com/Azure-Samples/azure-search-openai-demo.git
cd azure-search-openai-demo

# Install dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt

Basic E2E Test Execution

# First activate the virtual environment
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Run the e2e tests specifically
pytest tests/e2e.py -v

With Playwright Browser Visibility

# Install Playwright browsers if not already done
playwright install

# Run the e2e tests with visible browser
pytest tests/e2e.py -v --headed

Advanced Playwright Options

# Run in headless mode (default)
pytest tests/e2e.py -v

# Run with specific browser
pytest tests/e2e.py -v --browser chromium
pytest tests/e2e.py -v --browser firefox
pytest tests/e2e.py -v --browser webkit

# Run with debugging
pytest tests/e2e.py -v --headed --slowmo 1000

# Run specific test
pytest tests/e2e.py::test_chat -v

# Run with multiple workers (parallel execution)
pytest tests/e2e.py -v --numprocesses 2

# Generate test report
pytest tests/e2e.py -v --html=report.html

Environment Setup for E2E Tests

# The tests mock these environment variables
export AZURE_STORAGE_ACCOUNT="test-storage-account"
export AZURE_STORAGE_CONTAINER="test-storage-container"
export AZURE_STORAGE_RESOURCE_GROUP="test-storage-rg"
export AZURE_SUBSCRIPTION_ID="test-storage-subid"
export ENABLE_LANGUAGE_PICKER="false"
export USE_SPEECH_INPUT_BROWSER="false"
export USE_SPEECH_OUTPUT_AZURE="false"
export AZURE_SEARCH_INDEX="test-search-index"
export AZURE_SEARCH_SERVICE="test-search-service"
export AZURE_SPEECH_SERVICE_ID="test-id"
export AZURE_SPEECH_SERVICE_LOCATION="eastus"
export AZURE_OPENAI_SERVICE="test-openai-service"
export AZURE_OPENAI_CHATGPT_MODEL="gpt-4.1-mini"
export AZURE_OPENAI_EMB_MODEL_NAME="text-embedding-3-large"
export AZURE_OPENAI_EMB_DIMENSIONS="3072"

Alternative: Rename for Discovery

If you want the runTests tool to discover these tests automatically, you could:

Option 1: Rename the file

# Rename to follow pytest convention
mv tests/e2e.py tests/test_e2e.py
# or
mv tests/e2e.py tests/e2e_test.py

Option 2: Create a pytest configuration

# Create pytest.ini file
[tool:pytest]
testpaths = tests
python_files = test_*.py *_test.py e2e.py
python_classes = Test*
python_functions = test_*

Option 3: Use pytest collection configuration

# In pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py", "*_test.py", "e2e.py"]

Why This Naming Convention Makes Sense

The naming convention you're using (e2e.py) is intentional and practical for keeping these slower end-to-end tests separate from your fast unit tests. This approach:

Benefits

• Separation of Concerns: Prevents accidental inclusion in fast test suites
• CI/CD Pipeline Control: Allows for separate pipeline stages
• Clear Intent: Makes it obvious these are integration/e2e tests
• Performance: Avoids running slow tests during development
• Resource Management: E2E tests often require different setup/teardown

Trade-offs

• Tools relying on pytest's automatic discovery won't find them
• Requires explicit execution commands
• May need documentation for team members

Repository Context

Repository: Azure-Samples/azure-search-openai-demo
Description: A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.

Language Composition:

• Python: 65.7%
• TypeScript: 16.2%
• Bicep: 15.2%
• CSS: 1.9%
• PowerShell: 0.5%
• Shell: 0.5%

Recommended Workflow

For your development workflow with AnasJadoon31's repositories, consider this pattern:

# Fast development cycle
pytest tests/ -v --ignore=tests/e2e.py

# Full test suite including e2e
pytest tests/ -v

# E2E only (when needed)
pytest tests/e2e.py -v --headed

This approach maintains the separation you intended while providing flexibility for different testing scenarios.

[pamelafox]

@AnasJadoon31 I know all of that. (Was that LLM-generated?) My issue is about runTests tool, and whether it can find tests when explicitly given a filename that does not fit the test-discovery naming pattern. The pytest command can do so, so it seems reasonable that runTests tool could be extended to do so as well.

[eleanorjboyd]

Hi @pamelafox! If you run pytest from your terminal can it find and run these tests? The test behavior should behave just like the terminal so if you are able to get it running and confirm the args are the same between the terminal and running in VS Code then this will be a bug I can investigate. Thanks!

[pamelafox]

@eleanorjboyd I can run pytest tests/e2e.py and it runs them correctly. It won't find them if I just say pytest since I haven't named the file with test_/_test

[eleanorjboyd]

ok that makes sense- so you would need the tool to provide run args when it does it call since it would need to run pytest tests/e2e.py. I will investigate model being able to provide args for the python test runs and how this could work since I don't think it is supported at all right now

[wiperpaul]

Hey I also wanted to chime in here, i'm on version (1.105.1); I also had to tell chat to explicitly use the runtests tool otherwise it was trying to use the pytest terminal commands.

I also found it discovered the tests better after I added some explicit settings in settings.json. A follow up task used this in a few instances. The test blade and discovery was already configured for pytest and working before this.

I ran this a few times in different repos and sometimes a "Discovering tests..." step runs and performs a search for the test folders but often it just runs without any file paths or with the path of file you want to run the tests for.