Rename smoke-test to smoke-test-dotnet and add smoke-test-python scenario (#12390)

* Initial plan

* Rename smoke-test to smoke-test-dotnet and create smoke-test-python scenario

Co-authored-by: mitchdenny <[email protected]>

* Update agent scenarios README with smoke-test-dotnet and smoke-test-python documentation

Co-authored-by: mitchdenny <[email protected]>

* Fix SDK feature flag name to dotNetSdkInstallationEnabled

Co-authored-by: mitchdenny <[email protected]>

* Update dashboard port documentation to reflect random port assignment

Co-authored-by: mitchdenny <[email protected]>

* Update tests/agent-scenarios/README.md

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: mitchdenny <[email protected]>
Co-authored-by: Mitch Denny <[email protected]>
Co-authored-by: Copilot <[email protected]>

Author: 3 people

tests/agent-scenarios/README.md

@@ -72,3 +72,27 @@ The workflow will:
 Creates a basic Aspire starter application.
 
 **Prompt:** Create an aspire application starting by downloading the Aspire CLI and creating a starter app.
+
+### smoke-test-dotnet
+
+Performs a comprehensive smoke test of an Aspire PR build by installing the Aspire CLI, creating a .NET Blazor-based starter application, and verifying its functionality including the Dashboard, API service, and frontend.
+
+**Key features:**
+- Tests the native AOT build of the Aspire CLI
+- Creates and runs an Aspire starter app with Blazor frontend
+- Verifies Dashboard functionality and telemetry collection
+- Tests SDK install feature flag (`dotNetSdkInstallationEnabled`)
+- Captures screenshots for verification
+
+### smoke-test-python
+
+Performs a comprehensive smoke test of an Aspire PR build by installing the Aspire CLI, creating a Python starter application with Vite/React frontend, and verifying its functionality.
+
+**Key features:**
+- Tests the native AOT build of the Aspire CLI
+- Creates and runs an Aspire Python starter app (`aspire-py-starter`)
+- Tests Python backend API service and Vite/React frontend
+- Verifies Dashboard functionality and telemetry collection
+- Tests SDK install feature flag (`dotNetSdkInstallationEnabled`)
+- Tests hot reload for both Python and Vite
+- Captures screenshots for verification

tests/agent-scenarios/smoke-test/prompt.md renamed to tests/agent-scenarios/smoke-test-dotnet/prompt.md

@@ -37,6 +37,26 @@ aspire --version
 
 Expected output should show the version matching the PR build.
 
+### 1.1 Enable SDK Install Feature Flag
+
+Before proceeding, enable the `dotNetSdkInstallationEnabled` feature flag to force SDK installation for testing purposes. This ensures the Aspire CLI's SDK installation functionality is properly exercised.
+
+Set the configuration value:
+
+```bash
+aspire config set --global features.dotNetSdkInstallationEnabled true
+```
+
+Verify the configuration was set:
+
+```bash
+aspire config get --global features.dotNetSdkInstallationEnabled
+```
+
+Expected output: `true`
+
+**Note**: This feature flag forces the Aspire CLI to install the .NET SDK even if a compatible version is already available on the system. This is specifically for testing the SDK installation feature.
+
 ## Step 2: Create a New Aspire Starter Application
 
 Create a new Aspire application using the Blazor-based starter template. The application will be created in the current git workspace so it becomes part of the PR when the scenario completes.
@@ -106,33 +126,34 @@ The `aspire run` command will:
 **What to observe:**
 - The command should start the Aspire AppHost
 - You should see console output indicating:
-  - Dashboard starting (typically on http://localhost:18888)
+  - Dashboard starting with a randomly assigned port and access token
   - Resources being initialized
   - Services starting up
   - No critical errors in the startup logs
 
 ### 3.2 Wait for Startup
 
 Allow 30-60 seconds for the application to fully start. Monitor the console output for:
-- "Dashboard running at: http://localhost:XXXXX" message
+- "Dashboard running at: http://localhost:XXXXX" message with the access token
 - "Application started" or similar success messages
 - All resources showing as "Running" status
 
-**Tip:** The dashboard URL will be displayed in the console. Note this URL for later steps.
+**Tip:** The dashboard URL with access token will be displayed in the console output from `aspire run`. Note this complete URL (including the token parameter) for later steps. The port is randomly selected each time a new project is created.
 
 ## Step 4: Verify the Aspire Dashboard
 
 The Aspire Dashboard is the central monitoring interface. Let's verify it's accessible and functional.
 
 ### 4.1 Access the Dashboard
 
-Open the dashboard URL in a browser (typically http://localhost:18888).
+The dashboard URL with access token is displayed in the output from `aspire run`. Use this URL to access the dashboard.
 
 **Use browser automation tools to access and capture screenshots:**
 
 ```bash
-# Navigate to the dashboard
-playwright-browser navigate http://localhost:18888
+# Navigate to the dashboard using the URL from aspire run output
+# Example: DASHBOARD_URL="http://localhost:12345?token=abc123"
+playwright-browser navigate $DASHBOARD_URL
 ```
 
 **Take a screenshot of the dashboard:**
@@ -149,11 +170,11 @@ playwright-browser take_screenshot --filename dashboard-main.png
 **If browser automation fails, use curl for diagnostics:**
 
 ```bash
-# Check if dashboard is accessible
-curl -I http://localhost:18888
+# Check if dashboard is accessible using the URL from aspire run output
+curl -I $DASHBOARD_URL
 
 # Get the HTML content to diagnose issues
-curl http://localhost:18888
+curl $DASHBOARD_URL
 ```
 
 ### 4.2 Navigate Dashboard Sections
@@ -479,7 +500,7 @@ If issues occur during the smoke test:
 
 ### Startup Failures
 - Check Docker is running (if using containers)
-- Verify ports are not already in use (18888, 5000, 5001, etc.)
+- Verify ports are not already in use by other applications
 - Review console output for specific error messages
 - Check system resources (disk space, memory)