diff --git a/.gitignore b/.gitignore index a05922aa..a8137335 100644 --- a/.gitignore +++ b/.gitignore @@ -200,4 +200,8 @@ app_data.db # Don't commit the evaluation data *.evalset.json -.adk/ \ No newline at end of file +.adk/ + + +run-with-google-adk/libs/markdown +run-with-google-adk/libs/markdown-3.8.2.dist-info \ No newline at end of file diff --git a/docs/run_with_google_adk/advanced_topics.md b/docs/run_with_google_adk/advanced_topics.md new file mode 100644 index 00000000..aec4781f --- /dev/null +++ b/docs/run_with_google_adk/advanced_topics.md @@ -0,0 +1,10 @@ +# Advanced Topics + +### Improving Performance and Optimizing Costs +You can control the number of previous interactions sent to the LLM by setting the `MAX_PREV_USER_INTERACTIONS` variable in your `.env` file. A lower number (default is 3) reduces context size, which can improve performance and lower costs. + +### Integrating Custom MCP Servers +This agent can be extended to include your own MCP servers. For a detailed guide and examples, please refer to the `docs/development_guide.md` file. + +### Additional Features +The agent supports creating files and generating signed URLs for them via the artifact service. For example, you can ask the agent to "add the summary as markdown to summary_146.md" and then "create a link to file summary_146.md". diff --git a/docs/run_with_google_adk/agentspace_app_setup.md b/docs/run_with_google_adk/agentspace_app_setup.md new file mode 100644 index 00000000..d4113c6e --- /dev/null +++ b/docs/run_with_google_adk/agentspace_app_setup.md @@ -0,0 +1,261 @@ +# Agentspace App Setup Guide + +This guide walks you through creating and configuring a Google Agentspace application, which is a prerequisite for deploying your MCP Security Agent with Agentspace integration. + +## Overview + +Agentspace is Google's platform for building and deploying conversational AI agents. Before you can integrate your MCP Security Agent with Agentspace, you need to: + +1. Create an Agentspace application +2. Configure the necessary permissions and APIs +3. Obtain the required configuration values for your `.env` file + +## Prerequisites + +Before starting, ensure you have: +- A Google Cloud project with billing enabled +- Administrative access to your Google Cloud project +- Discovery Engine API enabled in your project +- Vertex AI API enabled in your project + +## Step 1: Enable Required APIs + +First, enable the necessary Google Cloud APIs: + +```bash +# Enable Discovery Engine API (required for Agentspace) +gcloud services enable discoveryengine.googleapis.com + +# Enable Vertex AI API (required for Agent Engine) +gcloud services enable aiplatform.googleapis.com + +# Verify APIs are enabled +gcloud services list --enabled --filter="name:discoveryengine.googleapis.com OR name:aiplatform.googleapis.com" +``` + +## Step 2: Set Up IAM Permissions + +Ensure your account has the necessary permissions: + +```bash +# Check current permissions +gcloud projects get-iam-policy $GOOGLE_CLOUD_PROJECT + +# Add necessary roles (if not already present) +gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \ + --member="user:your-email@domain.com" \ + --role="roles/discoveryengine.admin" + +gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \ + --member="user:your-email@domain.com" \ + --role="roles/aiplatform.user" +``` + +**Required Roles:** +- `roles/discoveryengine.admin` - Create and manage Agentspace apps +- `roles/aiplatform.user` - Deploy and manage Agent Engine resources +- `roles/iam.serviceAccountTokenCreator` - Generate access tokens for API calls + +## Step 3: Create Agentspace Application + +### 3.1 Access the Agentspace Console + +1. **Navigate to Discovery Engine Console** + - Go to [Google Cloud Console - Agentspace](https://console.cloud.google.com/gen-app-builder/engines) + - Ensure you're in the correct project + +2. **Create a New App** + - Click "Create App" + - Choose "Agentspace (Preview)" as the app type and click Create + - Select your preferred region (e.g., `global`, `us`, or `eu`) + +### 3.2 Configure Your Agentspace App + +**Basic Configuration:** +- **App Name**: Choose a descriptive App name (e.g., "MCP Security Agent") +- **App ID**: This will be auto-generated (e.g., `mcp-security-agent-app_1234567890`) +- **Location**: Choose the same region as your other resources + + +### 3.3 App Creation Results + +After creating your app, note these important values: + +1. **App Name**: The full app identifier (e.g., `mcp-security-agent-app_1234567890`) +2. **Project Number**: Your Google Cloud project number (numeric) +3. **Location**: The region where your app is deployed + +## Step 4: Obtain Configuration Values + +### 4.1 Get Project Number + +```bash +# Get your project number +gcloud projects describe $GOOGLE_CLOUD_PROJECT --format="value(projectNumber)" +``` + +### 4.2 Get App Information + +```bash +# List your Agentspace apps +gcloud alpha discovery-engine engines list --location=$LOCATION --project=$GOOGLE_CLOUD_PROJECT + +# Get specific app details +gcloud alpha discovery-engine engines describe YOUR_APP_NAME \ + --location=$LOCATION \ + --project=$GOOGLE_CLOUD_PROJECT +``` + +### 4.3 Update Your .env File + +Add the Agentspace configuration to your `.env` file: + +```bash +# Agentspace Configuration +AGENTSPACE_PROJECT_NUMBER=1234567890 # Your Google Cloud project number +AGENTSPACE_APP_NAME=mcp-security-agent-app_1234567890 # Agentspace app identifier +AGENTSPACE_AGENT_ID= # Populated when agent is registered with Agentspace +``` + +**Using Make Commands:** + +```bash +# Update project number +make env-update KEY=AGENTSPACE_PROJECT_NUMBER VALUE="1234567890" + +# Update app name +make env-update KEY=AGENTSPACE_APP_NAME VALUE="mcp-security-agent-app_1234567890" +``` + +## Step 5: Verify Agentspace Setup + +### 5.1 Test API Access + +Verify you can access the Agentspace APIs: + +```bash +# Test Discovery Engine API access +PROJECT_NUMBER=$(gcloud projects describe $(gcloud config get-value project) --format='value(projectNumber)') +LOCATION=global +curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + -H "X-Goog-User-Project: $(gcloud config get-value project)" \ + "https://discoveryengine.googleapis.com/v1alpha/projects/$PROJECT_NUMBER/locations/$LOCATION/collections/default_collection/engines" +``` + +### 5.2 Validate Environment + +Check your Agentspace configuration: + +```bash +# Validate Agentspace environment variables +make env-validate-agentspace +``` + +## Understanding Agentspace Integration + +### What Agentspace Actually Is + +Agentspace is **not a separate platform** - it's a web UI service within Google Cloud's Discovery Engine that provides a chat interface for interacting with Agent Engine deployments. + +### How It Works + +1. **Agent Engine**: Your agent is deployed to Vertex AI Agent Engine (the actual compute/logic layer) +2. **Agentspace App**: A Discovery Engine configuration that provides a web UI +3. **Registration**: You register your Agent Engine deployment with your Agentspace App +4. **Access**: Users interact with your agent through the Agentspace web interface + +### Simple Architecture + +``` +User → Agentspace Web UI → Agent Engine (your deployed agent) +``` + +**That's it!** Agentspace is essentially a UI wrapper around Agent Engine deployments. + +## Common Agentspace URLs + +After setup, you'll access your Agentspace app through these URLs: + +- **Main Console**: `https://console.cloud.google.com/ai/discovery-engine/engines/YOUR_APP_NAME/overview` +- **Agent Management**: `https://console.cloud.google.com/ai/discovery-engine/engines/YOUR_APP_NAME/agents` +- **Integration URL**: Available after agent deployment (use `make agentspace-url`) + +## Troubleshooting + +### Common Issues + +1. **"Permission denied" errors** + - Verify IAM roles are properly assigned + - Check that APIs are enabled + - Ensure you're using the correct project + +2. **"App not found" errors** + - Verify app name is correct + - Check that app was created in the expected region + - Confirm project number matches your Google Cloud project + +3. **API quota exceeded** + - Check Discovery Engine API quotas in Cloud Console + - Request quota increases if needed + +### Debug Commands + +```bash +# Check current project and permissions +gcloud config list +gcloud auth list + +# Verify API enablement +gcloud services list --enabled | grep -E "(discoveryengine|aiplatform)" + +# List Agentspace resources +gcloud alpha discovery-engine engines list --location=global + +# Check environment configuration +make config-show +``` + +## Security Considerations + +1. **Access Control** + - Use least-privilege IAM policies + - Regularly audit Agentspace access + - Monitor API usage and costs + +2. **Data Privacy** + - Review data handling policies + - Configure appropriate data retention + - Understand data residency requirements + +3. **API Security** + - Use service accounts for production + - Implement proper authentication + - Monitor API access logs + +## Next Steps + +After completing Agentspace app setup: + +1. **Deploy Agent Engine**: Follow the [Agent Engine Quickstart](agentspace_quickstart.md) +2. **Configure OAuth**: Set up OAuth authentication using the [OAuth Setup Guide](oauth_agentspace_setup.md) +3. **Register Agent**: Deploy your agent to Agentspace using `make agentspace-register` + +## Environment Variables Summary + +After completing this guide, your `.env` file should include: + +```bash +# Google Cloud Configuration +GOOGLE_CLOUD_PROJECT=your-project-id +GOOGLE_CLOUD_LOCATION=us-central1 + +# Agentspace Configuration +AGENTSPACE_PROJECT_NUMBER=1234567890 +AGENTSPACE_APP_NAME=mcp-security-agent-app_1234567890 +AGENTSPACE_AGENT_ID= # Populated after agent registration + +# Agent Engine Configuration (from previous setup) +AGENT_ENGINE_RESOURCE_NAME=projects/PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_ID +``` + +These values are required for the OAuth setup and agent registration processes that follow. \ No newline at end of file diff --git a/docs/run_with_google_adk/agentspace_quickstart.md b/docs/run_with_google_adk/agentspace_quickstart.md new file mode 100644 index 00000000..0b4ccffe --- /dev/null +++ b/docs/run_with_google_adk/agentspace_quickstart.md @@ -0,0 +1,42 @@ +# Quick Start: Deploy to AgentSpace (via Agent Engine) + +This path is for users who want to integrate the agent with the full AgentSpace ecosystem, enabling more advanced features and interactions. + +### Prerequisites + +- Complete the prerequisites from the [Cloud Run Quick Start](cloud_run_quickstart.md). +- Ensure your `.env` file is correctly configured with your project details. + +### Deployment and Registration Steps + +1. **Deploy to Agent Engine:** + This step provisions the agent in Vertex AI Agent Engine. + ```bash + make adk-deploy + ``` + After deployment, note the `AGENT_ENGINE_RESOURCE_NAME` from the output. + +2. **Update Environment:** + Add the resource name to your `.env` file. + ```bash + make env-update KEY=AGENT_ENGINE_RESOURCE_NAME VALUE= + ``` + +3. **Configure OAuth for AgentSpace:** + This is a crucial step to allow AgentSpace to securely communicate with your agent. + ```bash + make oauth-setup + ``` + This interactive command will guide you through creating an OAuth client, generating an authorization URL, and linking it to your AgentSpace project. This process generates the required `OAUTH_AUTH_ID`. + +4. **Register with AgentSpace:** + With OAuth configured, you can now register your agent. + ```bash + make agentspace-register + ``` + +5. **Verify and Test:** + Finally, verify the integration and test the agent's functionality. + ```bash + make agentspace-verify + make test-agent MSG="List available security tools." diff --git a/docs/run_with_google_adk/cloud_run_quickstart.md b/docs/run_with_google_adk/cloud_run_quickstart.md new file mode 100644 index 00000000..32d29d1f --- /dev/null +++ b/docs/run_with_google_adk/cloud_run_quickstart.md @@ -0,0 +1,28 @@ +# Quick Start: Deploy to Cloud Run + +This path is recommended for users who want a simple, standalone deployment of the agent as a web service. + +### Prerequisites + +- Ensure you have `gcloud` CLI installed and configured. +- Enable the required APIs for Cloud Run deployment. See [Cloud Run Source Deployment Docs](https://cloud.google.com/run/docs/deploying-source-code#before_you_begin). +- Set the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` variables in your `.env` file (`run-with-google-adk/agents/google_mcp_security_agent/.env`). + +### Deployment Steps + +1. **Deploy the Service:** + From the `run-with-google-adk` directory, run the following command: + ```bash + make cloudrun-deploy + ``` + This command packages the agent and deploys it as a service on Cloud Run. + +2. **Test the Service:** + Once deployed, you can test your service using: + ```bash + make cloudrun-test + ``` + This will provide the service URL and send a test request. + +> ⚠️ **Security Warning:** +> By default, the Cloud Run service is deployed with unauthenticated invocations enabled for initial testing. It is critical to **secure your service** by enabling IAM authentication. Follow the guide [here](https://cloud.google.com/run/docs/authenticating/developers). diff --git a/docs/run_with_google_adk/index.md b/docs/run_with_google_adk/index.md new file mode 100644 index 00000000..1edde0a0 --- /dev/null +++ b/docs/run_with_google_adk/index.md @@ -0,0 +1,11 @@ +# Running with Google ADK + +This section provides comprehensive guidance on deploying and managing the prebuilt ADK (Agent Development Kit) agent. Whether you're looking for a quick deployment to a standalone service or a full integration with AgentSpace, these guides will walk you through the process. + +## Table of Contents + +* [**Cloud Run Quickstart**](cloud_run_quickstart.md): For a simple, standalone deployment of the agent as a web service. +* [**AgentSpace Quickstart**](agentspace_quickstart.md): For integrating the agent with the full AgentSpace ecosystem. +* [**Local Setup Guide**](local_setup.md): For setting up and running the agent on your local machine for development. +* [**Makefile Reference**](makefile_reference.md): A detailed reference for all available `make` commands. +* [**Advanced Topics**](advanced_topics.md): Information on performance tuning, custom MCP integrations, and other advanced features. diff --git a/docs/run_with_google_adk/local_setup.md b/docs/run_with_google_adk/local_setup.md new file mode 100644 index 00000000..e7cf8046 --- /dev/null +++ b/docs/run_with_google_adk/local_setup.md @@ -0,0 +1,38 @@ +# Detailed Local Setup Guide + +For development purposes, you can run the agent entirely on your local machine. + +### Prerequisites +- `python` v3.11+ +- `pip` +- `gcloud` CLI + +### Setup and Run + +```bash +# Clone the repo +git clone https://github.com/google/mcp-security.git + +# Navigate to the agent directory +cd mcp-security/run-with-google-adk + +# Create and activate a virtual environment +python3 -m venv .venv +source .venv/bin/activate + +# Install dependencies +pip install -r requirements.txt + +# Create your .env file from the template +cp agents/google_mcp_security_agent/.env.template agents/google_mcp_security_agent/.env + +# Edit agents/google_mcp_security_agent/.env with your credentials and settings +# (e.g., GOOGLE_API_KEY, CHRONICLE_PROJECT_ID, etc.) + +# Authenticate for Google Cloud APIs +gcloud auth application-default login + +# Run the agent with the ADK Web UI +./scripts/run-adk-agent.sh adk_web +``` +You can now access the agent's web interface at `http://localhost:8000`. diff --git a/docs/run_with_google_adk/makefile_reference.md b/docs/run_with_google_adk/makefile_reference.md new file mode 100644 index 00000000..ccab29f3 --- /dev/null +++ b/docs/run_with_google_adk/makefile_reference.md @@ -0,0 +1,33 @@ +# Makefile Commands Reference + +This project uses a `Makefile` to simplify common operations. Run `make help` to see all available commands. + +### Environment Management +- `make env-check`: Validate required environment variables. +- `make config-show`: Display the current configuration (masks secrets). +- `make env-update KEY=... VALUE=...`: Update a variable in the `.env` file. + +### OAuth Management +- `make oauth-setup`: Run the complete OAuth setup workflow. +- `make oauth-client`: Guide for creating an OAuth client. +- `make oauth-uri`: Generate an OAuth authorization URI. +- `make oauth-link`: Link OAuth credentials to AgentSpace. +- `make oauth-verify`: Verify the OAuth configuration. + +### Agent Engine Deployment +- `make adk-deploy`: Deploy the agent to Agent Engine. +- `make test-agent`: Test the deployed agent with a message. +- `make agents-list`: List all deployed Agent Engine instances. +- `make agents-delete INDEX=...`: Delete a specific Agent Engine instance. + +### AgentSpace Integration +- `make agentspace-register`: Register the agent with AgentSpace. +- `make agentspace-update`: Update an existing AgentSpace registration. +- `make agentspace-verify`: Verify the AgentSpace integration. +- `make agentspace-url`: Get the AgentSpace UI URL. + +### Cloud Run Deployment +- `make cloudrun-deploy`: Deploy the agent to Cloud Run. +- `make cloudrun-test`: Test the deployed Cloud Run service. +- `make cloudrun-logs`: View logs for the Cloud Run service. +- `make cloudrun-url`: Get the URL for the Cloud Run service. diff --git a/docs/run_with_google_adk/oauth_agentspace_setup.md b/docs/run_with_google_adk/oauth_agentspace_setup.md new file mode 100644 index 00000000..c73cd972 --- /dev/null +++ b/docs/run_with_google_adk/oauth_agentspace_setup.md @@ -0,0 +1,543 @@ +# OAuth Setup for AgentSpace Integration + +This guide walks you through the complete process of setting up OAuth authentication for your MCP Security Agent with Google's AgentSpace platform. This enables your agent to access Google services on behalf of users while maintaining secure authentication. + +## Overview + +AgentSpace OAuth integration allows your deployed agent to: +- Access Google services (Drive, Calendar, etc.) on behalf of users +- Maintain secure authentication without exposing credentials +- Provide seamless user experience with proper authorization flows + +## Prerequisites + +Before starting, ensure you have: +- A Google Cloud project with billing enabled +- Agent Engine deployment completed (see [Agent Engine Quickstart](agentspace_quickstart.md)) +- **AgentSpace app created and configured** (see [AgentSpace App Setup](agentspace_app_setup.md)) +- Proper IAM permissions for OAuth client creation + +## Step 1: Create OAuth Client + +### 1.1 Generate OAuth Client Guide + +Run the guided setup to create your OAuth client: + +```bash +make oauth-client +``` + +**Expected Output:** +``` +Starting OAuth client creation guide... +DEBUG: Loading env file from: /Users/your-user/path/to/.env +============================================================ +OAuth Client Creation Guide +============================================================ + +Steps to create an OAuth 2.0 client: + +1. Open the Google Cloud Console: + https://console.cloud.google.com/apis/credentials + (Make sure you're in project: your-project-id) + +2. Click '+ CREATE CREDENTIALS' → 'OAuth client ID' + +3. If prompted to configure consent screen: + - Choose 'Internal' for organization use + - Fill in required fields + - Add scopes if needed + +4. For Application type, select 'Web application' + +5. Configure the client: + - Name: 'AgentSpace MCP Security Agent' (or similar) + - Authorized redirect URIs: + https://vertexaisearch.cloud.google.com/oauth-redirect + +6. Click 'CREATE' + +7. Download the client configuration: + - Click the download button (⬇) next to your new client + - Save as 'client_secret.json' in this directory + +Would you like to open the Google Cloud Console now? (y/n): +``` + +**What This Command Does:** + +1. **Environment Loading**: The script loads your `.env` file to read the current project configuration +2. **Project Detection**: It automatically detects your Google Cloud project ID from the environment variables +3. **Interactive Guide**: Provides a step-by-step guide with specific instructions for your project +4. **Console Link**: Offers to open the Google Cloud Console directly to the credentials page +5. **Specific Configuration**: Provides exact values you need to enter, including the critical redirect URI + +**Key Points About the Output:** + +- **Project-Specific URL**: The console URL includes your specific project ID, taking you directly to the right place +- **Redirect URI**: The most critical piece - `https://vertexaisearch.cloud.google.com/oauth-redirect` - this must be exact +- **Application Type**: Must be "Web application" for AgentSpace integration +- **File Location**: Tells you exactly where to save the downloaded `client_secret.json` file + +**Following the Interactive Prompts:** + +1. **Answer "y" to open the console**: If you respond with "y", the script will attempt to open your browser to the Google Cloud Console +2. **Answer "n" to continue manually**: If you respond with "n", you can manually navigate to the URL provided + +**After Creating the OAuth Client:** + +Once you complete the steps in the Google Cloud Console, you'll have: + +1. **OAuth Client ID**: A string like `1234567890-abc123def456.apps.googleusercontent.com` +2. **OAuth Client Secret**: A string like `GOCSPX-AbC123DeF456GhI789JkL012` +3. **Downloaded JSON file**: Contains both the client ID and secret in JSON format + +**What to Do with the client_secret.json:** + +The downloaded file will look like this: +```json +{ + "web": { + "client_id": "1234567890-abc123def456.apps.googleusercontent.com", + "project_id": "your-project-id", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://oauth2.googleapis.com/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "client_secret": "[redacted]", + "redirect_uris": ["https://vertexaisearch.cloud.google.com/oauth-redirect"] + } +} +``` + +**Important:** Save this file in your project's root directory (same level as the Makefile) and **never commit it to version control**. + +### 1.1.2 Extracting Credentials to .env File + +After downloading the `client_secret.json` file, you need to extract the OAuth credentials and add them to your `.env` file. + +**Method 1: Using Make Command (Recommended)** + +You can automatically extract and add the OAuth credentials using: + +```bash +make env-update KEY=OAUTH_CLIENT_ID VALUE="your-client-id-from-json" +make env-update KEY=OAUTH_CLIENT_SECRET VALUE="your-client-secret-from-json" +``` + +**Method 2: Manual Extraction** + +1. **Open the client_secret.json file** and locate these values: + ```json + { + "web": { + "client_id": "1234567890-abc123def456.apps.googleusercontent.com", + "client_secret": "[redacted]", + "token_uri": "https://oauth2.googleapis.com/token" + } + } + ``` + +2. **Add these variables to your `.env` file**: + ```bash + # OAuth Client Configuration + OAUTH_CLIENT_ID=1234567890-abc123def456.apps.googleusercontent.com + OAUTH_CLIENT_SECRET=[redacted] + OAUTH_TOKEN_URI=https://oauth2.googleapis.com/token + ``` + +**Where to Find Each Value:** + +- **OAUTH_CLIENT_ID**: Located at `web.client_id` in the JSON file +- **OAUTH_CLIENT_SECRET**: Located at `web.client_secret` in the JSON file +- **OAUTH_TOKEN_URI**: Located at `web.token_uri` in the JSON file (usually `https://oauth2.googleapis.com/token`) + +**Example of Complete .env OAuth Section:** + +After adding the OAuth credentials, your `.env` file should include: + +```bash +# OAuth Client Configuration (added manually) +OAUTH_CLIENT_ID=1002894625780-a65bv4jgplu1pk1ljc2pkki6kjsocqjr.apps.googleusercontent.com +OAUTH_CLIENT_SECRET=[redacted] +OAUTH_TOKEN_URI=https://oauth2.googleapis.com/token + +# OAuth Session Variables (auto-generated in next steps) +OAUTH_AUTH_ID=mcp-agent-20250704-104439 # Auto-generated unique identifier +OAUTH_AUTH_URI=https://accounts.google.com/o/oauth2/auth?client_id=... +``` + +**Important Note About OAUTH_AUTH_ID:** + +The `OAUTH_AUTH_ID` is **not** from Google or the JSON file. It's a **unique identifier that we make up** to name our OAuth authorization in AgentSpace. The system automatically generates one with the format `mcp-agent-YYYYMMDD-HHMMSS` (e.g., `mcp-agent-20250704-104439`) when you run `make oauth-uri` for the first time. + +- **You don't need to create this manually** - it's auto-generated +- **You can customize it** if you want a more descriptive name (e.g., `my-security-agent-oauth`) +- **It must be unique** within your AgentSpace project +- **It's used internally** by AgentSpace to track your OAuth authorization + +**Verification:** + +You can verify your OAuth configuration is loaded correctly: + +```bash +# Check current OAuth configuration (secrets will be masked) +make config-show + +# Validate OAuth environment variables +make env-validate-oauth +``` + +### 1.1.1 OAuth Consent Screen Configuration + +If this is your first time creating an OAuth client in your project, you'll be prompted to configure the OAuth consent screen. Here's what to expect: + +**Step 1: Choose User Type** +- **Internal**: For Google Workspace organizations - only users in your organization can use the app +- **External**: For any Google account - requires app verification for production use + +**Step 2: App Information** +Fill in the required fields: +- **App name**: "AgentSpace MCP Security Agent" or similar +- **User support email**: Your email address +- **App logo**: Optional, but helps with branding +- **App domain**: Your organization's domain (if applicable) +- **Developer contact information**: Your email address + +**Step 3: Scopes** +The most common scopes for AgentSpace integration: +- `https://www.googleapis.com/auth/cloud-platform` - Access to Google Cloud services +- `https://www.googleapis.com/auth/generative-language.retriever` - Vertex AI Search access + +**Step 4: Test Users** (External apps only) +- Add email addresses of users who can test your app before verification +- You can add up to 100 test users during development + +**Note**: For internal apps in Google Workspace organizations, users must be in the same organization. For external apps intended for broad use, you'll need to submit for verification once ready for production. + +### 1.2 Manual OAuth Client Creation + +If you prefer to create the client manually: + +1. **Open Google Cloud Console** + - Navigate to [Google Cloud Console - Credentials](https://console.cloud.google.com/apis/credentials) + - Ensure you're in the correct project (check project ID in your `.env` file) + +2. **Configure OAuth Consent Screen** (if not already done) + - Click "Configure Consent Screen" + - Choose "Internal" for organization use or "External" for broader access + - Fill in required application information: + - Application name: "AgentSpace MCP Security Agent" + - User support email: Your email + - Developer contact information: Your email + - Add necessary scopes (e.g., `https://www.googleapis.com/auth/cloud-platform`) + +3. **Create OAuth Client** + - Click "+ CREATE CREDENTIALS" → "OAuth client ID" + - Select "Web application" as the application type + - Configure the client: + - **Name**: "AgentSpace MCP Security Agent" + - **Authorized redirect URIs**: `https://vertexaisearch.cloud.google.com/oauth-redirect` + +4. **Download Client Secret** + - Click "CREATE" + - Download the client configuration as `client_secret.json` + - Save this file securely (do not commit to version control) + +## Step 2: Generate OAuth Authorization URI + +### 2.1 Using Make Target + +Generate the OAuth authorization URL: + +```bash +make oauth-uri +``` + +This will: +- Read your OAuth client configuration from your `.env` file +- Generate a properly formatted authorization URL with the default scopes: + - `https://www.googleapis.com/auth/cloud-platform` + - `https://www.googleapis.com/auth/generative-language.retriever` +- Update your `.env` file with the authorization URI + +**Customizing OAuth Scopes:** + +If you need different scopes, you can set them in your `.env` file before running `make oauth-uri`: + +```bash +# Custom scopes (comma-separated) +OAUTH_SCOPES=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/calendar.readonly +``` + +The system will use your custom scopes instead of the defaults. + +### 2.2 Manual URI Generation + +If you need to generate the URI manually, you can use Python: + +```python +import google_auth_oauthlib.flow + +# Load your client secret file +flow = google_auth_oauthlib.flow.Flow.from_client_secrets_file( + 'client_secret.json', + scopes=[ + 'https://www.googleapis.com/auth/cloud-platform', + 'https://www.googleapis.com/auth/generative-language.retriever' + ] # These are the same default scopes used by make oauth-uri +) + +# Set the redirect URI +flow.redirect_uri = 'https://vertexaisearch.cloud.google.com/oauth-redirect' + +# Generate the authorization URL +authorization_url, state = flow.authorization_url( + access_type='offline', + include_granted_scopes='true', + prompt='consent' +) + +print('Visit this URL to authorize:', authorization_url) +``` + +## Step 3: User Authorization + +### 3.1 Visit Authorization URL + +1. **Get the Authorization URL** + - From Step 2, copy the generated `OAUTH_AUTH_URI` from your `.env` file + - Or check the output from `make oauth-uri` + +2. **Complete Authorization** + - Visit the authorization URL in your browser + - Sign in with your Google account + - Review the requested permissions + - Click "Allow" to grant access + +3. **Note the Redirect** + - After authorization, you'll be redirected to the AgentSpace OAuth redirect URL + - The URL will contain an authorization code (this is handled automatically) + +## Step 4: Link OAuth to AgentSpace + +### 4.1 Environment Validation + +First, ensure your environment has all required OAuth variables: + +```bash +make env-validate-oauth +``` + +### 4.2 Link Authorization + +Link your OAuth configuration to AgentSpace: + +```bash +make oauth-link +``` + +This command will: +- Create an OAuth authorization record in AgentSpace +- Link your OAuth client to your AgentSpace project +- Configure the authorization for your agent + +### 4.3 Manual Linking Process + +The linking process performs the following API call: + +```bash +curl -X POST \ + -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + -H "Content-Type: application/json" \ + -H "X-Goog-User-Project: ${PROJECT_NUMBER}" \ +https://discoveryengine.googleapis.com/v1alpha/projects/${PROJECT_NUMBER}/locations/global/authorizations?authorizationId=${AUTH_ID} \ + -d '{ + "name": "projects/'"${PROJECT_NUMBER}"'/locations/global/authorizations/'"${AUTH_ID}"'", + "serverSideOauth2": { + "clientId": "'"${OAUTH_CLIENT_ID}"'", + "clientSecret": "'"${OAUTH_CLIENT_SECRET}"'", + "authorizationUri": "'"${OAUTH_AUTH_URI}"'", + "tokenUri": "'"${OAUTH_TOKEN_URI}"'" + } + }' +``` + +## Step 5: Register Agent with AgentSpace + +### 5.1 Agent Registration + +Register your agent with AgentSpace and link the OAuth authorization: + +```bash +make agentspace-register +``` + +This will: +- Create an agent record in AgentSpace +- Link your deployed Agent Engine resource +- Associate the OAuth authorization with your agent + +### 5.2 Update Existing Registration + +If you need to update an existing agent registration: + +```bash +make agentspace-update +``` + +## Step 6: Verify OAuth Setup + +### 6.1 Verify Configuration + +Check that your OAuth setup is working correctly: + +```bash +make oauth-verify +``` + +### 6.2 Verify AgentSpace Integration + +Verify the complete AgentSpace integration: + +```bash +make agentspace-verify +``` + +### 6.3 Get AgentSpace URL + +Get the URL to access your agent in AgentSpace: + +```bash +make agentspace-url +``` + +## Environment Variables Reference + +Your `.env` file should contain these OAuth-related variables: + +```bash +# OAuth Client Configuration (from client_secret.json) +OAUTH_CLIENT_ID=your-client-id.apps.googleusercontent.com # From JSON: web.client_id +OAUTH_CLIENT_SECRET=your-client-secret # From JSON: web.client_secret +OAUTH_TOKEN_URI=https://oauth2.googleapis.com/token # From JSON: web.token_uri + +# OAuth Session Variables (auto-generated by make commands) +OAUTH_AUTH_ID=mcp-agent-YYYYMMDD-HHMMSS # Auto-generated unique ID (can be customized) +OAUTH_AUTH_URI=https://accounts.google.com/o/oauth2/auth?... # Generated authorization URL + +# AgentSpace Configuration +AGENTSPACE_PROJECT_NUMBER=your-project-number +AGENTSPACE_APP_NAME=your-app-name +AGENTSPACE_AGENT_ID=your-agent-id + +# Agent Engine Configuration +AGENT_ENGINE_RESOURCE_NAME=projects/PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_ID +``` + +**Variable Sources:** +- **OAUTH_CLIENT_ID/SECRET/TOKEN_URI**: Come from the `client_secret.json` file you download from Google Cloud Console +- **OAUTH_AUTH_ID**: A unique name we create to identify this OAuth authorization in AgentSpace (auto-generated or custom) +- **OAUTH_AUTH_URI**: Generated by combining your client ID with proper scopes and redirect URI +- **AGENTSPACE_***: Configuration specific to your AgentSpace app +- **AGENT_ENGINE_RESOURCE_NAME**: The resource name of your deployed Agent Engine + +## Common OAuth Scopes + +Depending on your agent's functionality, you may need these scopes: + +- `https://www.googleapis.com/auth/cloud-platform` - Full Google Cloud access +- `https://www.googleapis.com/auth/generative-language.retriever` - Vertex AI Search access +- `https://www.googleapis.com/auth/drive.readonly` - Google Drive read access +- `https://www.googleapis.com/auth/calendar.readonly` - Google Calendar read access +- `https://www.googleapis.com/auth/gmail.readonly` - Gmail read access + +## Troubleshooting + +### Common Issues + +1. **"redirect_uri_mismatch" Error** + - Ensure the redirect URI in your OAuth client matches exactly: `https://vertexaisearch.cloud.google.com/oauth-redirect` + - Check for trailing slashes or typos + +2. **"invalid_client" Error** + - Verify your `OAUTH_CLIENT_ID` and `OAUTH_CLIENT_SECRET` are correct + - Ensure the OAuth client is properly configured in Google Cloud Console + +3. **Authorization Not Found** + - Check that `OAUTH_AUTH_ID` is unique and properly formatted + - Verify your project number is correct + +4. **Permission Denied** + - Ensure your Google Cloud account has proper permissions + - Check that Discovery Engine API is enabled + - Verify the service account has necessary IAM roles + +### Debug Commands + +Check your OAuth configuration: + +```bash +# Show current configuration (masks secrets) +make config-show + +# Validate OAuth-specific environment +make env-validate-oauth + +# List all AgentSpace agents +make agents-list + +# Verify AgentSpace integration +make agentspace-verify +``` + +## Security Best Practices + +1. **Client Secret Protection** + - Never commit `client_secret.json` to version control + - Store OAuth client secrets securely + - Rotate client secrets regularly + +2. **Scope Minimization** + - Only request the minimum scopes needed + - Review and audit scope usage regularly + +3. **Environment Management** + - Use separate OAuth clients for different environments + - Implement proper secret management for production + +4. **Monitoring** + - Monitor OAuth usage and token refresh patterns + - Set up alerts for authentication failures + +## Complete Setup Workflow + +Here's the complete workflow summary: + +```bash +# 0. Prerequisites (complete these first) +# - Create AgentSpace app (see AgentSpace App Setup guide) +# - Deploy Agent Engine (see Agent Engine Quickstart guide) + +# 1. Create OAuth client (follow guided setup) +make oauth-client + +# 2. Generate authorization URI +make oauth-uri + +# 3. Visit the authorization URI in browser and authorize + +# 4. Link OAuth to AgentSpace +make oauth-link + +# 5. Register agent with AgentSpace +make agentspace-register + +# 6. Verify setup +make oauth-verify +make agentspace-verify + +# 7. Get AgentSpace URL +make agentspace-url +``` + +After completing these steps, your agent will be accessible through AgentSpace with proper OAuth authentication, allowing users to securely access Google services through your MCP Security Agent. \ No newline at end of file diff --git a/docs/toc.md b/docs/toc.md index 3a018849..a1dd940c 100644 --- a/docs/toc.md +++ b/docs/toc.md @@ -2,6 +2,12 @@ * [Development Guide](development_guide.md) * [Usage Guide](usage_guide.md) +* [Running with Google ADK](run_with_google_adk/index.md) + * [Cloud Run Quickstart](run_with_google_adk/cloud_run_quickstart.md) + * [AgentSpace Quickstart](run_with_google_adk/agentspace_quickstart.md) + * [Local Setup](run_with_google_adk/local_setup.md) + * [Makefile Reference](run_with_google_adk/makefile_reference.md) + * [Advanced Topics](run_with_google_adk/advanced_topics.md) * [Servers](servers/index.md) * [Google Threat Intelligence](servers/gti_mcp.md) * [Security Command Center](servers/scc_mcp.md) diff --git a/run-with-google-adk/Makefile b/run-with-google-adk/Makefile new file mode 100644 index 00000000..3c8b605c --- /dev/null +++ b/run-with-google-adk/Makefile @@ -0,0 +1,497 @@ +# Makefile for ADK Runbooks project +# + +# Environment file is now loaded by the env_manager.py script + +# Python executable (use activated venv if available) +PYTHON := python3 + +# Multi-agent directory +MULTIAGENT_DIR := multi-agent + +# Cookiecutter directories +COOKIECUTTER_TEMPLATE := cookiecutter-adk-agent +COOKIECUTTER_BUILDER := cookiecutter-agent-builder + +# Agent management script +MANAGE_AGENTS := $(PYTHON) scripts/manage_agents.py + +# Environment management script +ENV_FILE_PATH := agents/google_mcp_security_agent/.env +ENV_MANAGER := $(PYTHON) scripts/env_manager.py + +.PHONY: help env-setup env-check env-update config-show oauth-client oauth-uri oauth-link oauth-verify oauth-setup docs agents-list agents-delete multi-agent-setup multi-agent-run multi-agent-web cookiecutter-setup cookiecutter-run cookiecutter-new-agent adk-deploy adk-redeploy agentspace-register agentspace-update agentspace-verify agentspace-delete agentspace-url cloudrun-deploy cloudrun-run cloudrun-test cloudrun-logs cloudrun-url cloudrun-delete gcloud-proxy test-agent + +# Put it first so that "make" without argument is like "make help". +help: + @echo "Available targets:" + @echo "" + @echo "Environment Management:" + @echo " env-setup - Create .env from .env.sample" + @echo " env-check - Validate required environment variables" + @echo " config-show - Display current configuration (masks secrets)" + @echo " env-update - Update environment variable" + @echo "" + @echo "OAuth Management:" + @echo " oauth-client - Guide for creating OAuth client" + @echo " oauth-uri - Generate OAuth authorization URI" + @echo " oauth-link - Link OAuth to AgentSpace" + @echo " oauth-verify - Verify OAuth configuration" + @echo " oauth-setup - Complete OAuth setup workflow" + @echo "" + @echo "Agent Management:" + @echo " agents-list - List all deployed Agent Engine instances" + @echo " agents-delete - Delete Agent Engine instance by index" + @echo " adk-deploy - Deploy agent to Agent Engine" + @echo " adk-redeploy - Redeploy agent to existing Agent Engine resource" + @echo " test-agent - Test deployed agent with a message" + @echo "" + @echo "AgentSpace Integration:" + @echo " agentspace-register - Register agent with AgentSpace" + @echo " agentspace-update - Update existing AgentSpace registration" + @echo " agentspace-verify - Verify AgentSpace integration" + @echo " agentspace-delete - Remove agent from AgentSpace" + @echo " agentspace-url - Get AgentSpace UI URL" + @echo "" + @echo "Cloud Run Deployment:" + @echo " cloudrun-deploy - Deploy agent to Cloud Run" + @echo " cloudrun-run - Run Cloud Run service locally" + @echo " cloudrun-test - Test deployed Cloud Run service" + @echo " cloudrun-logs - View Cloud Run service logs" + @echo " cloudrun-url - Get Cloud Run service URL" + @echo " cloudrun-delete - Delete Cloud Run service" + @echo "" + @echo "Development Tools:" + @echo " docs - Build documentation" + @echo " gcloud-proxy - Proxy to Cloud Run service locally" + @echo "" + @echo "Multi-Agent System:" + @echo " multi-agent-setup - Set up multi-agent environment" + @echo " multi-agent-run - Run the manager agent" + @echo " multi-agent-web - Run the multi-agent web UI" + @echo "" + @echo "Agent Templates:" + @echo " cookiecutter-setup - Set up cookiecutter agent builder" + @echo " cookiecutter-run - Run the cookiecutter web builder" + @echo " cookiecutter-new-agent - Create new agent from template" + @echo "" + @echo "Environment variables from .env will be automatically loaded." + @echo "" + @echo "Examples:" + @echo " make env-check" + @echo " make config-show" + @echo " make env-update KEY=GOOGLE_CLOUD_PROJECT VALUE=my-project" + @echo " make oauth-setup" + @echo " make agents-list" + @echo " make adk-deploy AGENT_DIR=. DISPLAY_NAME='Security Agent'" + @echo " make test-agent MSG='List security tools'" + @echo " make agentspace-register" + @echo " make agentspace-verify" + @echo " make agentspace-url" + +# Environment Management targets +env-setup: + @echo "Creating .env file from example..." + @if [ -f "agents/google_mcp_security_agent/.env.example" ]; then \ + cp agents/google_mcp_security_agent/.env.example agents/google_mcp_security_agent/.env; \ + echo "✓ .env file created successfully."; \ + echo "Please review and update the values in agents/google_mcp_security_agent/.env"; \ + else \ + echo "Error: agents/google_mcp_security_agent/.env.example not found."; \ + exit 1; \ + fi + +env-check: + @echo "Checking environment configuration..." + @$(ENV_MANAGER) check --env-file $(ENV_FILE_PATH) --deployment $(or $(DEPLOYMENT),base) + + +config-show: + @$(ENV_MANAGER) show --env-file $(ENV_FILE_PATH) $(if $(ALL),--all) + +env-update: +ifndef KEY + @echo "Error: KEY parameter required. Usage: make env-update KEY=VAR_NAME VALUE=var_value" + @exit 1 +endif +ifndef VALUE + @echo "Error: VALUE parameter required. Usage: make env-update KEY=VAR_NAME VALUE=var_value" + @exit 1 +endif + @$(ENV_MANAGER) update --env-file $(ENV_FILE_PATH) --key $(KEY) --value "$(VALUE)" + +# Validate environment for specific deployment types +env-validate-agent-engine: + @$(ENV_MANAGER) check --env-file $(ENV_FILE_PATH) --deployment agent_engine + +env-validate-agentspace: + @$(ENV_MANAGER) check --env-file $(ENV_FILE_PATH) --deployment agentspace + +env-validate-oauth: + @$(ENV_MANAGER) check --env-file $(ENV_FILE_PATH) --deployment oauth + +# OAuth Management targets +OAUTH_MANAGER := $(PYTHON) scripts/oauth_manager.py --env-file agents/google_mcp_security_agent/.env + +oauth-client: + @echo "Starting OAuth client creation guide..." + @$(OAUTH_MANAGER) guide + +oauth-uri: + @echo "Generating OAuth authorization URI..." + @$(OAUTH_MANAGER) generate $(if $(CLIENT_SECRET),--client-secret $(CLIENT_SECRET)) + +oauth-link: env-validate-oauth + @echo "Linking OAuth to AgentSpace..." + @$(OAUTH_MANAGER) link + +oauth-verify: + @echo "Verifying OAuth configuration..." + @$(OAUTH_MANAGER) verify + +oauth-setup: oauth-client + @echo "" + @echo "OAuth Setup Workflow:" + @echo "1. Follow the guide above to create an OAuth client" + @echo "2. Download client_secret.json to this directory" + @echo "3. Run: make oauth-uri" + @echo "4. Open the generated URL and authorize" + @echo "5. Run: make oauth-link" + +# Documentation targets +docs: + @echo "Building Sphinx documentation..." + @cd rules-bank && $(MAKE) html + +# Agent management targets +agents-list: + @echo "Listing Agent Engine instances..." +ifdef PROJECT +ifdef LOCATION + $(MANAGE_AGENTS) --project $(PROJECT) --location $(LOCATION) list $(if $(VERBOSE),-v) +else + $(MANAGE_AGENTS) --project $(PROJECT) list $(if $(VERBOSE),-v) +endif +else + $(MANAGE_AGENTS) list $(if $(VERBOSE),-v) +endif + +agents-delete: +ifndef INDEX + @echo "Error: INDEX parameter required. Usage: make agents-delete INDEX=1" + @exit 1 +endif + @echo "Deleting Agent Engine instance at index $(INDEX)..." +ifdef PROJECT +ifdef LOCATION + $(MANAGE_AGENTS) --project $(PROJECT) --location $(LOCATION) delete --index $(INDEX) $(if $(FORCE),--force) +else + $(MANAGE_AGENTS) --project $(PROJECT) delete --index $(INDEX) $(if $(FORCE),--force) +endif +else + $(MANAGE_AGENTS) delete --index $(INDEX) $(if $(FORCE),--force) +endif + +# Multi-agent system targets +multi-agent-setup: + @echo "Setting up multi-agent environment..." + @if [ ! -f "$(MULTIAGENT_DIR)/manager/.env" ]; then \ + echo "Creating .env file in $(MULTIAGENT_DIR)/manager/"; \ + echo "GOOGLE_API_KEY=your_api_key_here" > "$(MULTIAGENT_DIR)/manager/.env"; \ + echo "Please edit $(MULTIAGENT_DIR)/manager/.env with your API key"; \ + fi + @cd $(MULTIAGENT_DIR) && pip install -r ../requirements.txt + +multi-agent-run: + @echo "Running manager agent..." + @cd $(MULTIAGENT_DIR) && adk run manager + +multi-agent-web: + @echo "Starting multi-agent web UI..." + @cd $(MULTIAGENT_DIR) && adk web + +# Cookiecutter targets +cookiecutter-setup: + @echo "Setting up cookiecutter agent builder..." + @if [ ! -d "$(COOKIECUTTER_BUILDER)/venv" ]; then \ + echo "Creating virtual environment for cookiecutter builder..."; \ + cd $(COOKIECUTTER_BUILDER) && python3 -m venv venv; \ + fi + @echo "Installing dependencies..." + @cd $(COOKIECUTTER_BUILDER) && \ + source venv/bin/activate && \ + pip install -r requirements.txt && \ + pip install cookiecutter + @echo "Running migrations..." + @cd $(COOKIECUTTER_BUILDER) && \ + source venv/bin/activate && \ + python manage.py migrate + @echo "Cookiecutter builder setup complete!" + @echo "Run 'make cookiecutter-run' to start the web interface" + +cookiecutter-run: + @echo "Starting cookiecutter agent builder web interface..." + @echo "Access at http://localhost:8000" + @cd $(COOKIECUTTER_BUILDER) && \ + source venv/bin/activate && \ + python manage.py runserver + +cookiecutter-new-agent: +ifndef NAME + @echo "Error: NAME parameter required. Usage: make cookiecutter-new-agent NAME=my-agent" + @exit 1 +endif + @echo "Creating new agent '$(NAME)' from cookiecutter template..." + @if [ ! -f ~/.local/bin/cookiecutter ] && [ ! -f venv/bin/cookiecutter ]; then \ + echo "Installing cookiecutter..."; \ + pip install cookiecutter; \ + fi + @cookiecutter $(COOKIECUTTER_TEMPLATE) --no-input project_name="$(NAME)" project_slug="$(NAME)" || \ + cookiecutter $(COOKIECUTTER_TEMPLATE) + @echo "Agent '$(NAME)' created successfully!" + @echo "Check the new directory for your agent files." + +# ADK deployment targets +adk-deploy: + @# Use agents/google_mcp_security_agent directory if AGENT_DIR not specified + $(eval AGENT_DIR := $(or $(AGENT_DIR),agents/google_mcp_security_agent)) + @echo "Deploying agent from directory: $(AGENT_DIR)" + @if [ ! -d "$(AGENT_DIR)" ]; then \ + echo "Error: Directory $(AGENT_DIR) does not exist"; \ + exit 1; \ + fi + @# Install markdown + @echo "Installing markdown..." + @pip install markdown + @# Copy libs and server to agent directory for deployment + @echo "Copying libs to $(AGENT_DIR)..." + @cp -R libs "$(AGENT_DIR)/" + @echo "Copying server directory to $(AGENT_DIR)..." + @if [ -d "../server" ]; then \ + cp -R ../server "$(AGENT_DIR)/"; \ + echo "Server directory copied from ../server"; \ + echo "Removing .venv directories from copied server"; \ + find "$(AGENT_DIR)/server" -name ".venv" -type d -exec rm -rf {} + 2>/dev/null || true; \ + else \ + echo "ERROR: ../server directory not found!"; \ + echo "Expected server at: $$(pwd)/../server"; \ + echo "Available directories:"; \ + ls -la ../; \ + exit 1; \ + fi + @# Use environment variables with proper fallbacks + $(eval PROJECT := $(or $(PROJECT),$(shell grep "^GOOGLE_CLOUD_PROJECT=" $(ENV_FILE_PATH) | cut -d= -f2))) + $(eval REGION := $(or $(REGION),$(shell grep "^GOOGLE_CLOUD_LOCATION=" $(ENV_FILE_PATH) | cut -d= -f2),us-central1)) + $(eval DISPLAY_NAME := $(or $(DISPLAY_NAME),$(AGENT_DISPLAY_NAME),"Google Security Agent")) + $(eval STAGING_BUCKET := $(or $(STAGING_BUCKET),$(GCS_STAGING_BUCKET))) + @# Generate staging bucket if not provided + @BUCKET_TO_USE='$(STAGING_BUCKET)'; \ + if [ -z "$$BUCKET_TO_USE" ]; then \ + BUCKET_TO_USE="gs://agent-deploy-$(PROJECT)-$(shell date +%Y%m%d-%H%M%S)"; \ + echo "Generated staging bucket: $$BUCKET_TO_USE"; \ + fi; \ + echo "Deployment configuration:"; \ + echo " PROJECT: $(PROJECT)"; \ + echo " REGION: $(REGION)"; \ + echo " DISPLAY_NAME: $(DISPLAY_NAME)"; \ + echo " STAGING_BUCKET: $$BUCKET_TO_USE"; \ + echo " AGENT_DIR: $(AGENT_DIR)"; \ + echo ""; \ + echo "Deploying agent to Agent Engine..."; \ + adk deploy agent_engine \ + --project $(PROJECT) \ + --region $(REGION) \ + --staging_bucket "$$BUCKET_TO_USE" \ + --display_name '$(DISPLAY_NAME)' \ + $(if $(TRACE),--trace_to_cloud) \ + $(AGENT_DIR); + @# Clean up copied libs and server + @echo "Cleaning up copied libs from $(AGENT_DIR)..." + @rm -rf "$(AGENT_DIR)/libs" + @echo "Cleaning up copied server from $(AGENT_DIR)..." + @rm -rf "$(AGENT_DIR)/server" + @echo "" + @echo "✓ Agent deployed successfully!" + @echo "" + @echo "Next steps:" + @echo "1. Note the resource name from the output above" + @echo "2. Update .env with: make env-update KEY=AGENT_ENGINE_RESOURCE_NAME VALUE=" + @echo "3. Test the agent: make test-agent" + @echo "4. Register with AgentSpace: make agentspace-register" + +# AgentSpace management targets +AGENTSPACE_MANAGER := $(PYTHON) scripts/agentspace_manager.py + +agentspace-register: env-validate-agentspace + @echo "Registering agent with AgentSpace..." + @$(AGENTSPACE_MANAGER) register --env-file $(ENV_FILE_PATH) $(if $(FORCE),--force) + +agentspace-update: env-validate-agentspace + @echo "Updating AgentSpace registration..." + @$(AGENTSPACE_MANAGER) update --env-file $(ENV_FILE_PATH) + +agentspace-verify: + @echo "Verifying AgentSpace integration..." + @$(AGENTSPACE_MANAGER) verify --env-file $(ENV_FILE_PATH) + +agentspace-delete: + @echo "Removing agent from AgentSpace..." + @$(AGENTSPACE_MANAGER) delete --env-file $(ENV_FILE_PATH) $(if $(FORCE),--force) + +agentspace-url: + @$(AGENTSPACE_MANAGER) url --env-file $(ENV_FILE_PATH) + +# Cloud Run deployment targets +cloudrun-deploy: env-check + @echo "Deploying agent to Cloud Run..." + @echo "Deployment configuration:" + @echo " PROJECT: $$(grep GOOGLE_CLOUD_PROJECT $(ENV_FILE_PATH) | cut -d= -f2)" + @echo " REGION: $$(grep GOOGLE_CLOUD_LOCATION $(ENV_FILE_PATH) | cut -d= -f2)" + @echo " SERVICE: mcp-security-agent-service" + @echo "" + @# Run the deployment script + @bash ./scripts/cloudrun_deploy_run.sh deploy + @echo "" + @echo "✓ Cloud Run deployment complete!" + @echo "" + @echo "Next steps:" + @echo "1. Test the service: make cloudrun-test" + @echo "2. View logs: make cloudrun-logs" + @echo "3. Get service URL: make cloudrun-url" + +cloudrun-run: env-check + @echo "Running Cloud Run service locally..." + @cd .. && bash ./run-with-google-adk/scripts/cloudrun_deploy_run.sh run + +cloudrun-test: + @echo "Testing Cloud Run service..." + $(eval PROJECT := $(or $(PROJECT),$(GOOGLE_CLOUD_PROJECT))) + $(eval REGION := $(or $(REGION),$(GOOGLE_CLOUD_LOCATION),us-central1)) + $(eval SERVICE := mcp-security-agent-service) + @echo "Getting service URL..." + $(eval SERVICE_URL := $(shell gcloud run services describe $(SERVICE) --project $(PROJECT) --region $(REGION) --format='value(status.url)' 2>/dev/null)) + @if [ -z "$(SERVICE_URL)" ]; then \ + echo "Error: Could not get service URL. Is the service deployed?"; \ + exit 1; \ + fi + @echo "Service URL: $(SERVICE_URL)" + @echo "" + @echo "Testing with curl..." + @curl -s -X POST $(SERVICE_URL)/health || echo "Note: /health endpoint may not exist" + @echo "" + @echo "To test interactively, visit: $(SERVICE_URL)" + +cloudrun-logs: + @echo "Viewing Cloud Run logs..." + $(eval PROJECT := $(or $(PROJECT),$(shell grep "^GOOGLE_CLOUD_PROJECT=" $(ENV_FILE_PATH) | cut -d= -f2))) + $(eval REGION := $(or $(REGION),$(shell grep "^GOOGLE_CLOUD_LOCATION=" $(ENV_FILE_PATH) | cut -d= -f2),us-central1)) + $(eval SERVICE := mcp-security-agent-service) + gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=$(SERVICE)" \ + --project $(PROJECT) \ + --limit 50 \ + --format "table(timestamp,severity,textPayload)" + +cloudrun-url: + $(eval PROJECT := $(or $(PROJECT),$(GOOGLE_CLOUD_PROJECT))) + $(eval REGION := $(or $(REGION),$(GOOGLE_CLOUD_LOCATION),us-central1)) + $(eval SERVICE := mcp-security-agent-service) + @echo "Getting Cloud Run service URL..." + @gcloud run services describe $(SERVICE) \ + --project $(PROJECT) \ + --region $(REGION) \ + --format='value(status.url)' || echo "Service not found. Deploy first with: make cloudrun-deploy" + +cloudrun-delete: + $(eval PROJECT := $(or $(PROJECT),$(GOOGLE_CLOUD_PROJECT))) + $(eval REGION := $(or $(REGION),$(GOOGLE_CLOUD_LOCATION),us-central1)) + $(eval SERVICE := mcp-security-agent-service) + @echo "Deleting Cloud Run service..." + @echo " SERVICE: $(SERVICE)" + @echo " PROJECT: $(PROJECT)" + @echo " REGION: $(REGION)" + @echo "" + @read -p "Are you sure you want to delete this service? (y/N) " confirm; \ + if [ "$$confirm" = "y" ] || [ "$$confirm" = "Y" ]; then \ + gcloud run services delete $(SERVICE) \ + --project $(PROJECT) \ + --region $(REGION) \ + --quiet && \ + echo "✓ Service deleted successfully"; \ + else \ + echo "Deletion cancelled"; \ + fi + +# Cloud Run proxy targets +gcloud-proxy: +ifndef SERVICE + @echo "Error: SERVICE parameter required. Usage: make gcloud-proxy SERVICE=my-service" + @exit 1 +endif + @echo "Starting gcloud proxy for Cloud Run service: $(SERVICE)" + @# Set default values if not provided + $(eval PROJECT := $(or $(PROJECT),$(shell echo $$GOOGLE_CLOUD_PROJECT || echo $$GCP_PROJECT))) + $(eval REGION := $(or $(REGION),$(shell echo $$GOOGLE_CLOUD_REGION || echo $$GOOGLE_CLOUD_LOCATION || echo "us-central1"))) + $(eval PORT := $(or $(PORT),8080)) + @echo "Using PROJECT: $(PROJECT)" + @echo "Using REGION: $(REGION)" + @echo "Using PORT: $(PORT)" + @if [ -z "$(PROJECT)" ]; then \ + echo "Error: PROJECT not set. Use PROJECT=your-project or set GOOGLE_CLOUD_PROJECT env var"; \ + exit 1; \ + fi + @echo "Proxying $(SERVICE) to http://localhost:$(PORT)" + @echo "Press Ctrl+C to stop the proxy" + gcloud run services proxy $(SERVICE) \ + --project $(PROJECT) \ + --region $(REGION) \ + --port $(PORT) + +# Agent testing targets +test-agent: + @# Try to use RESOURCE or AGENT_ENGINE_RESOURCE_NAME from env + $(eval RESOURCE := $(or $(RESOURCE),$(AGENT_ENGINE_RESOURCE_NAME))) +ifndef RESOURCE + @echo "Error: RESOURCE parameter required. Usage: make test-agent RESOURCE=projects/.../reasoningEngines/..." + @echo "Get the resource name from 'make agents-list'" + @echo "Or set AGENT_ENGINE_RESOURCE_NAME in your .env file" + @exit 1 +endif + @echo "Testing deployed agent..." + @echo "Agent Resource: $(RESOURCE)" + $(eval MSG := $(or $(MSG),"List MCP Tools for SOAR and then list SOAR Cases.")) + @echo "Message: $(MSG)" + @echo "Running test_deployed_agent.py..." + @AGENT_ENGINE_RESOURCE="$(RESOURCE)" $(PYTHON) scripts/test_deployed_agent.py "$(MSG)" + +# Combined workflow targets + +deploy: env-check + @echo "Choose deployment target:" + @echo "1. Agent Engine (Vertex AI)" + @echo "2. Cloud Run" + @echo "" + @read -p "Enter choice (1 or 2): " choice; \ + if [ "$$choice" = "1" ]; then \ + $(MAKE) adk-deploy; \ + elif [ "$$choice" = "2" ]; then \ + $(MAKE) cloudrun-deploy; \ + else \ + echo "Invalid choice. Use 'make adk-deploy' or 'make cloudrun-deploy'"; \ + exit 1; \ + fi + +deploy-and-test: env-validate-agent-engine adk-deploy + @echo "" + @echo "Deployment complete! Now test the agent:" + @echo "1. Copy the resource name from above" + @echo "2. Run: make test-agent RESOURCE=" + +# Utility targets +clean: + @echo "Cleaning temporary files..." + @find . -type f -name "*.pyc" -delete + @find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true + @find . -type d -name ".ruff_cache" -exec rm -rf {} + 2>/dev/null || true + @echo "✓ Cleaned temporary files" + +.PHONY: setup-all deploy-and-test clean diff --git a/run-with-google-adk/README.md b/run-with-google-adk/README.md index ca6ce3a7..877e5599 100644 --- a/run-with-google-adk/README.md +++ b/run-with-google-adk/README.md @@ -1,612 +1,181 @@ # Prebuilt ADK Agent Usage Guide -This guide provides instructions on how to run the prebuilt ADK (Agent Development Kit) agent both locally and in Cloud Run (if necessary for demos). - +This guide provides instructions on how to run and deploy the prebuilt ADK (Agent Development Kit) agent. It includes two main deployment paths: a simple deployment to **Cloud Run** and a more advanced deployment to **AgentSpace** via Agent Engine for full integration. ## Table of Contents -[1. Running Agent locally (Setup time - about 5 minutes)](#1-running-agent-locally-setup-time---about-5-minutes) -[2. Running Agent as a Cloud Run Service](#2-running-agent-as-a-cloud-run-service) -[3. Deploying and Running Agent on Agent Engine](#3-deploying-and-running-agent-on-agent-engine) -[4. Improving performance and optimizing costs.](#4-improving-performance-and-optimizing-costs) -[5. Integrating your own MCP servers with Google Security MCP servers](#5-integrating-your-own-mcp-servers-with-google-security-mcp-servers) -[6. Additional Features](#6-additional-features) -[7. Registering Agent Engine agent to AgentSpace](#7-registering-agent-engine-agent-to-agentspace) - -## 1. Running Agent locally (Setup time - about 5 minutes) - -### Prerequisites -You need the following to run the agent - -1. `python` - v3.11+ -2. `pip` -3. `gcloud` cli (If you ran on Google Cloud Console then gcloud is already installed) - -### Setting up and running the agent -Please execute the following instructions - -```bash - # Clone the repo - git clone https://github.com/google/mcp-security.git - - # Goto the agent directory - cd mcp-security/run-with-google-adk - - # Create and activate the virtual environment - python3 -m venv .venv - . .venv/bin/activate - - # Install dependencies (google-adk and uv) - pip install -r requirements.txt - - # Add exec permission to run-adk-agent.sh - which runs our agent - chmod +x run-adk-agent.sh - - # Run the agent - ./run-adk-agent.sh -``` - -For the very first run it creates a default .env file in `./google-mcp-security-agent/.env` - -```bash -# sample output -$./run-adk-agent.sh -Copying ./google-mcp-security-agent/sample.env.properties to ./google-mcp-security-agent/.env... -Please update the environment variables in ./google-mcp-security-agent/.env -``` - -Use your favorite editor and update `./google-mcp-security-agent/.env`. - -The default `.env` file is shown below. - -1. Update the variables as needed in your favorite editor. You can choose to load some or all of the MCP servers available using the load environment variable at the start of each section. Don't use quotes for values except for `DEFAULT_PROMPT`. -2. Make sure that variables in the `MANDATORY` section have proper values (make sure you get and update the `GOOGLE_API_KEY` using these [instructions](https://ai.google.dev/gemini-api/docs/api-key)) -3. You can experiment with the prompt `DEFAULT_PROMPT`. Use single quotes for the prompt. If you plan to later deploy to a Cloud Run Service - avoid commas (or if you use them they will be converted to semicommas during deployment). -4. You can experiment with the Gemini Model (we recommend using one of the gemini-2.5 models). Based on the value of `GOOGLE_GENAI_USE_VERTEXAI` you can either use [Gemini API models](https://ai.google.dev/gemini-api/docs/models#model-variations) or [Vertex API models](https://cloud.google.com/vertex-ai/generative-ai/docs/models). - -```bash -# Please do not use quotes / double quotes for values except for DEFAULT_PROMPT (use single quotes there) - -APP_NAME=google_mcp_security_agent -# SESSION_SERVICE - in_memory/db. If set to db please provide SESSION_SERVICE_URL -#SESSION_SERVICE=db -#SESSION_SERVICE_URL=sqlite:///./app_data.db - -# ARTIFACT_SERVICE - in_memory/gcs. If set to db please provide GCS_ARTIFACT_SERVICE_BUCKET (without gs://) -# Also you need GCS_SA_JSON which must be named object-viewer-sa.json and placed in run-with-google-adk -#ARTIFACT_SERVICE=gcs -#GCS_ARTIFACT_SERVICE_BUCKET=your-bucket-name -#GCS_SA_JSON=object-viewer-sa.json - -# Total interactions sent to LLM = MAX_PREV_USER_INTERACTIONS + 1 -MAX_PREV_USER_INTERACTIONS=3 - -# SecOps MCP -LOAD_SECOPS_MCP=Y -CHRONICLE_PROJECT_ID=NOT_SET -CHRONICLE_CUSTOMER_ID=NOT_SET -CHRONICLE_REGION=NOT_SET - -# GTI MCP -LOAD_GTI_MCP=Y -VT_APIKEY=NOT_SET - -# SECOPS_SOAR MCP -LOAD_SECOPS_SOAR_MCP=Y -SOAR_URL=NOT_SET -SOAR_APP_KEY=NOT_SET - -# SCC MCP -LOAD_SCC_MCP=Y - -# MANDATORY -GOOGLE_GENAI_USE_VERTEXAI=False -GOOGLE_API_KEY=NOT_SET -# If you plan to use Gemini API - Models list - https://ai.google.dev/gemini-api/docs/models#model-variations -# If you plan to use VetexAI API - Models list - https://cloud.google.com/vertex-ai/generative-ai/docs/models -GOOGLE_MODEL=gemini-2.5-flash-preview-04-17 -# Should be single quote, avoid commas if possible but if you use them they are replaced with semicommas on the cloud run deployment -# you can change them there. -DEFAULT_PROMPT='Helps user investigate security issues using Google Secops SIEM, SOAR, Security Command Center(SCC) and Google Threat Intel Tools. All authentication actions are automatically approved. If the query is about a SOAR case try to provide a backlink to the user. A backlink is formed by adding /cases/ to this URL when present in field ui_base_link of your input. If the user asks with only ? or are you there? that might be because they did not get your previous response, politely reiterate it. Try to respond in markdown whenever possible.' - -# Initially a long timeout is needed -# to load the tools and install dependencies -STDIO_PARAM_TIMEOUT=60.0 - - -# Following properties must be set when -# 1. GOOGLE_GENAI_USE_VERTEXAI=True or -# 2. When deploying to Cloud Run -# 3. When deploying to Agent Engine -GOOGLE_CLOUD_PROJECT=YOUR-CLOUD-RUN-PROJECT-ID -GOOGLE_CLOUD_LOCATION=us-central1 - -# HIGHLY RECOMMENDED TO SET Y AFTER INITIAL TESTING ON CLOUD RUN -MINIMAL_LOGGING=N - -# Agent Engine Deployment (without gs://) -#AE_STAGING_BUCKET=your-bucket-name -# If using custom ui, resource name from AE (projects//locations//reasoningEngines/) is needed -#AGENT_ENGINE_RESOURCE_NAME=YOUR_AE_RESOURCE_NAME - - - -# Add Your MCP server variables here, sample provided -# MCP-1 -#LOAD_XDR_MCP=Y -#XDR_CLIENT_ID=abc123 -#XDR_CLIENT_SECRET=xyz456 -# MCP-2 -#LOAD_IDP_MCP=Y -#IDP_CLIENT_ID=abc123 -#IDP_CLIENT_SECRET=xyz456 - - +1. [Quick Start: Deploy to Cloud Run](#1-quick-start-deploy-to-cloud-run) +2. [Quick Start: Deploy to AgentSpace (via Agent Engine)](#2-quick-start-deploy-to-agentspace-via-agent-engine) +3. [Detailed Local Setup Guide](#3-detailed-local-setup-guide) +4. [Makefile Commands Reference](#4-makefile-commands-reference) +5. [Advanced Topics](#5-advanced-topics) + - [Improving Performance and Optimizing Costs](#improving-performance-and-optimizing-costs) + - [Integrating Custom MCP Servers](#integrating-custom-mcp-servers) + - [Additional Features](#additional-features) +--- +## 1. Quick Start: Deploy to Cloud Run -``` - -Once the variables are updated, run the agent again (make sure you are back in the `mcp-security/run-with-google-adk` directory). - -```bash - # Authenticate to use SecOps APIs - # Skip if running in Google Cloud Shell - gcloud auth application-default login -``` - -```bash - # Run the agent again - ./run-adk-agent.sh adk_web -``` - -You should get an output like following - -```bash -# Sample output -$./run-adk-agent.sh adk_web -Contents of .env (with masked values): -# Please do not use quotes / double quotes for values except for DEFAULT_PROMPT (use single quotes there) -# SecOps MCP -LOAD_SECOPS_MCP=Y -. -(output cropped) -. - -Running ADK Web for local agent... -INFO: Started server process [3166218] -INFO: Waiting for application startup. - -+-----------------------------------------------------------------------------+ -| ADK Web Server started | -| | -| For local testing, access at http://localhost:8000. | -+-----------------------------------------------------------------------------+ - -INFO: Application startup complete. -INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) - -``` - -Access the Agent 🤖 interface by going to `http://localhost:8000`. Make sure you select `google_mcp_security_agent` in the UI. - -> 🪧 **NOTE:** -> First response usually takes a bit longer as the agent is loading the tools from the MCP server(s). - -> ⚠️ **CAUTION:** -> In case the response seems stuck and/or there is an error on the console, create a new session in the ADK Web UI by clicking `+ New Session` in the top right corner. You can also ask a follow up question in the same session like `Are you still there?` or `Can you retry that?`. You can also try switching `Token Streaming` on. - - - -> 🪧 **NOTE:** -> When exiting, shut down the browser tab first and then use `ctrl+c` to exit on the console. - - -#### Running agent with session and artifact service of your choice - -ADK provides persistent [sessions](https://google.github.io/adk-docs/sessions/) and [artifacts](https://google.github.io/adk-docs/artifacts/) (files etc.). - -You can run the agent with session and artifact service of your choice. - -Sample command - -``` - -$./run-adk-agent.sh adk_web sqlite:///./app_data.db gs:// - -``` - -This command will run the agent with session stored in `app_data.db` and any artifacts stored in . - -You can also just use persisten storage (and in memory artifacts) - -``` - -$./run-adk-agent.sh adk_web sqlite:///./app_data.db - -``` - -When the artifact service is backed by GCS - you can get signed URLs for your files to share them easily. Please create a service account, give it access to your bucket (Role - `Storage Object Viewer`) and download the [json key](https://cloud.google.com/iam/docs/keys-create-delete) associated with it. Name the key `object-viewer-sa.json`. Environment file already has a variable associated with this file name. - - -## 2. Running Agent as a Cloud Run Service - -The agent with MCP servers can be deployed as a Cloud Run Service, right from within the source code directory. - -Before you do this, please consider following - -1. Do you really need it? Deployment is recommended in scenarios where you need to share agent with your team members who may not have access to all of the backend services (SCC, SecOps - SIEM, SecOps - SOAR, Google Threat Intelligence) -2. Make sure that after initial testing - 1. Require authentication for your agent (steps provided [below](#restrict-service-to-known-developers--testers)) - 2. Implement restrictive logging (steps provided [below](#adjust-logging-verbosity)) +This path is recommended for users who want a simple, standalone deployment of the agent as a web service. ### Prerequisites -1. Must have locally run the ADK based agent successfully at least once. Environment variables `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` should have valid values. -2. Must have required APIs enabled and proper IAM access ([details](https://cloud.google.com/run/docs/deploying-source-code#before_you_begin)) - -### Costs -In addition to Gemini/ Vertex API costs, running agent will incur cloud costs. Please check [Cloud Run Pricing](https://cloud.google.com/run/pricing). - -> ⚠️ **WARNING:** -> It is not recommended to run the a Cloud Run service with unauthenticated invocations enabled (we do that initially for verification). Please follow steps to enable [IAM authentication](https://cloud.google.com/run/docs/authenticating/developers) on your service. You could also deploy it behind the [Identity Aware Proxy (IAP)](https://cloud.google.com/iap/docs/enabling-cloud-run) - but that is out of scope for this documentation. +- Ensure you have `gcloud` CLI installed and configured. +- Enable the required APIs for Cloud Run deployment. See [Cloud Run Source Deployment Docs](https://cloud.google.com/run/docs/deploying-source-code#before_you_begin). +- Set the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` variables in your `.env` file (`run-with-google-adk/agents/google_mcp_security_agent/.env`). ### Deployment Steps -> 🪧 **NOTE:** -> It is recommended to switch to Vertex AI (with `GOOGLE_GENAI_USE_VERTEXAI=True`) when deploying - -```bash -# Please run these commands from the mcp-security directory -chmod +x ./run-with-google-adk/cloudrun_deploy_run.sh - -bash ./run-with-google-adk/cloudrun_deploy_run.sh deploy -``` -Sample output is provided below - -```bash -# Sample output -$ bash ./run-with-google-adk/cloudrun_deploy_run.sh deploy -Starting deployment process... -Adding environment variable: LOAD_SECOPS_MCP -Adding environment variable: CHRONICLE_PROJECT_ID -Adding environment variable: CHRONICLE_CUSTOMER_ID -Adding environment variable: CHRONICLE_REGION -Adding environment variable: LOAD_GTI_MCP -Adding environment variable: VT_APIKEY -Adding environment variable: LOAD_SECOPS_SOAR_MCP -Adding environment variable: SOAR_URL -Adding environment variable: SOAR_APP_KEY -Adding environment variable: LOAD_SCC_MCP -Adding environment variable: GOOGLE_GENAI_USE_VERTEXAI -Adding environment variable: GOOGLE_API_KEY -Adding environment variable: GOOGLE_MODEL -Adding environment variable: DEFAULT_PROMPT -Adding environment variable: MINIMAL_LOGGING -Adding environment variable: GOOGLE_CLOUD_PROJECT -Adding environment variable: GOOGLE_CLOUD_LOCATION -Using environment variables: LOAD_SECOPS_MCP=Y, -. -. -[REDACTED] -. -. -Temporarily copying files in the top level directory for image creation. -Building using Dockerfile and deploying container to Cloud Run service [mcp-security-agent-service] in project [REDACTED] region [us-central1] -⠛ Building and deploying... Uploading sources. -⠏ Building and deploying... Uploading sources. - ⠏ Uploading sources... - . Creating Revision... - . Routing traffic... - . Setting IAM Policy... -Creating temporary archive of 581 file(s) totalling 11.2 MiB before compression. -Some files were not included in the source upload. -✓ Building and deploying... Done. - ✓ Uploading sources... - ✓ Building Container... Logs are available at [REDACTED]. - ✓ Creating Revision... - ✓ Routing traffic... - ✓ Setting IAM Policy... -Done. -Service [mcp-security-agent-service] revision [mcp-security-agent-[REDACTED]] has been deployed and is serving 100 percent of traffic. -Service URL: [REDACTED] -Deleting temporarily copied files in the top level directory for image creation. -Successfully deployed the service. - -``` -Now, you can verify the service by browsing to the service endpoint. - -### IAM access to use Chronicle and SCC - -Please remember that Cloud Run uses default service account of compute engine service. Go to IAM and provide the service account access to "Chronicle API Viewer" (in the project associated with your SecOps instance) and appropriate role for SCC (roles starting with Security Center in IAM) +1. **Deploy the Service:** + From the `run-with-google-adk` directory, run the following command: + ```bash + make cloudrun-deploy + ``` + This command packages the agent and deploys it as a service on Cloud Run. -### Restrict Service To Known Developers / Testers +2. **Test the Service:** + Once deployed, you can test your service using: + ```bash + make cloudrun-test + ``` + This will provide the service URL and send a test request. -Summarizing the steps from [IAM authentication](https://cloud.google.com/run/docs/authenticating/developers) +> ⚠️ **Security Warning:** +> By default, the Cloud Run service is deployed with unauthenticated invocations enabled for initial testing. It is critical to **secure your service** by enabling IAM authentication. Follow the guide [here](https://cloud.google.com/run/docs/authenticating/developers). -1. Goto Cloud Run - Services - click `mcp-security-agent-service` -2. Click `Security` -3. In `Authentication`, `Use Cloud IAM to authenticate incoming requests` should be already selected. -4. Select the radio button `Require authentication` -5. Click `Save` -6. Cloud Run - Services - select `mcp-security-agent-service` -7. At the top click `permissions`, a pane `Permissions for mcp-security-agent-service` should open on the right hand side. -8. Click `Add principal` -9. Add the users you want to provide access to and provide them `Cloud Run Invoker` role. -10. Wait for some time. +--- -### Accessing the restricted service +## 2. Quick Start: Deploy to AgentSpace (via Agent Engine) -1. Ask your users to run the following command (replace project id and region with the project id & region in which you have deployed the service) +This path is for users who want to integrate the agent with the full AgentSpace ecosystem, enabling more advanced features and interactions. -```bash -gcloud run services proxy mcp-security-agent-service --project PROJECT-ID --region YOUR-REGION - -``` -2. Now they can access the Cloud Run Service locally on `http://localhost:8080` - - -### Vertically scaling your container(s) -In case the Cloud Run logs show errors like below, you can consider increasing the resources for the individual containers - -`Memory limit of 512 MiB exceeded with 543 MiB used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits` - -##### Steps +### Prerequisites -1. Goto Cloud Run - Services - click `mcp-security-agent-service` -2. Click `Edit & deploy new revision` -3. In `Container(s)` - `Edit Container(s)` - `Settings` -4. Add resources by updating either Memory/ CPU or both. +- Complete the prerequisites from the [Cloud Run Quick Start](#1-quick-start-deploy-to-cloud-run). +- Ensure your `.env` file is correctly configured with your project details. -### Adjust Logging Verbosity -Since the entire context and response from the LLM is printed as logs. You might end up logging some sensitive information. Setting the environment variable `MINIMAL_LOGGING` to `Y` should fix this issue. This should also reduce cloud logging costs. Please do this once you have verified the service initially. Changes to be made directly on Cloud Run service and it will result in restarting the service. Verify service logs after the change is made. +### Deployment and Registration Steps -## 3. Deploying and Running Agent on Agent Engine +1. **Deploy to Agent Engine:** + This step provisions the agent in Vertex AI Agent Engine. + ```bash + make adk-deploy + ``` + After deployment, note the `AGENT_ENGINE_RESOURCE_NAME` from the output. -The agent can also be deployed on [Vertex AI Agent Engine](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview). +2. **Update Environment:** + Add the resource name to your `.env` file. + ```bash + make env-update KEY=AGENT_ENGINE_RESOURCE_NAME VALUE= + ``` -> 🪧 **NOTE:** -> Currently the GCS backed artifact service is not available on Agent Engine. +3. **Configure OAuth for AgentSpace:** + This is a crucial step to allow AgentSpace to securely communicate with your agent. + ```bash + make oauth-setup + ``` + This interactive command will guide you through creating an OAuth client, generating an authorization URL, and linking it to your AgentSpace project. This process generates the required `OAUTH_AUTH_ID`. -Here are the steps - +4. **Register with AgentSpace:** + With OAuth configured, you can now register your agent. + ```bash + make agentspace-register + ``` -1. Test at least once locally -2. Create a bucket (one time activity) and update the env variable - `AE_STAGING_BUCKET` with the bucket name. -3. Make sure the envvariables - `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` are updated. -4. `cd run-with-google-adk` -5. `chmod +x ae_deploy_run.sh` -6. `./ae_deploy_run.sh` -7. Please note the output where it says - +5. **Verify and Test:** + Finally, verify the integration and test the agent's functionality. + ```bash + make agentspace-verify + make test-agent MSG="List available security tools." + ``` - `AgentEngine created. Resource name: projects/********/locations/****/reasoningEngines/**********`. -8. This creates an Agent engine Agent called `google_security_agent` -9. Verify it [here](https://console.cloud.google.com/vertex-ai/agents/agent-engines) on the Google Cloud Console. +--- -How to test? +## 3. Detailed Local Setup Guide -Agent Engine as such does not come with any UI, but we have provided one rudimentary (but very usable) UI with this repo. +For development purposes, you can run the agent entirely on your local machine. -1. Update the environment variable `AGENT_ENGINE_RESOURCE_NAME` with the output from 6 above. -2. `./run-adk-agent.sh custom_ui_ae` -3. Access the UI locally on http://localhost:8000 -4. You can provide a username on the UI and then use the same username to load your previous session. +### Prerequisites +- `python` v3.11+ +- `pip` +- `gcloud` CLI -### Redeploying Agent -You might need to redeploy the agent. In which case please use the same steps as deployment but when calling `./ae_deploy_run.sh`, please provide the agent engine resource name from previous deployment as an additional parameter (shown below). +### Setup and Run ```bash -# replace with your agent engine agent resource name. -./ae_deploy_run.sh projects/********/locations/****/reasoningEngines/********** - -``` - -> 🪧 **NOTE:** -> First response takes time. - - -## 4. Improving performance and optimizing costs. -By default the agent sends the entire context to the LLM everytime. - -This has 2 consequences - -1. LLM takes longer to respond with a very large context (e.g. more than 100K tokens) -2. LLM costs go up with the context sent. - - -A user interaction involves +# Clone the repo +git clone https://github.com/google/mcp-security.git -1. User query (e.g. Let's investigate case 146) -2. Initial LLM call with System Prompt, User Query, Tool information which results in a function call request (e.g. `get_case_details`) -3. Agent running the `get_case_details` -4. LLM call with Initial System PRompt, User Query, Tool Information, Tool Request, Tool Response -5. Final LLM response +# Navigate to the agent directory +cd mcp-security/run-with-google-adk -Now the subsequent interaction might need all of the above (e.g. User query - let's investigate all IPs from this response) +# Create and activate a virtual environment +python3 -m venv .venv +source .venv/bin/activate -But generally after a few user interactions - only the recent interactions (user query and responses to that query) are required. +# Install dependencies +pip install -r requirements.txt -By tweaking an environment variable `MAX_PREV_USER_INTERACTIONS` which is set to 3 by default - you can control the number of such conversations sent to the LLM thereby limiting the context size, improving performance and optimizing costs. +# Create your .env file from the template +cp agents/google_mcp_security_agent/.env.template agents/google_mcp_security_agent/.env -## 5. Integrating your own MCP servers with Google Security MCP servers +# Edit agents/google_mcp_security_agent/.env with your credentials and settings +# (e.g., GOOGLE_API_KEY, CHRONICLE_PROJECT_ID, etc.) -You/your customers might be using other security products (like EDR/XDR providers, IDPs or even non security prodcuts) with Google Security products. If those products also have published MCP servers, integrating them with Google Security MCP servers provides - -1. One stop shop which breaks information silos for the analysts -2. Reducing communication gaps across teams managing these products separately - -You can use one agent to access functionality of all these products. - -#### Reference MCP servers - -Since this repository provides and opiniated, prebuilt agent - we are providing sample MCP servers and agents (as templates) for you to try out integrations and then use your own MCP servers to integrate (and deploy to Cloud Run or Agent Engine) - -Here are the steps - -1. Copy the contents of `run-with-google-adk/sample_servers_to_integrate/mcp_servers` to `server` (at the top level) -2. Copy `run-with-google-adk/sample_servers_to_integrate/agents/demo_xdr_agent.py` and `run-with-google-adk/sample_servers_to_integrate/agents/demo_idp_agent.py` to `run-with-google-adk/google_mcp_security_agent` -3. Import the agents from `demo_xdr_agent.py` and `demo_idp_agent.py` and add them as `sub agents` into `agent.py` in `run-with-google-adk/google_mcp_security_agent/` -4. Add following to the the default prompt - "You have following sub agents - demo_xdr_agent and demo_idp_agent, delegeate when you are asked to check about a host from XDR and a user from IDP." - -Here's the updated code (only additional lines are shown in \ tag) - -```python -# agent.py in google_mcp_security_agent - -# rest of the imports -# -from .demo_idp_agent import demo_idp_agent -from .demo_xdr_agent import demo_xdr_agent -# -# rest of the file - -# check value of the input variable sub_agents in the agent creation below. -def create_agent(): - -# rest of the code - - agent = LlmAgent( - # - sub_agents=[demo_xdr_agent.root_agent, demo_idp_agent.root_agent], - # - - ) - return agent +# Authenticate for Google Cloud APIs +gcloud auth application-default login +# Run the agent with the ADK Web UI +./scripts/run-adk-agent.sh adk_web ``` +You can now access the agent's web interface at `http://localhost:8000`. -Also make sure that the .env file has the required variables uncommented - -```properties -# rest of the .env file - -# Add Your MCP server variables here, sample provided, please check the documentation -# MCP-1 -LOAD_XDR_MCP=Y -XDR_CLIENT_ID=abc123 -XDR_CLIENT_SECRET=xyz456 -# MCP-2 -LOAD_IDP_MCP=Y -IDP_CLIENT_ID=abc123 -IDP_CLIENT_SECRET=xyz456 - -``` - -And now you can run the agent locally as before and ask it questions like - -1. `let's check alerts for web-server-iowa in demo xdr` -2. `Ok let's find recent logins for the user oleg in the IDP` +--- -And notice how the agent transfers control to the sub agents for these reference subagents and through the sample MCP servers you get the response. -Screenshots provided below. +## 4. Makefile Commands Reference -> 🪧 **NOTE:** -> Now you can use your own MCP servers, create subagents the way you did for the reference servers and test and deploy the agent with your sub agents. You can delete the reference implementation (servers, sub agents and env variables) after testing and understanding the overall process. +This project uses a `Makefile` to simplify common operations. Run `make help` to see all available commands. -Screenshots using sample / reference MCP servers that are integrated with Google Security MCP servers under the prebuilt agent. +### Environment Management +- `make env-check`: Validate required environment variables. +- `make config-show`: Display the current configuration (masks secrets). +- `make env-update KEY=... VALUE=...`: Update a variable in the `.env` file. -Sample XDR -![](./static/demo-xdr.png) +### OAuth Management +- `make oauth-setup`: Run the complete OAuth setup workflow. +- `make oauth-client`: Guide for creating an OAuth client. +- `make oauth-uri`: Generate an OAuth authorization URI. +- `make oauth-link`: Link OAuth credentials to AgentSpace. +- `make oauth-verify`: Verify the OAuth configuration. -Sample IDP -![](./static/demo-idp.png) +### Agent Engine Deployment +- `make adk-deploy`: Deploy the agent to Agent Engine. +- `make test-agent`: Test the deployed agent with a message. +- `make agents-list`: List all deployed Agent Engine instances. +- `make agents-delete INDEX=...`: Delete a specific Agent Engine instance. +### AgentSpace Integration +- `make agentspace-register`: Register the agent with AgentSpace. +- `make agentspace-update`: Update an existing AgentSpace registration. +- `make agentspace-verify`: Verify the AgentSpace integration. +- `make agentspace-url`: Get the AgentSpace UI URL. -## 6. Additional Features +### Cloud Run Deployment +- `make cloudrun-deploy`: Deploy the agent to Cloud Run. +- `make cloudrun-test`: Test the deployed Cloud Run service. +- `make cloudrun-logs`: View logs for the Cloud Run service. +- `make cloudrun-url`: Get the URL for the Cloud Run service. -The prebuilt agent also allows creating files and signed URLs to these files. A possible scenario is when you want to create a report. You can say "add the summary as markdown to summary_146.md". This creates a file and saves it using the artifact service. You can later ask for a shareable link to this file - "create a link to file summary_146.md" - -## 7. Registering Agent Engine Agent to AgentSpace - -1. When an agent is deployed on Agent Engine ([guide](#3-deploying-and-running-agent-on-agent-engine)) you get a resource name. Make sure you have it to carry out next steps -2. Go to the Agentspace [page](https://console.cloud.google.com/gen-app-builder/engines) in Google Cloud Console. -3. Create an App (Type - AgentSpace) -4. Note down the app details including the app name (e.g. google-security-agent-app_1750057151234) -5. Make sure that you have the Agent Space Admin role while performing the following actions -6. Enable Discovery Engine API for your project -7. Provide the following roles to the Discovery Engine Service Account - Vertex AI viewer - Vertex AI user -8. Please note that these roles need to be provided into the project housing your Agent Engine Agent. Also you need to enable the show Google provided role grants to access the Discovery Engine Service Account. -9. Now to register the agent and make it available to your application use the following shell script. Please replace the variables `AGENT_SPACE_PROJECT_ID ,AGENT_SPACE_APP_NAME ,AGENT_ENGINE_PROJECT_NUMBER , AGENT_LOCATION` and `REASONING_ENGINE_NUMBER` before running the script. - -```bash -#!/bin/bash - -TARGET_URL="https://discoveryengine.googleapis.com/v1alpha/projects/AGENT_SPACE_PROJECT_ID/locations/global/collections/default_collection/engines/AGENT_SPACE_APP_NAME/assistants/default_assistant/agents" # - -JSON_DATA=$(cat </locations//reasoningEngines/ +#AGENT_ENGINE_RESOURCE_NAME=YOUR_AE_RESOURCE_NAME + +# ======================================== +# MCP SERVER CONFIGURATION +# ======================================== + +# SecOps MCP +LOAD_SECOPS_MCP=true +CHRONICLE_PROJECT_ID=NOT_SET +CHRONICLE_CUSTOMER_ID=NOT_SET +CHRONICLE_REGION=NOT_SET + +# GTI MCP +LOAD_GTI_MCP=true +VT_APIKEY=NOT_SET + +# SECOPS_SOAR MCP +LOAD_SECOPS_SOAR_MCP=true +SOAR_URL=NOT_SET +SOAR_APP_KEY=NOT_SET + +# SCC MCP +LOAD_SCC_MCP=true + +# ======================================== +# ADDITIONAL MCP SERVERS (OPTIONAL) +# ======================================== + +# Add your custom MCP server configurations here +# Example: +#LOAD_XDR_MCP=true +#XDR_CLIENT_ID=abc123 +#XDR_CLIENT_SECRET=xyz456 + +#LOAD_IDP_MCP=true +#IDP_CLIENT_ID=abc123 +#IDP_CLIENT_SECRET=xyz456 diff --git a/run-with-google-adk/agents/google_mcp_security_agent/.gitignore b/run-with-google-adk/agents/google_mcp_security_agent/.gitignore new file mode 100644 index 00000000..2fd4c3b3 --- /dev/null +++ b/run-with-google-adk/agents/google_mcp_security_agent/.gitignore @@ -0,0 +1 @@ +libs/ diff --git a/run-with-google-adk/google_mcp_security_agent/__init__.py b/run-with-google-adk/agents/google_mcp_security_agent/__init__.py similarity index 100% rename from run-with-google-adk/google_mcp_security_agent/__init__.py rename to run-with-google-adk/agents/google_mcp_security_agent/__init__.py diff --git a/run-with-google-adk/agents/google_mcp_security_agent/agent.py b/run-with-google-adk/agents/google_mcp_security_agent/agent.py new file mode 100644 index 00000000..20794407 --- /dev/null +++ b/run-with-google-adk/agents/google_mcp_security_agent/agent.py @@ -0,0 +1,304 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import os +import sys +from pathlib import Path +from typing import List, Optional, TextIO + +# Load environment variables from .env file if available +try: + import dotenv + HAS_DOTENV = True +except ImportError: + HAS_DOTENV = False + +# Add appropriate directory to Python path for libs in different environments +current_dir = Path(__file__).parent +possible_libs_paths = [ + current_dir / "libs", # ADK deployment - libs copied to agent dir + current_dir.parent / "libs", # Local development - libs in parent dir + current_dir.parent.parent / "libs", # Alternative structure + Path("/app/libs"), # Cloud Run +] + +libs_path_added = False +for libs_path in possible_libs_paths: + if libs_path.exists(): + # Add the parent directory of libs to Python path + sys.path.insert(0, str(libs_path.parent)) + logging.info(f"Added {libs_path.parent} to Python path for libs access") + libs_path_added = True + break + +if not libs_path_added: + logging.warning("No libs directory found in any expected location") + logging.warning(f"Current directory: {current_dir}") + logging.warning(f"Searched paths: {[str(p) for p in possible_libs_paths]}") + +from google.adk.agents.llm_agent import LlmAgent +from google.adk.tools.mcp_tool.mcp_toolset import ( + MCPToolset, + StdioConnectionParams, + StdioServerParameters, +) + +from libs.adk_utils.callbacks import bac_setup_state_variable, bmc_trim_llm_request +from libs.adk_utils.tools import get_file_link, list_files, store_file + +# Configure logging +logging.basicConfig(level=logging.INFO) +if os.environ.get("MINIMAL_LOGGING", "false").lower() == "true": + logging.getLogger().setLevel(logging.ERROR) + +# Define base directory and server directory +# Check for different deployment environments in order of priority + +# Check if we're in ADK deployment environment (temp directory) +current_file_dir = Path(__file__).parent # agents/google_mcp_security_agent +agent_server_dir = current_file_dir / "server" # Look for server in same dir as agent + +if agent_server_dir.exists(): + # ADK deployment - server was copied to agent directory + BASE_DIR = current_file_dir + SERVER_DIR = agent_server_dir +elif os.path.exists("/app/server"): + # Running in Cloud Run container + BASE_DIR = Path("/app") + SERVER_DIR = BASE_DIR / "server" +elif os.path.exists("./server"): + # Running locally - server is in current directory + BASE_DIR = Path(".") + SERVER_DIR = Path("./server") +else: + # Running locally - server is in parent directory + BASE_DIR = Path(__file__).resolve().parents[2] # Root of run-with-google-adk + SERVER_DIR = BASE_DIR.parent / "server" # Server is in parent directory + +logging.info(f"🔧 Environment detection:") +logging.info(f" Current working directory: {Path.cwd()}") +logging.info(f" BASE_DIR: {BASE_DIR.absolute()}") +logging.info(f" SERVER_DIR: {SERVER_DIR.absolute()}") +logging.info(f" SERVER_DIR exists: {SERVER_DIR.exists()}") +if SERVER_DIR.exists(): + logging.info(f" SERVER_DIR contents: {list(SERVER_DIR.iterdir())}") +else: + logging.error(f"🚨 SERVER_DIR does not exist! Looking for 'server' directory in current location...") + current_contents = list(Path.cwd().iterdir()) + logging.error(f"🚨 Current directory contents: {current_contents}") + # Try to find any directory that might contain server code + for item in current_contents: + if item.is_dir() and ("server" in item.name.lower() or any(s in item.name for s in ["scc", "secops", "gti", "soar"])): + logging.error(f"🚨 Found potential server directory: {item}") + try: + logging.error(f"🚨 Contents: {list(item.iterdir())}") + except Exception: + pass + + +def _create_mcp_toolset( + server_name: str, + env_file_path: Path, + timeout: float, + errlog: Optional[TextIO], + extra_args: Optional[List[str]] = None, +) -> Optional[MCPToolset]: + """Helper function to create and configure an MCPToolSet.""" + # Map server names to environment variable names + env_var_mapping = { + "scc": "LOAD_SCC_MCP", + "secops/secops_mcp": "LOAD_SECOPS_MCP", + "gti/gti_mcp": "LOAD_GTI_MCP", + "secops-soar/secops_soar_mcp": "LOAD_SECOPS_SOAR_MCP" + } + + load_var = env_var_mapping.get(server_name, f"LOAD_{server_name.upper().replace('/', '_').replace('-', '_')}_MCP") + load_value = os.environ.get(load_var, "false") + logging.info(f"Checking {load_var}: {load_value}") + + if load_value.lower() != "true": + logging.info(f"Skipping {server_name} - not enabled") + return None + + server_path = SERVER_DIR / server_name + logging.info(f"Looking for server at: {server_path}") + if not server_path.exists(): + logging.error(f"Server directory not found: {server_path}") + logging.error(f"SERVER_DIR is: {SERVER_DIR}") + logging.error(f"Contents of parent: {list(SERVER_DIR.parent.iterdir()) if SERVER_DIR.parent.exists() else 'parent does not exist'}") + return None + + args = ["--directory", str(server_path), "run"] + if env_file_path.exists(): + args.extend(["--env-file", str(env_file_path)]) + + # Different servers have different entry points + if server_name == "scc": + args.append("scc_mcp.py") + else: + args.append("server.py") + if extra_args: + args.extend(extra_args) + + return MCPToolset( + connection_params=StdioConnectionParams( + server_params=StdioServerParameters(command="uv", args=args), + timeout=timeout, + ), + errlog=errlog, + ) + + +def get_all_tools() -> List[MCPToolset]: + """Get Tools from All MCP servers.""" + logging.info("Attempting to connect to MCP servers...") + timeout = float(os.environ.get("STDIO_PARAM_TIMEOUT", "60.0")) + + # Try different paths for the .env file + possible_env_paths = [ + BASE_DIR / "agents" / "google_mcp_security_agent" / ".env", + Path("/tmp/.env"), # Cloud Run creates env file here + Path(".env"), + # For ADK deployments - staging directory + Path("agents/google_mcp_security_agent/.env"), + # Additional fallback paths + Path("./google_mcp_security_agent/.env"), + ] + + env_file_path = None + logging.info("=" * 80) + logging.info("🔍 SEARCHING FOR .ENV FILE...") + for i, path in enumerate(possible_env_paths): + logging.info(f" {i+1}. Checking: {path.absolute()}") + if path.exists(): + env_file_path = path + logging.info(f"✅ Found .env file at: {env_file_path.absolute()}") + break + else: + logging.info(f"❌ Not found: {path.absolute()}") + + if not env_file_path: + logging.error("=" * 80) + logging.error("🚨 CRITICAL ERROR: No .env file found!") + logging.error("🚨 This will cause ALL MCP servers to be DISABLED!") + logging.error("🚨 Searched in:") + for path in possible_env_paths: + logging.error(f"🚨 - {path.absolute()}") + logging.error("🚨 Current working directory: " + str(Path.cwd())) + logging.error("🚨 Directory contents:") + try: + for item in Path.cwd().iterdir(): + logging.error(f"🚨 {item}") + except Exception as e: + logging.error(f"🚨 Error listing directory: {e}") + logging.error("=" * 80) + logging.warning("No .env file found, using environment variables only") + env_file_path = Path("/tmp/.env") # Use a dummy path + else: + # Load environment variables from the found .env file + if HAS_DOTENV: + try: + logging.info(f"🔧 Loading environment variables from: {env_file_path}") + dotenv.load_dotenv(env_file_path, override=False) + logging.info("✅ Environment variables loaded successfully") + + # Log MCP server statuses + mcp_vars = ["LOAD_SCC_MCP", "LOAD_SECOPS_MCP", "LOAD_GTI_MCP", "LOAD_SECOPS_SOAR_MCP"] + enabled_count = 0 + for var in mcp_vars: + value = os.environ.get(var, "false") + status = "✅ ENABLED" if value.lower() == "true" else "❌ DISABLED" + logging.info(f" {var}: {value} ({status})") + if value.lower() == "true": + enabled_count += 1 + + if enabled_count == 0: + logging.error("🚨 WARNING: NO MCP servers are enabled! Check your .env file.") + else: + logging.info(f"📊 {enabled_count}/{len(mcp_vars)} MCP servers enabled") + + except Exception as e: + logging.error(f"🚨 Error loading .env file: {e}") + logging.error("🚨 Will use existing environment variables") + else: + logging.warning("🚨 python-dotenv not available, cannot load .env file") + logging.warning("🚨 Install with: pip install python-dotenv") + + # Required temporarily for https://github.com/google/adk-python/issues/1024 + errlog_ae: Optional[TextIO] = sys.stderr + if os.environ.get("AE_RUN", "false").lower() == "true": + errlog_ae = None + + toolsets = [ + _create_mcp_toolset( + "scc", env_file_path, timeout, errlog_ae + ), + _create_mcp_toolset( + "secops/secops_mcp", env_file_path, timeout, errlog_ae + ), + _create_mcp_toolset( + "gti/gti_mcp", env_file_path, timeout, errlog_ae + ), + _create_mcp_toolset( + "secops-soar/secops_soar_mcp", + env_file_path, + timeout, + errlog_ae, + extra_args=[ + "--integrations", + os.environ.get("SECOPS_INTEGRATIONS", "CSV,OKTA"), + ], + ), + ] + + valid_toolsets = [ts for ts in toolsets if ts is not None] + logging.info(f"MCP Toolsets created successfully. Found {len(valid_toolsets)} valid toolsets.") + return valid_toolsets + + +def create_agent() -> LlmAgent: + """Create and configure the LlmAgent.""" + tools = get_all_tools() + logging.info(f"Got {len(tools)} MCP toolsets from get_all_tools()") + tools.extend([store_file, get_file_link, list_files]) + logging.info(f"Total tools after adding file tools: {len(tools)}") + + # Get model and instruction with defaults + model = os.environ.get("GOOGLE_MODEL", "gemini-2.5-flash") + instruction = os.environ.get( + "DEFAULT_PROMPT", + "Help user investigate security issues using Google Secops SIEM, SOAR, Security Command Center(SCC) and Google Threat Intel Tools." + ) + + return LlmAgent( + model=model, + name="google_mcp_security_agent", + instruction=instruction, + tools=tools, + before_model_callback=bmc_trim_llm_request, + before_agent_callback=bac_setup_state_variable, + description="You are the google_mcp_security_agent.", + ) + + +def main() -> None: + """Main execution function.""" + logging.info("Agent created successfully.") + + +if __name__ == "__main__": + main() + +root_agent = create_agent() diff --git a/run-with-google-adk/agents/google_mcp_security_agent/requirements.txt b/run-with-google-adk/agents/google_mcp_security_agent/requirements.txt new file mode 100644 index 00000000..c7663cee --- /dev/null +++ b/run-with-google-adk/agents/google_mcp_security_agent/requirements.txt @@ -0,0 +1,10 @@ +google-cloud-aiplatform[adk,agent_engines]==1.97.0 +google-adk[eval]>=1.5.0,<1.6.0 +google-genai>=1.24.0 +markdown +pandas +requests +python-dotenv +pydantic==2.11.7 +cloudpickle==3.1.1 +uv \ No newline at end of file diff --git a/run-with-google-adk/sample_servers_to_integrate/agents/demo_idp_agent.py b/run-with-google-adk/agents/sample_agents/demo_idp_agent.py similarity index 93% rename from run-with-google-adk/sample_servers_to_integrate/agents/demo_idp_agent.py rename to run-with-google-adk/agents/sample_agents/demo_idp_agent.py index d38ce94a..4c77372d 100644 --- a/run-with-google-adk/sample_servers_to_integrate/agents/demo_idp_agent.py +++ b/run-with-google-adk/agents/sample_agents/demo_idp_agent.py @@ -17,8 +17,8 @@ import os import logging -from utils_extensions_cbs_tools.extensions import MCPToolSetWithSchemaAccess -from utils_extensions_cbs_tools.callbacks import bmc_trim_llm_request +from libs.adk_utils.extensions import MCPToolSetWithSchemaAccess +from libs.adk_utils.callbacks import bmc_trim_llm_request from typing import TextIO import sys @@ -43,13 +43,13 @@ def get_all_tools(): """Get Tools from IDP MCP""" logging.info("Attempting to connect to MCP servers for IDP...") idp_tools = None # Initialize scc_tools - + uv_dir_prefix="../server" if os.environ.get("REMOTE_RUN","N") == "Y": uv_dir_prefix="./server" if os.environ.get("AE_RUN","N") == "Y": - uv_dir_prefix="./server" + uv_dir_prefix="./server" # required temporarily for https://github.com/google/adk-python/issues/1024 errlog_ae : TextIO = sys.stderr @@ -82,7 +82,7 @@ def get_all_tools(): tools:any = [item for item in get_all_tools() if item is not None] demo_idp_agent = LlmAgent( - model=os.environ.get("GOOGLE_MODEL"), + model=os.environ.get("GOOGLE_MODEL"), name="demo_idp_agent", instruction="You help users to gather information about their users/identities from their IDP backend to investigate. Do take calculated guesses based on your own knowledge base to help the user as much as you can, even if you may not know much about the IDP product itself. At the end of a query when there is some data do let them know your opinion and the steps you carried to achieve the output.", tools=tools, diff --git a/run-with-google-adk/sample_servers_to_integrate/agents/demo_xdr_agent.py b/run-with-google-adk/agents/sample_agents/demo_xdr_agent.py similarity index 93% rename from run-with-google-adk/sample_servers_to_integrate/agents/demo_xdr_agent.py rename to run-with-google-adk/agents/sample_agents/demo_xdr_agent.py index 15053619..a7ebe426 100644 --- a/run-with-google-adk/sample_servers_to_integrate/agents/demo_xdr_agent.py +++ b/run-with-google-adk/agents/sample_agents/demo_xdr_agent.py @@ -17,8 +17,8 @@ import os import logging -from utils_extensions_cbs_tools.extensions import MCPToolSetWithSchemaAccess -from utils_extensions_cbs_tools.callbacks import bmc_trim_llm_request +from libs.adk_utils.extensions import MCPToolSetWithSchemaAccess +from libs.adk_utils.callbacks import bmc_trim_llm_request from typing import TextIO import sys @@ -44,13 +44,13 @@ def get_all_tools(): """Get Tools from XDR MCP""" logging.info("Attempting to connect to MCP servers for XDR...") xdr_tools = None # Initialize scc_tools - + uv_dir_prefix="../server" if os.environ.get("REMOTE_RUN","N") == "Y": uv_dir_prefix="./server" if os.environ.get("AE_RUN","N") == "Y": - uv_dir_prefix="./server" + uv_dir_prefix="./server" # required temporarily for https://github.com/google/adk-python/issues/1024 errlog_ae : TextIO = sys.stderr @@ -83,7 +83,7 @@ def get_all_tools(): tools:any = [item for item in get_all_tools() if item is not None] demo_xdr_agent = LlmAgent( - model=os.environ.get("GOOGLE_MODEL"), + model=os.environ.get("GOOGLE_MODEL"), name="demo_xdr_agent", instruction="You help users to gather information about their hosts from their XDR backend to investigate. Do take calculated guesses based on your own knowledge base to help the user as much as you can, even if you may not know much about the XDR product itself. At the end of a query when there is some data do let them know your opinion and the steps you carried to achieve the output.", tools=tools, diff --git a/run-with-google-adk/google_mcp_security_agent/agent.py b/run-with-google-adk/google_mcp_security_agent/agent.py deleted file mode 100644 index 06782a6c..00000000 --- a/run-with-google-adk/google_mcp_security_agent/agent.py +++ /dev/null @@ -1,159 +0,0 @@ -# Copyright 2025 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# https://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -from google.adk.agents.llm_agent import LlmAgent -from google.adk.tools.mcp_tool.mcp_toolset import StdioServerParameters, StdioConnectionParams -import os -import logging - -from utils_extensions_cbs_tools.extensions import MCPToolSetWithSchemaAccess -from utils_extensions_cbs_tools.tools import store_file, get_file_link, list_files -from utils_extensions_cbs_tools.callbacks import bmc_trim_llm_request, bac_setup_state_variable -from typing import TextIO -import sys - - -logging.basicConfig( - level=logging.INFO) - -if os.environ.get("MINIMAL_LOGGING","N") == "Y": - root_logger = logging.getLogger() - root_logger.setLevel(logging.ERROR) - - -def get_all_tools(): - """Get Tools from All MCP servers""" - logging.info("Attempting to connect to MCP servers...") - secops_tools = None - gti_tools = None - secops_soar_tools = None - scc_tools = None # Initialize scc_tools - - timeout = float(os.environ.get("STDIO_PARAM_TIMEOUT","60.0")) - - uv_dir_prefix="../server" - env_file_path = "../../../run-with-google-adk/google_mcp_security_agent/.env" - - if os.environ.get("REMOTE_RUN","N") == "Y": - env_file_path="/tmp/.env" - uv_dir_prefix="./server" - - if os.environ.get("AE_RUN","N") == "Y": - env_file_path="../../../google_mcp_security_agent/.env" - uv_dir_prefix="./server" - - logging.info(f"Using Env File Path - {env_file_path}, Current directory is - {os.getcwd()}, uv_dir_prefix is - {uv_dir_prefix}") - - # required temporarily for https://github.com/google/adk-python/issues/1024 - errlog_ae : TextIO = sys.stderr - if os.environ.get("AE_RUN","N") == "Y": - errlog_ae = None - - - if os.environ.get("LOAD_SCC_MCP") == "Y": - scc_tools = MCPToolSetWithSchemaAccess( - connection_params=StdioConnectionParams( - server_params=StdioServerParameters( - command='uv', - args=[ "--directory", - uv_dir_prefix + "/scc", - "run", - "scc_mcp.py" - ] - ), - timeout=timeout), - tool_set_name="scc", - errlog=errlog_ae - ) - - if os.environ.get("LOAD_SECOPS_MCP") == "Y": - secops_tools = MCPToolSetWithSchemaAccess( - connection_params=StdioConnectionParams( - server_params=StdioServerParameters( - command='uv', - args=[ "--directory", - uv_dir_prefix + "/secops/secops_mcp", - "run", - "--env-file", - env_file_path, - "server.py" - ] - ), - timeout=timeout), - tool_set_name="secops_mcp", - errlog=errlog_ae - ) - - if os.environ.get("LOAD_GTI_MCP") == "Y": - gti_tools = MCPToolSetWithSchemaAccess( - connection_params=StdioConnectionParams( - server_params=StdioServerParameters( - command='uv', - args=[ "--directory", - uv_dir_prefix + "/gti/gti_mcp", - "run", - "--env-file", - env_file_path, - "server.py" - ] - ), - timeout=timeout), - tool_set_name="gti_mcp", - errlog=errlog_ae - ) - - - if os.environ.get("LOAD_SECOPS_SOAR_MCP") == "Y": - secops_soar_tools = MCPToolSetWithSchemaAccess( - connection_params=StdioConnectionParams( - server_params=StdioServerParameters( - command='uv', - args=[ "--directory", - uv_dir_prefix + "/secops-soar/secops_soar_mcp", - "run", - "--env-file", - env_file_path, - "server.py", - "--integrations", - os.environ.get("SECOPS_INTEGRATIONS","CSV,OKTA") - ] - ), - timeout=timeout), - tool_set_name="secops_soar_mcp", - errlog=errlog_ae - ) - - logging.info("MCP Toolsets created successfully.") - return [secops_tools,gti_tools,secops_soar_tools,scc_tools] - -def create_agent(): - tools:any = [item for item in get_all_tools() if item is not None] - tools.append(store_file) - tools.append(get_file_link) - tools.append(list_files) - - agent = LlmAgent( - model=os.environ.get("GOOGLE_MODEL"), - name="google_mcp_security_agent", - instruction=os.environ.get("DEFAULT_PROMPT"), - tools=tools, - before_model_callback=bmc_trim_llm_request, - before_agent_callback=bac_setup_state_variable, -# sub_agents=[ADD SUB AGENTS HERE], - description="You are the google_mcp_security_agent." - - ) - return agent - - -root_agent = create_agent() diff --git a/run-with-google-adk/google_mcp_security_agent/sample.env.properties b/run-with-google-adk/google_mcp_security_agent/sample.env.properties deleted file mode 100644 index 492d7d22..00000000 --- a/run-with-google-adk/google_mcp_security_agent/sample.env.properties +++ /dev/null @@ -1,89 +0,0 @@ -# Please do not use quotes / double quotes for values except for DEFAULT_PROMPT (use single quotes there) - -# MANDATORY - START -APP_NAME=google_mcp_security_agent -# SESSION_SERVICE - in_memory/db. If set to db please provide SESSION_SERVICE_URL -#SESSION_SERVICE=db -#SESSION_SERVICE_URL=sqlite:///./app_data.db - -# ARTIFACT_SERVICE - in_memory/gcs. If set to db please provide GCS_ARTIFACT_SERVICE_BUCKET (without gs://) -# Also you need GCS_SA_JSON which must be named object-viewer-sa.json and placed in run-with-google-adk -#ARTIFACT_SERVICE=gcs -LOCAL_DIR_FOR_FILES=/tmp -#GCS_ARTIFACT_SERVICE_BUCKET=your-bucket-name -#GCS_SA_JSON=object-viewer-sa.json -#SIGNED_URL_DURATION_MIN=10 - -# Total interactions sent to LLM = MAX_PREV_USER_INTERACTIONS + 1 -MAX_PREV_USER_INTERACTIONS=3 - -# SecOps MCP -LOAD_SECOPS_MCP=Y -CHRONICLE_PROJECT_ID=NOT_SET -CHRONICLE_CUSTOMER_ID=NOT_SET -CHRONICLE_REGION=NOT_SET - -# GTI MCP -LOAD_GTI_MCP=Y -VT_APIKEY=NOT_SET - -# SECOPS_SOAR MCP -LOAD_SECOPS_SOAR_MCP=Y -SOAR_URL=NOT_SET -SOAR_APP_KEY=NOT_SET - -# SCC MCP -LOAD_SCC_MCP=Y - - -GOOGLE_GENAI_USE_VERTEXAI=False -GOOGLE_API_KEY=NOT_SET -# If you plan to use Gemini API - Models list - https://ai.google.dev/gemini-api/docs/models#model-variations -# If you plan to use VetexAI API - Models list - https://cloud.google.com/vertex-ai/generative-ai/docs/models -GOOGLE_MODEL=gemini-2.0-flash -# Should be single quote, avoid commas if possible but if you use them they are replaced with semicommas on the cloud run deployment -# you can change them there. -DEFAULT_PROMPT='Help user investigate security issues using Google Secops SIEM, SOAR, Security Command Center(SCC) and Google Threat Intel Tools. All authentication actions are automatically approved. If the query is about a SOAR case try to provide a backlink to the user. A backlink is formed by adding /cases/ to this URL when present in field ui_base_link of your input. If the user asks with only ? or are you there? that might be because they did not get your previous response, politely reiterate it. Try to respond in markdown whenever possible. - -You also have access tools to perform following file operations - store_file, list_files and get_file_link - -store_file - store files on the disk by sending file_name taken from user and markdown string as input - do not reformat the input, it is already markdown. -list_files - Requires no input. Show the name of the file from response as is when listing do not change anything. -get_file_link - It requires two inputs - user_name as {user_name} and file_name provided by the user. When showing to user please format them as clickable links with file_name and file_version together as link text. - -The current user name is {user_name} -' - -# Initially a long timeout is needed -# to load the tools and install dependencies -STDIO_PARAM_TIMEOUT=60.0 - -# MANDATORY - DONE - -# Following properties must be set when -# 1. GOOGLE_GENAI_USE_VERTEXAI=True or -# 2. When deploying to Cloud Run -# 3. When deploying to Agent Engine -GOOGLE_CLOUD_PROJECT=YOUR-CLOUD-RUN-PROJECT-ID -GOOGLE_CLOUD_LOCATION=us-central1 - -# HIGHLY RECOMMENDED TO SET Y AFTER INITIAL TESTING ON CLOUD RUN -MINIMAL_LOGGING=N - -# Agent Engine Deployment (without gs://) -#AE_STAGING_BUCKET=your-bucket-name -# If using custom ui, resource name from AE (projects//locations//reasoningEngines/) is needed -#AGENT_ENGINE_RESOURCE_NAME=YOUR_AE_RESOURCE_NAME - - - -# Add Your MCP server variables here, sample provided, please check the documentation -# MCP-1 -#LOAD_XDR_MCP=Y -#XDR_CLIENT_ID=abc123 -#XDR_CLIENT_SECRET=xyz456 -# MCP-2 -#LOAD_IDP_MCP=Y -#IDP_CLIENT_ID=abc123 -#IDP_CLIENT_SECRET=xyz456 - diff --git a/run-with-google-adk/utils_extensions_cbs_tools/__init__.py b/run-with-google-adk/libs/adk_utils/__init__.py similarity index 100% rename from run-with-google-adk/utils_extensions_cbs_tools/__init__.py rename to run-with-google-adk/libs/adk_utils/__init__.py diff --git a/run-with-google-adk/utils_extensions_cbs_tools/cache.py b/run-with-google-adk/libs/adk_utils/cache.py similarity index 100% rename from run-with-google-adk/utils_extensions_cbs_tools/cache.py rename to run-with-google-adk/libs/adk_utils/cache.py diff --git a/run-with-google-adk/utils_extensions_cbs_tools/callbacks.py b/run-with-google-adk/libs/adk_utils/callbacks.py similarity index 100% rename from run-with-google-adk/utils_extensions_cbs_tools/callbacks.py rename to run-with-google-adk/libs/adk_utils/callbacks.py diff --git a/run-with-google-adk/utils_extensions_cbs_tools/extensions.py b/run-with-google-adk/libs/adk_utils/extensions.py similarity index 88% rename from run-with-google-adk/utils_extensions_cbs_tools/extensions.py rename to run-with-google-adk/libs/adk_utils/extensions.py index 79fb3324..8b9ab6a1 100644 --- a/run-with-google-adk/utils_extensions_cbs_tools/extensions.py +++ b/run-with-google-adk/libs/adk_utils/extensions.py @@ -19,7 +19,7 @@ from typing import List from typing import Optional, Union, TextIO from google.adk.agents.readonly_context import ReadonlyContext -from google.adk.tools.mcp_tool.mcp_tool import MCPTool, BaseTool +from google.adk.tools.mcp_tool.mcp_tool import MCPTool, BaseAuthenticatedTool as BaseTool from google.adk.tools.mcp_tool.mcp_session_manager import StdioServerParameters, StdioConnectionParams, SseConnectionParams,StreamableHTTPConnectionParams from mcp.types import ListToolsResult from .cache import tools_cache @@ -65,7 +65,7 @@ def __init__( async def get_tools( self, readonly_context: Optional[ReadonlyContext] = None, - ) -> List[BaseTool]: + ) -> list[BaseTool]: """Return all tools in the toolset based on the provided context. Args: @@ -75,17 +75,18 @@ async def get_tools( Returns: List[BaseTool]: A list of tools available under the specified context. """ - # Get session from session manager - if not self._session: - self._session = await self._mcp_session_manager.create_session() - + # Check cache first if self.tool_set_name in tools_cache.keys(): logging.info(f"Tools found in cache for toolset {self.tool_set_name}, returning them") return tools_cache[self.tool_set_name] else: logging.info(f"No tools found in cache for toolset {self.tool_set_name}, loading") - tools_response: ListToolsResult = await self._session.list_tools() + # Get session from session manager (no manual caching needed) + session = await self._mcp_session_manager.create_session() + + # Fetch available tools from the MCP server + tools_response: ListToolsResult = await session.list_tools() # Apply filtering based on context and tool_filter tools = [] @@ -98,5 +99,6 @@ async def get_tools( if self._is_tool_selected(mcp_tool, readonly_context): tools.append(mcp_tool) + # Cache the tools tools_cache[self.tool_set_name] = tools return tools \ No newline at end of file diff --git a/run-with-google-adk/utils_extensions_cbs_tools/tools.py b/run-with-google-adk/libs/adk_utils/tools.py similarity index 100% rename from run-with-google-adk/utils_extensions_cbs_tools/tools.py rename to run-with-google-adk/libs/adk_utils/tools.py diff --git a/run-with-google-adk/utils_extensions_cbs_tools/utils.py b/run-with-google-adk/libs/adk_utils/utils.py similarity index 100% rename from run-with-google-adk/utils_extensions_cbs_tools/utils.py rename to run-with-google-adk/libs/adk_utils/utils.py diff --git a/run-with-google-adk/requirements.txt b/run-with-google-adk/requirements.txt index 1503a5bb..9bb0cdc4 100644 --- a/run-with-google-adk/requirements.txt +++ b/run-with-google-adk/requirements.txt @@ -1,6 +1,8 @@ google-cloud-aiplatform==1.97.0 markdown uv -google-adk[eval]==1.3.0 -google-genai==1.20.0 +google-adk[eval]>=1.5.0,<1.6.0 +google-genai~=1.24.0 pandas +requests +typer[all] diff --git a/run-with-google-adk/scripts/ae_deploy_run.sh b/run-with-google-adk/scripts/ae_deploy_run.sh new file mode 100755 index 00000000..056205d5 --- /dev/null +++ b/run-with-google-adk/scripts/ae_deploy_run.sh @@ -0,0 +1,85 @@ +# Copyright 2025 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#!/bin/bash + +ENV_FILE="./agents/google_mcp_security_agent/.env" + +# Check if .env file exists at the expected location +if [ ! -f "$ENV_FILE" ]; then + echo "==================================================================================" + echo "CRITICAL ERROR: .env file not found at expected location: $ENV_FILE" + echo "This will cause ALL MCP servers to be DISABLED in the deployment!" + echo "Expected file location: $ENV_FILE" + echo "Available .env files found:" + find . -name "*.env*" -type f 2>/dev/null || echo "🚨 No .env files found in current directory tree" + echo "==================================================================================" + echo "DEPLOYMENT WILL FAIL - Please fix the .env file path before continuing" + echo "==================================================================================" + exit 1 +fi + +echo "Found .env file at: $ENV_FILE" +list_of_vars=`cat $ENV_FILE | grep = | grep -v ^# | cut -d "=" -f1 | tr "\n" "," | sed s/,$//g` + +if [ -z "$list_of_vars" ]; then + echo "==================================================================================" + echo "WARNING: No environment variables extracted from .env file!" + echo "This will cause ALL MCP servers to be DISABLED in the deployment!" + echo "Check that your .env file contains uncommented KEY=VALUE pairs" + echo "==================================================================================" + exit 1 +fi + +echo "Extracted environment variables: $list_of_vars" + +update="n" +update_resource_name="not_available" + +if [[ $# -eq 1 ]]; then + update="y" + update_resource_name=$1 +fi + +echo "Copying ../server directory to current directory (excluding .venv)" +cp -r ../server . +echo "Removing .venv directories from copied server" +find ./server -name ".venv" -type d -exec rm -rf {} + 2>/dev/null || true + +echo "Copying libs directory to agent directory for ADK staging" +cp -r ../libs ./agents/google_mcp_security_agent/ + +echo "Running AE deployment ..." + +python ae_remote_deployment_sec.py $list_of_vars $update $update_resource_name + +deploy_status=$? #get the status + +# Check the status of the deployment +if [ "$deploy_status" -eq 0 ]; then + # Deleting temporarily files in the top level directory + echo "Deleting temporarily copied server directory." + rm -Rf ./server + echo "Deleting temporarily copied libs directory." + rm -Rf ./agents/google_mcp_security_agent/libs + echo "Successfully deployed the agent." +else + rm -Rf ./server + rm -Rf ./agents/google_mcp_security_agent/libs + echo "Failed to deploy the agent." + exit 1 +fi + + + diff --git a/run-with-google-adk/ae_remote_deployment_sec.py b/run-with-google-adk/scripts/ae_remote_deployment_sec.py similarity index 53% rename from run-with-google-adk/ae_remote_deployment_sec.py rename to run-with-google-adk/scripts/ae_remote_deployment_sec.py index de97425d..5c704b1f 100644 --- a/run-with-google-adk/ae_remote_deployment_sec.py +++ b/run-with-google-adk/scripts/ae_remote_deployment_sec.py @@ -17,7 +17,25 @@ import sys -dotenv.load_dotenv("./google_mcp_security_agent/.env") +# Load .env file with error checking +env_file_path = "./agents/google_mcp_security_agent/.env" +print(f"Attempting to load .env file from: {env_file_path}") + +if not os.path.exists(env_file_path): + print("=" * 80) + print("CRITICAL ERROR: .env file not found!") + print(f"Expected location: {env_file_path}") + print("This will cause ALL MCP servers to be DISABLED!") + print("Available .env files:") + for root, dirs, files in os.walk("."): + for file in files: + if file.endswith('.env') or '.env' in file: + print(f" Found: {os.path.join(root, file)}") + print("=" * 80) + sys.exit(1) + +dotenv.load_dotenv(env_file_path) +print(f"✅ Successfully loaded .env file from: {env_file_path}") import vertexai import sys @@ -39,6 +57,18 @@ env_vars = sys.argv[1] env_vars_list=env_vars.split(",") +print(f"Raw environment variables string from command line: '{env_vars}'") +print(f"Parsed environment variables list: {env_vars_list}") + +if not env_vars or env_vars.strip() == "" or env_vars_list == ['']: + print("=" * 80) + print("CRITICAL ERROR: No environment variables provided!") + print("This means ALL MCP servers will be DISABLED!") + print("The .env file processing failed in the calling script.") + print("Check that the .env file exists and contains valid KEY=VALUE pairs.") + print("=" * 80) + sys.exit(1) + update=sys.argv[2] update_resource_name=sys.argv[3] @@ -46,8 +76,31 @@ for key in env_vars_list: if key not in ["GOOGLE_CLOUD_PROJECT","GOOGLE_CLOUD_LOCATION"]: # These are not allowed as env variables on the AE, filtering them. - env_vars_to_send[key] = os.environ.get(key) - + value = os.environ.get(key) + env_vars_to_send[key] = value + if value is None: + print(f"⚠️ WARNING: Environment variable '{key}' is None/missing!") + else: + # Only show first few chars of sensitive values + display_value = value[:10] + "..." if len(value) > 10 else value + print(f"✅ Environment variable '{key}': {display_value}") + +# Check specifically for MCP loader variables +mcp_vars = [k for k in env_vars_to_send.keys() if k.startswith('LOAD_') and k.endswith('_MCP')] +enabled_mcp = [k for k in mcp_vars if env_vars_to_send.get(k, '').lower() == 'true'] + +print(f"📊 MCP Status Summary:") +print(f" Total MCP loader variables found: {len(mcp_vars)}") +print(f" Enabled MCP servers: {len(enabled_mcp)}") +print(f" Enabled servers: {enabled_mcp}") + +if len(enabled_mcp) == 0: + print("=" * 80) + print("CRITICAL WARNING: NO MCP SERVERS ARE ENABLED!") + print("The deployed agent will have no security tools available!") + print("This is likely not what you want!") + print("=" * 80) + # override env_vars_to_send['GOOGLE_GENAI_USE_VERTEXAI'] = 'True' @@ -61,7 +114,7 @@ # # remote deplyment and first run # # remote run from vertexai import agent_engines -from google_mcp_security_agent import agent +from agents.google_mcp_security_agent import agent print(f"env_vars_to_send => {env_vars_to_send}") @@ -69,12 +122,12 @@ print("Creating a new Agent Engine Agent") remote_app = agent_engines.create( # Mostly for agent engine Console. - display_name="google_security_agent",description="Allows security actions on various google security products", + display_name="google_security_agent",description="Allows security actions on various google security products", agent_engine=agent.root_agent, requirements="requirements.txt", extra_packages=[ - "./google_mcp_security_agent", # a directory - "./utils_extensions_cbs_tools", # a directory + "./agents/google_mcp_security_agent", # a directory + "./libs/adk_utils", # a directory "./server", # a directory #"./temp", #"./object-viewer-sa.json", # a file @@ -96,19 +149,19 @@ else: import datetime now = datetime.datetime.now() - print(f"Updating the existing Agent Engine Agent {update_resource_name}") + print(f"Updating the existing Agent Engine Agent {update_resource_name}") remote_app = agent_engines.update( resource_name=update_resource_name, # Mostly for agent engine Console. - display_name="google_security_agent",description=f"Allows security actions on various google security products, updated {now.strftime("%Y_%m_%d_%H_%M_%S_%f")}", + display_name="google_security_agent",description=f"Allows security actions on various google security products, updated {now.strftime("%Y_%m_%d_%H_%M_%S_%f")}", agent_engine=agent.root_agent, requirements="requirements.txt", extra_packages=[ - "./google_mcp_security_agent", # a directory - "./utils_extensions_cbs_tools", # a directory + "./agents/google_mcp_security_agent", # a directory + "./libs/adk_utils", # a directory "./server", # a directory "./temp", "./object-viewer-sa.json", # a file ], env_vars=env_vars_to_send # send all required variables to agent engine. - ) \ No newline at end of file + ) diff --git a/run-with-google-adk/scripts/agentspace_deploy.sh b/run-with-google-adk/scripts/agentspace_deploy.sh new file mode 100755 index 00000000..12f2e152 --- /dev/null +++ b/run-with-google-adk/scripts/agentspace_deploy.sh @@ -0,0 +1,158 @@ +#!/bin/bash +# +# AgentSpace Deployment Script for Google MCP Security Agent +# This script registers an agent with AgentSpace using environment variables +# + +# Script directory +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${SCRIPT_DIR}/../google_mcp_security_agent/.env" + +# Load environment variables +if [ -f "$ENV_FILE" ]; then + # Export all variables from .env file + export $(grep -v '^#' "$ENV_FILE" | xargs) +else + echo "Error: Environment file not found: $ENV_FILE" + echo "Please run 'make env-template' and configure your .env file" + exit 1 +fi + +# Validate required environment variables +REQUIRED_VARS=( + "AGENTSPACE_PROJECT_ID" + "AGENTSPACE_PROJECT_NUMBER" + "AGENTSPACE_APP_NAME" + "AGENT_ENGINE_RESOURCE_NAME" + "GOOGLE_CLOUD_LOCATION" +) + +MISSING_VARS=() +for VAR in "${REQUIRED_VARS[@]}"; do + if [ -z "${!VAR}" ]; then + MISSING_VARS+=("$VAR") + fi +done + +if [ ${#MISSING_VARS[@]} -ne 0 ]; then + echo "Error: Missing required environment variables:" + for VAR in "${MISSING_VARS[@]}"; do + echo " - $VAR" + done + echo "" + echo "Please set these variables in: $ENV_FILE" + exit 1 +fi + +# Extract reasoning engine ID from full resource name +# Format: projects/PROJECT_NUMBER/locations/LOCATION/reasoningEngines/ENGINE_ID +REASONING_ENGINE_ID=$(echo "$AGENT_ENGINE_RESOURCE_NAME" | sed 's/.*reasoningEngines\///') + +# Use environment variables with defaults +AGENT_DISPLAY_NAME="${AGENT_DISPLAY_NAME:-Google Security Agent}" +AGENT_DESCRIPTION="${AGENT_DESCRIPTION:-Allows security operations on Google Security Products}" +AGENT_TOOL_DESCRIPTION="${AGENT_TOOL_DESCRIPTION:-Various Tools from SIEM, SOAR and SCC}" +AGENTSPACE_COLLECTION="${AGENTSPACE_COLLECTION:-default_collection}" +AGENTSPACE_ASSISTANT="${AGENTSPACE_ASSISTANT:-default_assistant}" + +# Construct API endpoint +TARGET_URL="https://discoveryengine.googleapis.com/v1alpha/projects/${AGENTSPACE_PROJECT_NUMBER}/locations/global/collections/${AGENTSPACE_COLLECTION}/engines/${AGENTSPACE_APP_NAME}/assistants/${AGENTSPACE_ASSISTANT}/agents" + +# Build JSON payload +JSON_DATA=$(cat </dev/null || echo "$JSON_DATA" +echo "" + +# Check if we're doing a dry run +if [ "$1" == "--dry-run" ]; then + echo "DRY RUN: Would send the above request" + exit 0 +fi + +echo "Sending POST request..." +echo "" + +# Perform the POST request using curl +RESPONSE=$(curl -s -w "\n%{http_code}" -X POST \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + -H "X-Goog-User-Project: ${AGENTSPACE_PROJECT_ID}" \ + -d "$JSON_DATA" \ + "$TARGET_URL" 2>&1) + +# Extract HTTP status code and response body +HTTP_CODE=$(echo "$RESPONSE" | tail -n1) +RESPONSE_BODY=$(echo "$RESPONSE" | sed '$d') + +echo "HTTP Status Code: $HTTP_CODE" +echo "" + +if [ "$HTTP_CODE" == "200" ]; then + echo "✓ Agent registered successfully!" + echo "" + echo "Response:" + echo "$RESPONSE_BODY" | jq . 2>/dev/null || echo "$RESPONSE_BODY" + + # Extract agent ID from response + AGENT_ID=$(echo "$RESPONSE_BODY" | jq -r '.name' 2>/dev/null | sed 's/.*agents\///') + if [ -n "$AGENT_ID" ] && [ "$AGENT_ID" != "null" ]; then + echo "" + echo "Agent ID: $AGENT_ID" + echo "" + echo "To save this ID to your environment:" + echo "make env-update KEY=AGENTSPACE_AGENT_ID VALUE=$AGENT_ID" + fi +else + echo "✗ Failed to register agent" + echo "" + echo "Response:" + echo "$RESPONSE_BODY" | jq . 2>/dev/null || echo "$RESPONSE_BODY" + exit 1 +fi + +echo "" +echo "Next steps:" +echo "1. Open AgentSpace in Google Cloud Console" +echo "2. From left menu, click 'Integration'" +echo "3. Open the URL provided" +echo "4. From left menu, select your agent" diff --git a/run-with-google-adk/scripts/agentspace_manager.py b/run-with-google-adk/scripts/agentspace_manager.py new file mode 100755 index 00000000..3f46cb72 --- /dev/null +++ b/run-with-google-adk/scripts/agentspace_manager.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +""" +AgentSpace Manager for Google MCP Security Agent + +This script manages AgentSpace operations including registration, updates, +verification, and deletion of agents in AgentSpace. +""" + +import json +import sys +from pathlib import Path +from typing import Any, Dict, Optional, Tuple + +import google.auth +from google.auth.transport import requests as google_requests +import requests +import typer +from typing_extensions import Annotated + +# Add parent directory to path for imports +sys.path.append(str(Path(__file__).parent.parent)) +from scripts.env_manager import EnvManager + +app = typer.Typer( + add_completion=False, + help="Manage AgentSpace operations for the Google MCP Security Agent.", +) + +DISCOVERY_ENGINE_API_BASE = "https://discoveryengine.googleapis.com/v1alpha" + + +class AgentSpaceManager: + """Manages AgentSpace configuration and operations.""" + + def __init__(self, env_file: Path): + """ + Initialize the AgentSpace manager. + + Args: + env_file: Path to the environment file. + """ + self.env_manager = EnvManager(env_file) + self.env_vars = self.env_manager.env_vars + self.creds, self.project = google.auth.default() + + def _get_access_token(self) -> Optional[str]: + """Get Google Cloud access token.""" + if not self.creds.valid: + self.creds.refresh(google_requests.Request()) + return self.creds.token + + def _validate_environment(self) -> Tuple[bool, list]: + """Validate required environment variables for AgentSpace operations.""" + required_vars = [ + "AGENTSPACE_PROJECT_ID", + "AGENTSPACE_PROJECT_NUMBER", + "AGENTSPACE_APP_NAME", + "AGENT_ENGINE_RESOURCE_NAME", + "GOOGLE_CLOUD_LOCATION", + ] + missing = [var for var in required_vars if not self.env_vars.get(var)] + return not missing, missing + + def _make_request( + self, method: str, url: str, **kwargs: Any + ) -> Optional[requests.Response]: + """Make an authenticated request to the Discovery Engine API.""" + access_token = self._get_access_token() + if not access_token: + return None + + headers = { + "Authorization": f"Bearer {access_token}", + "Content-Type": "application/json", + "X-Goog-User-Project": self.env_vars["AGENTSPACE_PROJECT_ID"], + } + headers.update(kwargs.pop("headers", {})) + + try: + response = requests.request(method, url, headers=headers, **kwargs) + response.raise_for_status() + return response + except requests.exceptions.RequestException as e: + typer.secho(f"✗ API request failed: {e}", fg=typer.colors.RED) + if e.response is not None: + typer.echo(f" Response: {e.response.text}") + return None + + def _get_agent_api_url(self, agent_id: Optional[str] = None) -> str: + """Construct the API URL for AgentSpace agents.""" + project_number = self.env_vars["AGENTSPACE_PROJECT_NUMBER"] + app_name = self.env_vars["AGENTSPACE_APP_NAME"] + collection = self.env_vars.get("AGENTSPACE_COLLECTION", "default_collection") + assistant = self.env_vars.get("AGENTSPACE_ASSISTANT", "default_assistant") + + url = ( + f"{DISCOVERY_ENGINE_API_BASE}/projects/{project_number}/" + f"locations/global/collections/{collection}/engines/{app_name}/" + f"assistants/{assistant}/agents" + ) + if agent_id: + url += f"/{agent_id}" + return url + + def _build_agent_config(self) -> Dict[str, Any]: + """Build the agent configuration payload.""" + config = { + "displayName": self.env_vars.get( + "AGENT_DISPLAY_NAME", "Google Security Agent" + ), + "description": self.env_vars.get( + "AGENT_DESCRIPTION", + "Allows security operations on Google Security Products", + ), + "adk_agent_definition": { + "tool_settings": { + "tool_description": self.env_vars.get( + "AGENT_TOOL_DESCRIPTION", "Various Tools from SIEM, SOAR and SCC" + ) + }, + "provisioned_reasoning_engine": { + "reasoning_engine": self.env_vars["AGENT_ENGINE_RESOURCE_NAME"] + }, + }, + } + if oauth_auth_id := self.env_vars.get("OAUTH_AUTH_ID"): + config["adk_agent_definition"]["authorizations"] = [ + f"projects/{self.env_vars['AGENTSPACE_PROJECT_NUMBER']}/locations/global/authorizations/{oauth_auth_id}" + ] + return config + + def register_agent(self, force: bool = False) -> bool: + """Register agent with AgentSpace.""" + typer.echo("Registering agent with AgentSpace...") + is_valid, missing = self._validate_environment() + if not is_valid: + typer.secho( + f"✗ Missing required variables: {', '.join(missing)}", + fg=typer.colors.RED, + ) + return False + + if self.env_vars.get("AGENTSPACE_AGENT_ID") and not force: + typer.secho( + "⚠️ Agent already registered. Use --force to re-register.", + fg=typer.colors.YELLOW, + ) + return False + + api_url = self._get_agent_api_url() + agent_config = self._build_agent_config() + + response = self._make_request("POST", api_url, json=agent_config) + if response and response.status_code == 200: + result = response.json() + agent_name = result.get("name", "") + agent_id = agent_name.split("/")[-1] if agent_name else "" + typer.secho("✓ Agent registered successfully!", fg=typer.colors.GREEN) + typer.echo(f" Agent Name: {agent_name}") + typer.echo(f" Agent ID: {agent_id}") + if agent_id: + self.env_manager.update_env("AGENTSPACE_AGENT_ID", agent_id) + return True + return False + + def update_agent(self) -> bool: + """Update existing AgentSpace agent configuration.""" + typer.echo("Updating AgentSpace agent...") + agent_id = self.env_vars.get("AGENTSPACE_AGENT_ID") + if not agent_id: + typer.secho( + "✗ No agent registered yet. Run 'register' first.", fg=typer.colors.RED + ) + return False + + api_url = self._get_agent_api_url(agent_id) + agent_config = self._build_agent_config() + + response = self._make_request("PATCH", api_url, json=agent_config) + if response and response.status_code == 200: + typer.secho("✓ Agent updated successfully!", fg=typer.colors.GREEN) + return True + return False + + def verify_agent(self) -> bool: + """Verify AgentSpace agent configuration and status.""" + typer.echo("Verifying AgentSpace configuration...") + is_valid, missing = self._validate_environment() + if not is_valid: + typer.secho( + f"✗ Missing required variables: {', '.join(missing)}", + fg=typer.colors.RED, + ) + return False + + agent_id = self.env_vars.get("AGENTSPACE_AGENT_ID") + if not agent_id: + typer.secho("⚠️ No agent registered yet.", fg=typer.colors.YELLOW) + return False + + api_url = self._get_agent_api_url(agent_id) + response = self._make_request("GET", api_url) + if response and response.status_code == 200: + typer.secho("✓ AgentSpace agent verified successfully!", fg=typer.colors.GREEN) + return True + return False + + def delete_agent(self, force: bool = False) -> bool: + """Delete agent from AgentSpace.""" + agent_id = self.env_vars.get("AGENTSPACE_AGENT_ID") + if not agent_id: + typer.secho("✗ No agent registered to delete.", fg=typer.colors.RED) + return True + + if not force and not typer.confirm( + f"Are you sure you want to delete agent {agent_id}?" + ): + typer.echo("Cancelled.") + return False + + api_url = self._get_agent_api_url(agent_id) + response = self._make_request("DELETE", api_url) + if response and response.status_code in [200, 204]: + typer.secho("✓ Agent deleted successfully!", fg=typer.colors.GREEN) + self.env_manager.update_env("AGENTSPACE_AGENT_ID", "") + return True + return False + + def display_url(self) -> None: + """Display AgentSpace UI URL.""" + project_id = self.env_vars.get("AGENTSPACE_PROJECT_ID") + app_name = self.env_vars.get("AGENTSPACE_APP_NAME") + if not all([project_id, app_name]): + typer.secho("✗ Cannot generate URL - missing configuration.", fg=typer.colors.RED) + return + + url = f"https://console.cloud.google.com/gen-ai-studio/agentspace/apps/{app_name}?project={project_id}" + typer.echo("AgentSpace UI URL:") + typer.echo("=" * 80) + typer.echo(url) + typer.echo("=" * 80) + + +@app.command() +def register( + force: Annotated[ + bool, typer.Option("--force", help="Force re-registration if agent exists.") + ] = False, + env_file: Annotated[ + Path, typer.Option(help="Path to the environment file.") + ] = Path("google_mcp_security_agent/.env"), +) -> None: + """Register the agent with AgentSpace.""" + manager = AgentSpaceManager(env_file) + if not manager.register_agent(force): + raise typer.Exit(code=1) + + +@app.command() +def update( + env_file: Annotated[ + Path, typer.Option(help="Path to the environment file.") + ] = Path("google_mcp_security_agent/.env"), +) -> None: + """Update the existing AgentSpace agent configuration.""" + manager = AgentSpaceManager(env_file) + if not manager.update_agent(): + raise typer.Exit(code=1) + + +@app.command() +def verify( + env_file: Annotated[ + Path, typer.Option(help="Path to the environment file.") + ] = Path("google_mcp_security_agent/.env"), +) -> None: + """Verify the AgentSpace agent configuration and status.""" + manager = AgentSpaceManager(env_file) + if not manager.verify_agent(): + raise typer.Exit(code=1) + + +@app.command() +def delete( + force: Annotated[ + bool, typer.Option("--force", help="Force deletion without confirmation.") + ] = False, + env_file: Annotated[ + Path, typer.Option(help="Path to the environment file.") + ] = Path("google_mcp_security_agent/.env"), +) -> None: + """Delete the agent from AgentSpace.""" + manager = AgentSpaceManager(env_file) + if not manager.delete_agent(force): + raise typer.Exit(code=1) + + +@app.command() +def url( + env_file: Annotated[ + Path, typer.Option(help="Path to the environment file.") + ] = Path("google_mcp_security_agent/.env"), +) -> None: + """Display the AgentSpace UI URL.""" + manager = AgentSpaceManager(env_file) + manager.display_url() + + +if __name__ == "__main__": + app() diff --git a/run-with-google-adk/cloudrun_deploy.py b/run-with-google-adk/scripts/cloudrun_deploy.py old mode 100644 new mode 100755 similarity index 90% rename from run-with-google-adk/cloudrun_deploy.py rename to run-with-google-adk/scripts/cloudrun_deploy.py index 3012719c..40d0a484 --- a/run-with-google-adk/cloudrun_deploy.py +++ b/run-with-google-adk/scripts/cloudrun_deploy.py @@ -13,13 +13,17 @@ # limitations under the License. import os +import sys + +# Add the run-with-google-adk directory to Python path for libs imports +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "run-with-google-adk")) import uvicorn from fastapi import FastAPI from google.adk.cli.fast_api import get_fast_api_app # Get the directory where main.py is located -AGENT_DIR = os.path.dirname(os.path.abspath(__file__))+"/run-with-google-adk" +AGENT_DIR = os.path.dirname(os.path.abspath(__file__))+"/run-with-google-adk/agents" # Example session DB URL (e.g., SQLite) SESSION_SERVICE_URI = None if os.environ.get("SESSION_SERVICE","in_memory") == "db": diff --git a/run-with-google-adk/cloudrun_deploy_run.sh b/run-with-google-adk/scripts/cloudrun_deploy_run.sh similarity index 78% rename from run-with-google-adk/cloudrun_deploy_run.sh rename to run-with-google-adk/scripts/cloudrun_deploy_run.sh index 35bdbd70..3c83ea4d 100755 --- a/run-with-google-adk/cloudrun_deploy_run.sh +++ b/run-with-google-adk/scripts/cloudrun_deploy_run.sh @@ -12,17 +12,17 @@ # See the License for the specific language governing permissions and # limitations under the License. -# this script runs from the top level directory (mcp-security when deploying and /app when running in container) +# this script runs from the run-with-google-adk directory (mcp-security when deploying and /app when running in container) #!/bin/bash -ENV_FILE="./run-with-google-adk/google_mcp_security_agent/.env" +ENV_FILE="./agents/google_mcp_security_agent/.env" # Function to create .env file create_env_file() { local env_file="$1" shift # Remove the first argument ($env_file) - + # Check if the .env file already exists if [ -f "$env_file" ]; then echo "Warning: $env_file already exists. Overwriting." @@ -77,7 +77,7 @@ if [ "$1" = "deploy" ]; then fi if [ "$key" = "GOOGLE_CLOUD_PROJECT" ]; then GOOGLE_CLOUD_PROJECT="$value" - fi + fi if [[ $env_vars == "" ]]; then env_vars="$key=$value" @@ -86,8 +86,8 @@ if [ "$1" = "deploy" ]; then fi env_vars="$env_vars,$key=$value" #echo "$line" - done < "$ENV_FILE" - + done < "$ENV_FILE" + #echo $env_vars @@ -113,7 +113,7 @@ else: print(prepared_text) EOF - default_prompt=$(python $PYTHON_SCRIPT_PATH) + default_prompt=$(python $PYTHON_SCRIPT_PATH) env_vars="$env_vars,DEFAULT_PROMPT=$default_prompt,GCS_SA_JSON=object-viewer-sa1.json" # Check if any environment variables were found @@ -130,50 +130,53 @@ EOF # Copying files in the top level directory as required by cloud run deployment echo "Temporarily copying files in the top level directory for image creation." - if [[ -e "./run-with-google-adk/object-viewer-sa.json" ]]; then - cp ./run-with-google-adk/object-viewer-sa.json object-viewer-sa1.json + if [[ -e "./object-viewer-sa.json" ]]; then + cp ./object-viewer-sa.json ../object-viewer-sa1.json + fi + cp ./scripts/cloudrun_deploy_run.sh .. + cp ./scripts/cloudrun_deploy.py .. + cp ./Dockerfile .. + if [[ -e "./.dockerignore" ]]; then + cp ./.dockerignore .. fi - cp ./run-with-google-adk/cloudrun_deploy_run.sh . - cp ./run-with-google-adk/cloudrun_deploy.py . - cp ./run-with-google-adk/Dockerfile . - cp ./run-with-google-adk/.dockerignore . + # Copy server directory for MCP tools + echo "Copying server directory for MCP tools..." + # Server directory is already in parent directory, no need to copy - - # Deploy the service with the dynamically constructed environment variables gcloud run deploy mcp-security-agent-service \ - --source . \ + --source .. \ --region "$GOOGLE_CLOUD_LOCATION" \ --project "$GOOGLE_CLOUD_PROJECT" \ --allow-unauthenticated \ --set-env-vars="$env_vars" \ --memory 2Gi - + deploy_status=$? #get the status # Check the status of the deployment if [ "$deploy_status" -eq 0 ]; then - # Deleting temporarily files in the top level directory - echo "Deleting temporarily copied files in the top level directory for image creation." - rm ./cloudrun_deploy_run.sh - rm ./cloudrun_deploy.py - rm ./Dockerfile - rm ./.dockerignore - if [[ -e "./run-with-google-adk/object-viewer-sa.json" ]]; then - rm object-viewer-sa1.json + # Deleting temporarily files in the top level directory + echo "Deleting temporarily copied files in the top level directory." + rm -f ../cloudrun_deploy_run.sh + rm -f ../cloudrun_deploy.py + rm -f ../Dockerfile + rm -f ../.dockerignore + if [[ -e "../object-viewer-sa1.json" ]]; then + rm -f ../object-viewer-sa1.json fi - echo "Successfully deployed the service." else - rm ./cloudrun_deploy_run.sh - rm ./cloudrun_deploy.py - rm ./Dockerfile - rm ./.dockerignore - if [[ -e "./run-with-google-adk/object-viewer-sa.json" ]]; then - rm object-viewer-sa1.json + # Clean up even on failure + rm -f ../cloudrun_deploy_run.sh + rm -f ../cloudrun_deploy.py + rm -f ../Dockerfile + rm -f ../.dockerignore + if [[ -e "../object-viewer-sa1.json" ]]; then + rm -f ../object-viewer-sa1.json fi echo "Failed to deploy the service." - #exit 1 + exit 1 fi elif [ "$1" = "run" ]; then diff --git a/run-with-google-adk/scripts/env_manager.py b/run-with-google-adk/scripts/env_manager.py new file mode 100755 index 00000000..f0223ce9 --- /dev/null +++ b/run-with-google-adk/scripts/env_manager.py @@ -0,0 +1,320 @@ +#!/usr/bin/env python3 +""" +Environment Manager for Google MCP Security Agent + +This script manages environment variables for the Google MCP Security Agent, +including validation, template generation, and configuration display. +""" + +import os +import sys +import uuid +from pathlib import Path +from typing import Dict, List, Optional, Tuple + +import typer +from typing_extensions import Annotated + +app = typer.Typer( + add_completion=False, + help="Manage environment configuration for the Google MCP Security Agent.", +) + +# Define base directory +BASE_DIR = Path(__file__).resolve().parent.parent + + +class EnvManager: + """Manages environment configuration for the Google MCP Security Agent.""" + + # Define required variables for different deployment scenarios + REQUIRED_VARS = { + "base": ["APP_NAME", "GOOGLE_CLOUD_PROJECT", "GOOGLE_CLOUD_LOCATION"], + "mcp_servers": { + "secops": [ + "CHRONICLE_PROJECT_ID", + "CHRONICLE_CUSTOMER_ID", + "CHRONICLE_REGION", + ], + "gti": ["VT_APIKEY"], + "secops_soar": ["SOAR_URL", "SOAR_APP_KEY"], + "scc": [], # No additional vars needed + }, + "agent_engine": [], + "agentspace": [ + "AGENTSPACE_PROJECT_ID", + "AGENTSPACE_PROJECT_NUMBER", + "AGENTSPACE_APP_NAME", + "AGENT_ENGINE_RESOURCE_NAME", + ], + "oauth": ["OAUTH_CLIENT_ID", "OAUTH_CLIENT_SECRET", "OAUTH_AUTH_ID"], + "vertex_ai": ["GOOGLE_CLOUD_PROJECT", "GOOGLE_CLOUD_LOCATION"], + "gemini_api": ["GOOGLE_API_KEY"], + "cloudrun": [], # Cloud Run deployment doesn't require MCP configs + } + + # Sensitive variables that should be masked + SENSITIVE_VARS = [ + "GOOGLE_API_KEY", + "VT_APIKEY", + "SOAR_APP_KEY", + "OAUTH_CLIENT_SECRET", + "XDR_CLIENT_SECRET", + "IDP_CLIENT_SECRET", + ] + + def __init__(self, env_file: Path): + """ + Initialize the environment manager. + + Args: + env_file: Path to the environment file. + """ + self.env_file = Path(env_file) + print(f"DEBUG: Loading env file from: {self.env_file.resolve()}") + self.env_vars = self._load_env() + + def _is_uuid(self, value: str) -> bool: + """Check if a string is a valid UUID.""" + try: + uuid.UUID(value) + return True + except ValueError: + return False + + def _load_env(self) -> Dict[str, str]: + """Load environment variables from file and system environment.""" + env_vars = dict(os.environ) + + if self.env_file.exists(): + with open(self.env_file, "r") as f: + for line in f: + line = line.strip() + if line and not line.startswith("#") and "=" in line: + key, value = line.split("=", 1) + if value.startswith("'") and value.endswith("'"): + value = value[1:-1] + elif value.startswith('"') and value.endswith('"'): + value = value[1:-1] + env_vars[key.strip()] = value.strip() + return env_vars + + def validate(self, deployment_type: str = "base") -> Tuple[bool, List[str]]: + """ + Validate required environment variables for a deployment type. + + Args: + deployment_type: The deployment scenario to validate against. + + Returns: + A tuple containing a boolean indicating validity and a list of missing variables. + """ + missing_vars = [] + required = self.REQUIRED_VARS + + # Base requirements + missing_vars.extend( + v for v in required["base"] if not self.env_vars.get(v) + ) + + # MCP server requirements + for server, variables in required["mcp_servers"].items(): + if self.env_vars.get(f"LOAD_{server.upper()}_MCP") == "true": + for var in variables: + value = self.env_vars.get(var) + if not value or value == "NOT_SET": + missing_vars.append(var) + elif var == "CHRONICLE_CUSTOMER_ID" and not self._is_uuid(value): + missing_vars.append(f"{var} (must be a valid UUID4)") + elif var == "SOAR_URL" and not value.startswith("https://"): + missing_vars.append(f"{var} (must start with https://)") + elif var == "SOAR_APP_KEY" and not self._is_uuid(value): + missing_vars.append(f"{var} (must be a valid UUID)") + + # Deployment-specific requirements + if deployment_type in required: + missing_vars.extend( + v for v in required[deployment_type] if not self.env_vars.get(v) + ) + + # LLM provider requirements + if self.env_vars.get("GOOGLE_GENAI_USE_VERTEXAI") == "True": + missing_vars.extend( + v for v in required["vertex_ai"] if not self.env_vars.get(v) + ) + else: + missing_vars.extend( + v + for v in required["gemini_api"] + if not self.env_vars.get(v) or self.env_vars.get(v) == "NOT_SET" + ) + + return not missing_vars, sorted(list(set(missing_vars))) + + def mask_value(self, key: str, value: str) -> str: + """Mask sensitive values for display.""" + if key in self.SENSITIVE_VARS and value and value != "NOT_SET": + return "*" * len(value) + return value + + def display_config(self, show_all: bool = False) -> None: + """Display current configuration with sensitive values masked.""" + typer.echo("=" * 60) + typer.echo("Google MCP Security Agent Configuration") + typer.echo("=" * 60) + typer.echo() + + categories = { + "Core Configuration": [ + "APP_NAME", + "GOOGLE_CLOUD_PROJECT", + "GOOGLE_CLOUD_LOCATION", + "GOOGLE_MODEL", + "GOOGLE_GENAI_USE_VERTEXAI", + ], + "MCP Servers": [ + "LOAD_SECOPS_MCP", + "CHRONICLE_PROJECT_ID", + "CHRONICLE_CUSTOMER_ID", + "CHRONICLE_REGION", + "LOAD_GTI_MCP", + "VT_APIKEY", + "LOAD_SECOPS_SOAR_MCP", + "SOAR_URL", + "SOAR_APP_KEY", + "LOAD_SCC_MCP", + ], + "Agent Engine": [ + "AGENT_ENGINE_RESOURCE_NAME", + "GCS_STAGING_BUCKET", + "AGENT_DISPLAY_NAME", + "AGENT_DESCRIPTION", + ], + "AgentSpace": [ + "AGENTSPACE_PROJECT_ID", + "AGENTSPACE_PROJECT_NUMBER", + "AGENTSPACE_APP_NAME", + "AGENTSPACE_COLLECTION", + "AGENTSPACE_ASSISTANT", + "AGENTSPACE_AGENT_ID", + ], + "OAuth": [ + "OAUTH_CLIENT_ID", + "OAUTH_CLIENT_SECRET", + "OAUTH_AUTH_ID", + "OAUTH_REDIRECT_URI", + "OAUTH_TOKEN_URI", + ], + } + + for category, variables in categories.items(): + typer.echo(f"{category}:") + typer.echo("-" * 40) + displayed_any = False + for var in variables: + value = self.env_vars.get(var) + if value or show_all: + masked_value = self.mask_value(var, value) if value else "" + typer.echo(f" {var}: {masked_value}") + displayed_any = True + if not displayed_any: + typer.echo(" ") + typer.echo() + + def update_env(self, key: str, value: str) -> None: + """Update a single environment variable in the .env file.""" + self.env_vars[key] = value + self._save_env() + typer.echo(f"✓ Updated {key}") + + def _save_env(self) -> None: + """Save environment variables back to the .env file.""" + lines = self.env_file.read_text().splitlines() if self.env_file.exists() else [] + + updated_keys = set() + new_lines = [] + + for line in lines: + stripped = line.strip() + if stripped and not stripped.startswith("#") and "=" in stripped: + line_key = stripped.split("=", 1)[0].strip() + if line_key in self.env_vars: + if line_key == "DEFAULT_PROMPT": + new_lines.append(f"{line_key}='{self.env_vars[line_key]}'") + else: + new_lines.append(f"{line_key}={self.env_vars[line_key]}") + updated_keys.add(line_key) + else: + new_lines.append(line) + else: + new_lines.append(line) + + for key, value in self.env_vars.items(): + if key not in updated_keys and not key.startswith("_") and key.isupper(): + if key == "DEFAULT_PROMPT": + new_lines.append(f"\n{key}='{value}'") + else: + new_lines.append(f"\n{key}={value}") + + self.env_file.write_text("\n".join(new_lines) + "\n") + + +@app.command() +def check( + deployment: Annotated[ + str, + typer.Option( + help="Deployment type to validate.", + case_sensitive=False, + ), + ] = "base", + env_file: Annotated[ + Path, + typer.Option(help="Path to the environment file."), + ] = BASE_DIR / "agents" / "google_mcp_security_agent" / ".env", +) -> None: + """Validate the environment configuration for a given deployment type.""" + manager = EnvManager(env_file) + is_valid, missing = manager.validate(deployment) + if not is_valid: + typer.secho( + f"✗ Environment validation failed for {deployment}", fg=typer.colors.RED + ) + typer.echo(f" Missing variables: {', '.join(missing)}") + raise typer.Exit(code=1) + else: + typer.secho(f"✓ Environment valid for {deployment}", fg=typer.colors.GREEN) + + +@app.command() +def show( + all_vars: Annotated[ + bool, + typer.Option("--all", help="Show all variables, including unset ones."), + ] = False, + env_file: Annotated[ + Path, + typer.Option(help="Path to the environment file."), + ] = BASE_DIR / "agents" / "google_mcp_security_agent" / ".env", +) -> None: + """Display the current configuration, masking sensitive values.""" + manager = EnvManager(env_file) + manager.display_config(all_vars) + + +@app.command() +def update( + key: Annotated[str, typer.Option(help="Environment variable key to update.")], + value: Annotated[str, typer.Option(help="New value for the variable.")], + env_file: Annotated[ + Path, + typer.Option(help="Path to the environment file."), + ] = BASE_DIR / "agents" / "google_mcp_security_agent" / ".env", +) -> None: + """Update a variable in the environment file.""" + manager = EnvManager(env_file) + manager.update_env(key, value) + + +if __name__ == "__main__": + app() diff --git a/run-with-google-adk/scripts/manage_agents.py b/run-with-google-adk/scripts/manage_agents.py new file mode 100755 index 00000000..eb58bea7 --- /dev/null +++ b/run-with-google-adk/scripts/manage_agents.py @@ -0,0 +1,254 @@ +#!/usr/bin/env python3 +""" +Utility to manage (list and delete) remote agents in Google Agent Engine +""" + +import argparse +import sys +from typing import List, Optional +import vertexai +from vertexai import agent_engines +from google.cloud import aiplatform +from datetime import datetime +import os + +# ANSI color codes for better output +class Colors: + HEADER = '\033[95m' + BLUE = '\033[94m' + CYAN = '\033[96m' + GREEN = '\033[92m' + YELLOW = '\033[93m' + RED = '\033[91m' + ENDC = '\033[0m' + BOLD = '\033[1m' + +def print_header(text): + print(f"\n{Colors.HEADER}{Colors.BOLD}{'=' * 80}{Colors.ENDC}") + print(f"{Colors.HEADER}{Colors.BOLD}{text.center(80)}{Colors.ENDC}") + print(f"{Colors.HEADER}{Colors.BOLD}{'=' * 80}{Colors.ENDC}\n") + +def format_timestamp(timestamp): + """Format timestamp to readable string""" + if timestamp: + dt = datetime.fromtimestamp(timestamp) + return dt.strftime("%Y-%m-%d %H:%M:%S") + return "N/A" + +def initialize_vertexai(project: str, location: str): + """Initialize Vertex AI with project and location""" + try: + vertexai.init(project=project, location=location) + aiplatform.init(project=project, location=location) + print(f"{Colors.GREEN}✅ Initialized Vertex AI - Project: {project}, Location: {location}{Colors.ENDC}") + except Exception as e: + print(f"{Colors.RED}❌ Failed to initialize Vertex AI: {str(e)}{Colors.ENDC}") + sys.exit(1) + +def list_agents(project: str, location: str, verbose: bool = False) -> List[dict]: + """List all Agent Engine instances""" + print_header("Listing Agent Engine Instances") + + initialize_vertexai(project, location) + + try: + # Use the vertexai API to list reasoning engines + from google.cloud.aiplatform_v1beta1 import ReasoningEngineServiceClient + from google.api_core import client_options + + # Create client with regional endpoint + endpoint = f"{location}-aiplatform.googleapis.com" + client_opts = client_options.ClientOptions(api_endpoint=endpoint) + client = ReasoningEngineServiceClient(client_options=client_opts) + + parent = f"projects/{project}/locations/{location}" + agents = client.list_reasoning_engines(parent=parent) + + agent_list = [] + agents_list = list(agents) # Convert to list to get count + + if not agents_list: + print(f"{Colors.YELLOW}No Agent Engine instances found.{Colors.ENDC}") + return [] + + print(f"Found {Colors.CYAN}{len(agents_list)}{Colors.ENDC} Agent Engine instance(s):\n") + + for i, agent in enumerate(agents_list, 1): + agent_info = { + 'resource_name': agent.name, + 'display_name': agent.display_name, + 'create_time': agent.create_time, + 'update_time': agent.update_time, + 'state': agent.state.name if hasattr(agent, 'state') else 'UNKNOWN', + } + agent_list.append(agent_info) + + print(f"{Colors.CYAN}{i}. {agent.display_name}{Colors.ENDC}") + print(f" Resource: {agent.name}") + print(f" Created: {format_timestamp(agent.create_time.timestamp() if agent.create_time else None)}") + print(f" Updated: {format_timestamp(agent.update_time.timestamp() if agent.update_time else None)}") + + if verbose: + print(f" State: {agent_info['state']}") + # Try to get additional details + try: + full_agent = agent_engines.get(agent.name) + print(f" Type: {type(full_agent).__name__}") + except Exception as e: + print(f" {Colors.YELLOW}Could not fetch additional details: {str(e)}{Colors.ENDC}") + + print() + + return agent_list + + except Exception as e: + print(f"{Colors.RED}❌ Error listing agents: {str(e)}{Colors.ENDC}") + return [] + +def delete_agent(resource_name: str, project: str, location: str, force: bool = False) -> bool: + """Delete a specific Agent Engine instance""" + print_header("Deleting Agent Engine Instance") + + initialize_vertexai(project, location) + + try: + # Get the agent first to confirm it exists + from google.cloud.aiplatform_v1beta1 import ReasoningEngineServiceClient + from google.api_core import client_options + + # Create client with regional endpoint + endpoint = f"{location}-aiplatform.googleapis.com" + client_opts = client_options.ClientOptions(api_endpoint=endpoint) + client = ReasoningEngineServiceClient(client_options=client_opts) + + print(f"Fetching agent: {resource_name}") + agent = client.get_reasoning_engine(name=resource_name) + + print(f"\n{Colors.YELLOW}Agent Details:{Colors.ENDC}") + print(f" Name: {agent.display_name}") + print(f" Resource: {agent.name}") + print(f" Created: {format_timestamp(agent.create_time.timestamp() if agent.create_time else None)}") + + if not force: + confirm = input(f"\n{Colors.RED}Are you sure you want to delete this agent? (yes/no): {Colors.ENDC}") + if confirm.lower() != 'yes': + print(f"{Colors.YELLOW}Deletion cancelled.{Colors.ENDC}") + return False + + print(f"\n{Colors.YELLOW}Deleting agent...{Colors.ENDC}") + # Delete with force=True to remove child resources + from google.cloud.aiplatform_v1beta1 import DeleteReasoningEngineRequest + request = DeleteReasoningEngineRequest( + name=resource_name, + force=True # This will delete child resources (sessions, memories) + ) + client.delete_reasoning_engine(request=request) + print(f"{Colors.GREEN}✅ Agent deleted successfully!{Colors.ENDC}") + return True + + except Exception as e: + print(f"{Colors.RED}❌ Error deleting agent: {str(e)}{Colors.ENDC}") + return False + +def delete_agent_by_index(index: int, project: str, location: str, force: bool = False) -> bool: + """Delete an agent by its index in the list""" + agents = list_agents(project, location, verbose=False) + + if not agents: + return False + + if index < 1 or index > len(agents): + print(f"{Colors.RED}❌ Invalid index. Please choose between 1 and {len(agents)}{Colors.ENDC}") + return False + + agent = agents[index - 1] + return delete_agent(agent['resource_name'], project, location, force) + +def main(): + parser = argparse.ArgumentParser( + description="Manage Google Agent Engine instances", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # List all agents in the default project + python manage_agents.py list + + # List agents in a specific project and location + python manage_agents.py list --project my-project --location us-central1 + + # List agents with verbose output + python manage_agents.py list -v + + # Delete an agent by resource name + python manage_agents.py delete --resource projects/123/locations/us-central1/reasoningEngines/456 + + # Delete an agent by index from the list + python manage_agents.py delete --index 2 + + # Force delete without confirmation + python manage_agents.py delete --index 1 --force + """ + ) + + # Get defaults from environment or gcloud config + default_project = os.getenv('GOOGLE_CLOUD_PROJECT', '') + if not default_project: + try: + import subprocess + result = subprocess.run(['gcloud', 'config', 'get-value', 'project'], + capture_output=True, text=True, check=True) + default_project = result.stdout.strip() + except: + default_project = None + + default_location = os.getenv('GOOGLE_CLOUD_LOCATION', 'us-central1') + + # Global options + parser.add_argument('--project', '-p', + default=default_project, + help=f'Google Cloud project ID (default: {default_project or "from gcloud config"})') + parser.add_argument('--location', '-l', + default=default_location, + help=f'Google Cloud location (default: {default_location})') + + # Subcommands + subparsers = parser.add_subparsers(dest='command', help='Commands') + + # List command + list_parser = subparsers.add_parser('list', help='List all Agent Engine instances') + list_parser.add_argument('--verbose', '-v', action='store_true', + help='Show verbose output') + + # Delete command + delete_parser = subparsers.add_parser('delete', help='Delete an Agent Engine instance') + delete_group = delete_parser.add_mutually_exclusive_group(required=True) + delete_group.add_argument('--resource', '-r', + help='Full resource name of the agent to delete') + delete_group.add_argument('--index', '-i', type=int, + help='Index of the agent from the list to delete') + delete_parser.add_argument('--force', '-f', action='store_true', + help='Force deletion without confirmation') + + args = parser.parse_args() + + # Check if project is set + if not args.project: + print(f"{Colors.RED}❌ Error: No project specified. Use --project or set GOOGLE_CLOUD_PROJECT{Colors.ENDC}") + sys.exit(1) + + # Execute command + if args.command == 'list': + list_agents(args.project, args.location, args.verbose) + elif args.command == 'delete': + if args.resource: + success = delete_agent(args.resource, args.project, args.location, args.force) + else: # args.index + success = delete_agent_by_index(args.index, args.project, args.location, args.force) + + sys.exit(0 if success else 1) + else: + parser.print_help() + sys.exit(1) + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/run-with-google-adk/scripts/oauth_manager.py b/run-with-google-adk/scripts/oauth_manager.py new file mode 100755 index 00000000..c9ddb4a6 --- /dev/null +++ b/run-with-google-adk/scripts/oauth_manager.py @@ -0,0 +1,351 @@ +#!/usr/bin/env python3 +""" +OAuth Manager for Google MCP Security Agent + +This script manages OAuth operations for AgentSpace integration, +including client creation guidance, authorization URI generation, +and linking OAuth to AgentSpace. +""" + +import os +import sys +import json +import subprocess +import webbrowser +from pathlib import Path +from typing import Dict, Optional, Tuple +from urllib.parse import urlencode +import argparse +from datetime import datetime + +# Add parent directory to path for imports +sys.path.append(str(Path(__file__).parent.parent)) +from scripts.env_manager import EnvManager + + +class OAuthManager: + """Manages OAuth configuration and operations.""" + + OAUTH_CONSOLE_URL = "https://console.cloud.google.com/apis/credentials" + REDIRECT_URI = "https://vertexaisearch.cloud.google.com/oauth-redirect" + TOKEN_URI = "https://oauth2.googleapis.com/token" + + def __init__(self, env_file: str = '.env'): + """Initialize the OAuth manager.""" + self.env_manager = EnvManager(env_file) + self.env_vars = self.env_manager.env_vars + + def guide_client_creation(self) -> None: + """Guide user through OAuth client creation process.""" + print("=" * 60) + print("OAuth Client Creation Guide") + print("=" * 60) + print() + + project = self.env_vars.get('GOOGLE_CLOUD_PROJECT') + if not project: + print("⚠️ Warning: GOOGLE_CLOUD_PROJECT not set in .env") + print(" Please set it first: make env-update KEY=GOOGLE_CLOUD_PROJECT VALUE=") + print() + + print("Steps to create an OAuth 2.0 client:") + print() + print("1. Open the Google Cloud Console:") + print(f" {self.OAUTH_CONSOLE_URL}") + if project: + print(f" (Make sure you're in project: {project})") + print() + + print("2. Click '+ CREATE CREDENTIALS' → 'OAuth client ID'") + print() + + print("3. If prompted to configure consent screen:") + print(" - Choose 'Internal' for organization use") + print(" - Fill in required fields") + print(" - Add scopes if needed") + print() + + print("4. For Application type, select 'Web application'") + print() + + print("5. Configure the client:") + print(" - Name: 'AgentSpace MCP Security Agent' (or similar)") + print(" - Authorized redirect URIs:") + print(f" {self.REDIRECT_URI}") + print() + + print("6. Click 'CREATE'") + print() + + print("7. Download the client configuration:") + print(" - Click the download button (⬇) next to your new client") + print(" - Save as 'client_secret.json' in this directory") + print() + + # Offer to open the console + response = input("Would you like to open the Google Cloud Console now? (y/n): ") + if response.lower() == 'y': + webbrowser.open(self.OAUTH_CONSOLE_URL) + print("\n✓ Opened Google Cloud Console in your browser") + + print("\nOnce you've downloaded the client_secret.json file:") + print("Run: make oauth-uri") + + def generate_auth_uri(self, client_secret_path: Optional[str] = None) -> str: + """Generate OAuth authorization URI.""" + # Use provided path or look for client_secret.json + if not client_secret_path: + client_secret_path = 'client_secret.json' + + if not Path(client_secret_path).exists(): + print(f"Error: Client secret file not found: {client_secret_path}") + print("\nPlease run 'make oauth-client' to create an OAuth client first.") + sys.exit(1) + + # Load client configuration + with open(client_secret_path, 'r') as f: + client_data = json.load(f) + + # Extract client configuration + if 'web' in client_data: + client_config = client_data['web'] + elif 'installed' in client_data: + client_config = client_data['installed'] + else: + raise ValueError("Invalid client_secret.json format") + + client_id = client_config['client_id'] + client_secret = client_config['client_secret'] + auth_uri = client_config.get('auth_uri', 'https://accounts.google.com/o/oauth2/v2/auth') + + # Update environment variables + self.env_manager.update_env('OAUTH_CLIENT_ID', client_id) + self.env_manager.update_env('OAUTH_CLIENT_SECRET', client_secret) + self.env_manager.update_env('OAUTH_TOKEN_URI', self.TOKEN_URI) + + # Get or generate auth ID + auth_id = self.env_vars.get('OAUTH_AUTH_ID') + if not auth_id: + auth_id = f"mcp-agent-{datetime.now().strftime('%Y%m%d-%H%M%S')}" + self.env_manager.update_env('OAUTH_AUTH_ID', auth_id) + print(f"Generated AUTH_ID: {auth_id}") + + # Get redirect URI and scopes + redirect_uri = self.env_vars.get('OAUTH_REDIRECT_URI', self.REDIRECT_URI) + scopes = self.env_vars.get('OAUTH_SCOPES', + 'https://www.googleapis.com/auth/cloud-platform,' + 'https://www.googleapis.com/auth/generative-language.retriever' + ).split(',') + + # OAuth parameters + params = { + 'client_id': client_id, + 'redirect_uri': redirect_uri, + 'response_type': 'code', + 'scope': ' '.join(scopes), + 'access_type': 'offline', + 'prompt': 'consent' + } + + # Generate authorization URL + auth_url = f"{auth_uri}?{urlencode(params)}" + + # Save to environment + self.env_manager.update_env('OAUTH_AUTH_URI', auth_url) + + print("\n✓ OAuth configuration updated in .env") + print("\nOAuth Authorization URI:") + print("=" * 80) + print(auth_url) + print("=" * 80) + print("\nNext steps:") + print("1. Open the above URL in your browser") + print("2. Sign in and grant permissions") + print("3. After redirect, run: make oauth-link") + + return auth_url + + def link_to_agentspace(self) -> bool: + """Link OAuth authorization to AgentSpace.""" + # Validate required variables + required = [ + 'AGENTSPACE_PROJECT_NUMBER', + 'OAUTH_CLIENT_ID', + 'OAUTH_CLIENT_SECRET', + 'OAUTH_AUTH_ID', + 'OAUTH_AUTH_URI' + ] + + missing = [var for var in required if not self.env_vars.get(var)] + if missing: + print("Error: Missing required environment variables:") + for var in missing: + print(f" - {var}") + print("\nPlease set these variables in your .env file") + return False + + project_number = self.env_vars['AGENTSPACE_PROJECT_NUMBER'] + auth_id = self.env_vars['OAUTH_AUTH_ID'] + + # Prepare API request + api_url = ( + f"https://discoveryengine.googleapis.com/v1alpha/" + f"projects/{project_number}/locations/global/authorizations" + f"?authorizationId={auth_id}" + ) + + payload = { + "name": f"projects/{project_number}/locations/global/authorizations/{auth_id}", + "serverSideOauth2": { + "clientId": self.env_vars['OAUTH_CLIENT_ID'], + "clientSecret": self.env_vars['OAUTH_CLIENT_SECRET'], + "authorizationUri": self.env_vars['OAUTH_AUTH_URI'], + "tokenUri": self.env_vars.get('OAUTH_TOKEN_URI', self.TOKEN_URI) + } + } + + # Create the authorization + print(f"Creating OAuth authorization '{auth_id}' in AgentSpace...") + + try: + # Get access token + token_result = subprocess.run( + ['gcloud', 'auth', 'print-access-token'], + capture_output=True, + text=True, + check=True + ) + access_token = token_result.stdout.strip() + + # Make API request + import requests + headers = { + 'Authorization': f'Bearer {access_token}', + 'Content-Type': 'application/json', + 'X-Goog-User-Project': project_number + } + + response = requests.post(api_url, json=payload, headers=headers) + + if response.status_code == 200: + print("\n✓ OAuth authorization created successfully!") + print(f" Authorization ID: {auth_id}") + print("\nNext steps:") + print("1. Deploy your agent: make adk-deploy") + print("2. Register with AgentSpace: make agentspace-register") + return True + else: + print(f"\n✗ Failed to create authorization: {response.status_code}") + print(f" Response: {response.text}") + return False + + except subprocess.CalledProcessError as e: + print(f"\n✗ Failed to get access token: {e}") + print(" Make sure you're authenticated: gcloud auth login") + return False + except Exception as e: + print(f"\n✗ Error creating authorization: {e}") + return False + + def verify_oauth(self) -> bool: + """Verify OAuth configuration and authorization.""" + print("Verifying OAuth configuration...") + print() + + # Check client configuration + client_vars = ['OAUTH_CLIENT_ID', 'OAUTH_CLIENT_SECRET'] + client_ok = all(self.env_vars.get(var) for var in client_vars) + + if client_ok: + print("✓ OAuth client configured") + print(f" Client ID: {self.env_vars['OAUTH_CLIENT_ID'][:20]}...") + else: + print("✗ OAuth client not configured") + print(" Run: make oauth-client") + return False + + # Check authorization + if self.env_vars.get('OAUTH_AUTH_ID'): + print(f"✓ OAuth authorization ID: {self.env_vars['OAUTH_AUTH_ID']}") + else: + print("✗ OAuth authorization not created") + print(" Run: make oauth-uri") + return False + + # Check AgentSpace link + if self.env_vars.get('AGENTSPACE_PROJECT_NUMBER'): + # Try to verify the authorization exists + project_number = self.env_vars['AGENTSPACE_PROJECT_NUMBER'] + auth_id = self.env_vars['OAUTH_AUTH_ID'] + + try: + token_result = subprocess.run( + ['gcloud', 'auth', 'print-access-token'], + capture_output=True, + text=True, + check=True + ) + access_token = token_result.stdout.strip() + + import requests + api_url = ( + f"https://discoveryengine.googleapis.com/v1alpha/" + f"projects/{project_number}/locations/global/authorizations/{auth_id}" + ) + + headers = { + 'Authorization': f'Bearer {access_token}', + 'X-Goog-User-Project': project_number + } + + response = requests.get(api_url, headers=headers) + + if response.status_code == 200: + print("✓ OAuth linked to AgentSpace") + else: + print("✗ OAuth not linked to AgentSpace") + print(" Run: make oauth-link") + return False + + except Exception: + print("⚠️ Could not verify AgentSpace link") + else: + print("⚠️ AgentSpace not configured") + + print("\n✓ OAuth configuration verified") + return True + + +def main(): + """Main entry point for CLI usage.""" + parser = argparse.ArgumentParser(description='Manage OAuth configuration') + parser.add_argument('action', choices=['guide', 'generate', 'link', 'verify'], + help='Action to perform') + parser.add_argument('--client-secret', help='Path to client_secret.json') + parser.add_argument('--env-file', default='.env', help='Environment file to use') + + args = parser.parse_args() + + manager = OAuthManager(args.env_file) + + if args.action == 'guide': + manager.guide_client_creation() + + elif args.action == 'generate': + manager.generate_auth_uri(args.client_secret) + + elif args.action == 'link': + if manager.link_to_agentspace(): + sys.exit(0) + else: + sys.exit(1) + + elif args.action == 'verify': + if manager.verify_oauth(): + sys.exit(0) + else: + sys.exit(1) + + +if __name__ == '__main__': + main() \ No newline at end of file diff --git a/run-with-google-adk/run-adk-agent.sh b/run-with-google-adk/scripts/run-adk-agent.sh similarity index 79% rename from run-with-google-adk/run-adk-agent.sh rename to run-with-google-adk/scripts/run-adk-agent.sh index 89541668..f7025d2a 100755 --- a/run-with-google-adk/run-adk-agent.sh +++ b/run-with-google-adk/scripts/run-adk-agent.sh @@ -15,8 +15,8 @@ #!/bin/bash # Define the file paths -SAMPLE_ENV_FILE="./google_mcp_security_agent/sample.env.properties" -ENV_FILE="./google_mcp_security_agent/.env" +ENV_FILE="./agents/google_mcp_security_agent/.env" +ENV_TEMPLATE_FILE="./agents/google_mcp_security_agent/.env.example" # Function to mask environment variable values mask_env_value() { @@ -95,30 +95,37 @@ case "$COMMAND" in adk_web) # If .env exists, display its contents with masked values and run the command show_env_masked + + # Copy libs to agent directory if not present (needed for imports) + if [ ! -d "agents/google_mcp_security_agent/libs" ]; then + echo "Copying libs to agent directory for local testing..." + cp -r libs agents/google_mcp_security_agent/ + fi + # Handle adk_web command based on argument count if [ "$#" -eq 1 ]; then echo "Running ADK Web for local agent..." - adk web + adk web agents elif [ "$#" -eq 2 ]; then echo "Running ADK Web with session service URI: $2" - adk web --session_service_uri "$2" + adk web agents --session_service_uri "$2" elif [ "$#" -eq 3 ]; then echo "Running ADK Web with session service URI: $2 and artifact service URI: $3" - adk web --session_service_uri "$2" --artifact_service_uri "$3" + adk web agents --session_service_uri "$2" --artifact_service_uri "$3" else echo "Error: Incorrect number of arguments for 'adk_web'." usage fi ;; custom_ui) - show_env_masked + show_env_masked # Ensure that 'custom_ui' and 'custom_ui_ae' are only called with 1 argument. if [ "$#" -ne 1 ]; then echo "Error: 'custom_ui' expects no additional arguments." usage fi echo "Running Custom UI for local agent ..." - uvicorn main:app --reload + uvicorn ui.main:app --reload ;; custom_ui_ae) show_env_masked @@ -128,28 +135,28 @@ case "$COMMAND" in usage fi echo "Running Custom UI for Agent Engine Backend ..." - uvicorn main_ae:app --reload + uvicorn ui.main_ae:app --reload ;; env_files) - # Check for the existence of the files - if [ ! -f "$SAMPLE_ENV_FILE" ] && [ ! -f "$ENV_FILE" ]; then - echo "Error: Missing both $SAMPLE_ENV_FILE and $ENV_FILE files." - exit 1 - elif [ -f "$SAMPLE_ENV_FILE" ] && [ ! -f "$ENV_FILE" ]; then - echo "Copying $SAMPLE_ENV_FILE to $ENV_FILE..." - cp "$SAMPLE_ENV_FILE" "$ENV_FILE" - echo "Please update the environment variables in $ENV_FILE" - exit 0 + # Check for the existence of the .env file + if [ ! -f "$ENV_FILE" ]; then + if [ -f "$ENV_TEMPLATE_FILE" ]; then + echo "Copying $ENV_TEMPLATE_FILE to $ENV_FILE..." + cp "$ENV_TEMPLATE_FILE" "$ENV_FILE" + echo "Please update the environment variables in $ENV_FILE" + exit 0 + else + echo "Error: Missing both $ENV_TEMPLATE_FILE and $ENV_FILE files." + exit 1 + fi else echo "Environment file ok, please check the usage below" usage - fi + fi ;; *) # Default case for unknown commands when an argument *was* provided. echo "Error: Unknown command '$COMMAND'." - usage + usage ;; esac - - diff --git a/run-with-google-adk/scripts/test_deployed_agent.py b/run-with-google-adk/scripts/test_deployed_agent.py new file mode 100644 index 00000000..c5b9bf8e --- /dev/null +++ b/run-with-google-adk/scripts/test_deployed_agent.py @@ -0,0 +1,76 @@ +import os +import sys +import vertexai.agent_engines + +# Get message from command line argument or environment variable +message = None +if len(sys.argv) > 1: + message = " ".join(sys.argv[1:]) +else: + message = os.getenv("TEST_MESSAGE", "List MCP Tools") + +# Get deployed agent from Agent Engine +resource_name = os.getenv("AGENT_ENGINE_RESOURCE") +print(f">>> 🤖 Connecting to agent: {resource_name}") +deployed_agent = vertexai.agent_engines.get(resource_name) +print(">>> ✅ Connected successfully") + +# List Agent Engine sessions +response = deployed_agent.list_sessions(user_id="test_user") +if response: + sessions = response.get("sessions") +else: + sessions = None + +# Use existing session if available, to showcase session state +if sessions: + print(f">>> 🗃️ Found {len(sessions)} existing session(s)") + # Use the first session + session = sessions[0] + print(f">>> 🗂️ Using session: {session['id']}") + session = deployed_agent.get_session(user_id="test_user", session_id=session['id']) + print(f">>> 🧠 Session state: {session['state']}") +else: + # Create an Agent Engine session + print(">>> 📝 Creating new session...") + session = deployed_agent.create_session(user_id="test_user") + print(f">>> 📝 Created session: {session['id']}") + print(f">>> 🧠 Session state: {session['state']}") + +# Use the session to chat with the agent +print(f">>> 💬 Sending message: {message}") +print(">>> 🤖 Agent response:") +response_received = False +for event in deployed_agent.stream_query( + user_id="test_user", + session_id=session['id'], + message=message, + ): + print(f">>> Raw event: {event}") # Debug: show raw event structure + if 'content' in event and 'parts' in event['content'] and event['content']['parts']: + content = event['content']['parts'][0] + if content: + print(f"<<< 🤖 {content}") + response_received = True + elif 'content' in event and event['content']: + # Handle different response structure + content = event['content'] + if isinstance(content, dict) and 'parts' in content: + for part in content['parts']: + if part and isinstance(part, dict) and 'text' in part: + print(f"<<< 🤖 {part['text']}") + response_received = True + elif part: + print(f"<<< 🤖 {part}") + response_received = True + else: + print(f"<<< 🤖 {content}") + response_received = True + +if not response_received: + print(">>> ❌ No response received from agent") + +# Get session and see updated state +session = deployed_agent.get_session(user_id="test_user", session_id=session['id']) +print(f">>> 🧠 Final session state: {session['state']}") +print(">>> ✨ Test completed") \ No newline at end of file diff --git a/run-with-google-adk/scripts/test_deployed_agent_simple.py b/run-with-google-adk/scripts/test_deployed_agent_simple.py new file mode 100644 index 00000000..d605d5ba --- /dev/null +++ b/run-with-google-adk/scripts/test_deployed_agent_simple.py @@ -0,0 +1,36 @@ +import os +import sys +import vertexai.agent_engines + +# Get message from command line argument or environment variable +message = None +if len(sys.argv) > 1: + message = " ".join(sys.argv[1:]) +else: + message = os.getenv("TEST_MESSAGE", "List MCP Tools") + +# Get deployed agent from Agent Engine +resource_name = os.getenv("AGENT_ENGINE_RESOURCE") +print(f">>> 🤖 Connecting to agent: {resource_name}") +deployed_agent = vertexai.agent_engines.get(resource_name) +print(">>> ✅ Connected successfully") + +# Query the agent directly without sessions +print(f">>> 💬 Sending message: {message}") +print(">>> 🤖 Agent response:") + +try: + # Try direct query without session + response = deployed_agent.query(message=message) + print(response) +except Exception as e: + print(f"❌ Error querying agent: {e}") + print("\n>>> Trying invoke method instead...") + try: + # Alternative: try invoke method + response = deployed_agent.invoke({"message": message}) + print(response) + except Exception as e2: + print(f"❌ Error invoking agent: {e2}") + +print(">>> ✨ Test completed") \ No newline at end of file diff --git a/run-with-google-adk/main.py b/run-with-google-adk/ui/main.py similarity index 100% rename from run-with-google-adk/main.py rename to run-with-google-adk/ui/main.py diff --git a/run-with-google-adk/main_ae.py b/run-with-google-adk/ui/main_ae.py similarity index 100% rename from run-with-google-adk/main_ae.py rename to run-with-google-adk/ui/main_ae.py diff --git a/run-with-google-adk/static/app.js b/run-with-google-adk/ui/static/app.js similarity index 100% rename from run-with-google-adk/static/app.js rename to run-with-google-adk/ui/static/app.js diff --git a/run-with-google-adk/static/demo-idp.png b/run-with-google-adk/ui/static/demo-idp.png similarity index 100% rename from run-with-google-adk/static/demo-idp.png rename to run-with-google-adk/ui/static/demo-idp.png diff --git a/run-with-google-adk/static/demo-xdr.png b/run-with-google-adk/ui/static/demo-xdr.png similarity index 100% rename from run-with-google-adk/static/demo-xdr.png rename to run-with-google-adk/ui/static/demo-xdr.png diff --git a/run-with-google-adk/static/index.html b/run-with-google-adk/ui/static/index.html similarity index 100% rename from run-with-google-adk/static/index.html rename to run-with-google-adk/ui/static/index.html diff --git a/run-with-google-adk/static/index_script.js b/run-with-google-adk/ui/static/index_script.js similarity index 100% rename from run-with-google-adk/static/index_script.js rename to run-with-google-adk/ui/static/index_script.js diff --git a/run-with-google-adk/static/landing.html b/run-with-google-adk/ui/static/landing.html similarity index 100% rename from run-with-google-adk/static/landing.html rename to run-with-google-adk/ui/static/landing.html diff --git a/run-with-google-adk/static/landing_script.js b/run-with-google-adk/ui/static/landing_script.js similarity index 100% rename from run-with-google-adk/static/landing_script.js rename to run-with-google-adk/ui/static/landing_script.js diff --git a/run-with-google-adk/sample_servers_to_integrate/mcp_servers/demo_idp/idp_mcp_server.py b/server/demo_idp/idp_mcp_server.py similarity index 100% rename from run-with-google-adk/sample_servers_to_integrate/mcp_servers/demo_idp/idp_mcp_server.py rename to server/demo_idp/idp_mcp_server.py diff --git a/run-with-google-adk/sample_servers_to_integrate/mcp_servers/demo_xdr/xdr_mcp_server.py b/server/demo_xdr/xdr_mcp_server.py similarity index 100% rename from run-with-google-adk/sample_servers_to_integrate/mcp_servers/demo_xdr/xdr_mcp_server.py rename to server/demo_xdr/xdr_mcp_server.py