Skip to content

[Feature Request]: Track Creator & Timestamp Metadata for Servers, Tools, and Resources #137

New issue
New issue
Closed
Feature
#776
Closed
[Feature Request]: Track Creator & Timestamp Metadata for Servers, Tools, and Resources#137
Feature
#776
Assignees
crivetimihai
Labels
enhancementNew feature or requestfrontendFrontend development (HTML, CSS, JavaScript)observabilityObservability, logging, monitoringpythonPython / backend development (FastAPI)triageIssues / Features awaiting triage
Milestone
Release 0.6.0
@crivetimihai

Description

@crivetimihai
crivetimihai
opened on Jun 20, 2025

🧭 Epic
Title: Comprehensive Metadata Attribution for All MCP Gateway Entities
Goal: Track and display comprehensive metadata for all entities (Gateway, Virtual Server, Tool, Prompt, Resource) including creation details, source information, and audit trails for enhanced traceability and governance.
Why now: As MCP Gateway deployments scale across teams and environments, comprehensive metadata tracking becomes essential for security auditing, compliance, debugging, and operational workflows.

🧭 Type of Feature

  • Enhancement to existing functionality
  • New feature or capability
  • Security Related (for auditability)

🙋‍♂️ User Story 1

As a: gateway admin
I want: to see comprehensive metadata for any entity in detail view
So that: I can trace origin, audit access, and understand entity lifecycle

✅ Acceptance Criteria

Scenario: View comprehensive metadata in detail view
  Given I open the Admin UI
  When I click "View" on any tool, server, gateway, prompt, or resource
  Then I see a metadata section containing:
    - Created by (username)
    - Created at (timestamp)
    - Created from IP address
    - Creation source (UI/API/Import/Federation)
    - Last modified by (username)
    - Last modified at (timestamp)
    - Last modified from IP
    - Modification source
    - Import batch ID (if applicable)
    - Federation source gateway (if applicable)
    - Version/revision number

🙋‍♂️ User Story 2

As a: API consumer
I want: comprehensive metadata in JSON responses for all entities
So that: I can build audit trails, track provenance, and implement custom governance

✅ Acceptance Criteria

Scenario: Fetch comprehensive metadata via API
  When I GET /tools, /servers, /gateways, /prompts, or /resources
  Then each object includes metadata fields:
    - created_by, created_at, created_from_ip, created_via
    - modified_by, modified_at, modified_from_ip, modified_via
    - import_batch_id, federation_source, version

🙋‍♂️ User Story 3

As a: security auditor
I want: to track all creation and modification events with full context
So that: I can investigate security incidents and ensure compliance

✅ Acceptance Criteria

Scenario: Audit trail for security investigation
  Given an entity was created or modified
  When I access its metadata
  Then I can see complete provenance including:
    - Who performed the action (authenticated user)
    - When it happened (precise timestamp)
    - From where (IP address, user agent)
    - How it was done (UI interaction, API call, bulk import, federation sync)
    - What changed (if modification)

📐 Design Sketch

🗃️ Schema Changes

Add comprehensive metadata columns to all entity tables:

-- Common metadata columns for all entities
-- Creation metadata
ALTER TABLE tools ADD COLUMN created_by TEXT;
ALTER TABLE tools ADD COLUMN created_at TIMESTAMP DEFAULT now();
ALTER TABLE tools ADD COLUMN created_from_ip INET;
ALTER TABLE tools ADD COLUMN created_via TEXT; -- 'ui', 'api', 'import', 'federation'
ALTER TABLE tools ADD COLUMN created_user_agent TEXT;

-- Modification metadata  
ALTER TABLE tools ADD COLUMN modified_by TEXT;
ALTER TABLE tools ADD COLUMN modified_at TIMESTAMP DEFAULT now();
ALTER TABLE tools ADD COLUMN modified_from_ip INET;
ALTER TABLE tools ADD COLUMN modified_via TEXT;
ALTER TABLE tools ADD COLUMN modified_user_agent TEXT;

-- Source tracking metadata
ALTER TABLE tools ADD COLUMN import_batch_id TEXT; -- UUID for bulk imports
ALTER TABLE tools ADD COLUMN federation_source TEXT; -- Source gateway for federated entities
ALTER TABLE tools ADD COLUMN version INTEGER DEFAULT 1; -- For change tracking

-- Apply same schema to all entity tables
-- (repeat for resources, prompts, servers, gateways tables)

🔄 Comprehensive Entity Coverage

Entities requiring metadata tracking:

  • Tools - Individual MCP tools from servers
  • Resources - MCP resources (files, data sources)
  • Prompts - Prompt templates
  • Servers - Virtual MCP servers (compositions)
  • Gateways - Peer gateway instances (federation)

Additional metadata considerations:

  • Request ID - Link to specific request that created/modified entity
  • Session ID - Track admin UI sessions for user activity correlation
  • Bulk Operation ID - Group related bulk import/export operations
  • Parent Entity ID - Track hierarchical relationships (e.g., tool -> server)
  • Configuration Hash - Detect configuration drift
  • Validation Status - Track entity validation state

Migration Strategy: Backfill historical entries with reasonable defaults (created_by: 'system', created_via: 'migration', created_from_ip: null)

🧩 Pydantic Schema Changes

Add comprehensive metadata to all Read schemas (ToolRead, ServerRead, ResourceRead, PromptRead, GatewayRead):

# Creation metadata
created_by: Optional[str] = Field(None, description="Username who created this entity")
created_at: Optional[datetime] = Field(None, description="When entity was created")
created_from_ip: Optional[str] = Field(None, description="IP address of creator")
created_via: Optional[str] = Field(None, description="Creation method: ui|api|import|federation")
created_user_agent: Optional[str] = Field(None, description="User agent of creation request")

# Modification metadata
modified_by: Optional[str] = Field(None, description="Username who last modified this entity")
modified_at: Optional[datetime] = Field(None, description="When entity was last modified")
modified_from_ip: Optional[str] = Field(None, description="IP address of last modifier")
modified_via: Optional[str] = Field(None, description="Modification method: ui|api|import|federation")
modified_user_agent: Optional[str] = Field(None, description="User agent of modification request")

# Source tracking
import_batch_id: Optional[str] = Field(None, description="UUID of bulk import batch")
federation_source: Optional[str] = Field(None, description="Source gateway for federated entities")
version: Optional[int] = Field(1, description="Entity version for change tracking")

# Optional extended metadata
request_id: Optional[str] = Field(None, description="Request ID that created/modified entity")
session_id: Optional[str] = Field(None, description="Admin UI session ID")
parent_entity_id: Optional[str] = Field(None, description="Parent entity for hierarchical tracking")

📊 Metadata Display Schema

Create dedicated metadata response schema for detail views:

class EntityMetadata(BaseModel):
    """Comprehensive metadata for any MCP Gateway entity"""
    
    # Core identification
    entity_id: str = Field(description="Entity UUID")
    entity_type: str = Field(description="Entity type: tool|resource|prompt|server|gateway")
    
    # Creation tracking
    creation: CreationMetadata = Field(description="Creation metadata")
    
    # Modification tracking  
    last_modification: Optional[ModificationMetadata] = Field(None, description="Last modification metadata")
    
    # Source and lineage
    source: SourceMetadata = Field(description="Source and lineage information")
    
    # Operational metadata
    operational: OperationalMetadata = Field(description="Operational status and metrics")

class CreationMetadata(BaseModel):
    created_by: str
    created_at: datetime
    created_from_ip: Optional[str]
    created_via: str  # ui|api|import|federation|system
    created_user_agent: Optional[str]

class ModificationMetadata(BaseModel):
    modified_by: str
    modified_at: datetime
    modified_from_ip: Optional[str] 
    modified_via: str
    modified_user_agent: Optional[str]

class SourceMetadata(BaseModel):
    import_batch_id: Optional[str]
    federation_source: Optional[str]
    parent_entity_id: Optional[str]
    configuration_hash: Optional[str]

class OperationalMetadata(BaseModel):
    version: int
    validation_status: str  # valid|invalid|pending
    last_accessed_at: Optional[datetime]
    access_count: Optional[int]

🔄 Comprehensive Metadata Workflow

📝 Entity Creation Flow

  1. Request Context Capture

    • Extract authenticated user: user: str = Depends(require_auth)
    • Capture client IP: request.client.host or via X-Forwarded-For headers
    • Extract User-Agent: request.headers.get("user-agent")
    • Determine source: ui (Admin UI), api (direct API), import (bulk), federation (peer sync)
    • Generate request ID for traceability

  2. Metadata Population

    • created_by = authenticated username
    • created_at = current UTC timestamp
    • created_from_ip = client IP address
    • created_via = request source type
    • created_user_agent = browser/client identification
    • version = 1 (initial version)
    • import_batch_id = UUID if part of bulk operation
    • federation_source = source gateway ID if federated

  3. Entity Persistence

    • Save entity with complete metadata
    • Log creation event for audit trail
    • Return entity with metadata in response

✏️ Entity Modification Flow

  1. Update Context Capture

    • Same context extraction as creation
    • Increment version number
    • Preserve original creation metadata

  2. Modification Tracking

    • modified_by = authenticated username
    • modified_at = current UTC timestamp
    • modified_from_ip = client IP address
    • modified_via = request source type
    • modified_user_agent = browser/client identification

  3. Change Recording

    • Log what changed (for audit trail)
    • Update modification metadata
    • Preserve creation metadata unchanged

🖥️ UI Display Strategy

List Views: Show minimal metadata (created_by, created_at) in tooltip or expandable row
Detail Views: Dedicated "Metadata" section with comprehensive information organized by:

  • Creation Info - Who, when, from where, how
  • Modification History - Last change details
  • Source & Lineage - Import batch, federation source, parent relationships
  • Operational Status - Version, validation, usage stats

Metadata Section Layout:

┌─ Metadata ──────────────────────────────────────┐
│ Creation:     admin @ 2024-01-15 10:30:00 UTC    │
│ Source:       Admin UI (192.168.1.100)           │
│ Last Change:  alice @ 2024-01-16 14:22:00 UTC    │
│ Version:      3                                   │
│ Import Batch: bulk-2024-01-15-001 (if applicable)│
│ Federation:   gateway-prod-east (if applicable)  │
└─────────────────────────────────────────────────┘

🧭 Implementation Tasks

🗄️ Database & Schema

Task Description Priority Notes
[ ] Create Alembic migration for metadata columns High Add all metadata columns to tools, resources, prompts, servers, gateways tables
[ ] Add database indexes for metadata queries Medium Index on created_by, created_at, modified_at for performance
[ ] Create metadata-specific database functions Low Stored procedures for common metadata queries
[ ] Design metadata retention policy Medium How long to keep detailed metadata vs. summary data

🛠️ Backend Services

Task Description Priority Notes
[ ] Create metadata capture utility functions High Extract IP, User-Agent, determine source type
[ ] Update all create endpoints with metadata capture High tools, resources, prompts, servers, gateways
[ ] Update all update endpoints with modification tracking High Preserve creation metadata, track modifications
[ ] Add bulk import metadata tracking Medium Track import batch ID and bulk operation context
[ ] Add federation metadata tracking Medium Track source gateway for federated entities
[ ] Create metadata service layer Medium Centralized metadata management and queries
[ ] Add audit logging for metadata events Medium Log creation/modification events for compliance

📊 API & Schemas

Task Description Priority Notes
[ ] Update Pydantic Read schemas with metadata fields High All entity read schemas
[ ] Create dedicated metadata response schemas High EntityMetadata, CreationMetadata, etc.
[ ] Add metadata endpoints Medium GET /entities/{id}/metadata for detailed metadata
[ ] Update OpenAPI documentation Medium Document all new metadata fields
[ ] Add metadata filtering to list endpoints Low Filter by created_by, created_via, date ranges

🖥️ Admin UI

Task Description Priority Notes
[ ] Add metadata section to all detail views High Show comprehensive metadata in "View" modals
[ ] Update list views with basic metadata Medium Tooltips or expandable rows for created_by/created_at
[ ] Add metadata-based filtering and search Medium Filter entities by creator, source, date ranges
[ ] Create metadata history view Low Show modification history for entities
[ ] Add metadata export functionality Low Export metadata reports for auditing

🔧 Infrastructure & Utilities

Task Description Priority Notes
[ ] Create request context middleware High Capture IP, User-Agent, determine source automatically
[ ] Add metadata validation Medium Ensure metadata consistency and completeness
[ ] Create metadata migration script High Backfill existing entities with default metadata
[ ] Add metadata configuration options Medium Control what metadata to capture via environment variables
[ ] Create metadata cleanup utilities Low Archive old metadata, cleanup incomplete records

🧪 Testing

Task Description Priority Notes
[ ] Unit tests for metadata capture High Test metadata population in all scenarios
[ ] Integration tests for metadata API High Test metadata in create/read/update workflows
[ ] UI tests for metadata display Medium Test metadata sections in detail views
[ ] Performance tests with metadata Medium Ensure metadata doesn't impact performance
[ ] Security tests for metadata access High Ensure metadata follows same auth rules as entities

📚 Documentation

Task Description Priority Notes
[ ] Update API documentation High Document all metadata fields and endpoints
[ ] Create metadata user guide Medium How to use metadata features in admin UI
[ ] Document metadata privacy considerations High IP address storage, data retention policies
[ ] Create metadata troubleshooting guide Low Common issues and solutions

🔗 MCP Standards Check

  • ✔️ Metadata-only; no effect on MCP wire protocol
  • ✔️ All metadata fields are optional; backward-compatible
  • ✔️ Does not alter existing auth or business logic
  • ✔️ Preserves MCP server functionality and tool execution
  • ✔️ Metadata is gateway-specific, not propagated to MCP clients
  • ✔️ Does not affect MCP JSON-RPC message format

🔒 Security & Privacy Considerations

🛡️ Data Protection

  • IP Address Storage: Consider GDPR/privacy implications of storing IP addresses
  • User Agent Storage: May contain personally identifiable information
  • Retention Policy: Define how long to retain detailed metadata
  • Access Control: Metadata should follow same permissions as parent entity
  • Data Anonymization: Option to anonymize metadata after retention period

🔐 Security Features

  • Audit Trail: Complete history of who did what, when, and from where
  • Access Tracking: Monitor unusual access patterns via metadata
  • Change Attribution: Link all changes to authenticated users
  • Source Validation: Verify entity source claims against metadata
  • Federation Security: Track trusted vs. untrusted federation sources

🚀 Future Enhancements

📊 Advanced Metadata Features

  • Change History: Full audit log with before/after state tracking
  • Metadata Versioning: Schema evolution for metadata fields
  • Custom Metadata: Allow users to add custom metadata fields
  • Automated Tagging: Auto-tag entities based on creation patterns
  • Metadata Analytics: Usage patterns, popular creators, trend analysis

🔄 Integration Features

  • SIEM Integration: Export metadata to security information systems
  • Compliance Reporting: Generate compliance reports from metadata
  • Workflow Integration: Trigger actions based on metadata changes
  • External Identity: Link to external identity providers for richer user context

🌐 Federation Features

  • Cross-Gateway Metadata: Sync metadata across federated gateways
  • Metadata Conflicts: Handle metadata conflicts in federation scenarios
  • Trust Scoring: Rate federation sources based on metadata quality
  • Provenance Tracking: Full lineage tracking across gateway federation

🔧 Implementation Guide

Based on codebase analysis, here are the specific files and implementation steps:

📁 Files to Modify

1. Database Models (mcpgateway/db.py)

Current Models: Tool, Resource, Prompt, Server, Gateway
Required Changes: Add metadata columns to all entity models

Example for Tool model (lines 300-435):

# Add these columns to Tool class:
created_by: Mapped[Optional[str]] = mapped_column(String)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utc_now)
created_from_ip: Mapped[Optional[str]] = mapped_column(String)
created_via: Mapped[Optional[str]] = mapped_column(String)  # 'ui', 'api', 'import', 'federation'
created_user_agent: Mapped[Optional[str]] = mapped_column(Text)

modified_by: Mapped[Optional[str]] = mapped_column(String)
modified_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utc_now, onupdate=utc_now)
modified_from_ip: Mapped[Optional[str]] = mapped_column(String)
modified_via: Mapped[Optional[str]] = mapped_column(String)
modified_user_agent: Mapped[Optional[str]] = mapped_column(Text)

import_batch_id: Mapped[Optional[str]] = mapped_column(String)  # UUID for bulk imports
federation_source: Mapped[Optional[str]] = mapped_column(String)  # Source gateway
version: Mapped[int] = mapped_column(Integer, default=1)

Apply same pattern to: Resource (573-764), Prompt (778-946), Server (948-1090), Gateway (1092-1135)

2. Alembic Migration

Create new migration in mcpgateway/alembic/versions/:

alembic revision -m "add_comprehensive_metadata_to_all_entities"

Migration content (follow pattern from cc7b95fec5d9_add_tags_support_to_all_entities.py):

def upgrade() -> None:
    """Add comprehensive metadata columns to all entity tables."""
    tables = ['tools', 'resources', 'prompts', 'servers', 'gateways']
    
    for table in tables:
        # Creation metadata (nullable=True for backwards compatibility)
        op.add_column(table, sa.Column('created_by', sa.String(), nullable=True))
        op.add_column(table, sa.Column('created_from_ip', sa.String(), nullable=True))
        op.add_column(table, sa.Column('created_via', sa.String(), nullable=True))
        op.add_column(table, sa.Column('created_user_agent', sa.Text(), nullable=True))
        
        # Modification metadata (nullable=True for backwards compatibility)
        op.add_column(table, sa.Column('modified_by', sa.String(), nullable=True))
        op.add_column(table, sa.Column('modified_from_ip', sa.String(), nullable=True))
        op.add_column(table, sa.Column('modified_via', sa.String(), nullable=True))
        op.add_column(table, sa.Column('modified_user_agent', sa.Text(), nullable=True))
        
        # Source tracking (nullable=True for backwards compatibility)
        op.add_column(table, sa.Column('import_batch_id', sa.String(), nullable=True))
        op.add_column(table, sa.Column('federation_source', sa.String(), nullable=True))
        op.add_column(table, sa.Column('version', sa.Integer(), nullable=False, server_default='1'))
        
        # Indexes for performance
        try:
            op.create_index(f'idx_{table}_created_by', table, ['created_by'])
            op.create_index(f'idx_{table}_created_at', table, ['created_at'])
            op.add_column(table, sa.Column('modified_at', sa.DateTime(timezone=True), nullable=True))
            op.create_index(f'idx_{table}_modified_at', table, ['modified_at'])
        except Exception:
            pass  # SQLite compatibility

3. Pydantic Schemas (mcpgateway/schemas.py)

Update Read schemas (lines 802-838 for ToolRead, similar for others):

# Add to ToolRead, ResourceRead, PromptRead, ServerRead, GatewayRead:
created_by: Optional[str] = Field(None, description="Username who created this entity")
created_at: Optional[datetime] = Field(None, description="When entity was created")
created_from_ip: Optional[str] = Field(None, description="IP address of creator")
created_via: Optional[str] = Field(None, description="Creation method: ui|api|import|federation")
created_user_agent: Optional[str] = Field(None, description="User agent of creation request")

modified_by: Optional[str] = Field(None, description="Username who last modified this entity")
modified_at: Optional[datetime] = Field(None, description="When entity was last modified")
modified_from_ip: Optional[str] = Field(None, description="IP address of last modifier")
modified_via: Optional[str] = Field(None, description="Modification method")
modified_user_agent: Optional[str] = Field(None, description="User agent of modification request")

import_batch_id: Optional[str] = Field(None, description="UUID of bulk import batch")
federation_source: Optional[str] = Field(None, description="Source gateway for federated entities")
version: Optional[int] = Field(1, description="Entity version for change tracking")

4. Request Context Middleware

Create new file: mcpgateway/middleware/metadata_capture.py

from fastapi import Request
from typing import Optional

class MetadataCapture:
    @staticmethod
    def extract_request_metadata(request: Request) -> dict:
        """Extract metadata from request for entity creation/modification."""
        return {
            "from_ip": request.client.host if request.client else None,
            "user_agent": request.headers.get("user-agent"),
            "via": "ui" if "/admin/" in str(request.url) else "api",
        }

5. API Endpoints (mcpgateway/main.py & mcpgateway/admin.py)

Update create/update functions to capture metadata:

Example for create_tool (line 1248):

async def create_tool(
    tool: ToolCreate, 
    request: Request,  # Add this
    db: Session = Depends(get_db), 
    user: str = Depends(require_auth)
) -> ToolRead:
    # Capture metadata
    metadata = MetadataCapture.extract_request_metadata(request)
    
    # Pass to service with metadata
    return await tool_service.create_tool(
        db=db, 
        tool=tool, 
        created_by=user,
        **metadata
    )

Update admin endpoints (mcpgateway/admin.py line 1813):

async def admin_add_tool(
    request: Request,
    db: Session = Depends(get_db),
    user: str = Depends(require_auth),
) -> JSONResponse:
    # Add metadata capture similar to above

6. Service Layer Updates

Update services (mcpgateway/services/tool_service.py, etc.) to accept and store metadata:

async def create_tool(
    self,
    db: Session,
    tool: ToolCreate,
    created_by: str,
    from_ip: Optional[str] = None,
    user_agent: Optional[str] = None,
    via: str = "api",
    import_batch_id: Optional[str] = None,
    federation_source: Optional[str] = None,
) -> ToolRead:
    # Create tool with metadata
    db_tool = Tool(
        **tool.model_dump(),
        created_by=created_by,
        created_from_ip=from_ip,
        created_user_agent=user_agent,
        created_via=via,
        import_batch_id=import_batch_id,
        federation_source=federation_source,
    )

7. Admin UI Updates (mcpgateway/templates/admin.html)

Update the View modal (around line 1389) to show metadata:

Add metadata section to tool details modal:

<!-- Add to existing tool modal around line 1389 -->
<div class="mt-6 border-t pt-4">
    <h4 class="text-lg font-semibold text-gray-800 dark:text-gray-200 mb-4">Metadata</h4>
    <div class="grid grid-cols-2 gap-4 text-sm">
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Created By:</span>
            <span class="ml-2" id="tool-created-by"></span>
        </div>
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Created At:</span>
            <span class="ml-2" id="tool-created-at"></span>
        </div>
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Created From:</span>
            <span class="ml-2" id="tool-created-from"></span>
        </div>
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Created Via:</span>
            <span class="ml-2" id="tool-created-via"></span>
        </div>
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Last Modified By:</span>
            <span class="ml-2" id="tool-modified-by"></span>
        </div>
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Last Modified At:</span>
            <span class="ml-2" id="tool-modified-at"></span>
        </div>
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Version:</span>
            <span class="ml-2" id="tool-version"></span>
        </div>
        <div>
            <span class="font-medium text-gray-600 dark:text-gray-400">Import Batch:</span>
            <span class="ml-2" id="tool-import-batch"></span>
        </div>
    </div>
</div>

8. JavaScript Updates (mcpgateway/static/admin.js)

Update viewTool function (line 4778) to populate metadata:

async function viewTool(toolId) {
    // ... existing code ...
    
    // Populate metadata fields
    document.getElementById('tool-created-by').textContent = tool.createdBy || 'N/A';
    document.getElementById('tool-created-at').textContent = tool.createdAt || 'N/A';
    document.getElementById('tool-created-from').textContent = tool.createdFromIp || 'N/A';
    document.getElementById('tool-created-via').textContent = tool.createdVia || 'N/A';
    document.getElementById('tool-modified-by').textContent = tool.modifiedBy || 'N/A';
    document.getElementById('tool-modified-at').textContent = tool.modifiedAt || 'N/A';
    document.getElementById('tool-version').textContent = tool.version || '1';
    document.getElementById('tool-import-batch').textContent = tool.importBatchId || 'N/A';
}

9. Authentication Context (mcpgateway/utils/verify_credentials.py)

Current auth function (line 216): require_auth already handles both scenarios:

  • When AUTH_REQUIRED=true: Returns authenticated username
  • When AUTH_REQUIRED=false: Returns "anonymous" (line 282, 308)

Usage: Existing pattern works gracefully - user: str = Depends(require_auth) provides either:

  • Authenticated username (e.g., "admin", "alice")
  • "anonymous" when auth is disabled

No code changes needed - the metadata implementation will automatically work with both auth modes.

10. Bulk Import/Export Services

Update import service (mcpgateway/services/import_service.py) to track import batch IDs
Update federation to track federation sources

🧪 Testing Updates

Add tests for:

  • Metadata capture in create/update operations
  • Metadata display in admin UI
  • Metadata filtering and search
  • Migration script execution

🔄 Implementation Order

  1. Database migration - Add columns
  2. Models - Update ORM models
  3. Schemas - Update Pydantic schemas
  4. Middleware - Add metadata capture
  5. Services - Update business logic
  6. API endpoints - Update create/update endpoints
  7. Admin UI - Update templates and JavaScript
  8. Testing - Add comprehensive tests

🛡️ Auth-Agnostic Implementation

🔧 Graceful Auth Handling

The implementation automatically works with both authentication modes:

✅ When AUTH_REQUIRED=true (Default)

# require_auth returns actual username
user = "admin"  # or "alice", "bob", etc.
created_by = user  # "admin"

✅ When AUTH_REQUIRED=false (Development/Open APIs)

# require_auth returns "anonymous"
user = "anonymous"
created_by = user  # "anonymous"

📊 Metadata Display Examples

Authenticated Environment:

Created By: admin
Created At: 2024-01-15 10:30:00 UTC
Created From: 192.168.1.100
Created Via: UI

Anonymous Environment:

Created By: anonymous
Created At: 2024-01-15 10:30:00 UTC  
Created From: 192.168.1.100
Created Via: UI

🧪 Testing Both Scenarios

Test with auth enabled:

def test_metadata_with_auth():
    app.dependency_overrides[require_auth] = lambda: "test_user"
    # Test metadata capture
    assert tool.created_by == "test_user"

Test with auth disabled:

def test_metadata_without_auth():
    app.dependency_overrides[require_auth] = lambda: "anonymous"
    # Test metadata capture
    assert tool.created_by == "anonymous"

⚙️ Configuration Impact

No special configuration needed - the implementation adapts automatically based on:

  • AUTH_REQUIRED environment variable
  • Existing require_auth dependency behavior
  • Current auth patterns in the codebase

This ensures the metadata feature works seamlessly across all deployment scenarios without breaking existing functionality.

🔄 Backwards Compatibility Strategy

🛡️ Zero-Downtime Migration

The implementation ensures existing gateways can upgrade without data loss or functionality breaks:

📊 Database Migration Strategy

All metadata columns are nullable=True:

-- Existing entities will have NULL values initially
ALTER TABLE tools ADD COLUMN created_by VARCHAR NULL;
ALTER TABLE tools ADD COLUMN created_at TIMESTAMP NULL;
-- ... etc for all metadata columns

Version column with default:

-- Existing entities get version=1 automatically
ALTER TABLE tools ADD COLUMN version INTEGER NOT NULL DEFAULT 1;

🔧 Code Compatibility

Pydantic schemas handle NULL gracefully:

# All metadata fields are Optional with None defaults
created_by: Optional[str] = Field(None, description="Username who created this entity")
created_at: Optional[datetime] = Field(None, description="When entity was created")
# Existing entities: created_by = None, created_at = None
# New entities: created_by = "admin", created_at = "2024-01-15T10:30:00Z"

🖥️ UI Display for Legacy Entities

Admin UI handles missing metadata gracefully:

<span class="ml-2" id="tool-created-by">{{ tool.created_by || 'N/A' }}</span>
<span class="ml-2" id="tool-created-at">{{ tool.created_at || 'Unknown' }}</span>

JavaScript display logic:

// Handles both existing (null) and new entities
document.getElementById('tool-created-by').textContent = tool.createdBy || 'Legacy Entity';
document.getElementById('tool-created-at').textContent = tool.createdAt || 'Pre-metadata';

📈 Migration Scenarios

✅ Scenario 1: Fresh Installation

  • All new entities get complete metadata
  • Full audit trail from day one

✅ Scenario 2: Existing Gateway Upgrade

  • Existing entities: Show "N/A" or "Legacy Entity" for missing metadata
  • New entities: Get full metadata tracking
  • Mixed display: Legacy and new entities coexist seamlessly

✅ Scenario 3: Bulk Import/Historical Data

  • Import services can optionally populate metadata for historical accuracy
  • Migration scripts can backfill known information (e.g., system user, approximate dates)

🧪 Backwards Compatibility Testing

Test existing entity access:

def test_legacy_entity_access():
    # Create entity without metadata (simulates pre-migration state)
    tool = Tool(name="legacy_tool", url="http://example.com")
    db.add(tool)
    db.commit()
    
    # Verify API still works
    response = client.get(f"/tools/{tool.id}")
    assert response.status_code == 200
    
    # Verify metadata fields show as None/null
    data = response.json()
    assert data["createdBy"] is None
    assert data["createdAt"] is None

Test mixed entity display:

def test_mixed_entity_display():
    # Legacy entity (no metadata)
    legacy_tool = Tool(name="legacy", url="http://example.com")
    
    # New entity (with metadata)  
    new_tool = create_tool_with_metadata(
        name="new_tool",
        created_by="admin",
        created_at=datetime.utcnow()
    )
    
    # Both should display without errors
    tools = client.get("/tools").json()
    assert len(tools) == 2

🔧 Optional Metadata Backfill

For enhanced audit trails, optionally backfill known data:

-- Backfill system-created entities
UPDATE tools SET 
    created_by = 'system',
    created_via = 'migration',
    version = 1
WHERE created_by IS NULL;

-- Backfill from existing created_at timestamps if available
UPDATE tools SET created_at = created_at WHERE created_at IS NOT NULL;

🎯 Key Benefits

  1. Zero service interruption during upgrades
  2. Gradual metadata adoption - new entities get tracking immediately
  3. Clear legacy identification - easy to spot pre-metadata entities
  4. Optional enhancement - metadata backfill can be done post-migration
  5. Future-proof - all new operations include full metadata

This strategy ensures that existing production gateways can upgrade safely while immediately benefiting from metadata tracking for all new operations.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestfrontendFrontend development (HTML, CSS, JavaScript)observabilityObservability, logging, monitoringpythonPython / backend development (FastAPI)triageIssues / Features awaiting triage

Type

Feature

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions