LLM-as-a-Judge for evaluating whether a "grant-xxx-permission" tool call is "reasonable safe"

[Mossaka]

For example, the domain passed to the grant-network-permission should not look suspicious... This LLM-as-a-Judge could enhance the security posture of Wassette to add additional layer of defense. It also helps in case wassette is running in auto-approve mode.

[Mossaka]

/scout please evaluate different approaches to achieve this.

Agentic Scout triggered by this comment

[github-actions]

🔍 Scout Research Report

Triggered by @Mossaka

Executive Summary

Based on research into LLM-as-a-Judge approaches for evaluating "grant-xxx-permission" tool calls in Wassette, I've identified several viable approaches ranging from simple rule-based systems to sophisticated LLM-powered validation. The key insight is that a multi-layered defense strategy combining pattern matching, heuristics, and LLM evaluation offers the best security posture, especially for auto-approve modes.

Click to expand detailed findings

Research Findings

1. LLM-as-a-Judge Architecture Patterns

Based on Microsoft's Azure AI Content Safety documentation and security best practices, there are three primary approaches:

Approach A: Synchronous LLM Evaluation (Azure Content Safety Model)

• Description: Real-time evaluation of permission requests using an LLM judge before granting permissions
• How it works:
  • Intercept permission grant calls (e.g., grant-network-permission)
  • Send the request parameters (domain, access type, component context) to an LLM
  • LLM evaluates based on configured severity thresholds and safety rules
  • Block or allow based on LLM response
• Pros: Most comprehensive, can detect sophisticated threats, contextual understanding
• Cons: Latency overhead (100-500ms), requires API access, cost per evaluation
• Best for: High-security scenarios, auto-approve mode with human oversight

Microsoft Reference: Azure AI Content Safety uses this pattern for real-time filtering with configurable severity levels (low/medium/high) across categories like hate, violence, sexual content, and jailbreak detection.

Approach B: Hybrid Pattern Matching + LLM Escalation

• Description: Fast pattern-based filtering with LLM escalation for ambiguous cases
• How it works:
  1. First Layer: Fast rule-based checks (regex, domain reputation lists, known-bad patterns)
  2. Second Layer: Heuristic scoring (domain age, TLD reputation, entropy analysis)
  3. Third Layer: LLM evaluation only for borderline cases
• Pros: Low latency for most cases (1-10ms), cost-effective, still catches sophisticated attacks
• Cons: More complex implementation, requires tuning thresholds
• Best for: Production environments balancing security and performance

Industry Pattern: This mirrors the Azure AI Foundry's "prompt shields" approach that combines pattern detection with LLM-based jailbreak detection.

Approach C: Post-Grant Monitoring with LLM Auditing

• Description: Allow permissions but monitor usage with LLM-powered auditing
• How it works:
  • Grant permissions immediately (fast path)
  • Log all permission usage with context
  • Background LLM process reviews logs and flags suspicious patterns
  • Revoke permissions retroactively if threats detected
• Pros: Zero latency impact, learns from usage patterns, can detect novel attacks
• Cons: Reactive rather than preventive, potential for damage before detection
• Best for: Development environments, trusted contexts, research scenarios

2. Domain-Specific Security Validation Techniques

For grant-network-permission specifically, research indicates several validation layers:

Pattern-Based Validation (Fast Layer):

• Known malicious domain lists (blocklists)
• Suspicious TLDs (.tk, .ml, .ga, uncommon country codes)
• Homograph attacks (e.g., gооgle.com with Cyrillic 'о')
• IP address patterns (private ranges, bogon IPs)
• Port validation (non-standard ports for known services)

Heuristic Scoring (Medium Layer):

• Domain reputation scoring (using public APIs like VirusTotal, Google Safe Browsing)
• Domain age verification (newly registered domains are suspicious)
• WHOIS privacy analysis (privacy-protected domains more suspicious)
• SSL/TLS certificate validation
• DNS resolution consistency checks

LLM Evaluation Criteria (Slow Layer):

• Contextual appropriateness (does a "time" component need network access?)
• Request justification analysis (parse component metadata for legitimacy)
• Pattern recognition for social engineering
• Semantic analysis of domain names (e.g., "api-login-microsoft-verify.com")

3. Implementation Approaches for Wassette

Based on Wassette's architecture (Rust-based MCP server with WebAssembly components), here are concrete implementation strategies:

Recommended: Hybrid Approach Implementation

// Conceptual architecture
pub struct PermissionJudge {
    fast_validator: PatternValidator,      // Regex, blocklists
    heuristic_scorer: HeuristicScorer,     // Domain reputation APIs
    llm_evaluator: Option<LLMEvaluator>,   // Azure OpenAI or Claude
    config: JudgeConfig,
}

pub struct JudgeConfig {
    auto_approve_threshold: f32,  // 0.0-1.0, higher = safer
    llm_escalation_threshold: f32, // When to invoke LLM
    blocklist_path: PathBuf,
    allowed_domains_path: PathBuf,
}

// Evaluation flow
impl PermissionJudge {
    pub async fn evaluate(&self, request: GrantPermissionRequest) 
        -> Result<JudgeDecision> {
        
        // Layer 1: Fast validation (1-5ms)
        if self.fast_validator.is_blocked(&request) {
            return Ok(JudgeDecision::Deny("Blocked by pattern match"));
        }
        
        // Layer 2: Heuristic scoring (10-100ms)
        let score = self.heuristic_scorer.score(&request).await?;
        
        if score >= self.config.auto_approve_threshold {
            return Ok(JudgeDecision::Approve);
        }
        
        if score < self.config.llm_escalation_threshold {
            return Ok(JudgeDecision::Deny("Low heuristic score"));
        }
        
        // Layer 3: LLM evaluation (100-500ms)
        if let Some(llm) = &self.llm_evaluator {
            let llm_decision = llm.evaluate(&request, score).await?;
            return Ok(llm_decision);
        }
        
        // Fallback: deny if LLM unavailable
        Ok(JudgeDecision::RequestHumanReview)
    }
}

Integration Points in Wassette:

1. Hook into grant-network-permission, grant-storage-permission, grant-environment-variable-permission tools
2. Add --judge-mode CLI flag: off, fast, hybrid, llm-only
3. Add --auto-approve-threshold configuration
4. Store decisions in policy database for audit trail

LLM Prompt Engineering for Security Evaluation

Based on Microsoft's Semantic Kernel prompt injection protection patterns:

You are a security evaluator for a WebAssembly component runtime.
Your task is to evaluate whether granting a permission is "reasonably safe."

COMPONENT CONTEXT:
- Component ID: {component_id}
- Component Name: {component_name}
- Component Description: {component_description}
- Source URI: {source_uri}

PERMISSION REQUEST:
- Type: {permission_type}
- Details: {permission_details}

HEURISTIC SCORE: {heuristic_score}/1.0

EVALUATION CRITERIA:
1. Contextual Appropriateness: Does this permission match the component's purpose?
2. Domain Reputation: Is the target domain/resource trustworthy?
3. Suspicious Patterns: Any signs of social engineering or obfuscation?
4. Risk Assessment: What's the worst-case scenario if this is malicious?

Respond with JSON:
{
  "decision": "approve" | "deny" | "review",
  "confidence": 0.0-1.0,
  "reasoning": "brief explanation",
  "risk_level": "low" | "medium" | "high"
}

IMPORTANT: Err on the side of caution. Any uncertainty should result in "review".

4. Azure AI Content Safety Integration Option

For production deployments, Azure AI Content Safety provides enterprise-grade LLM evaluation:

Features Relevant to Wassette:

• Prompt Shields: Detects jailbreak attempts and indirect prompt injection
• Custom Categories: Define domain-specific safety rules (e.g., "suspicious-domains")
• Configurable Severity: Low/Medium/High thresholds per category
• Real-time API: Sub-second evaluation latency
• Annotation Mode: Get safety signals without blocking (useful for logging)

Implementation Pattern:

use azure_ai_content_safety::ContentSafetyClient;

pub struct AzureContentSafetyJudge {
    client: ContentSafetyClient,
    config: AzureConfig,
}

impl LLMEvaluator for AzureContentSafetyJudge {
    async fn evaluate(&self, request: &GrantPermissionRequest, score: f32) 
        -> Result<JudgeDecision> {
        
        let analysis_request = AnalyzeTextRequest {
            text: format!(
                "Evaluate network permission grant to domain: {}\nComponent: {}\nPurpose: {}",
                request.details.host,
                request.component_id,
                request.component_metadata.description
            ),
            categories: vec!["Jailbreak", "Custom-SuspiciousDomain"],
            output_type: "EightSeverityLevels",
        };
        
        let result = self.client.analyze_text(analysis_request).await?;
        
        if result.jailbreak_detected || result.custom_category_severity >= 4 {
            Ok(JudgeDecision::Deny("Azure Content Safety flagged request"))
        } else {
            Ok(JudgeDecision::Approve)
        }
    }
}

5. Security Considerations & OWASP Alignment

Based on OWASP LLM security guidance referenced in Microsoft docs:

Threat: Prompt Injection via Component Metadata

• Risk: Malicious component includes instructions in its description to convince the LLM judge to approve
• Mitigation:
  • Use structured prompts with clear role boundaries (system message vs. user input)
  • HTML-encode component metadata before sending to LLM (Semantic Kernel pattern)
  • Implement "Allow Unsafe Content" flags with explicit trust decisions
  • Cross-validate LLM decisions with heuristic scores

Threat: Model Manipulation

• Risk: Attacker discovers prompt patterns that consistently fool the judge
• Mitigation:
  • Combine LLM with deterministic checks (defense in depth)
  • Regular red-team testing of judge prompts
  • Monitor judge decisions for suspicious patterns (e.g., always approving)
  • Version control judge prompts and A/B test improvements

Threat: Judge Availability

• Risk: LLM API outage leaves system unable to process permissions
• Mitigation:
  • Graceful degradation to heuristic-only mode
  • Cached decisions for repeated requests
  • Fallback to "request human review" rather than auto-approve/deny

Recommendations

Immediate Next Steps (MVP)

1. Implement Fast Layer (Week 1):

   • Add blocklist support to Wassette (read from blocked_domains.txt)
   • Implement regex-based pattern matching for obvious threats
   • Add to existing grant-network-permission tool

2. Add Heuristic Layer (Week 2):

   • Integrate domain reputation API (start with Google Safe Browsing API - free tier)
   • Implement domain age checking via WHOIS
   • Calculate composite risk score

3. Prototype LLM Judge (Week 3):

   • Start with OpenAI API or Azure OpenAI
   • Implement prompt template for security evaluation
   • Add --judge-mode=llm flag to enable opt-in
   • Store all decisions in audit log

Production Considerations

1. Configuration:

   • Make judge behavior configurable via policy.yaml
   • Support per-component override (some components need network, others don't)
   • Environment-based profiles (development vs. production thresholds)

2. Observability:

   • Prometheus metrics for judge decisions (approve/deny/review counts)
   • Structured logging of all evaluations with context
   • Dashboard for monitoring judge effectiveness

3. Human-in-the-Loop:

   • When judge returns "review", prompt user for approval
   • Learn from human decisions to improve heuristics
   • Allow users to provide feedback on judge decisions

4. Performance:

   • Cache LLM decisions for identical requests (TTL: 24h)
   • Parallel evaluation of multiple permission requests
   • Circuit breaker for LLM API failures

5. Cost Management:

   • Track LLM API costs per evaluation
   • Set monthly budget limits
   • Optimize escalation threshold to minimize LLM calls

Advanced Features (Future)

1. Anomaly Detection: Learn normal permission patterns per component type
2. Reputation System: Track components that repeatedly request suspicious permissions
3. Community Blocklists: Share blocklists across Wassette instances
4. Intent Analysis: Parse component WIT interfaces to understand permission needs
5. Sandboxed Testing: Test components in isolated environment before granting real permissions

Key Sources

1. Azure AI Content Safety Overview - Comprehensive guide to content filtering and prompt shields
2. Security Planning for LLM Applications - Threat modeling for LLM systems including prompt injection mitigation
3. Prompt Shields for User Prompts - Detecting jailbreak attempts and role-play attacks
4. Semantic Kernel Prompt Injection Protection - HTML encoding and trust boundaries for LLM inputs
5. Responsible AI Practices for Azure OpenAI - Four-stage approach: Identify, Measure, Mitigate, Operate

Suggested Implementation Priority

Phase 1: Foundation (2-3 weeks)

1. Add fast pattern-based validation (blocklists, regex)
2. Implement domain reputation checking (heuristic layer)
3. Create audit logging infrastructure

Phase 2: LLM Integration (2 weeks)

1. Implement LLM judge with OpenAI/Azure OpenAI
2. Add escalation logic (heuristic → LLM → human)
3. Build judge configuration system

Phase 3: Production Hardening (2 weeks)

1. Add caching and performance optimizations
2. Implement circuit breakers and graceful degradation
3. Build monitoring dashboards

Phase 4: Advanced Features (ongoing)

1. Anomaly detection
2. Community blocklists
3. Intent-based analysis

---

Total Estimated Effort: 6-8 weeks for production-ready implementation with 1 engineer

Recommendation: Start with Approach B (Hybrid Pattern) as it provides the best balance of security, performance, and cost-effectiveness. The fast layer handles 80%+ of cases instantly, while the LLM layer provides sophisticated analysis for the ambiguous 20%.

AI generated by Scout