[Schema Consistency] Schema Consistency Issues: Cross-Schema Analysis 2025-11-27

[github-actions[bot]]

🔍 Schema Consistency Check - 2025-11-27

Summary

This analysis focused on cross-schema consistency by comparing the three JSON schemas that define agentic workflow configurations. The analysis revealed critical inconsistencies in MCP tool definitions, network configurations, and permission scopes between the main workflow schema and the included file schema.

Key Metrics:

• Inconsistencies Found: 6 critical/moderate + 3 low-severity
• Categories Analyzed: Schema $defs, top-level properties, shared field types
• Strategy Used: Strategy-004 (Cross-Schema Consistency & Type Analysis)
• New Strategy: No (proven strategy, last used 2025-11-01)
• Effectiveness: Very High

Critical Discovery: The included_file_schema.json contains contaminated MCP tool definitions that incorrectly allow HTTP-only properties in stdio-type tools, potentially enabling invalid configurations in shared workflow files.

Full Report Details

Critical Issues

1. ⚠️ stdio_mcp_tool contaminated with HTTP fields

Schema: included_file_schema.json
Location: $defs/stdio_mcp_tool/properties
Severity: HIGH

The stdio MCP tool definition incorrectly includes HTTP-only properties:

Schema	stdio_mcp_tool Properties
Main (correct)	allowed, args, command, container, entrypointArgs, env, network, registry, type, version
Included (incorrect)	Same as above PLUS headers, url

Impact:

• Shared/imported files could specify invalid configurations
• IDE autocomplete suggests invalid properties
• External validation tools accept invalid configs
• No runtime error until compilation

Example invalid config that would pass schema validation:

# In .github/workflows/shared/mcp/mytool.md
mcp-servers:
  mytool:
    type: stdio
    command: mytool
    url: (redacted)    # ❌ INVALID for stdio
    headers:                   # ❌ INVALID for stdio
      X-Api-Key: secret

Root Cause: Likely copy-paste error during schema creation. The http_mcp_tool definition was probably used as a template and not properly cleaned for stdio.

Recommendation: Remove headers and url from included_file_schema.json $defs/stdio_mcp_tool

---

2. 🔒 Network field missing firewall support in included schema

Schema: included_file_schema.json
Location: properties/network
Severity: MEDIUM-HIGH

The included file schema doesn't support AWF (Agent Workflow Firewall) configuration that's available in the main schema.

Main schema supports:

network:
  allowed: [defaults, python]
  firewall:
    version: v1.0.0
    log-level: debug
    args: [--verbose]

Included schema only supports:

network:
  allowed: [defaults, python]
  # NO firewall property

Impact:

• Shared included files cannot configure firewall settings
• Limits security capabilities for modular workflows
• Inconsistent feature availability between main and included files
• Workflows must duplicate network config instead of importing

Use Case Example:
A team wants to create shared/secure-network.md with standard firewall settings for all workflows, but cannot due to schema limitation.

Recommendation: Add firewall property to included_file_schema.json network definition to match main schema

---

3. 🔐 Permissions field inconsistencies

Schema: Both main_workflow_schema.json and included_file_schema.json
Location: properties/permissions
Severity: MEDIUM

Multiple inconsistencies in permission field definitions:

3a. String shortcuts missing in included schema

Feature	Main Schema	Included Schema
String shortcuts	✅ read-all, write-all, read, write	❌ Only object format
Object format	✅ Supported	✅ Supported
all shorthand	✅ all: read	❌ Not supported

Impact: Included files must use verbose object syntax:

# Main workflow can use:
permissions: read-all

# But included files must use:
permissions:
  contents: read
  issues: read
  pull-requests: read
  # ... (must list all)

3b. Permission scope mismatches

Scope	Main Schema	Included Schema
models	read | none	read | write | none
metadata	❌ Not defined	✅ read | write | none
all	✅ read only	❌ Not defined

Critical Question: Why does included schema allow models: write when main schema explicitly restricts to read|none?

Impact:

• Included files could request write access to models (likely invalid)
• Main workflows cannot use metadata scope
• Validation inconsistency between workflow types

Recommendation:

1. Align models permission enum (should be read|none in both)
2. Either add metadata to main OR remove from included
3. Add all: read shorthand to included schema

---

Moderate Issues

4. 🔧 Network field pattern inconsistency

Location: properties/network/oneOf[0] for "defaults" value
Severity: LOW

• Main schema: Uses enum: ["defaults"]
• Included schema: Uses const: "defaults"

Both are valid JSON Schema but inconsistent style reduces maintainability.

Recommendation: Standardize on const for single values (more semantically correct)

---

5. 🔀 MCP standalone schema is type-ambiguous

Schema: mcp_config_schema.json
Severity: MEDIUM

The standalone MCP config schema contains properties from BOTH HTTP and stdio types:

{
  "properties": {
    "command": {},      // stdio-only
    "args": {},         // stdio-only
    "container": {},    // stdio-only
    "url": {},          // HTTP-only
    "headers": {},      // HTTP-only
    "type": {},         // discriminator
    "network": {}       // shared
  }
}

Issue: Schema allows configurations like:

type: stdio
command: mytool
url: (redacted)  # Contradictory!

Root Cause: Schema represents a union of types but lacks proper oneOf discrimination

Recommendation: Refactor to use oneOf based on type discriminator:

{
  "oneOf": [
    {
      "type": "object",
      "properties": {
        "type": {"const": "http"},
        "url": {},
        "headers": {}
      }
    },
    {
      "type": "object",
      "properties": {
        "type": {"const": "stdio"},
        "command": {},
        "args": {}
      }
    }
  ]
}

---

6. 📐 Engine field organizational differences

Severity: LOW

• Main schema: Uses $ref: "#/$defs/engine_config" (modular)
• Included schema: Inline oneOf definition (duplicated)

Analysis: Both functionally equivalent. Main schema approach is better for DRY principle and reusability.

Recommendation: Consider using $ref in included schema for consistency

---

Positive Findings ✅

1. http_mcp_tool perfectly consistent between main and included schemas

   • Same properties: allowed, headers, registry, type, url
   • Same required: ["url"]
   • No inconsistencies found

2. Tools field consistent across schemas

   • Both use object type
   • Similar nested structure

3. Real workflows don't exploit inconsistencies

   • Analyzed 78 workflows in .github/workflows/*.md
   • Analyzed 20+ shared files in .github/workflows/shared/
   • All use valid configurations
   • No workflows use problematic string permission shortcuts
   • No workflows mix HTTP/stdio properties

4. Schema internal references valid

   • All $ref references resolve correctly
   • No broken references found
   • All $defs are used (no dead definitions)

---

Architecture Insights

Schema Structure Comparison

Feature	Main Schema	Included Schema	MCP Schema
Top-level properties	35	12	12
$defs	4	3	0
Supports	Full workflows	Shared configs	Standalone MCP
Use case	Primary	Imports	External tools

Common $defs

Both main and included schemas share:

• http_mcp_tool ✅ (identical)
• stdio_mcp_tool ⚠️ (inconsistent - included has extra fields)

Unique to main schema:

• engine_config
• github_token

Unique to included schema:

• safe_job

Shared Top-Level Properties

Properties present in both main and included:

• description ✅
• engine ⚠️ (different structure but compatible)
• mcp-servers ⚠️
• network ⚠️ (included missing firewall)
• permissions ⚠️ (type inconsistencies)
• runtimes ✅
• safe-outputs ✅
• secret-masking ✅
• services ✅
• steps ✅
• tools ✅

---

Recommendations

Priority 1 - Critical (Must Fix)

1. Fix stdio_mcp_tool in included_file_schema.json

   • Remove headers property
   • Remove url property
   • File: pkg/parser/schemas/included_file_schema.json:$defs.stdio_mcp_tool

2. Align models permission scope

   • Change included schema from read|write|none to read|none
   • File: pkg/parser/schemas/included_file_schema.json:properties.permissions.oneOf[0].properties.models
   • Verify no workflows use models: write

3. Document or fix metadata scope mismatch

   • Either add to main schema OR remove from included schema
   • Investigate: Is metadata a valid GitHub permission scope?

Priority 2 - High (Should Fix)

4. Add firewall support to included schema

   • Copy firewall definition from main schema
   • Enable AWF configuration in shared files
   • File: pkg/parser/schemas/included_file_schema.json:properties.network.oneOf[1].properties

5. Add permission string shortcuts to included schema

   • Add read-all, write-all, read, write enum values
   • Add all: read property to permission object
   • File: pkg/parser/schemas/included_file_schema.json:properties.permissions

6. Refactor MCP standalone schema

   • Implement oneOf discrimination by type field
   • Separate HTTP and stdio property sets
   • File: pkg/parser/schemas/mcp_config_schema.json

Priority 3 - Medium (Improve)

7. Standardize const vs enum usage

   • Use const for single-value constraints
   • Replace enum: ["defaults"] with const: "defaults"
   • Applies to network field

8. Add schema consistency tests

   • Unit test to verify $defs match across schemas
   • Test that shared properties have same types
   • Catch regressions automatically

9. Document intentional differences

   • If included schema is intentionally more restrictive, document why
   • Add comments in schema files explaining design decisions

Priority 4 - Low (Consider)

10. Consider schema inheritance
    • Use JSON Schema composition (allOf, $ref) to reduce duplication
    • Single source of truth for shared definitions
    • Example: Both schemas could reference a common $defs file

---

Testing Impact

Workflows to Validate

Before deploying schema changes, test against:

1. All 78 production workflows in .github/workflows/*.md
2. All 20+ shared files in .github/workflows/shared/
3. MCP tool definitions in .github/workflows/shared/mcp/*.md

Validation Commands

# Validate all workflows against updated schema
for file in .github/workflows/*.md; do
  ./gh-aw validate "$file"
done

# Validate shared files
for file in .github/workflows/shared/**/*.md; do
  ./gh-aw validate "$file"
done

# Check for workflows using problematic patterns
grep -r "models: write" .github/workflows/
grep -r "permissions: read-all" .github/workflows/shared/

Expected Impact

✅ No breaking changes expected - Real workflows already use valid configurations
⚠️ Stricter validation - Some previously-accepted-but-invalid configs will now fail
✅ Better IDE support - Autocomplete will suggest correct properties only

---

Strategy Performance

Strategy Used: Strategy-004 (Cross-Schema Consistency & Type Analysis)
Findings: 6 critical/moderate + 3 low-severity = 9 total
Effectiveness: Very High
Should Reuse: Yes - excellent for detecting cross-schema drift

Methodology

1. Extracted and compared $defs across all three schemas
2. Compared top-level properties for type consistency
3. Deep-dived shared fields (network, permissions, tools, engine)
4. Validated findings against real workflow usage
5. Analyzed architectural patterns and design decisions

Unique Value

This strategy is the only one focused on inter-schema consistency. Other strategies analyze:

• Individual field validation (Strategy-001, 002, 003)
• Documentation gaps (Strategy-007, 008, 016)
• Runtime behavior (Strategy-017, 018)
• Evolution over time (Strategy-015)

Complementary strategies:

• Run Strategy-004 every 4-6 weeks to catch schema drift
• Pair with Strategy-008 (schema completeness) for full schema quality audit
• Follow up with Strategy-009 (end-to-end tracing) to verify fixes

---

Next Steps

1. ✅ Create GitHub discussion with these findings
2. 📝 Create tracking issues for Priority 1-2 recommendations
3. 🔧 Fix stdio_mcp_tool contamination (highest priority)
4. 📚 Document intentional schema differences (if any)
5. 🧪 Add schema consistency tests to CI/CD
6. 🔄 Re-run Strategy-004 after fixes to verify

---

Appendix: Schema File Locations

• Main workflow schema: pkg/parser/schemas/main_workflow_schema.json
• Included file schema: pkg/parser/schemas/included_file_schema.json
• MCP config schema: pkg/parser/schemas/mcp_config_schema.json
• Parser implementation: pkg/parser/frontmatter.go
• Compiler implementation: pkg/workflow/compiler.go
• Documentation: docs/src/content/docs/reference/frontmatter.md

---

Analysis Completed: 2025-11-27
Strategy ID: strategy-004
Run Count: 3
Overall Success Rate: Very High

AI generated by Schema Consistency Checker

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Schema Consistency Checker.
🗺️ A newer treasure map awaits ye at Discussion #5254.
Fair winds, matey! 🏴‍☠️