[IMPROVE] Add skill quality validator for tags, workflow completeness, prerequisite consistency, and safety gates

[optimization2026]

Summary

This repository has grown into a large cybersecurity Agent Skills catalog. At this size, quality control becomes as important as skill creation.

Recent audit findings show two important classes of mechanical issues:

1. weak routing metadata, such as filename-split tags
2. workflow completeness gaps, such as prerequisites listed without concrete workflow usage

I propose adding a lightweight skill quality validator that can run locally and in CI.

The goal is not to block contributors with heavy bureaucracy.

The goal is to catch mechanical quality drift early.

Proposed Solution

Add a validator such as:

tools/skill_quality_validator.py

It could check every skills/*/SKILL.md file for:

frontmatter quality
tag quality
required section presence
workflow completeness
prerequisite consistency
reference/script/asset existence
framework mapping consistency
high-risk safety gates

Example command:

python tools/skill_quality_validator.py skills/

Example output:

PASS  analyzing-powershell-script-block-logging
WARN  weak tag: analyzing
WARN  weak tag: block
FAIL  prerequisite listed but not used in workflow: boto3
FAIL  high-risk skill missing authorization / scope gate

Validation Categories

1. Frontmatter validation

Check that required fields exist and are useful:

name
description
domain
subdomain
tags
version
framework mappings when present

The validator should flag missing or empty fields.

2. Tag quality

Tags should help an agent select the right skill.

Weak examples:

analyzing
with
logs
block
security
assessment

Better examples:

powershell
script-block-logging
event-id-4104
windows-forensics
cloudtrail
iam-anomaly
ssl-tls
volatility
sigma
kerberoasting

A simple first version could use:

weak tag denylist
minimum tag length
domain-specific tag recommendations
duplicate tag detection

3. Workflow completeness

A skill should not only describe a topic. It should guide execution.

Check for sections such as:

When to Use
Prerequisites
Workflow
Verification
Output Format

The validator can start with warnings, not failures.

4. Prerequisite consistency

If a skill lists a tool or library, the workflow should use or explain it.

Example:

Prerequisites:
  boto3

Workflow:
  no boto3 usage
  no AWS API call pattern
  no script

This should produce a warning or failure depending on policy.

5. Reference / script / asset consistency

If SKILL.md references:

references/foo.md
scripts/bar.py
assets/template.md

those files should exist.

6. Framework mapping validation

Framework mappings should be auditable.

The validator could check:

known ID format
deprecated/revoked IDs when lists are available
duplicate mappings
mapping fields present but empty

7. High-risk safety gates

For dual-use or high-risk domains, the validator should check for scope and authorization language.

High-risk examples:

red teaming
penetration testing
malware analysis
credential access
phishing simulation
C2
exploit validation
adversarial AI

Expected safety fields or language:

authorized environment
owned systems
lab environment
defensive investigation
scope confirmation
evidence handling
do not target third-party systems without permission

Optional AISOP companion

A machine-readable workflow companion could define the validator contract.

Example strict AISOP V1.0.0 sketch:

[
  {
    "role": "system",
    "content": {
      "protocol": "AISOP V1.0.0",
      "axiom_0": "Human_Sovereignty_and_Wellbeing",
      "id": "cyber_skill_quality_validator",
      "name": "Cybersecurity Skill Quality Validator",
      "version": "1.0.0",
      "summary": "Validate cybersecurity Agent Skills for routing metadata, workflow completeness, prerequisite consistency, mapping quality, and safety gates.",
      "flow_format": "mermaid",
      "loading_mode": "node",
      "tools": ["filesystem"],
      "params": {
        "skills_root": "string"
      },
      "system_prompt": "{system_prompt}"
    }
  },
  {
    "role": "user",
    "content": {
      "instruction": "RUN aisop.main",
      "user_input": "{user_input}",
      "aisop": {
        "main": "graph TD\n    scan[Scan skills] --> frontmatter[Validate frontmatter]\n    frontmatter --> tags[Validate tags]\n    tags --> sections[Validate required sections]\n    sections --> prereq[Validate prerequisites]\n    prereq --> resources[Validate referenced files]\n    resources --> mappings[Validate framework mappings]\n    mappings --> safety[Validate high-risk safety gates]\n    safety --> report[Generate report]\n    report --> end_node((End))"
      },
      "functions": {
        "scan": {
          "step1": "Scan skills_root for folders containing SKILL.md.",
          "output_mapping": "skill_files",
          "constraints": [
            "Only validate actual skill folders.",
            "Each skill folder should contain one SKILL.md entrypoint."
          ]
        },
        "frontmatter": {
          "step1": "Parse YAML frontmatter and check required metadata fields.",
          "output_mapping": "frontmatter_findings",
          "constraints": [
            "Name and description must exist.",
            "Domain and tags should support routing.",
            "Framework mapping fields should be parseable when present."
          ]
        },
        "tags": {
          "step1": "Check whether tags are meaningful cybersecurity discovery terms.",
          "step2": "Flag generic verbs, stopwords, filename fragments, and ambiguous routing terms.",
          "output_mapping": "tag_findings",
          "constraints": [
            "Tags should help an agent select the skill.",
            "Tags should describe cybersecurity concepts, techniques, tools, platforms, artifacts, or evidence types."
          ]
        },
        "sections": {
          "step1": "Check whether the skill body contains When to Use, Prerequisites, Workflow, Verification, and Output guidance.",
          "output_mapping": "section_findings",
          "constraints": [
            "Workflow should contain concrete ordered steps.",
            "Verification should define how the agent knows the task is complete."
          ]
        },
        "prereq": {
          "step1": "Compare listed tools and libraries against workflow steps, scripts, and references.",
          "output_mapping": "prerequisite_findings",
          "constraints": [
            "A prerequisite should be used or explained.",
            "Do not list libraries that never appear in the workflow."
          ]
        },
        "resources": {
          "step1": "Check that referenced files in references, scripts, and assets exist.",
          "output_mapping": "resource_findings",
          "constraints": [
            "A skill should not point to missing supporting files.",
            "Scripts referenced by SKILL.md should exist."
          ]
        },
        "mappings": {
          "step1": "Validate framework mappings for known syntax and obvious consistency.",
          "output_mapping": "mapping_findings",
          "constraints": [
            "Framework mappings should be auditable.",
            "Invalid or empty mapping fields should be reported."
          ]
        },
        "safety": {
          "step1": "Identify high-risk or dual-use skills.",
          "step2": "Check whether they include authorization, scope, lab-only, defensive-use, or evidence-handling gates.",
          "output_mapping": "safety_findings",
          "constraints": [
            "High-risk skills should include explicit safety boundaries.",
            "The validator should strengthen governance, not add offensive detail."
          ]
        },
        "report": {
          "step1": "Generate PASS, WARN, and FAIL findings grouped by skill and category.",
          "output_mapping": "validator_report"
        },
        "end_node": {
          "step1": "Return validator_report."
        }
      }
    }
  }
]

Why this helps

This would help maintain quality as the catalog grows:

1. better agent routing
2. fewer stub workflows
3. more consistent contribution review
4. safer handling of high-risk skills
5. easier CI-based quality checks
6. stronger trust in framework mappings

Non-goals

This should not:

1. replace human review
2. block all contributions immediately
3. force every skill into a rigid template
4. add offensive capability
5. change the Agent Skills format

A good first version could run in warning mode only.

Later, the project could decide which checks should become CI failures.