Skip to content

compliance: scope edge-agentic checker (rename to edge_agentic.py, required ruleset/model args) #397

Description

@Palanivelg

Follow-up to PR #346 (thread #346 (comment)), flagged by @viraatc as fine to address in a follow-up.

compliance/checker.py is edge-agentic-specific (BFCL metric keys, qwen3.6-27b / mlperf-edge-current defaults, hardcoded attestation note) under a generic top-level compliance/ name.

Proposed:

  • Rename to compliance/edge_agentic.py (or move next to the edge ruleset).
  • Make ruleset_name/model_name required args instead of defaults.
  • Name the model's golden_accuracy keys to match the scorer's breakdown keys (overall_accuracy, normalized_single_turn_score) so the _ACCURACY_METRIC_KEYS mapping disappears.
  • Generalizing into a real framework can wait until a second benchmark needs one; coordinate with feat(compliance): MLPerf TEST04 caching audit — extensible AuditTest framework #332.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions