Soft policies: LLM-evaluated policy rules

[dlorenc]

Overview

Extend the policy system to support LLM-evaluated "soft policies" alongside existing OPA/Rego hard policies. Some rules are too semantic to express in Rego — soft policies let you express intent in natural language.

Configuration

.docstore/policy/no_breaking_changes.agent.yaml:

name: no_breaking_changes
model: claude-haiku-4-5
cache: on_diff_hash       # cache result, invalidate on new commits
prompt: |
  Review this diff. Does it introduce any breaking changes to the
  public API (removed exports, changed signatures, removed fields)?
  Respond with JSON: {"allow": false, "reason": "..."} or {"allow": true}

Files ending in .agent.yaml are loaded alongside .rego files by the policy engine.

How it works

1. Policy engine detects .agent.yaml files alongside .rego files
2. On evaluation, runs the LLM with the diff + relevant context from assembleMergeInput()
3. Parses {"allow": bool, "reason": string} response
4. Returns same PolicyResult shape as Rego policies
5. Result cached on diff hash to avoid redundant calls

Why

Rego can only express rules over structured data. Semantic rules — "no breaking API changes", "matches our style guide", "safe to deploy on Fridays" — require judgment that LLMs handle well but Rego cannot encode. Soft policies fill this gap while integrating seamlessly with the existing policy evaluation and merge flow.

Scope

• Policy engine: detect and load .agent.yaml policy files
• LLM evaluation path in policy engine (reuse agent runner from issue Built-in prompt-configured agent reviewer roles #251)
• Response parsing + PolicyResult mapping
• Caching layer keyed on (policy_name, diff_hash)
• Include soft policy results in BranchStatusResponse with a type: "soft" marker
• Document the format and limitations (non-determinism, latency)