Skip to content

Commit eecca51

Browse files
authored
refactor(do): remove fable from the model selection policy (#896)
Owner directive: "our document for what agent to use should remove fable entirely" — removed, not demoted. - skills/meta/do/SKILL.md: drop the Fable-5 variant row, the manual-pick set and ordering, the cross-provider escalation target, and the felt-quality clause comparing fable to sol; reword the sentences that depended on them. - scripts/build-dispatch.py: drop fable from VALID_MODELS, ANTHROPIC_MODELS, and the manual-override gate (now sonnet alone). - hooks/routing-decision-recorder.py: drop fable from the valid-model set. - scripts/validate_component_models.py: ALLOWED_MODELS is {sonnet, opus}. - docs/PHILOSOPHY.md, agent-frontmatter-template.md, component-audit-checklists.md: same removal in live instructions. - Tests updated to match, including SUPPLIED_CLAUDE_POINTS. Opus-4.8 and Sonnet-5 prior-measurement rows are kept as manual-only: they are the Anthropic lane's only DeepSWE data (Opus 5 is unbenchmarked) and cover cost, latency, context-window, and fan-out constraints. Dated historical records keep their fable references (docs/what-didnt-work.md, scripts/routing-ab-results/) — editing them would falsify experiment records; neither is a live routing instruction.
1 parent 5971206 commit eecca51

10 files changed

Lines changed: 34 additions & 42 deletions

File tree

agents/system-upgrade-engineer/references/component-audit-checklists.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ For each agent in scope, check these fields in order:
3333

3434
```bash
3535
# 1. Check model field is current
36-
grep "^model:" agents/*.md | grep -v "sonnet\|opus\|fable"
36+
grep "^model:" agents/*.md | grep -v "sonnet\|opus"
3737

3838
# 2. Check allowed-tools for deprecated tools
3939
grep -A 10 "allowed-tools:" agents/*.md | grep "deprecated_tool_name"
@@ -207,7 +207,7 @@ Do instead: Scope audit to the component types identified in the Change Manifest
207207

208208
```bash
209209
# Agents: stale model name
210-
grep "^model:" agents/*.md | grep -v "sonnet\|opus\|fable"
210+
grep "^model:" agents/*.md | grep -v "sonnet\|opus"
211211

212212
# Hooks: syntax errors
213213
for f in hooks/*.py; do python3 -m py_compile "$f" 2>&1 | grep -v "^$" && echo "ERROR: $f"; done

docs/PHILOSOPHY.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -344,7 +344,7 @@ The biggest risk: rationalization. "Already done" (assumption). "Code looks corr
344344

345345
Owner model-selection policy (ADR `model-selection-policy`; operational table in `skills/meta/do/SKILL.md`, Model Selection):
346346

347-
**Harness-native routing:** each harness defaults to its own provider's model lane. Cross-provider dispatch is manual-only, never automatic. Start low, escalate on miss — high tiers cost 3-6x per Pass@1 point where measured. Plan budget ($200/month per provider) makes cost a first-class constraint. Three decision axes: the current session model (the harness runs Opus 5, the owner-directed Anthropic-lane default at every task class), DeepSWE Pass@1 (agentic completion) where a model has been measured, and owner-observed felt quality (fable > sol noticeable, opus > gpt-5.5 marginal); ties resolve in favor of felt quality.
347+
**Harness-native routing:** each harness defaults to its own provider's model lane. Cross-provider dispatch is manual-only, never automatic. Start low, escalate on miss — high tiers cost 3-6x per Pass@1 point where measured. Plan budget ($200/month per provider) makes cost a first-class constraint. Three decision axes: the current session model (the harness runs Opus 5, the owner-directed Anthropic-lane default at every task class), DeepSWE Pass@1 (agentic completion) where a model has been measured, and owner-observed felt quality (opus > gpt-5.5 marginal); ties resolve in favor of felt quality.
348348

349349
| Task class | Anthropic lane (Claude Code) | OpenAI lane (Codex CLI) |
350350
|---|---|---|
@@ -354,7 +354,7 @@ Owner model-selection policy (ADR `model-selection-policy`; operational table in
354354
| High-risk | `opus` / `high` (unmeasured) | `gpt-5.6-sol` / `xhigh` (71 / $4.70) |
355355
| Max-power | `opus` / `xhigh` (unmeasured) | `gpt-5.6-sol` / `max` (73 / $8.39) |
356356

357-
The `/do` table is canonical and records the full DeepSWE Pass@1 / cost / tokens / steps data, including the prior Fable-5 / Opus-4.8 / Sonnet-5 measurements kept for manual picks. Opus 5 has no DeepSWE run yet, so its pts/USD is uncomputable and the selection rests on the session-model and owner-directive axes; effort still follows start-low-escalate-on-miss. Max-power requires `manual_model_override=true` in both lanes. Fable/sonnet are manual-only. Legacy `gpt-5.5` and non-default GPT-5.6 points are manual-only. Haiku is retired (routing was Haiku pre-#777; self-route since — `scripts/routing-ab-results/self-route-v1/VERDICT.md`). Defaults, not limits: escalate when cheaper output misses the bar; for anything that ships, intelligence > taste > cost, with cost a tie-breaker only. Fan-out uses the lane's low-risk point; one synthesis agent may run one tier higher.
357+
The `/do` table is canonical and records the full DeepSWE Pass@1 / cost / tokens / steps data, including the prior Opus-4.8 / Sonnet-5 measurements kept for manual picks. Opus 5 has no DeepSWE run yet, so its pts/USD is uncomputable and the selection rests on the session-model and owner-directive axes; effort still follows start-low-escalate-on-miss. Max-power requires `manual_model_override=true` in both lanes. Sonnet is manual-only. Legacy `gpt-5.5` and non-default GPT-5.6 points are manual-only. Haiku is retired (routing was Haiku pre-#777; self-route since — `scripts/routing-ab-results/self-route-v1/VERDICT.md`). Defaults, not limits: escalate when cheaper output misses the bar; for anything that ships, intelligence > taste > cost, with cost a tie-breaker only. Fan-out uses the lane's low-risk point; one synthesis agent may run one tier higher.
358358

359359
**Coordinator model.** The main-thread coordinator routes and evaluates, never executes — its cost is input-dominated and Pass@1 measures execution it never does. Anthropic harness → opus (Opus 5, the session model); OpenAI harness → gpt-5.6-terra/high; downgrade the anthropic coordinator to sonnet only as a deliberate plan-limit measure, set via harness config (`/model`), not per-turn. Full rule: `skills/meta/do/SKILL.md`, Model Selection.
360360

hooks/routing-decision-recorder.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,6 @@
117117
{
118118
"sonnet",
119119
"opus",
120-
"fable",
121120
"codex",
122121
"gpt-5.5",
123122
"gpt-5.6-sol",

hooks/tests/test_routing_decision_recorder.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -445,26 +445,26 @@ def test_gpt_56_model_effort_is_persisted_in_route_evidence(self, db_env, monkey
445445
assert context["recent"][0]["model"] == "gpt-5.6-sol@xhigh"
446446

447447
def test_claude_model_effort_is_persisted_in_route_evidence(self, db_env, monkeypatch):
448-
"""Claude model@effort (fable@high) survives the marker round-trip (advisory effort)."""
448+
"""Claude model@effort (sonnet@high) survives the marker round-trip (advisory effort)."""
449449
a = _load(A_PATH, "rdr_claude_effort")
450450
monkeypatch.setattr(a, "append_pending_outcome", lambda *_a, **_k: None)
451451
monkeypatch.setattr(a, "claim_dispatch", lambda *_a, **_k: True)
452452
event = _agent_event(skill="go-patterns", session="claude-effort")
453453
event["tool_input"]["prompt"] = (
454454
"[do-route] agent=python-general-engineer skill=go-patterns complexity=complex "
455-
"model=fable effort=high health=-\nReview the implementation."
455+
"model=sonnet effort=high health=-\nReview the implementation."
456456
)
457457
with patch("sys.exit"), patch("sys.stdin.read", return_value=json.dumps(event)):
458458
a.main()
459459

460460
decisions = [event for event in _read_events(db_env) if event["type"] == "decision"]
461-
assert decisions[0]["model"] == "fable@high"
461+
assert decisions[0]["model"] == "sonnet@high"
462462

463463
sys.path.insert(0, str(LIB_DIR))
464464
import learning_db_v2 as ldb
465465

466466
context = ldb.get_evidence_route_context("python-general-engineer:go-patterns")
467-
assert context["recent"][0]["model"] == "fable@high"
467+
assert context["recent"][0]["model"] == "sonnet@high"
468468

469469
def test_no_telemetry_row_when_marker_absent(self, db_env, tmp_path, monkeypatch):
470470
# No [do-route] marker => no decision row AND no envelope row.

scripts/build-dispatch.py

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -124,9 +124,9 @@
124124
GPT_56_MODELS = ("gpt-5.6-sol", "gpt-5.6-terra", "gpt-5.6-luna")
125125
GPT_56_EFFORTS = ("low", "medium", "high", "xhigh", "max")
126126
LEGACY_GPT_55 = "gpt-5.5"
127-
VALID_MODELS = ("sonnet", "opus", "fable", "codex", LEGACY_GPT_55, *GPT_56_MODELS)
127+
VALID_MODELS = ("sonnet", "opus", "codex", LEGACY_GPT_55, *GPT_56_MODELS)
128128
VALID_PROVIDERS = ("anthropic", "openai", "other")
129-
ANTHROPIC_MODELS = ("fable", "opus", "sonnet")
129+
ANTHROPIC_MODELS = ("opus", "sonnet")
130130

131131
# DeepSWE Pass@1 / cost benchmark defaults per provider lane.
132132
# `deterministic` deliberately has no model: use scripts.
@@ -220,8 +220,8 @@ def resolve_model_selection(decision: dict, provider: str = "anthropic") -> tupl
220220
221221
Harness-aware: ``provider`` selects the automatic policy table.
222222
Anthropic lane defaults select Opus 5 at every task class (owner
223-
directive + current session model); fable and sonnet are manual-only,
224-
kept for felt-quality, context-window, and latency constraints.
223+
directive + current session model); sonnet is manual-only, kept for
224+
cost, context-window, and latency constraints.
225225
OpenAI lane defaults select GPT-5.6 Sol/Terra. Effort is recorded in
226226
the marker for all models; for Claude lanes it is advisory (the harness
227227
Agent tool does not accept per-call effort).
@@ -286,10 +286,10 @@ def resolve_model_selection(decision: dict, provider: str = "anthropic") -> tupl
286286
raise InputError("legacy gpt-5.5 requires manual_model_override=true")
287287
return model, effort
288288

289-
# Claude models (fable, opus, sonnet) and codex wrapper.
289+
# Claude models (opus, sonnet) and codex wrapper.
290290
# Effort is optional and advisory for Claude lanes — recorded in the
291291
# marker (model@effort) for telemetry but not passed to the Agent tool.
292-
if model in ("fable", "sonnet"):
292+
if model == "sonnet":
293293
if not manual:
294294
raise InputError(
295295
f"'{model}' requires manual_model_override=true "

scripts/tests/test_build_dispatch.py

Lines changed: 9 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -549,11 +549,6 @@ def test_non_default_openai_model_choices_require_manual_override(overrides):
549549
# ---------------------------------------------------------------------------
550550

551551
SUPPLIED_CLAUDE_POINTS = {
552-
("fable", "max"): (70, 21.63, 119_000, 88),
553-
("fable", "xhigh"): (70, 13.41, 80_000, 68),
554-
("fable", "high"): (69, 9.18, 57_000, 59),
555-
("fable", "medium"): (65, 6.09, 40_000, 48),
556-
("fable", "low"): (60, 3.76, 25_000, 38),
557552
("opus-4.8", "max"): (59, 13.22, 135_000, 120),
558553
("opus-4.8", "xhigh"): (54, 8.01, 86_000, 95),
559554
("opus-4.8", "high"): (52, 4.28, 50_000, 73),
@@ -599,12 +594,12 @@ def test_anthropic_policy_points_are_unmeasured_and_effort_rises_with_risk():
599594
previous = order.index(effort)
600595

601596

602-
def test_fable_max_still_costs_more_than_fable_xhigh():
603-
"""Recorded prior measurement: fable[max] matches fable[xhigh] Pass@1 at higher cost."""
604-
fmax = SUPPLIED_CLAUDE_POINTS[("fable", "max")]
605-
fxhigh = SUPPLIED_CLAUDE_POINTS[("fable", "xhigh")]
606-
assert fmax[0] == fxhigh[0], "precondition: same Pass@1"
607-
assert fmax[1] > fxhigh[1], "precondition: max costs more"
597+
def test_sonnet_max_still_costs_more_per_point_than_sonnet_xhigh():
598+
"""Recorded prior measurement: sonnet[max] buys 4 Pass@1 points for 2.2x the cost."""
599+
smax = SUPPLIED_CLAUDE_POINTS[("sonnet", "max")]
600+
sxhigh = SUPPLIED_CLAUDE_POINTS[("sonnet", "xhigh")]
601+
assert smax[0] > sxhigh[0], "precondition: max scores higher"
602+
assert smax[1] / smax[0] > sxhigh[1] / sxhigh[0], "precondition: max costs more per point"
608603

609604

610605
def test_opus_max_requires_manual_override():
@@ -614,15 +609,14 @@ def test_opus_max_requires_manual_override():
614609

615610

616611
def test_prior_measurements_retained_for_manual_picks():
617-
"""Historical Fable-5 / Opus-4.8 / Sonnet-5 points stay recorded, not deleted."""
618-
assert SUPPLIED_CLAUDE_POINTS[("fable", "low")] == (60, 3.76, 25_000, 38)
612+
"""Historical Opus-4.8 / Sonnet-5 points stay recorded, not deleted."""
619613
assert SUPPLIED_CLAUDE_POINTS[("opus-4.8", "max")] == (59, 13.22, 135_000, 120)
620614
assert SUPPLIED_CLAUDE_POINTS[("sonnet", "high")] == (48, 7.43, 87_000, 147)
621615

622616

623-
@pytest.mark.parametrize("model", ("fable", "sonnet"))
617+
@pytest.mark.parametrize("model", ("sonnet",))
624618
def test_off_policy_claude_models_require_manual_override(model):
625-
"""Opus 5 is the default; fable and sonnet are the manual-only picks."""
619+
"""Opus 5 is the default; sonnet is the manual-only pick."""
626620
with pytest.raises(bd.InputError, match="manual_model_override"):
627621
bd.build_marker(_decision(model=model))
628622
# With manual_override they work fine

scripts/tests/test_validate_component_models.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,14 +52,14 @@ def test_do_skill_is_exempt(tmp_path: Path) -> None:
5252
assert violations == []
5353

5454

55-
def test_sonnet_opus_and_fable_are_allowed(tmp_path: Path) -> None:
55+
def test_sonnet_and_opus_are_allowed(tmp_path: Path) -> None:
5656
repo = tmp_path / "repo"
5757
agents_dir = repo / "agents"
5858
agents_dir.mkdir(parents=True)
5959
(agents_dir / "demo-agent.md").write_text("---\nmodel: opus\n---\n", encoding="utf-8")
6060
skill_dir = repo / "skills" / "demo-skill"
6161
skill_dir.mkdir(parents=True)
62-
(skill_dir / "SKILL.md").write_text("---\nmodel: fable\n---\n", encoding="utf-8")
62+
(skill_dir / "SKILL.md").write_text("---\nmodel: sonnet\n---\n", encoding="utf-8")
6363
workflow_dir = repo / "skills" / "workflow" / "references"
6464
workflow_dir.mkdir(parents=True)
6565
(workflow_dir / "demo-pipeline.md").write_text("---\nmodel: sonnet\n---\n", encoding="utf-8")

scripts/validate_component_models.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111

1212
REPO_ROOT = Path(__file__).resolve().parent.parent
1313
MODEL_RE = re.compile(r"^model:\s*(.+)$", re.MULTILINE)
14-
ALLOWED_MODELS = {"sonnet", "opus", "fable"}
14+
ALLOWED_MODELS = {"sonnet", "opus"}
1515
EXEMPT_COMPONENTS: set[tuple[str, str]] = {
1616
("skill", "do"),
1717
}
@@ -58,7 +58,7 @@ def validate_models() -> list[Violation]:
5858
component_type=component_type,
5959
component=component,
6060
file=str(path.relative_to(REPO_ROOT)),
61-
issue=f"model '{model}' is not allowed; use 'sonnet', 'opus', or 'fable' (except /do)",
61+
issue=f"model '{model}' is not allowed; use 'sonnet' or 'opus' (except /do)",
6262
)
6363
)
6464
return violations

skills/meta/agent-creator/references/agent-frontmatter-template.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ user_invocable: false
8080
| `name` | yes | string | lowercase, hyphens, numbers |
8181
| `description` | yes | quoted string | 60–120 chars; no "Use when:" prefix |
8282
| `color` | no | string | blue, green, yellow, red, purple, orange |
83-
| `model` | no | string | sonnet, opus, fable |
83+
| `model` | no | string | sonnet, opus |
8484
| `routing.triggers` | yes | list | 3–6 natural-speech phrases |
8585
| `routing.pairs_with` | no | list | agent names that exist on disk |
8686
| `routing.complexity` | yes | enum | Low, Medium, High (case-sensitive) |

0 commit comments

Comments
 (0)