feat: verification + eval harness + context budget + model-agnostic runner - #1
Merged
Merged
Conversation
…rt header Wire check_citations.py into the live pipeline. Dead OPEN sources trigger re-search or claim demotion instead of silently shipping. medium blocks below 0.70 floor; deep requires zero unresolved red flags.
Questions span all 6 genres and 3 depths. aggregate.py joins scorecards with runs.csv into a quality-per-dollar table. Numbers come from real runs.
validate_structure.py enforces report naming, source frontmatter, csv columns, findings pattern. CI runs structure + budget + citation-fixture on every PR. No tokens.
Measures token load per phase against a window; --ci fails on SKILL.md or always-floor bloat. Current catalog ~154k tokens total, ~50k always-floor.
Resolves the ~210 data-portal URLs the weekly endpoint sync skips; reports dead/unknown for pruning and a verified-core vs community split.
Provider interface (strong/mid/cheap tiers) drives the 7 phases. DryRunProvider runs end-to-end and produces schema-valid output; Claude/OpenAI adapters stubbed.
…роксика в --ci - validate_structure.check_report: return len(dated) вместо `read_text() and len(dated)` (возвращало str вместо int + лишнее чтение файла) - providers.DryRunProvider.complete: безопасная первая строка через next(iter(...), "") — пустой/пробельный промпт больше не падает с IndexError - context_budget --ci: ловит дрейф CHARS_PER_TOKEN относительно score_run.py и падает на пропавших ALWAYS_LOAD-файлах (раньше молча занижался floor) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Набор инструментов вокруг качества deep-research: рантайм-верификация цитат, бенчмарк-харнесс, бюджет контекста, детерминированный CI и model-agnostic раннер.
Что внутри (6 фич-коммитов)
feat(verify)feat(eval)aggregate.py: scorecards → таблица «качество на доллар»feat(ci)validate_structure.py— проверка именования отчётов, frontmatter источников, колонок csv, паттерна findings + детерминированный workflow (без токенов)feat(budget)context_budget.py— токены контекста по фазам vs окно;--ciпадает на раздувании SKILL.md / always-floorfeat(catalog)validate_stat_sources.py— проверка ~210 data-portal URL, отчёт dead/unknownfeat(runner)Ревью-правки (7-й коммит)
После high-effort код-ревью (7 углов, adversarial verify) исправлено:
validate_structure.check_reportвозвращалstrвместоint(read_text() and len()) + лишнее чтение файла →return len(dated)DryRunProvider.completeпадал сIndexErrorна пустом промпте → безопасная первая строкаcontext_budget --ciтеперь ловит дрейфCHARS_PER_TOKENотносительноscore_run.pyи падает на пропавших always-load файлах (раньше floor молча занижался)Ложные срабатывания ревью (ключ
configв aggregate.py — верный; чужойcatalog-sync.yml— вне диапазона) отклонены.|| trueна структурном гейте оставлен осознанно (фикстура пока без plan.md/отчёта — задокументировано в воркфлоу).Проверено
python scripts/context_budget.py --ci→Budgets OK, exit 0; дрейф проксика ловится (exit 1)python eval/validate_structure.py --strict→ exit 1 на неполной фикстуре (ожидаемо)DryRunProvider: пустой/пробельный/нормальный промпт — без краха🤖 Generated with Claude Code