fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs#3923
Open
hancheolcho wants to merge 1 commit into
Open
fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs#3923hancheolcho wants to merge 1 commit into
hancheolcho wants to merge 1 commit into
Conversation
…val hangs schema_compliance() validates each prediction with an unbounded call to schema_conform_with_format_checker (jsonschema Draft202012Validator + FormatChecker). A single pathological schema/instance (deeply-recursive $ref, or a format regex with catastrophic backtracking) spins the CPU indefinitely, so the scoring phase never returns and the whole evaluation hangs with no error. Wrap the per-sample validation in a wall-clock SIGALRM timeout. On timeout the sample is treated as non-compliant (returns False), matching how the existing except-branch already handles un-processable schemas, instead of hanging. Budget defaults to 40s and is overridable via JSONSCHEMA_BENCH_TIMEOUT_S. If SIGALRM is unavailable (non-main thread / unsupported platform) validation runs unguarded so behavior never regresses to a crash.
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
jsonschema_benchcan hang an entire evaluation indefinitely during scoring.schema_compliance()validates each prediction with an unbounded call toschema_conform_with_format_checker(jsonschemaDraft202012Validator+FormatChecker). A single pathological schema/instance — e.g. a deeply-recursive$ref, or aformatregex with catastrophic backtracking — spins the CPU forever, so the scoring phase never returns. Generation completes, but the process then sits at ~100% CPU with no progress and no error/traceback.Fix
Wrap the per-sample validation in a wall-clock
SIGALRMtimeout (_run_with_timeout). On timeout the sample is treated as non-compliant (return False) — the same outcome the existingexcept Exceptionbranch already produces for un-processable schemas — instead of hanging.JSONSCHEMA_BENCH_TIMEOUT_Senvironment variable.SIGALRMis unavailable (non-main thread, or an unsupported platform), validation runs unguarded, so behavior never regresses to a crash.lm_eval/tasks/jsonschema_bench/metrics.py; samples that already validate within the budget are scored exactly as before.Notes
jsonschema_benchtask is affected.