Fix KeyError in truthful_qa_generative prompt function

Duc Hoang · Duc Hoang · commit f82255e624e1 · 2025-12-10T17:52:17.000-08:00
Remove access to mc1_targets field which only exists in multiple_choice subset, not generation subset. This was causing KeyError when running truthfulqa:gen task.
diff --git a/src/lighteval/tasks/default_prompts.py b/src/lighteval/tasks/default_prompts.py
@@ -2491,7 +2491,6 @@ def truthful_qa_generative(line, task_name: str = None):  # BLEU and combination
         query=line["question"].strip(),
         choices=correct_answers + incorrect_answers,
         gold_index=list(range(len(correct_answers))),
-        specific={"len_mc1": len(line["mc1_targets"]["choices"])},
     )
 
 

Original file line number	Diff line number	Diff line change
`@@ -2491,7 +2491,6 @@ def truthful_qa_generative(line, task_name: str = None): # BLEU and combination`
`2491`	`2491`	`query=line["question"].strip(),`
`2492`	`2492`	`choices=correct_answers + incorrect_answers,`
`2493`	`2493`	`gold_index=list(range(len(correct_answers))),`
`2494`		`- specific={"len_mc1": len(line["mc1_targets"]["choices"])},`
`2495`	`2494`	`)`
`2496`	`2495`
`2497`	`2496`