Missing confidence interval

[yoavkatz]

Even when running on multiple examples, sometime confidence interval as missing:

You can remove this warning by passing 'verification_mode=all_checks' instead.
warnings.warn(
LiteLLM Inference (watsonx/meta-llama/llama-3-2-1b-instruct): 100%|█████████████████████████████████████████| 10/10 [00:02<00:00, 4.27it/s]
/Users/yoavkatz/miniforge3/envs/fme/lib/python3.10/site-packages/scipy/stats/_resampling.py:144: RuntimeWarning: invalid value encountered in divide
a_hat = 1/6 * sum(nums) / sum(dens)(3/2)
/Users/yoavkatz/miniforge3/envs/fme/lib/python3.10/site-packages/scipy/stats/_resampling.py:144: RuntimeWarning: invalid value encountered in divide
a_hat = 1/6 * sum(nums) / sum(dens)(3/2)
/Users/yoavkatz/miniforge3/envs/fme/lib/python3.10/site-packages/scipy/stats/_resampling.py:144: RuntimeWarning: invalid value encountered in divide
a_hat = 1/6 * sum(nums) / sum(dens)**(3/2)
Sample input and output for template 'templates.my_entailment_as_fields' and num_demos '3':
source prediction processed_prediction
0 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
1 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
2 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
3 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
4 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
5 [{'role': 'system', 'content': 'Indicate wheth... contradiction contradiction
6 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
7 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
8 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral
9 [{'role': 'system', 'content': 'Indicate wheth... neutral neutral

	template	num_demos	f1_micro	ci_low	ci_high
0	templates.my_entailment_as_question	0	0.12	nan	nan
1	templates.my_entailment_as_question	3	0.22	nan	nan
2	templates.my_entailment_as_fields	0	0	nan	nan
3	templates.my_entailment_as_fields	3	0.4	nan	nan

[yoavkatz]

To replicate:

python examples/evaluate_different_templates.py