Skip to content

fix(afrixnli): use Jinja braces in prompt_1 doc_to_text so premise and hypothesis are substituted #3932

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AmosBunde wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(afrixnli): use Jinja braces in prompt_1 doc_to_text so premise and hypothesis are substituted #3932

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_amh.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_eng.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_ewe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_fra.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_hau.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_ibo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_kin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_lin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_lug.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_orm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_sna.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_sot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_swa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_twi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_wol.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_xho.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_yor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/direct/prompt_1/afrixnli_zul.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
6 changes: 3 additions & 3 deletions lm_eval/tasks/afrixnli/gen_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ def __init__(self, value):
def prompt_func(mode, lang):
prompt_map = {
"prompt_1": "Please identify whether the premise entails or contradicts the hypothesis in the following premise "
"and hypothesis. The answer should be exact entailment, contradiction, or neutral.\n\nPremise: {premise}\nHypothesis: {hypothesis}\n\n"
"and hypothesis. The answer should be exact entailment, contradiction, or neutral.\n\nPremise: {{premise}}\nHypothesis: {{hypothesis}}\n\n"
"Is it entailment, contradiction, or neutral?",
"prompt_3": f"Given the following premise and hypothesis in {lang}, identify if the premise entails, contradicts, "
f"or is neutral towards the hypothesis. Please respond with exact 'entailment', 'contradiction', or 'neutral'. \n\n"
Expand Down Expand Up @@ -56,7 +56,7 @@ def gen_lang_yamls(output_dir: str, overwrite: bool, mode: str) -> None:
"swa": "Swahili",
}

for lang in languages.keys():
for lang, lang_name in languages.items():
try:
file_name = f"afrixnli_{lang}.yaml"
task_name = f"afrixnli_{lang}_{mode}"
Expand All @@ -70,7 +70,7 @@ def gen_lang_yamls(output_dir: str, overwrite: bool, mode: str) -> None:
"include": yaml_template,
"task": task_name,
"dataset_name": lang,
"doc_to_text": prompt_func(mode, languages[lang]),
"doc_to_text": prompt_func(mode, lang_name),
}
else:
yaml_details = {
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_amh.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_ewe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_fra.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_hau.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_ibo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_kin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_lin.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_lug.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_orm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_sna.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_sot.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_swa.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
4 changes: 2 additions & 2 deletions lm_eval/tasks/afrixnli/translate/prompt_1/afrixnli_translate_twi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ doc_to_text: 'Please identify whether the premise entails or contradicts the hyp
contradiction, or neutral.


Premise: {premise}
Premise: {{premise}}

Hypothesis: {hypothesis}
Hypothesis: {{hypothesis}}


Is it entailment, contradiction, or neutral?'
Expand Down
Loading