Adding templates to the DSL

[BrentBlanckaert]

Templates are handled in the same file as the translations. So using to following will now also proces templates:

python -m tested.nat_translation ./exercise/simple-example/templates/suite.yaml en

[BrentBlanckaert]

Some examples of what is currently possible:

Usage of object template

translations:
  animal:
    en: "animals"
    nl: "dieren"
tabs:
  - tab: '{{translations.animal}}'
    templates:
      default_template:
        expression:
          javascript: "{{translations.animal}}_javascript({{ templates.input }})"
          typescript: "{{translations.animal}}_typescript({{ templates.input }})"
          java: "Submission.{{translations.animal}}_java({{ templates.input }})"
          python: !natural_language
            en: "{{translations.animal}}_python_en({{ templates.input }})"
            nl: "{{translations.animal}}_python_nl({{ templates.input }})"
        return: "{{ templates.output }}"
    testcases:
      - placeholder:
          name: "default_template"
          data:
            input: !natural_language
              en: "Sardines"
              nl: "Sardienen"
            output: !natural_language
              en: "fish"
              nl: "vis"
      - placeholder:
          name: "default_template"
          data:
            input: !natural_language
              en: "Tiger"
              nl: "Tijger"
            output: !natural_language
              en: "mammal"
              nl: "zoogdier"

Usage of list template

"units":
- "unit": "echo"
  "translations":
    "monkeys":
      en: "monkeys"
      nl: "apen"
  "templates":
    "drinks":
      - "expression": "drinks({{ templates.input1 }})"
        "return": "{{ templates.ok }}"
      - "expression": "drinks({{ templates.input2 }})"
        "return": "{{ templates.ok }}"
      - "expression": "drinks({{ templates.input3 }})"
        "return": "{{ templates.ok }}"
  "scripts":
    "placeholder":
      "name": "drinks"
      "data":
        "ok": "OK!"
        "input1": "Beer"
        "input2": "Soda"
        "input3": "Water"

Usage of scalar template

"units":
- "unit": "IO"
  "translations":
    "monkeys":
      en: "monkeys"
      nl: "apen"
  "templates":
    "heartbeats": !natural_language
      "en": "{{templates.animals}} have {{templates.billion_heartbeats_per_lifetime}} billion heartbeats"
      "nl": "{{templates.animals}} hebben {{templates.billion_heartbeats_per_lifetime}} miljard hartslagen"
  "scripts":
  - "stdin": |-
      horses
      44
      40
    "stdout": "horses have 0.93 billion heartbeats"
  - "stdin": |-
      {{translations.monkeys}}
      190
      40
      15
    "stdout":
      "placeholder":
        "name": "heartbeats"
        "data":
          "animals": "{{translations.monkeys}}"
          "billion_heartbeats_per_lifetime": 1.50

[BrentBlanckaert]

New suggestion for templates

Usage of object template

Basic usage

tabs:
  - tab: 'animals'
    templates:
      default_animal:
        expression:
          java: "Submission.animals({{ input }})"
          python: "animals({{ input }})"
        return: "{{ output }}"
    testcases:
      - template:
          name: "default_animal"
          parameters:
            - input: "Sardines"
              output: "fish"
            - input: "Tiger"
              output: "mammal"
      - template: 
          name: "default_animal"
          parameters: # Parameters kan bijvoorbeeld ook een object zijn.
            input: "Whale"
            output: "mammal"

A templates map can be defined in the exact same places as a translations map (globally, in a tab and in a context). The templates map will be a map with keys that will be the names for the templates. Under that name you define the actual template.

In the test-suite, you can then define a template. A template will always need a name. This will be used to match the defined keys in the templates maps. In a template, you can also specify parameters (not manditory). In this case input and output are specified. In the found templates the placesholders with the matching placeholders will be replaced with the defined parameters.

With repeats

tabs:
  - tab: 'random'
    templates:
      default_random:
        expression: "random()"
        return: "{{ output }}"
    testcases:
      - template: 
          name: "default_random"
          parameters:
            - output: 0
              repeat: 100
            - output: 9
              repeat: 100

Default template

For a default template, you should define it globally.

default_template:
  expression: "calculator({{ expr }})"
  return: "{{ result }}"
templates:
  print_calc:
    expression: "print_steps({{ expr }})"
    return: "{{ result }}"
tabs:
  - tab: 'calculator'
    testcases:
      - parameters:
          - expr: "2 + 10"
            result: 12
          - expr: "9 + 5"
            result: 14
          - expr: "99 + 3"
            result: 102
          - expr: "4 * 13"
            result: 52
      - template: 
          name: "print_calc"
          parameters:
            - expr: "2 + 10"
              result: "2 + 10 = 12"
            - expr: "9 + 5"
              result: "9 + 5 = 14"
            - expr: "99 + 3"
              result: "99 + 3 = 102"
            - expr: "4 * 13"
              result: "4 * 13 = 4 * 10 + 4 * 3 = 40 + 12 = 52"

To access the default template, you just have to specify the parameters.

Using translations

translations:
  animals:
    en: "animals"
    nl: "dieren"
tabs:
  - tab: '{{ animals }}'
    templates:
      default_animal:
        expression:
          java: "Submission.{{ translations.animals }}({{ templates.animals }})"
          python: "{{translations.animals}}({{ templates.animals }})"
        return: "{{ output }}"
    testcases:
      - template:
          name: "default_animal"
          parameters:
            - input: "Sardines"
              output: "fish"
            - input: "Tiger"
              output: "mammal"

This is a very rare case in which the template uses a key that is also defined in a translations map. In this case, namespaces should be used (throwing an error?).

Usage of list template

"units":
- "unit": "drinks"
  "templates":
    "default_drinks":
      - "expression": "drinks({{ input1 }})"
        "return": "{{ ok }}"
      - "expression": "drinks({{ input2 }})"
        "return": "{{ ok }}"
      - "expression": "drinks({{ input3 }})"
        "return": "{{ ok }}"
  "scripts":
    "placeholder":
      "name": "default_drinks"
      "data":
        "ok": "OK!"
        "input1": "Beer"
        "input2": "Soda"
        "input3": "Water"

Usage of scalar template

Not sure if usage in stdin, stdout, stderr, return, expressions, etc., should be supported or not.

"units":
- "unit": "IO"
  "translations":
    "monkeys":
      en: "monkeys"
      nl: "apen"
  "templates":
    "default_heartbeats": !natural_language
      "en": "{{animals}} have {{billion_heartbeats_per_lifetime}} billion heartbeats"
      "nl": "{{animals}} hebben {{billion_heartbeats_per_lifetime}} miljard hartslagen"
  "scripts":
  - "stdin": |-
      horses
      44
      40
    "stdout": "horses have 0.93 billion heartbeats"
  - "stdin": |-
      {{monkeys}}
      190
      40
      15
    "stdout":
      "placeholder":
        "name": "default_heartbeats"
        "data":
          "animals": "{{monkeys}}"
          "billion_heartbeats_per_lifetime": 1.50