Performance-Contingent Payout

[slussinator]

Dear Community,

I would like to build an experiment with performance-contingent payout. The more items correct, the higher the likelihood of winning a prize. I have done this with another experiment, where estimates made by participants were compared to true values.

My problem here is that subjects are asked a number of questions and are asked to pick one out of two possible answers (one of the answers is correct). I can't assign a "true value" because depending on condition, the true answer is either on the right or the left side. I hope this makes sense. Attached find a minimal experiment to hopefully make clearer what I am asking.

import alfred3 as al
exp = al.Experiment()

list_AC = [
    {"name": "AC_1_A", "toplab": "Rekapituliert nach Abschluss eines Aufgabenabschnitts, was gelaufen ist."},
    {"name": "AC_2_A", "toplab": "Macht stets deutlich, was man im Gegenzug für die Erfüllung bestimmter Aufgaben erwarten kann."},
    {"name": "AC_3_F", "toplab": "Schafft es, Probleme als Herausforderungen darzustellen, die gemeinsam überwunden werden können."},
    {"name": "AC_4_F", "toplab": "Bedenkt stets, welche moralischen und ethischen Konsequenzen Entscheidungen haben werden."},
    {"name": "AC_5_A", "toplab": "Organisiert zuverlässig, was getan werden muss, und formuliert klar, wer für welche Aufgabe zuständig ist."},
    {"name": "AC_6_F", "toplab": "Betont immer wieder, dass mit der Arbeit ein wichtiger Beitrag zu einer besseren Lebensqualität von Menschen mit Amputationen geleistet wird."},
    {"name": "AC_7_F", "toplab": "Tauscht sich mit dem Team aus und lässt deren Argumente in den Entscheidungsprozess einfließen"},
    ]

@exp.setup  
def setup(exp):
    exp.stud_randomizer = al.ListRandomizer(
        ("m_transact1", 2),
        ("m_transform1", 2),
        name="stud_randomizer",
        exp=exp,# hier wird das verwendete Experiment definiert. In diesem Fall eben unser exp
        respect_version=True
    )
    exp.condition = exp.stud_randomizer.get_condition()
    exp.log.info(exp.condition)


class m_AttentionCheck_transact1(al.Page):  
    title = "Aufmerksamkeitsprüfung"

    def on_exp_access(self):
        self += al.Card(
            body="Bitte wählen Sie aus auf welche Führungsperson die folgenden Verhaltensweisen jeweils am ehesten zutreffen")
        for i in list_AC:
            # The following two lines will skip the remaining iterations when the
            # item list runs out of items, giving the code way more flexibility.
            item = i
            self += al.SingleChoiceButtons(
                "Führungsperson Herr A.",
                "Führungsperson Herr F.",
                toplab=item['toplab'],
                name=item['name'],
                width="wide",
                button_width="140px",
                force_input=True,
                font_size="normal",
                position="center"
            )

class m_AttentionCheck_transform1(al.Page): 
    title = "Aufmerksamkeitsprüfung"

    def on_exp_access(self):
        self += al.Card(
            body="Bitte wählen Sie aus auf welche Führungsperson die folgenden Verhaltensweisen jeweils am ehesten zutreffen")
        for i in list_AC:
            # The following two lines will skip the remaining iterations when the
            # item list runs out of items, giving the code way more flexibility.
            item = i
            self += al.SingleChoiceButtons(
                "Führungsperson Herr F.",
                "Führungsperson Herr A.",
                toplab=item['toplab'],
                name=item['name'],
                width="wide",
                button_width="140px",
                force_input=True,
                font_size="normal",
                position="center"
            )

@exp.member
class PreferenceSection(al.Section):  
    name = "preference_section"

    def on_exp_access(self):
        self += al.AutoForwardPage(name="pseudo", timeout="0s")

        if self.exp.condition == ("m_transact1" or "m_transact2"):
            self += m_AttentionCheck_transact1(name="m_AttentionCheck_transact1")
        elif self.exp.condition == ("m_transform1" or "m_transform2"):
            self += m_AttentionCheck_transform1(name="m_AttentionCheck_transform1")

if __name__ == "__main__":
    exp.run()

[ctreffe]

Hey @slussinator,

I assume that you need an immediate evaluation of your participants' scores to determine whether they win a prize at the end of an experimental session in the laboratory. Otherwise, you could always opt to evaluate the performance scores later on in your data analysis (which you are probably doing anyway) and only subsequently inform the winners.

For an immediate evaluation, however, you will need to access and evaluate participants' inputs in one of the adequate page hooks (either on_close(), on_each_hide(), or on_first_hide():

def on_close(self):  # Add this method to both of your pages, as only one will be added to the experiment

    attention_score = 0  # First, create a new scoring variable
    
    if self.exp.condition == ("m_transact1" or "m_transact2"):  # Implement separate evaluations for your different conditions
        
        if self.exp.values.get("AC_1_A") == 1:  # You need to adjust this to the correct setting in the selected condition
            attention_score += 1

        if self.exp.values.get("AC_2_A") == 1:  # You need to adjust this to the correct setting in the selected condition
            attention_score += 1

        ...  # Continue with if statements for the other attention check items

   elif self.exp.condition == ("m_transform1" or "m_transform2"):

        if self.exp.values.get("AC_1_A") == 2:  # You need to adjust this to the correct setting in the selected condition
            attention_score += 1

        if self.exp.values.get("AC_2_A") == 2:  # You need to adjust this to the correct setting in the selected condition
            attention_score += 1

        ...  # Continue with if statements for the other attention check items

    self += al.Value(attention_score, name="attention_score")

The above pseudo code example should give you an idea on how to implement the evaluation in the on_close() hook. According to the SingleChoiceButtons Documenation, participants' answers should be saved as either 1 or 2 (numeric variable). The example uses basic if-statements, to check for the correct answer to each item and adds to the attention_score variable in case of an correct answer. In the above example, I save the final score to the experiment data (which saves you from having to evaluate the items again in your data analysis). You can either use this variable at the end of your experiment to determine the winning chances, or you could add the appropriate code for the winning chances directly to this on_close() hook.

Here I created code that can be copied & pasted to be used in both pages. Alternatively, you could always add only the selected condition's code to both page's on_close() hook and save the top-level if-statement within each method.

[slussinator]

Thank you so much! This solves my problem completely :)