Experiment closed for 2 days, despite enough slots available and session timeout set to 2h

[slussinator]

Dear community,

an Experiment that has been running for a while now seems to be full, despite slots remaining. I have adjusted the session time-out to 2h a while ago, such that the slots should not be blocked by experiments started (and not finished) more than two hours ago. Further, the experiment has last been accessed this morning around nine (by me), indicating that there is no more traffic and apparently all slots are filled. I don't understand why. I am not sure how better describe the problem, as posting a minimal experiment would not help. I am somewhat under time-pressure as data collection is preregistered to stop by May 30th.

I thank you for your tips and advice!

[ctreffe]

Hey @slussinator,

I think solving your problem will require a more detailed look into your current data. Could you e-mail both your main experiment data and your plugin data (if there is any) to me? Maybe @jobrachem and I can take a look and come back to you as soon as possible.

[slussinator]

I have sent the data via Email

[slussinator]

I just saw a new subject finishing the study... Maybe there is no problem whatsoever? There are still 17 subjects missing and an increase in sample size planned.

[ctreffe]

If I remember correctly, you changed the session timeout by adding the relevant setting to your experiment's config.conf in mortimer. Could you also post the exact text you added to the configuration field? So far, I do not see any problems with the randomizer data. Therefore, I suspect that there may be a typo or some missing entry in your config.

[slussinator]

I did not change any settings in the config.conf. I added a line in the exp.setup, as so:

@exp.setup 
def setup(exp):
    exp.session_timeout = 60 * 60 * 2

[slussinator]

At the moment it is possible to access the experiment, so it seems, that the randomizer is running correctly. I have also checked whether the exp.session_timeout works, in a minimal experiment. It seemed to work as expected. Not sure if there is some sort of superimposition on the exp.session_timeout going on, when the experiment runs on mortimer. Another possibility is that so many people are constantly clicking on the experiment, without filling it out, that the slots are constantly congested. Though I would suspect the slots to fill up faster then. Maybe SurveyCircle, despite denying this, is constantly accessing the experiment and filling up the slots... Either way, it seems impossible to get in the last 20 participants in a reasonable time

[ctreffe]

Hey @slussinator,

With the information from your last comment, I was able to narrow down the cause of your problems with the randomizer. Actually, the problems are not caused by alfred, but by SurveyCircle. I was under the impression you were conducting a study using Prolific, therefore, I did not realize that all the unfinished sessions in your experimental data were caused by non-user calls to your experiment, as you suspected.

Taking a look into your experimental data, I noticed a lot of calls to the experiment, that most likely do not originate from a regular browser since no JavaScript is executed. I infer this from these sessions' stored value for session_timeout, which is not changed according to your personal settings in the experiment's setup hook (as you described above). So, you are absolutely right: for some reason, SurveyCircle (or some other entity interacting with the link to the experiment) sends a lot of non-user calls to your experiment, I would think as a form of "keepalive check". These calls keep clogging the randomizer.

I see two possible solutions to this problem:

1. You could place a second one-page experiment between SurveyCircle and your actual experiment. This is the approach I would definitely recommend for future experiments. You can just make a single page experiment with an autoforward page to your actual experiment. Since the autoforward page employs JavaScript (@jobrachem, correct me if I'm wrong), the autoforward will only work if actual browsers call the startlink and the autoforward page can then forward users to the actual experiment. The keepalive calls we see in your current experiment will then be absorbed by this "buffer experiment", which is completely invisible to regular users.

2. Since you probably cannot change the start link in a running study, I would recommend you reduce the session_timeout to 30 or even 20 minutes. While this increases the chance of kicking a real participant from her or his session, this will still be the exception, as most participants won't make 30-minute breaks when working on your experiment. Reducing the session_timeout will therefore alleviate your problems with the randomizer.

[jobrachem]

I think in general it is good practice to have a "harmless" landing page on an experiment and start things like registration and randomization on the second page or later.

[slussinator]

Thank you @ctreffe! And @jobrachem, the landing page is a brilliant idea. Thank you :)

[ctreffe]

Good idea, @jobrachem 👍

Just a quick question, as there is no suitable code example showing this in the documentation: Would you create the whole randomizer within your second page's hook, as in the following example?

@exp.member
class SecondPage(al.Page):

    def on_first_show(self):
        randomizer = al.ListRandomizer(("cond1", 10), ("cond2", 10), exp=self.exp)
        self.exp.condition = randomizer.get_condition()

[slussinator]

I was just about to ask this, as the @exp.setup works independent of its position in the script

[jobrachem]

Oh boy, just saw that I apparently completely forgot to reply here. You both are right @ctreffe and @slussinator, with a landing page randomization should take place on_first_how of the second page, or maybe on_first_hide of the landing page.