We’ve updated our attention and comprehension check guidance

We want our researchers to be confident that their study participants are truly paying attention to what is being asked of them and understand the requirements of the research. Although we do everything we can to ensure our sample as a whole exhibits high levels of attention and comprehension, we also advocate the use of checks within researchers' surveys as an extra line of defence.

In response to feedback from our researcher community, we’ve now updated our guidance on the use of attention checks in studies on Prolific. You can find our new policy in detail here. Below is a brief overview of the major changes that have been made to the guidance.

Log in or sign up to Prolific today and launch your study to thousands of participants in minutes

Nonsensical items will now be allowed as an attention check

These are checks where there is only one objectively correct answer or response. For example, the statement “I swim across the Atlantic Ocean to get to work everyday” has an objectively correct answer (disagreement with the statement) as nobody could swim across the Atlantic Ocean to get to work every day.

To qualify as valid the check must also meet the following criteria:

• Questions must use a scale response (i.e. a Likert type scale)
• Questions must not assume prior knowledge. For example the statement “I often think about the 1980 Berlin Olympics” requires a participant to know where the Olympics were held that year and would therefore not be valid
• Participants should not be given a 'neutral' response option (i.e., no middle scale value)
• Correct responses should either be extreme agreement or extreme disagreement, however any response indicating disagreement or agreement should be accepted - participants who answer in the opposite manner to the objectively correct answer have failed
• If it is possible to justify an answer to your check that isn't they way you intended then the check is not suitable

As with our existing attention check guidance, participants can be rejected for failing a single nonsensical item if your study is under 5 minutes long. If the study is longer than 5 minutes then they must fail at least two checks to be rejected.

We now have specific guidance on the use of comprehension checks in your study

Comprehension checks are used to test whether a participant has understood critical information that is integral to completing your study successfully. You should only use this type of check if, without it, the task couldn’t be completed properly.

If you would like to use a comprehension check in your study, it must meet the following criteria:

• Participants must be free to re-read the key information at the time the comprehension check is presented
• Participants must be given at least two chances to get a correct answer
• These checks cannot involve free-text responses
• Comprehension checks must be given at the start of the study so participants are not screened out after having put in significant time and effort
• If a participant fails a comprehension check twice then they should be immediately asked to return their submission by closing the survey and clicking 'Stop Without Completing' on Prolific
• Participants should never be rejected on the basis of these checks. If participants who have failed comprehension checks are appearing as 'awaiting review' then please contact the support team for help in returning the submissions.

All of our existing guidance related to attention checks still exists. Make sure to read the full guidance to ensure that your checks are valid for use on Prolific.

Log in or sign up to Prolific today and launch your study to thousands of participants in minutes