Prolific's participant pool – the present and the future

Last week, we have shared some insights about the people behind Prolific, namely our Co-founder Katia. But Prolific is more than the team – online research would not be possible without our participants! Questions about our participant pool come up quite frequently. Our Data Analyst Jim Lumsden provides answers to the most frequently asked questions below. If you also have questions about Prolific's participant pool which are not covered in this post, just drop us a message!

How do you recruit your participants?
Believe it or not, every one of the 170k participants that have created an account so far, discovered Prolific through word of mouth. We’ve never advertised and we don’t head-hunt. We do, however, offer participants an incentive for referring their friends (10% of their friend’s first cash-out), and sites such as moneysavingexpert.com and reddit.com have picked us up as a good way to earn a bit of spare cash.

How do you check if your participants are actually who they say they are?
We have several processes in place to prevent fraudulent accounts. Including but not limited to the following:

1. Every account needs a unique non-VOIP phone number to verify.
2. We restrict signups based on IP and ISP (e.g., we allow common residential ISPs but block low-trustworthy IP/ISPS)
3. We limit to the number of accounts that can use the same IP address and the machine to prevent duplicate accounts.
4. We limit the number of unique IPs per study.
5. PayPal and Circle accounts for getting paid must be unique to a participant account. This means that in order to have 2 participant accounts that get paid, you would also need to have 2 PayPal accounts. PayPal and Circle also have steps to prevent duplicate accounts.
6. We take any data quality reports very seriously and whenever researchers have suspicions about accounts they can report the relevant participant IDs to us we investigate the individual accounts as well as any shared patterns between them.
7. We analyse our internal data to monitor for unusual usage patterns.

Can I collect a representative sample using Prolific’s participant pool?
We’re investigating the size and types of representative samples that Prolific can support and, assuming you cross-stratify on age, sex and simple ethnic group, we can currently support UK representative samples up to Ns of ~1000, and slightly smaller for the US.

At the moment (as of September 2018) 28,000 participants have completed a study on Prolific in the last 3 months: These participants are pretty diverse in age, ethnicity, sexuality, political orientation, occupation, hobbies and much more! This means we’re confident that your sample will capture a range wide of individuals.

In short, if you want to collect representative samples on Prolific you can… and we may even have a new feature coming out soon to help you with that 😉

How can I access niche samples on Prolific? (For example, I want to investigate people who are depressed – how can I access them on Prolific? Is this going to be cheaper or more efficient than a lab study?)
Prolific allows you to reach out to niche segments of the population. In the case of depression, about 20% of our platform have an ongoing mental health condition and 5% are currently taking antidepressants: that's ~1800 people you can invite to take part in your study. If you’re looking for men currently on a weight-reduction diet, then we’ve got 850 participants who can help you out. Or, if you’re looking for Teachers who love Twitter, we’ve got 600+ participants that meet your needs.

Accessing niche samples via the web is almost always going to be easier than trying to conduct a local laboratory study, and will save you time. But bare in mind that the nicher your sample, the longer it’ll take for you to reach your intended sample size. Furthermore, if your inclusion criteria are 100% inviolable, then be sure to include some backup checks in the study itself. For example, if you need participants taking antidepressants then asking them to confirm the drug-name and dosage is a good way of checking their legitimacy.

Are you currently doing anything to improve the data quality of the pool?
In short, yes! We are distinctly aware that without trustworthy participants Prolific will cease to be a valuable tool to researchers, and we are currently devoting considerable brainpower and development time to the problem. We’re hoping to perform more automated analysis of participant responses to catch outliers, tighten up our prescreening, improve our study-balancing tools, examine participant site-behaviour and make it easier for researchers to flag potentially dodgy participants to our team to review. In short, we’re doing everything we can to ensure our participants are NEAT (Naive, Engaged, Attentive and Trustworthy).

What are the future plans for the participant pool? (Such as: enlargement, representativeness, diversity, etc.)
Simply put, our goal is to keep growing. We’d love to reach less-internet savvy segments of society, break into new countries and be able to support larger representative samples. That said, it is essential that growth does not come at the cost of a loss in data quality; and we will continue to develop measures to preserve the reliability and validity of our pool in the months and years to come.

Come discuss this blog post with our community!