How to improve your data quality from online research
|
25
February 2022
17
August 2018
|
25
February 2022
17
August 2018
|
25
February 2022
17
August 2018
Last week we published a blog post detailing how Prolific verifies that our participants aren’t bots and that individuals aren't able to open multiple accounts. Thing is, when it comes to participant data, “not being a bot” is about the lowest bar you can set. This week, we want to say a bit more on the topic of data quality and give you some practical tips for minimising the amount of poor quality data in your dataset. We’ll also discuss some of the internal measures we have in place to catch participants who are dishonest.
Online data collection is revolutionizing the way we do research, yet it brings new risks regarding data quality. In the lab we can meet our participants face-to-face and monitor them while they complete the study. Online, we cannot be so sure that our participants are human, are who they say they are, are completing the study properly or are even paying attention.
At Prolific, we think of malicious participants as falling into roughly four groups: bots, liars, cheats and slackers.
Bots are autonomous or semi-autonomous pieces of software designed to complete online surveys with very little human intervention. Bots are often distinguished by random or very low effort/nonsensical free text responses. Thankfully, due to their obviously non-human answers, there are several methods for detecting bots in your data (see Dupuis et al., 2018). Unfortunately bad-acting humans can prove trickier to detect...
Liars (or to use the technical term: malingerers) submit false prescreening information in order to gain access to the largest numbers of studies possible, and consequently maximise their earning potential. The impact of liars on your data quality depends on two things:
The third group are cheats: participants that deliberately submit false information to your study. Importantly, cheaters are not always intending to be dishonest: some may genuinely be confused about the data you’re trying to collect, or whether they’ll get paid even if they don’t do “well”. This can happen because they fear your study’s rewards are tied to their performance (i.e., they believe they will only be paid if they get 100% on a test, so they google the correct answers). Alternatively, they might think you only want a certain kind of response (i.e., therefore always giving very positive/enthusiastic responses) or use aides (pen and paper) to artificially perform much better than reality. The final kind of cheats are participants who don’t take your survey seriously, perhaps completing it with their friends, or while drunk. To clarify: Liars provide false demographic information to gain access to your study. Cheats provide false information within the study itself. A participant can be both a liar and a cheat, but their effects on data quality are different.
The fourth group are slackers. These participants are inattentive and typically aren’t focused on maximising their earnings. Rather, they are simply unmotivated to provide any genuine data for the price you’re paying. Slackers encompass a broad group: from participants that don’t read instructions properly, to participants that are completing your study while watching TV. They may input random answers, gibberish or low-effort free text. Importantly, slackers are not always dishonest: some may just consider the survey reward too low to be worth their full attention.
It’s worth highlighting that these groups are not independent. A liar can use bots, slackers can cheat, etc. In fact, it’s likely there’s a lot of overlap, because most bad actors don’t really care what methods they use to earn rewards, so long as they’re maximising their income!
So, what can you do about it?
We at Prolific have banned our fair share of malicious accounts, and we’ve learned a thing or two along the way. The list below is not exhaustive, but provides some practical advice for designing your study and screening your data that will boost your confidence in the responses you collect. If this seems overwhelming to you, then don’t worry! We are doing a lot of work on our side to improve the quality of the pool, and ultimately you shouldn’t encounter many untrustworthy participants.
As we’ve already said, we have technological measures in place to prevent bots so you’re extremely unlike to find any in your dataset.
Again, we constantly analyse the answer sets of our participants looking for unusual combinations, impossible answers and other tell-tale signs of malingering.
However you clean your data, we strongly recommend that you preregister your data-screening criteria to increase reviewer confidence that you have not p-hacked.
One of the most important factors in determining data quality is the study’s reward. A recent study of Mechanical Turk participants concluded that fair pay and realistic completion times had a large impact on the quality of data they were willing to provide. On Prolific, it’s vital that trust goes both ways, and properly rewarding participants for their time is a large part of that. We enforce a minimum hourly reward of 5.00 GBP. But depending on the effort required by your study, this may not be sufficient to foster high levels of engagement and provide good data quality. Consider:
That said, paying more isn’t always a good idea! Consider that:
While participants are ultimately responsible for the quality of data they provide, you as the researcher need to set them up to do their best.
Firstly, talk to us. We will always ban participants using bots or lying in their prescreeners. You should reject submissions where you believe this to have occurred, and send us any evidence you have gathered. We’re on high alert right now in light of recent data quality issues on MTurk, and data quality is our top priority, so please reach out to us if you have any concerns, queries or suggestions.
In cases of cheating or slacking, we ask that you give participants some initial leeway. If they’ve clearly made some effort or attempted to engage with the task for a significant period of time but their data is not of sufficient quality, then consider approving them, but excluding them from your analysis. If the participant has clearly made little effort, failed multiple attention checks or has lied their way into your study, then rejection is appropriate. Please read our article on valid and invalid rejection reasons for more guidance.
Finally, if you found this blog post helpful, then watch this space over the next few months for more advice on how to make the most out of your research on Prolific.
Come discuss this blog post with our community!
Today Prolific is turning 5 years old – Happy Birthday to us! 🥳 It's been a remarkable journey so far. 3000+ researchers from science and industry have used Prolific last year, we have 45,000 quarterly active participants, and we've seen 200% year-on-year growth. But we're only getting started. In this post, I'll tell you a little bit about our journey, give credit where it's due!, and tell you about our exciting plans for the future.
Fresh out of YC's Summer 2019 batch, we want to share some of our most interesting learnings. If you're a startup founder or enthusiast and want to learn about product-market fit, growth experimentation and culture setting, you're in the right place!