LLM use in research: A study into mitigation strategies

AI is changing the nature of research. It’s a challenge faced by everyone in the online research community.

A recent study explored how AI is affecting crowdsourcing platforms. It looks at how we can keep researchers one step ahead of the challenges AI presents - and how they can embrace its opportunities.

Simple measures reduce LLM use by almost 50%

In this study, participants on the Prolific platform were asked to write a summary of a large chunk of text. The method required to do this was hard for humans - but easy for Large Language Models (LLMs).

Two distinct strategies were tested to reduce the number of participants using LLMs.

1: The ‘request’ strategy

In this approach, participants were explicitly asked not to use LLMs.

2: The ‘hurdle’ strategy

For this strategy, hurdles were created to deter participants from using LLMs. The original abstract text was converted to an image, or copy-pasting was disabled entirely.

Findings showed that:

• Around 30% used LLMs.
• Simple measures reduced LLM usage by almost 50%.

Challenges and nuances

The methods weren’t perfect. And there were some challenges in the analysis.

• The question intentionally makes using AI seem attractive - a complex task made easy by AI. It didn’t represent a typical question asked on Prolific.
• It’s very hard to detect who uses AI online, as others are finding. OpenAI recently discontinued its own tool for detecting AI-written text as it wasn’t accurate enough.

Embracing AI evolution: Why LLM use isn’t always cheating

LLM-based tools and how people use them are evolving fast.

It would be a mistake to define LLM use as foul play for every study run online. In some research, LLMs assisting crowd workers might even be beneficial. Also, as more people use LLMs to complete everyday tasks, the distinction between ‘synthetic’ and ‘human’ data will get increasingly blurred.

So, rather than focus on shunning LLMs from crowd work, it’s worth investigating how researchers can embrace them. There are many ways for crowd workers to use LLMs. And different approaches may have different effects on the output of the research.

Prolific helps researchers navigate rapid AI change

The truth matters in research. The best decisions - and biggest discoveries - are built on the highest-quality data.

Every online research platform has to manage rapid change brought on by AI. But we’re working hard to help people navigate that - through this study, and new platform features.

We’ll soon launch a new addition to the Prolific platform that will help us better detect participants who use LLMs. Based on our own machine learning models and informed by participant behaviour, this feature is 96% effective.

And we have more in the pipeline to help researchers deliver world-changing work.

You can read more about the mitigation methods tested as part of the study in the full paper.