Groups of Laypeople Reliably Rate Stories As Effectively as Fact-Checkers

Crowds Can Wise Up To Fake News

Experiment with Facebook-flagged content shows groups of laypeople reliably rate stories as effectively as fact-checkers do.

In the face of grave concerns about misinformation, social media networks and news organizations often employ fact-checkers to sort the real from the false. But fact-checkers can only assess a small portion of the stories floating around online.

A new study by MIT researchers suggests an alternate approach: Crowdsourced accuracy judgments from groups of normal readers can be virtually as effective as the work of professional fact-checkers.

“One problem with fact-checking is that there is just way too much content for professional fact-checkers to be able to cover, especially within a reasonable time frame,” says Jennifer Allen, a PhD student at the MIT Sloan School of Management and co-author of a newly published paper detailing the study.

But the current study, examining over 200 news stories that Facebook’s algorithms had flagged for further scrutiny, may have found a way to address that problem, by using relatively small, politically balanced groups of lay readers to evaluate the headlines and lead sentences of news stories.

“We found it to be encouraging,” says Allen. “The average rating of a crowd of 10 to 15 people correlated as well with the fact-checkers’ judgments as the fact-checkers correlated with each other. This helps with the scalability problem because these raters were regular people without fact-checking training, and they just read the headlines and lead sentences without spending the time to do any research.”

That means the crowdsourcing method could be deployed widely — and cheaply. The study estimates that the cost of having readers evaluate news this way is about $0.90 per story.

“There’s no one thing that solves the problem of false news online,” says David Rand, a professor at MIT Sloan and senior co-author of the study. “But we’re working to add promising approaches to the anti-misinformation tool kit.”

The paper, “Scaling up Fact-Checking Using the Wisdom of Crowds,” is being published today in Science Advances. The co-authors are Allen; Antonio A. Arechar, a research scientist at the MIT Human Cooperation Lab; Gordon Pennycook, an assistant professor of behavioral science at University of Regina’s Hill/Levene Schools of Business; and Rand, who is the Erwin H. Schell Professor and a professor of management science and brain and cognitive sciences at MIT, and director of MIT’s Applied Cooperation Lab.

A critical mass of readers

To conduct the study, the researchers used 207 news articles that an internal Facebook algorithm identified as being in need of fact-checking, either because there was reason to believe they were problematic or simply because they were being widely shared or were about important topics like health. The experiment deployed 1,128 U.S. residents using Amazon’s Mechanical Turk platform.

Those participants were given the headline and lead sentence of 20 news stories and were asked seven questions — how much the story was “accurate,” “true,” “reliable,” “trustworthy,” “objective,” “unbiased,” and “describ[ing] an event that actually happened” — to generate an overall accuracy score about each news item.

At the same time, three professional fact-checkers were given all 207 stories —asked to evaluate the stories after researching them. In line with other studies on fact-checking, although the ratings of the fact-checkers were highly correlated with each other, their agreement was far from perfect. In about 49 percent of cases, all three fact-checkers agreed on the proper verdict about a story’s facticity; around 42 percent of the time, two of the three fact-checkers agreed; and about 9 percent of the time, the three fact-checkers each had different ratings.

Intriguingly, when the regular readers recruited for the study were sorted into groups with the same number of Democrats and Republicans, their average ratings were highly correlated with the professional fact-checkers’ ratings — and with at least a double-digit number of readers involved, the crowd’s ratings correlated as strongly with the fact-checkers as the fact-checkers’ did with each other.

“These readers weren’t trained in fact-checking, and they were only reading the headlines and lead sentences, and even so they were able to match the performance of the fact-checkers,” Allen says.–nc-state-live-/–nc-state-live-/,49270925.html

Leave a Reply

Your email address will not be published. Required fields are marked *