U-M Tracker Measures Reliability of News on Facebook, Twitter

• By Dian Schaffhauser
• 10/16/18

As the deadline for voting in the mid-term elections approaches, people now have access to a new online tool to help monitor the prevalence of fake news on the big social media sites. The University of Michigan Center for Social Media Responsibility, housed in the School of Information, created Iffy Quotient as a "platform health metric" to track "iffy" news in real time.

As of the day this story was written, Facebook had a score of 3.6 and Twitter 4.4 on the index. During the 2016 election, according to the Center, the Iffy Quotient had grown dramatically for both social media sites, up to 7.8 for Facebook on Nov. 5, 2016 and 5.6 for Twitter.

Here's how the quotient is calculated:

1) NewsWhip, a social media engagement tracking firm, examines URLs on more than 400,000 sites daily and gathers data on which of those sites have engagements on Facebook and Twitter.

2) Iffy Quotient taps NewsWhip for the top 5,000 most popular URLs on the two social media platforms.

3) Next, Iffy Quotient classifies those domain names based on whether they have been flagged by Media Bias/Fact Check, an independent website that categorizes sources based on their reliability and bias.

4) The U-M service divides the URLs into three buckets:

• "Iffy" for sites that appear on the "Questionable Sources" or "Conspiracy" lists;
• "OK" for sites on any other list, such as "Left-Bias," "Right-Bias" or "Satire"; and
• "Unknown," for sites on no list (a count that is inching downward with time).

For example, Fox News and the Drudge Report are classified as OK, with their "right bias," according to a whitepaper on the project, while Breitbart and TruthFeed are considered iffy because they use questionable sources.

5) The Iffy Quotient is calculated by dividing the URLs from iffy sites by the total number of sites to derive a percentage. The current Facebook count, for example, means that 3.6 percent of the top sites on that platform are sharing dubious information. However, to smooth the graph, the site applies a seven-day moving average. A "raw" quotient is computed for each date, but the index reports an average of the raw Iffy Quotients for the previous seven days.

A dashboard displays the Iffy Quotients dating back to 2016, and it will be updated regularly, university officials said.

Besides showing the bump up in misinformation around the 2016 election, the service also indicated that both Facebook and Twitter have made progress since early 2017 on efforts to dampen misinformation — but one (Facebook) has succeeded more than the other. While the sites' "Iffy coefficients" were about the same in most of 2018, Facebook's is currently a bit lower. In fact, the platform has returned to its early 2016 levels. Twitter, on the other hand, hasn't shrunk much, and it's nearly twice as high as it was in early 2016.

"By contrast with the current environment of accountability by 'gotcha' examples of bad outcomes, the Iffy Quotient tells us something about the overall performance of the platforms," said Paul Resnick, founder and acting director of the center, and associate dean for research at the School of Information, in a statement. "The platforms can track metrics internally with their own data, but hesitate to report them externally. By publishing continuously, we can provide accountability when things get worse and credibility for claims of progress."

The service isn't foolproof, Resnick warned. For example, even shaky sites sometimes publish solid information that will still be classified as unreliable. Also, NewsWhip can miss popular URLs, especially those from "newer fly-by-night sites," which may not be listed on Media Bias/Fact Check yet. Finally, NewsWhip's URLs include some unrelated to news or public affairs; those are expected to be filtered out in a future release of the tool.

Because of these limitations, the researchers involved in the project advised keeping the index in perspective. "Trends in the Iffy Quotient over time are meaningful," Resnick said. "The absolute number...not so much."

People interested in tracking dramatic changes in the index can sign up on the site to be notified. Registered users will also receive alerts when the Center adds new sources and other social platforms to the tool.

A whitepaper about the project is openly available on the university website.