We live in an age of misinformation in which it is becoming increasingly difficult to discern truth from falsehood. Part of the blame rests with technological advances, conceived in the advent of internet bubbles; neural networks which can fabricate deep-fake videos and bots which flood the internet with fake news. In other cases the blame lies with individual people who intentionally spread misleading or outright deceitful messages, because they personally benefit from confusion about what is real and what is not. One of these people is the current President of the United States. Mr. Trump routinely discredits news sources which are widely regarded as trustworthy. Some examples of news outlets he labelled as “fake news media” are The New York Times, CNN, and The Washington Post. It’s hardly a coincidence that there is a large overlap between sources which are deemed fake and those which disagree with Mr. Trump’s political views.
To amplify his -often misleading- statements, Mr. Trump regularly takes to Twitter as a platform. For example, he has falsely claimed that mail-in ballots lead to a fraudulent election (they do not), and that he fired Marine Corps general James Mattis (he resigned). In Twitter’s defense, the company has labelled several tweets of Mr. Trump as containing false information. Besides the accuracy of Mr. Trump’s tweets, the style of writing is often exceptionally un-presidential with myriad exclamation marks and random capitalization. Misleading content written in an outlandish style has become his distinctive signature, and has incentivized many to start parody Twitter accounts which jokingly mimic Mr. Trump. This has created yet another blurred boundary between truth and fiction on top of an already opaque situation. At times, I have been hard pressed to determine whether a tweet was real or a parody. That’s when I decided to see if a machine learning algorithm would be able to make this distinction for me.
I scraped 1000 tweets from Mr. Trump himself and 500 tweets from two parody accounts (@realDonaldTrFan and @RealDonalDrumpf) to train a machine learning algorithm (see the Github repository for details) to tell the difference between the real and the parody tweets. The trained classifier performed remarkably well and was able to correctly classify 88% of new tweets as either real or fake. Unfortunately it doesn’t quite reach human-level performance yet. My girlfriend was kind and bored enough to sit behind a laptop and classify 100 tweets for me, and had 96% correct. Only twice did I hear “He actually **expletives** said that?!”.
I took the trained classifier and put it into a Twitter bot; @real_fakeTrump was born. The bot shows you a tweet from either Mr. Trump himself or one of the parody accounts, you then have to guess if it’s real or fake. In the thread the bot then tells you its own prediction, the correct answer, and a link to the original tweet. Let’s dive in and look at a couple of examples, first a tweet which the classifier correctly identified then two where it was mistaken. The latter case is the most interesting because they are either parody tweets that could be real or tweets from Mr. Trump that are so absurd they cannot be distinguished from parody.
Let’s give it a go shall we? What do you think of this one:
1. 8 Proven Ways to Use Chatbots for Marketing (with Real Examples)
2. How to Use Texthero to Prepare a Text-based Dataset for Your NLP Project
If you thought this tweet was fake, you were sadly mistaken. The classifier correctly identified this tweet as real (but only with 55% certainty), it is indeed from Mr. Trump himself. Let’s try another one:
This is an interesting example of where the classifier made a mistake; it identified this tweet as fake but it was actually from Mr. Trump. This means that this tweet is very close to parody regarding choice of words and sentence construction. However, because the algorithm builds a high-dimensional model to make its predictions it is unfortunately not possible to pin down exactly how it reached its conclusion. OK, last one:
You might have guessed that this one is fake, it was indeed from the parody account @RealDonalDrumpf. The classifier, however, thought this one was so close to Mr. Trump’s style of tweeting that it predicted it was real.
At the moment, deep-fake videos and fake news articles written by bots are relatively easy to spot, although in some cases it is already impossible. Take for example the college student who created a productivity blog with content that was generated by the language model GPT-3. Nobody realized the blogs weren’t written by a human and it reached the top of Hacker News. As technology advances, it will become increasingly difficult to discern truth from fiction. Ironically, we might need to depend on AI to make this distinction for us.
