How Twitter Uses Big Data And Artificial Intelligence (AI)

How Twitter Uses Big Data And Artificial Intelligence (AI)

Twitter is a social media platform where 328 million monthly active users microblog (share 280-character updates) with their followers. It’s a cross between instant messaging and blogging—or social messaging, but it’s also been crucial for news reporting, event promotion, marketing and business. Whether you’re a fan or not, Twitter has become the ninth largest social network in the world and Cortex, Twitter’s in-house engineering team, has turned to the power of artificial intelligence (AI) to help enhance the platform’s user experience.

How Twitter uses AI in practice

Billionaire Mark Cuban said in 2017 he purchased Twitter stock because “they finally got their act together with artificial intelligence.” In 2016, Twitter purchased Magic Pony Technology to bulk up Cortex, who wants to “build the most advanced AI platform in the world, at Twitter scale, to apply the most complex AI algorithms to our most challenging datasets, seamlessly.”

One of the ways Twitter uses artificial intelligence is to determine what tweet recommendations to suggest on users’ timelines with the goal of highlighting the most relevant tweets for every individual. Prior to this shift, Twitter would show its users tweets in reverse chronological order. Today, the algorithms scan and score thousands of tweets per second to rank them for every user’s feed.

The social media platform also deployed AI to fight inappropriate and racist content on its platform as authorities in the UK and Germany increase measures and fines to prevent hate speech, fake news and illegal content on social media. In the first six months of 2017, Twitter took down nearly 300,000 terrorist accounts which were identified by its AI tools; in fact, 95 percent of suspended terrorism-related accounts were identified by algorithms rather than human users.

Artificial intelligence tools are also supporting small tweaks to Twitter that improve the overall user experience. For example, its image cropping tools are using AI to automatically crop images in a much more appealing way and are monitoring live video feeds and categorising them based on subject matter to improve their searchability and to help the algorithms identify videos users might be interested in seeing in their feeds.

Technical details

Twitter’s ranking algorithm has taken in lots of data, processed it through deep neural networks and learned over time what content would be relevant for each individual user. All tweets are scored on a ranking model that is used to determine the probability if a user would value that content in their feed. The ranking model considers the content of the tweet itself including if it is accompanied with an image or video and how many retweets or likes it has received; the author of the tweet to see if you had any past interactions with the author and the strength of your connection to the author; and considers the type and tone of tweets you have a history of liking in the past and how this tweet resembles others that you seemed to appreciate. The higher the relevancy score, the higher on your feed you will see the tweet and the probability the tweet will appear in the “In case you missed it” module.

As Twitter continues to refine its algorithms, it has to balance the speed as well as quality of the algorithm’s review process to meet the requirements of the platform for real-time updates. In addition to the speed and quality of predictions, algorithms are assessed on their resource demands and maintainability over time.

The company uses IBM Watson and its natural language processing skill-set to track and remove abusive messages since the AI tech of Watson not only understands natural language but can infer intonation and extract meaning from images quickly—it can analyse millions of tweets in a second.

Twitter’s AI tools “crop using saliency” to show the most interesting aspect of images whether they are faces or not. In order to train the tool, the Twitter team defined what is most salient by using data from academics that studied eye tracking. Then, in order to optimise the tool in real time on the site, they used AI to train a quicker version of the tool to speed up the crops. First, they trained a smaller network by using the first program that was good, but slow. Then, the software engineers streamlined the process by removing the less important visual cues on images.

In order to train its algorithm to recognise what’s happening on a live feed, Cortex used deep learning. They trained a large neural network to recognise the content on video from a large number of examples. Humans watched videos and tagged them with several keywords to identify what they saw. So, a video of a dog wasn’t just tagged with the keyword dog, but also animal, canine, mammal and more. That data was then used to train the algorithm so it could then identify content in video.

Ideas and insights you can steal

As Twitter’s team illustrates, tools powered by artificial intelligence can be used to improve or enhance your products and services. The collaboration between humans and artificial intelligence help to accentuate the best attributes of each.



Written by

Bernard Marr

Bernard Marr is a world-renowned futurist, influencer and thought leader in the field of business and technology. He is the author of 18 best-selling books, writes a regular column for Forbes and advises and coaches many of the world’s best-known organisations. He has 2 million social media followers and was ranked by LinkedIn as one of the top 5 business influencers in the world and the No 1 influencer in the UK.

Some Of Our Customers

Connect with Bernard Marr