The Next Frontier Of Artificial Intelligence: Building Machines That Read Your Emotions
2 July 2021
The blueprint for today’s artificial intelligence, neural network and machine learning technology is the human brain, simply because it is the most effective tool for solving problems that we know of.

However, there is a big part of the puzzle missing – the aspect of human intelligence that we know as emotional intelligence, or empathy. Emotional intelligence is what allows us to take the feelings and considerations of other people into account in the solutions we make. And so far, progress here has been limited. Alexa, helpful as she may be in some circumstances, will not consider your feelings, or those of others affected by her actions, as she keeps your smart home running smoothly.
But all that could be about to change – in fact, has to change, if AI is to reach its potential as a tool for assisting in our business and day-to-day lives. Recently, emotion-focused AI developer Affectiva became one of the few small businesses to be asked to join the Partnership on AI to Benefit People and Society. The interest of the “grand masters” of AI which make up the partnership – Google, Microsoft, Facebook, etc – in Affectiva’s growing business is a sure sign that this overlooked aspect of AI is starting to get the attention it deserves.
Affectiva founder and CEO Rana el Kaliouby talked to me about her company’s work to develop what she calls “multi-modal emotion AI”. There may already be a growing understanding of how sentiment analysis can help machines understand how humans are feeling, and adapt their behaviour accordingly. But a great deal of valuable data which communicate our emotional state is lost if machines cannot also read our expressions, gestures, speech patterns, tone of voice or body language.
“There’s research showing that if you’re smiling and waving or shrugging your shoulders, that’s 55% of the value of what you’re saying – and then another 38% is in your tone of voice.
“Only seven per cent is in the actual choice of words you’re saying, so if you think about it like that, in the existing sentiment analysis market which looks at keywords and works out which specific words are being used on Twitter, you’re only capturing 7% of how humans communicate emotion, and the rest is basically lost in cyberspace.”
Affectiva’s Emotion AI technology is already in use by 1,400 brands worldwide, including Amazon and Kellogg’s, many of whom use it to judge the emotional effect of adverts by asking viewers to switch on their cameras while video plays. The facial images are analysed with deep learning algorithms which accurately classify them according to the feelings of the viewer.
Another application of the technology could be important in ensuring that automated cars – particularly level 2 and 3 autonomous vehicles – can safely assess the condition of the driver (for example – are they awake and sober?) before switching from automated to manual driving mode.
“Emotional intelligence is how you understand yourself and the people around you, and it is just as important as cognitive, or rational intelligence, to how we make decisions,” el Kaliouby tells me.
“People who don’t have that often really struggle with operating effectively in a world where other people live.”
So while it’s true that machines have been, by our own standards, largely sociopathic until now – unable to comprehend feelings and take them into account – that could be set to change.
“I’m a big believer that in the next three to five years, this is going to be ubiquitous, and our devices will have ‘emotion chips’ and a variety of sensors for the state of people around it – it’s a natural evolution from the mouse and keyboard we used to use, to touch interfaces, and most recently the way devices have become conversational. The next stage of that evolution is for devices to become perceptual.”
Taking a seat on the Partnership of course obliges Affectiva to deeply assess the moral and ethical implications of the tech they are developing, but el Kaliouby says that has always been a cornerstone of her company’s activity.
“Privacy is a very big issue, because emotions are obviously very personal data and you can easily see how it could be abused to manipulate people – for example to manipulate voters. We have always been very definite that we are big on consent, and for everything we do, people have to opt in.”
The company has even gone out of its way to avoid working with personal data gained in non-consensual circumstances – for example in security or military applications, even if those fields clearly offer opportunities.
“Bias is another issue … bias can be built into the data, there are algorithms that detect not just your emotions but your age, gender, ethnicity – we are very thoughtful and careful about how we use that data.”
We will surely hear a lot more about machines that can read our emotions, so stay tuned.
Related Articles
What Does ChatGPT Really Mean For Businesses?
ChatGPT, a new artificial intelligence chatbot, has taken the internet by storm. This tool is the latest example of AI-based tools that are augmenting the way we do business.[...]
ChatGPT: Everything You Really Need To Know
Unless you’ve been living off-grid for the past few weeks, you’ve probably noticed that the internet is abuzz with excitement about ChatGPT.[...]
The 10 Best Examples Of Low-Code And No-Code AI
Artificial intelligence (AI) can transform just about any business – from providing customers with products and services they really want to streamlining internal processes.[...]
The Top 4 Telecom Trends In 2023
The next 12 months look set to be a bumpy year for many global markets, and communications service providers (CSPs) will not be entirely insulated from the volatility.[...]
The Top 5 Healthcare Trends In 2023
The world is a very different place than it was ten years ago, and nowhere is this more evident than in healthcare.[...]
The Top 10 Tech Trends In 2023 Everyone Must Be Ready For
As a futurist, it’s my job to look ahead — so every year, I cover the emerging tech trends that will be shaping our digital world in the next 12 months.[...]
Stay up-to-date
- Get updates straight to your inbox
- Join my 1 million newsletter subscribers
- Never miss any new content
Social Media