Is AI’s next leap understanding emotion? $50M for Hume says yes

Why understanding human emotion is key to providing better AI experiences

Carrying on emotionally aware voice conversations with human users may like a simple enough task for an AI assistant in the year 2024, but it is actually a massively complex, nuanced, and difficult undertaking, as Hume AI doesn’t just want to understand if users are feeling “happy,” “sad,” “angry,” “afraid” or any of the five-to-seven “universal” human emotions across cultures categorized from facial expressions by PhD psychologist Paul Ekman.

No, Hume AI seeks to understand more nuanced and often multidimensional emotions of its human users. On its website, the startup lists 53 different emotions it is capable of detecting from a user, including:

1. Admiration
2. Adoration
3. Aesthetic Appreciation
4. Amusement
5. Anger
6. Annoyance
7. Anxiety
8. Awe
9. Awkwardness
10. Boredom
11. Calmness
12. Concentration
13. Confusion
14. Contemplation
15. Contempt
16. Contentment
17. Craving
18. Desire
19. Determination
20. Disappointment
21. Disapproval
22. Disgust
23. Distress
24. Doubt
25. Ecstasy
26. Embarrassment
27. Empathic Pain
28. Enthusiasm
29. Entrancement
30. Envy
31. Excitement
32. Fear
33. Gratitude
34. Guilt
35. Horror
36. Interest
37. Joy
38. Love
39. Nostalgia
40. Pain
41. Pride
42. Realization
43. Relief
44. Romance
45. Sadness
46. Sarcasm
47. Satisfaction
48. Shame
49. Surprise (negative)
50. Surprise (positive)
51. Sympathy
52. Tiredness
53. Triumph

Hume AI’s theory is that by developing AI models capable of a more granular understanding and expression of human emotion, it can better serve users — as a “willing ear” to listen and work through their feelings, but also providing more realistic and satisfying customer support, information retrieval, companionship, brainstorming, collaboration on knowledge work, and much more.

As Cowen told VentureBeat in an email sent via a spokesperson from Hume AI:

“Emotional intelligence includes the ability to infer intentions and preferences from behavior. That’s the very core of what AI interfaces are trying to achieve: inferring what users want and carrying it out. So in a very real sense, emotional intelligence is the single most important requirement for an AI interface.

With voice AI, you have access to more cues of user intentions and preferences. Studies show that vocal modulations and the tune rhythm and timbre of speech are a richer conduit for our preferences and intentions than language alone (e.g., see https://pure.uva.nl/ws/files/73486714/02699931.2022.pdf).

Understanding vocal cues is a key component of emotional intelligence. It makes our AI better at predicting human preferences and outcomes, knowing when to speak, knowing what to say, and knowing how to say it in the right tone of voice.”

How Hume AI’s EVI detects emotions from vocal changes

How does Hume AI’s EVI pick up on the cues of user intentions and preferences from vocal modulations of users? The AI model was trained on “controlled experimental data from hundreds of thousands of people around the world,” according to Cowen.

On its website, Hume notes: “The models were trained on human intensity ratings of large-scale, experimentally controlled emotional expression data” from methods described in two scientific research papers published by Cowen and his colleagues: “Deep learning reveals what vocal bursts express in different cultures” from December 2022 and “Deep learning reveals what facial expressions mean to people in different cultures” from this month.

The first study included “16,000 people from the United States, China, India, South Africa, and Venezuela” and had a subset of them listen to and record “vocal bursts,” or non-word sounds like chuckles and “uh huhs” and assign them emotions for the researchers. The participants were also asked this subset to record their own vocal bursts, then had another subset listen to those and categorize those emotions, as well.

The second study included 5,833 participants from the same five countries above, plus Ethiopia, and had them take a survey on a computer in which they analyzed up to 30 different “seed images” from a database of 4,659 facial expressions. Participants were asked to mimic the facial expression they saw on the computer and categorize the emotion conveyed by the expression from a list of 48 emotions, scaled 1-100 in terms of intensity. Here’s a video composite from Hume AI showing “hundreds of thousands facial expressions and vocal bursts from India, South Africa, Venezuela, the United States, Ethiopia, and China” used in its facial study.

Hume AI took the resulting photos and audio of participants in both studies and trained its own deep neural networks on them.

Hume’s EVI itself told me in an interview I conducted with it (disclaimer that it’s not a person and its answers may not always be accurate, as with most conversational AI assistants and chatbots) that Hume’s team “collected the largest, most diverse library of human emotional expressions ever assembled. We’re talking over a million participants from all around the world, engaged in all kinds of real-life interactions.”

According to Cowen, the vocal audio data from participants in Hume AI’s studies was also used to create a “speech prosody model, which measures the tune, rhythm, and timbre of speech and is incorporated into EVI” and which convey up to “48 distinct dimensions of emotional meaning.”

You can see — and hear — an interactive example of Hume AI’s speech prosody model here with 25 different vocal patterns.

The speech prosody model is what powers the bar graphs of different emotions and their proportions displayed helpfully and in what I found to be a thoroughly engaging manner on the right hand sidebar of Hume’s EVI online demo site.