As part of my doctoral study, I aimed to develop an emotionally intelligent AI conversational agent. To achieve this, I began by investigating methods for real-time human emotional state detection using technology. One such tool I explored was DeepFace by Facebook’s AI team, which employs deep learning to analyze facial expressions from a live camera feed to assess emotional states.
Using Python, I built a system with a browser interface using Flask. The system relied on the deepface Python library for emotional detection, OpenCV for handling the image stream, SocketIO for client-server communication, and numpy for data manipulation. The main challenge was processing the images from the stream to enable emotional detection and managing client-side behaviors with JavaScript, including configuring the browser for video streaming and displaying real-time emotional states with HTML.
Normally, DeepFace requires training a model on various emotional states, but due to time and resource constraints, I used a pre-trained model such as the freely available VGG and OpenFace. The trained model’s weights were saved in an “emotion_model.h5” file, a common format for deep learning model weights. Deployment was done through Flask’s sandbox server to simplify the process.
Initially, the system could only report a single emotion at a time from the live video stream. However, for my research, I aimed to report multiple emotions simultaneously with visual representations of each emotion’s intensity. JavaScript was used to calculate emotion percentiles and simple CSS styling enhanced the visual presentation. See the completed interface below:
Below, you’ll find the logic of the app I developed. Keep in mind that to run or deploy the app, you’ll need to ensure that all dependencies are installed, organize the directory accordingly, and have your own pre-trained model ready, as the provided files are not structured for direct duplication. The script is provided solely for understanding the system’s logic.
Reflections
After integrating a deepface-based emotional state reporting system using facial recognition within my AI conversational agent, I have decided to augment, rather than replace, this method with an additional tool for assessing users’ emotional states. This decision was influenced by the realisation that the facial recognition system, while innovative, sometimes struggles to accurately capture the full spectrum of emotional nuances during human interactions. Our initial testing indicated that not all individuals express their emotions vividly through facial expressions, especially in serious conversations, and that the deepface system may not consistently detect subtle emotional cues.
To address this, I am introducing a self-reporting tool that allows users to convey their emotional state using a scale. This provides a more straightforward and precise way for users to express their feelings. I believe this dual approach will enhance the system’s ability to understand and respond to user emotions more effectively.
