Researchers detect complex emotions by combining multiple optical signals
About Optica
17 February 2026
Researchers detect complex emotions by combining multiple optical signals
Portable system that combines fiber-based physiological sensing and facial thermal imaging could help monitor mental health and emotional states
Caption: Researchers developed a new way to recognize human emotions by combining fiber-based physiological signals with thermal images of the face.
Credit: Rui Min, Beijing Normal UniversityWASHINGTON — Researchers have developed a new way to recognize human emotions by combining fiber-based physiological signals with thermal images of the face. The portable emotional recognition system could eventually be used to support at-home mental health monitoring, improve driver safety and make technology more responsive to human emotions.
“Unlike many existing approaches, our method does not rely on facial expressions, which can be consciously controlled or exaggerated,” said research team leader Rui Min from Beijing Normal University at Zhuhai (BNU Zhuhai) in China. “Instead, it focuses on natural bodily responses that occur automatically. By combining multiple types of optical sensing and higher-level physiological interpretation, our system can more accurately capture emotion than those using a single type of input.”
In the Optica Publishing Group journal Biomedical Optics Express, the researchers describe the multimodal emotion recognition system, which achieved an accuracy of about 93 percent when identifying fear, happiness and relaxation in volunteers. It detects small heartbeat- and breathing-related chest movements using a wearable polymer optical fiber cardiorespiratory sensor and combines this data with subtle changes in facial temperature captured using a thermal imaging camera.
Caption: The sensor detects breathing and heartbeats by measuring how subtle chest movements change the path of light in the fiber. These optical changes were recorded with a smartphone.
Credit: Rui Min, Beijing Normal University“Early identification of emotional stress could be particularly valuable in helping people take action before problems become more serious,” said the paper’s first author Yingshuo Bao, a doctoral student under the supervision of Min. “For example, it could enable cars that can sense when a driver is becoming stressed or anxious and adjust alerts or assistance features to help improve safety.”
Combined data provides fuller picture
The researchers have been actively exploring how optical fiber sensors can be combined with other sensing devices to address more complex biomedical problems. This led them to investigate whether integrating fiber-based physiological signals with facial thermal information could provide a more complete description of emotional responses.
“While emotions are closely linked to changes in multiple physiological systems — particularly facial blood flow and cardiopulmonary activity — most existing emotion detection methods still rely on a single type of signal, which limits robustness and accuracy,” said Min. “The key challenge in developing a multimodal system was figuring out how to capture both modalities reliably and fuse them at the data level so that their strengths reinforce each other.”
Caption: Volunteers watched short video clips designed to evoke specific emotions while heartbeat- and breathing-related chest movements were captured with the sensor and facial temperature changes were captured with a thermal imaging camera.
Credit: Rui Min, Beijing Normal UniversityThe system uses a polymer optical fiber cardiorespiratory sensor that the researchers previously developed. The sensor sits in a soft chest strap, allowing comfortable, reliable monitoring without any electrical contacts. It detects breathing and heartbeats by measuring how subtle chest movements change the path of light in the fiber. These optical changes are recorded as image frames using a smartphone and later processed to extract detailed breathing and heart-related patterns.
As the cardiorespiratory sensor is working, the system’s thermal camera also records video of the face. By tracking average temperature changes in specific facial regions, the images can be used to detect subtle thermal variations associated with emotional responses. By using supervised machine learning algorithms to extract, align and integrate features from the fiber-based cardiopulmonary signals and facial thermal data, the researchers were able to more fully capture body changes linked to emotion than is possible using one type of signal.
From light signals to feelings
To demonstrate the new system, the researchers asked volunteers to watch short video clips designed to evoke emotions such as fear, happiness and relaxation while chest motion and facial temperature were recorded. They tested each sensing modality individually and in combination, finding that when both modalities were fused, the approach’s overall reliability for identifying each emotion — considering both misses and false alarms — was very high (F1-scores were 0.93). Most errors involved occasional overlap between fear and happiness.
Caption: The fiber-based cardiorespiratory sensor is incorporated into a wearable chest strap.
Credit: Rui Min, Beijing Normal UniversityNext, the researchers plan to extend and strengthen the system by integrating additional sensing modalities to enable even more comprehensive physiological monitoring. They also plan to test the effectiveness of the level of individual modalities under a wider range of conditions, such as different body movements and environmental settings. After optimization, large-scale and longer-term measurements will be needed to validate the system.
Paper: Y. Bao, S. Yan, J. Huang, Z. Wang, K. Xiao, R. Min, “POF cardiorespiratory sensor combined with a thermal imaging temperature sensor for multimodal emotion recognition,” Biomed. Opt. Express, 17, 1377-1392 (2026).
DOI: 10.1364/BOE.586279
About Optica Publishing Group
Optica Publishing Group is a division of the society, Optica, Advancing Optics and Photonics Worldwide. It publishes the largest collection of peer-reviewed and most-cited content in optics and photonics, including 19 prestigious journals, the society’s flagship member magazine, and papers and videos from over 1200 conferences. With over 505,000 journal articles, conference papers and videos to search, discover and access, its publications portfolio represents the full range of research in the field from around the globe.
About Biomedical Optics Express
Biomedical Optics Express serves the biomedical optics community with rapid, open-access, peer-reviewed papers related to optics, photonics and imaging in biomedicine. The journal scope encompasses fundamental research, technology development, biomedical studies and clinical applications. It is published monthly by Optica Publishing Group and edited by Ruikang (Ricky) Wang, University of Washington, USA. For more information, visit Biomedical Optics Express.
Media Contact
