Artificial Thermomechanical Receptor for In-Sensor Multimodal Fusion

As wearable devices evolve in complexity, there's a surging demand for advanced multimodal sensing technologies. Especially the acquisition of thermodynamic information is essential, which allows human-like perception, sensor calibration, and hazard detection. While previous studies have succeeded in selectively capturing temperature and mechanical signals, these methods typically required individual measuring units of target stimuli, leading to increased structural complexity, and necessitating extra computational processes for signal matching.<br/>To address these limitations of traditional multimodal sensors, we introduce the neuromorphic sensor for in-sensor multimodal fusion. This innovative design is inspired by the sensory nerve system's efficient computing architecture, which represents multimodal information by interleaving signals across time. To replicate this neural-like computing architecture, we utilized a stretchable memristive nanowire network (the Ag@Cu₂O core-shell nanowires) as the sensing material. This random network, with numerous memristive junctions of a metal-metal oxide-metal configuration, experiences a binary phase transition related to material and network geometry during the memristive switching process. In the high-resistance state, the conductivity pathway primarily goes through a dense network of Cu₂O shells, while in the low-resistance state, it predominantly goes through a sparse network of conductive filaments. This unique feature empowers active alteration in sensing capability, allowing for the selective extraction of thermal and mechanical signals through a singular resistance measurement unit. Furthermore, by carefully manipulating the signal measurement voltage pulse scenario, thermomechanical information is integrated into a singular interleaving response profile. It enables the spatiotemporal synchronization of thermal and mechanical information itself without the necessity of an external computing unit.<br/>Owing to its design simplicity, our neuromorphic multimodal sensor is much more miniaturized than previous work, achieving a sensing channel size of under 0.1mm² and a device thickness of less than 40 μm. This advancement not only improves its conformability on curved surfaces but also ensures rapid response to external stimuli. To demonstrate the practical application of realizing human-like perception, we integrated our sensor with the deep neural network and recognized random objects. In-sensor multimodal fusion, which captures complex physical characteristics, shows superior recognition accuracy compared to datasets acquired by measuring single information (thermal information or mechanical information). These findings open new opportunities for wearable electronics with wearability, and functionality and pave the way for intelligent multimodal sensor systems.