Reconfigurable Floating Gate MoS₂ Phototransistor for In-Sensor Multimodal Fusion

In-sensor computing overcomes the restriction of transmitting massive data between sensors and processing units, thereby enabling more effective edge computing. Multimodal perception can capture more precise and comprehensive information compared to unimodal approaches. However, current sensory systems typically merge multimodal signals at computing terminals following parallel processing and transmission, which results in the potential loss of spatial association information and requires timestamps to maintain temporal coherence for time-series data. The floating gate MoS₂ phototransistors with reconfigurable photoresponsivity offer the agile optoelectronic properties to realize bioinspired in-sensor multimodal fusion for extracting spatial and spatiotemporal association, which effectively enhances the imformation processing at sensory terminals. For nonvolatile mode, we realize an optimal spatial estimation by combining visual-tactile signals through the light-assisted Fowler-Nordheim tunneling effect. For volatile mode, with ample trapping centers in MoS₂phototransistors, we extract spatiotemporal information by real-time capturing and fusing visual-audio signals, realizing a dance-music synchronization recognition task without the additional timestamping process. This in-sensor multimodal fusion approach provides the potential to simplify the multimodal integration system, extending the in-sensor computing paradigm.