Dynamic Machine Vision with Retinomorphic Photomemristor-Reservoir Computing

Dynamic machine vision requires recognizing the past and predicting the future of a moving object based on present vision. Current image sensing and machine vision technologies accomplish this by analyzing massive frame-by-frame image sequences in multiple hardware blocks and complex software algorithms, engendering redundant data flows, high energy consumption, and latency. Photomemristors, or optoelectronic memristors, originally proposed for photosensing, processing, and memory functions [1], are ideal candidates for dynamic machine vision tasks. In recent years, photomemristors have been studied in neuromorphic vision and processing systems for static image classification [2-5] and human action recognition [5]. However, motion recognition and prediction within a compact dynamic sensing system, which is crucial for dynamic machine vision technology, has been elusive until very recently. Here, I will summarize a brief history of photomemristor, and show our recent works on photomemristors for sensing, processing, and memory. Then I will focus on the implementation of motion recognition and prediction in recurrent photomemristor networks [6]. In the retinomorphic photomemristor reservoir computing system, a retinomorphic photomemristor array, working as a dynamic vision reservoir, embeds past motion frames as hidden states into the single present frame through inherent dynamic memory. The informative present frame facilitates accurate recognition of past and prediction of future motions with machine learning algorithms. This in-sensor motion processing capability eliminates redundant data flows and promotes real-time perception of moving objects. Potential applications of the retinomorphic photomemristor-reservoir computing system include video analysis, autonomous driving, and robotic vision.<br/><br/>References<br/>1. H. Tan et al. <i>Adv. Mater.</i> <b>27</b>, 2797–2803, 2015.<br/>2. H. Tan et al. <i>Nat. Commun.</i> <b>12</b>, 1120, 2021.<br/>3. F. Zhou et al. <i>Nat. Nanotechnol.</i> <b>14</b>, 776–782, 2019.<br/>4. Y. Meng et al. <i>Sci. Adv.</i> <b>6</b>, eabc6389, 2020.<br/>5. Y. Sun et al. <i>Adv. Intell. </i><i>Syst.</i> <b>5</b>, 2200196, 2023.<br/>6. H. Tan et al. <i>Nat. Commun.</i> <b>14</b>, 2169, 2023.