Explore the vast range of titles available.

HighlightsThe neuromorphic vision sensors for near-sensor and in-sensor computing of visual information are implemented using optoelectronic synaptic circuits and single-device optoelectronic synapses, respectively.This review focuses on the recent progress, working mechanisms, and image pre-processing techniques about two types of neuromorphic vision sensors based on near-sensor and in-sensor vision computing methodologies.The latest developments in bio-inspired neuromorphic vision sensors can be summarized in 3 keywords: smaller, faster, and smarter. (1) Smaller: Devices are becoming more compact by integrating previously separated components such as sensors, memory, and processing units. As a prime example, the transition from traditional sensory vision computing to in-sensor vision computing has shown clear benefits, such as simpler circuitry, lower power consumption, and less data redundancy. (2) Swifter: Owing to the nature of physics, smaller and more integrated devices can detect, process, and react to input more quickly. In addition, the methods for sensing and processing optical information using various materials (such as oxide semiconductors) are evolving. (3) Smarter: Owing to these two main research directions, we can expect advanced applications such as adaptive vision sensors, collision sensors, and nociceptive sensors. This review mainly focuses on the recent progress, working mechanisms, image pre-processing techniques, and advanced features of two types of neuromorphic vision sensors based on near-sensor and in-sensor vision computing methodologies.

Rapid developments in the Internet of Things and Artificial Intelligence trigger higher requirements for image perception and learning of external environments through visual systems. However, limited by von Neumann's bottleneck, the physical separation of sense, memory, and processing units in a conventional personal computer‐based vision system tend to consume a significant amount of energy, time latency, and additional hardware costs. By integrating computational tasks of multiple functionalities into the sensors themselves, the emerging bio‐inspired neuromorphic visual systems provide an opportunity to overcome these limitations. With high speed, ultralow power and strong adaptability, it is highly desirable to develop a neuromorphic vision system that is based on highly precise in‐sensor computing devices, namely retinomorphic devices. We here present a timely review of retinomorphic devices for visual in‐sensor computing. We begin with several types of physical mechanisms of photoelectric sensors that can be constructed for artificial vision. The potential applications of retinomorphic hardware are, thereafter, thoroughly summarized. We also highlight the possible strategies to existing challenges and give a brief perspective of retinomorphic architecture for in‐sensor computing. image

Bioinspired neuromorphic machine vision system (NMVS) that integrates retinomorphic sensing and neuromorphic computing into one monolithic system is regarded as the most promising architecture for visual perception. However, the large intensity range of natural lights and complex illumination conditions in actual scenarios always require the NMVS to dynamically adjust its sensitivity according to the environmental conditions, just like the visual adaptation function of the human retina. Although some opto‐sensors with scotopic or photopic adaption have been developed, NMVSs, especially fully flexible NMVSs, with both scotopic and photopic adaptation functions are rarely reported. Here we propose an ion‐modulation strategy to dynamically adjust the photosensitivity and time‐varying activation/inhibition characteristics depending on the illumination conditions, and develop a flexible ion‐modulated phototransistor array based on MoS2/graphdiyne heterostructure, which can execute both retinomorphic sensing and neuromorphic computing. By controlling the intercalated Li+ ions in graphdiyne, both scotopic and photopic adaptation functions are demonstrated successfully. A fully flexible NMVS consisting of front‐end retinomorphic vision sensors and a back‐end convolutional neural network is constructed based on the as‐fabricated 28 × 28 device array, demonstrating quite high recognition accuracies for both dim and bright images and robust flexibility. This effort for fully flexible and monolithic NMVS paves the way for its applications in wearable scenarios. A flexible phototransistor array that can execute both retinomorphic sensing and neuromorphic computing is developed, demonstrating both scotopic and photopic adaptation functions. Based on this device array, a fully flexible neuromorphic machine vision system consisting of front‐end retinomorphic vision sensors and back‐end convolutional neural networks is constructed, demonstrating high recognition accuracies for dim and bright images.

The development of all‐in‐one devices for artificial visual systems offers an attractive solution in terms of energy efficiency and real‐time processing speed. In recent years, the proliferation of smart sensors in the growth of Internet‐of‐Things (IoT) has led to the increasing importance of in‐sensor computing technology, which places computational power at the edge of the data‐flow architecture. In this study, a prototype visual sensor inspired by the human retina is proposed, which integrates ferroelectricity and photosensitivity in two‐dimensional (2D) α‐In2Se3 material. This device mimics the functions of photoreceptors and amacrine cells in the retina, performing optical reception and memory computation functions through the use of electrical switching polarization in the channel. The gate‐tunable linearity of excitatory and inhibitory functions in photon‐induced short‐term plasticity enables to encode and classify 12 000 images in the Mixed National Institute of Standards and Technology (MNIST) dataset with remarkable accuracy, achieving ≈94%. Additionally, in‐sensor convolution image processing through a network of phototransistors, with five convolutional kernels electrically pre‐programmed into the transistors is demonstrated. The convoluted photocurrent matrices undergo straightforward arithmetic calculations to produce edge and feature‐enhanced scenarios. The findings demonstrate the potential of ferroelectric α‐In2Se3 for highly compact and efficient retinomorphic hardware implementation, regardless of ambipolar transport in the channel. This work demonstrates a retinomorphic prototype that integrates ferroelectricity and photosensitivity within a two‐dimensional (2D) α‐In2Se3 material, emulating simultaneous functions in the human retina, that is, perceptive light‐sensing, memory, and computation. The optoelectronic memory efficiently classifies 12 000 images with a satisfactory precision of 94% and replicates five convolutional kernels using a network of 3 × 3 phototransistors for the image processing task.

Photonic integrated circuits offer miniaturized solutions for multimodal spectroscopic sensory systems by leveraging the simultaneous interaction of light with temperature, chemicals, and biomolecules, among others. The multimodal spectroscopic sensory data is complex and has huge data volume with high redundancy, thus requiring high communication bandwidth associated with high communication power consumption to transfer the sensory data. To circumvent this high communication cost, the photonic sensor and processor are brought into intimacy and propose a photonic multimodal in‐sensor computing system using an integrated silicon photonic convolutional processor. A microring resonator crossbar array is used as the photonic processor to implement convolutional operation with 5‐bit accuracy, validated through image edge detection tasks. Further integrating the processor with a photonic spectroscopic sensor, the in situ processing of multimodal spectroscopic sensory data is demonstrated, achieving the classification of protein species of different types and concentrations at various temperatures. A classification accuracy of 97.58% across 45 different classes is achieved. The multimodal in‐sensor computing system demonstrates the feasibility of integrating photonic processors and photonic sensors to enhance the data processing capability of photonic devices at the edge. The work demonstrates a photonic multimodal in‐sensor computing system by combining a photonic sensor and a photonic processor. By integrating the photonic sensor with a photonic processor performing convolutional operation, the complex spectroscopic data is convolved for classification with a high accuracy, proving the feasibility of photonic in‐sensor computing.

The optoelectronic resistive random‐access memory (RRAM) with the integrated function of perception, storage and intrinsic randomness displays promising applications in the hardware level in‐sensor image cryptography. In this work, 2D hexagonal boron nitride based optoelectronic RRAM is fabricated with semitransparent noble metal (Ag or Au) as top electrodes, which can simultaneous capture color image and generate physically unclonable function (PUF) key for in‐sensor color image cryptography. Surface plasmons of noble metals enable the strong light absorption to realize an efficient modulation of filament growth at nanoscale. Resistive switching curves show that the optical stimuli can impede the filament aggregation and promote the filament annihilation, which originates from photothermal effects and photogenerated hot electrons in localized surface plasmon resonance of noble metals. By selecting noble metals, the optoelectronic RRAM array can respond to distinct wavelengths and mimic the biological dichromatic cone cells to perform the color perception. Due to the intrinsic and high‐quality randomness, the optoelectronic RRAM can produce a PUF key in every exposure cycle, which can be applied in the reconfigurable cryptography. The findings demonstrate an effective strategy to build optoelectronic RRAM for in‐sensor color image cryptography applications. The optoelectronic Au/h‐BN/Au and Ag/h‐BN/Au resistive random‐access memory (RRAM) array is developed to achieve high‐quality in‐sensor color image cryptography. Resistance switching characteristics of RRAM devices can be modulated by optical stimuli due to plasmonic effects of noble metals. The double‐binary physically unclonalbe function keys can be simultaneously generated in the optoelectronic RRAM array via reading the high resistance state.

Conventional software‐based encryption faces mounting limitations in power efficiency and security, inspiring the development of emerging neuromorphic computing hardware encryption. This study presents a hardware‐level multi‐dimensional encryption paradigm utilizing optoelectronic neuromorphic devices with low energy consumption of 3.3 fJ, exhibiting great potential in motion detection, in‐sensor computing and multilevel encrypted information communication. By encoding ASCII characters into unique optical pulse sequences defined by wavelength, duration, and pulse number, the device transforms digital information into physically obfuscated electrical responses, thereby establishing a secure encryption mechanism. Based on neuromorphic response of optoelectronic device, convolutional neural network was trained to decrypt signals with recognition accuracy of 97.4% for legitimate users while maintaining robustness against unauthorized access (∼2.88% accuracy). To address complex real‐world scenarios of maritime communication, dual‐authentication “friend‐or‐foe” identification system was constructed with two‐layer authentication. The neuromorphic optoelectronic system combines motion perception, real‐time flag semaphore recognition via reservoir computing with multi‐band photonic encryption, showing great potential in next‐generation neuromorphic maritime communication.

Organic electrochemical transistors (OECTs) hold potential for in‐sensor computing and wearable healthcare systems. Nevertheless, their inherent limitations in stretchability and conformability hinder their scalability and practical deployment. In a recent study, Liu et al. introduce a wearable in‐sensor computing platform that leverages stretchable OECTs, exhibiting over 50% elongation capability while preserving stable operational performance. This innovation enables the development of wearable systems that can accurately acquire biosignals. The large‐scale manufacturing and application of wearable medical devices have extensive research significance. Liu et al. report a wearable sensor‐based in‐sensor computing platform based on stretchable organic electrochemical transistors that provide more than 50% stretchability while maintaining stable performance, enabling a wearable system prepared with it to achieve high‐precision biosignal acquisition.

The importance of haptic in‐sensor computing devices has been increasing. Herein, a haptic sensor with a hierarchical structure is successfully fabricated via the sacrificial template method, using carbon nanotube‐polydimethylsiloxane (CNT‐PDMS) nanocomposites for in‐sensor computing applications. The CNT‐PDMS nanocomposite sensors, with different sensitivities, are obtained by varying the amount of CNTs. The input stimuli are transformed into higher‐dimensional information, enabling a new path for the CNT‐PDMS nanocomposite application, which is implemented on a robotic hand as an in‐sensor computing device by applying a reservoir computing paradigm. The nonlinear output data obtained from the sensors are trained using linear regression and used to classify nine different objects used in everyday life with an object recognition accuracy of >80% for each object. This approach can enable tactile sensation in robots while reducing the computational cost. Using a porous carbon nanotube‐polydimethylsiloxane nanocomposite, a sensor array integrated with a physical reservoir computing paradigm capable of in‐sensor computing is demonstrated. The device is able to classify between nine objects with an accuracy above 80%, opening the possibility for low‐power sensing/computing for future robotics.

The massive influx of continuous, real‐time environmental data demands highly energy‐efficient and low‐latency sensory processing. Conventional artificial sensory systems are limited by severe data transfer overhead issues due to physically separated processing and memory units, coupled with analog‐to‐digital converters. To resolve these issues, neuromorphic sensory platforms inspired by the biological nervous system have emerged as an innovative paradigm. This Review comprehensively investigates the structural evolution and current research trends of neuromorphic near‐sensor and in‐sensor computing systems. Initially, the fundamental physical mechanisms underlying artificial neurons and synapses are systematically analyzed. Furthermore, the distinct operating principles of optical, mechanical, and chemical sensors corresponding to the five human senses are discussed. To establish a clear structural framework, we systematically categorize neuromorphic‐integrated sensory systems into near‐sensor and in‐sensor computing architectures based on their level of integration. Near‐sensor processing minimizes data movement through system‐level integration, whereas in‐sensor computing executes stimulus transduction and state evolution simultaneously at the device level. Based on this classification, we extensively discuss recent research trends of near‐sensor and in‐sensor computing tailored to each of the five human senses. Ultimately, by identifying domain‐specific bottlenecks, this article provides strategic material and architectural guidelines for realizing fully integrated, next‐generation artificial cognitive systems.