Soft Robots and Neuromorphic Circuits—Utilizing Reinforcement Learning to Move Towards All-Embracing Intelligence

Besides drinking nectar, the hummingbird needs to hunt for insects to consume enough nutrients. However, its long beak would prevent the hummingbird from effectively catching its prey. Evolution’s solution is to allow the bottom part of the beak to curl up and snap shut. The inherent passive intelligence of a system, in this case the hummingbird’s beak, is what we call embodied intelligence. The field of soft robotic systems has approached embodied intelligence by designing inherently adaptive actuators, which use compliant, soft materials. These soft robotic systems are promising candidates for use in non-conventional environments, such as the exploration of coral reefs, and for complex applications, such as fruit picking. However, the behavior of these soft robotic systems is typically pre-programmed and subsequently is not capable of actively adapting the system’s behavior, yet the hummingbird needs to make active decisions to be able to catch its prey.

We want a centralized intelligent system, for example our brain, to work in tandem with our soft robot. Typical learning systems, such as conventional AI, are not suitable because they are too big and weigh too much, or require remote communication with the robot, thus limiting its applications. Organic neuromorphic circuits could tackle this problem, for which organic electrochemical transistors (OECT) have shown to be promising components. These transistors have the ability to control stable conductance states over a large range, which can be used to emulate the synaptic weight in artificial neural networks. This ability is an important component in achieving local learning for circumventing the von Neumann bottleneck, reducing operation power and reducing the size and weight of the centralized intelligent system. Moreover, OECTs can be made flexible to avoid limiting the soft robotic design space. Conversely, the embodied intelligent robot confines the extensiveness of its control, as some behavior is inherent to the robot and does not have to be learned.

Our goal is to integrate local neuromorphic circuits in soft robotic systems. We present a reinforcement learning algorithm that makes use of two networks: the actor and pseudo-environment. We show that with this algorithm we can achieve memory over multiple cycles, which is required for reinforcement learning, but difficult to achieve in OECTs. We lose the memory of the initial conductance of the transistor, while changing its conductance. Moreover, we can see that the behavior of the algorithm consists of two phases: an explorative and exploitative phase. Both phases are important for the robot to reach its optimal behavior. With this algorithm we create a tool that allows communication between an embodied intelligent and centralized intelligent system, thereby moving closer to all-embracing intelligence that the hummingbird shows when hunting for its prey.