SNS V11.28: Quantum Noise in Spiking Neural Networks

The SNS V11.28 introduces a novel approach to computation by leveraging physical entropy, including thermal noise and quantum effects, as a computational feature rather than a limitation. This architecture utilizes memristors for analog in-memory computing and quantum dot single-electron transistors to inject true randomness into the learning process, validated by the NIST SP 800-22 Suite. Instead of traditional backpropagation, it employs biologically plausible learning rules such as active inference and e-prop, aiming to operate at the edge of chaos for maximum information transmission. The architecture targets significantly lower energy consumption compared to GPUs, with aggressive efficiency goals, though it’s currently in the simulation phase with no hardware yet available. This matters because it presents a potential path to more energy-efficient and scalable neural network architectures by harnessing the inherent randomness of quantum processes.

The exploration of Stochastic Neuromorphic Systems (SNS) presents an intriguing shift from traditional neural network approaches, particularly in the context of bypassing deterministic limitations like the Memory Wall and high energy costs. By leveraging physical entropy, including thermal noise and quantum effects, this architecture aims to treat these elements not as obstacles but as integral features of computation. The use of memristors for analog in-memory computing and quantum dot single-electron transistors as sources of true randomness marks a significant departure from conventional deterministic floating-point operations. This approach could potentially revolutionize how we think about computation, especially in terms of energy efficiency and overcoming traditional bottlenecks.

At the algorithmic level, the system’s reliance on biologically plausible learning rules such as Active Inference, Tsallis Entropy, and e-prop for recurrent spiking neural networks (SNNs) represents a novel approach to learning. These methods aim to minimize variational free energy and operate at the edge of chaos, ensuring maximum information transmission. This is a stark contrast to the classic backpropagation method, which relies on “backprop through time.” The emphasis on self-organized criticality, where the system operates at the brink of chaos, is particularly fascinating as it suggests a dynamic balance that could optimize learning and adaptability.

The efficiency targets set by this architecture are ambitious, with the goal of achieving less than 5 picojoules per synaptic operation on-chip and under 500 picojoules per SOP system-wide, including cooling. This is considerably lower than the energy consumption of current GPU systems when factoring in overheads. The potential for such energy efficiency is a significant draw for those interested in the physics of large language models (LLMs) and neural networks. If successful, this could lead to more sustainable and cost-effective computational models, especially as the demand for AI and machine learning continues to grow.

Despite the promising theoretical framework, the reality check shows that this is still in the simulation phase, with no physical hardware yet available. The tape-out is planned for a future version, and the current work is being conducted through simulations using tools like Nengo and Python. The explicit distancing from claims of artificial general intelligence (AGI) or consciousness is a prudent step, focusing instead on functional metrics. The mention of potential bio-hybrid systems involving organoids is an exciting but ethically complex avenue that remains in the early stages of review. The integration of quantum noise for stochastic resonance to escape local minima in loss landscapes is an exciting prospect, but it remains to be seen whether Active Inference can scale effectively at the hardware level compared to traditional methods like backpropagation on GPUs. This exploration could redefine the boundaries of computational efficiency and capability.

Read the original article here