NOMA: Dynamic Neural Networks with Compiler Integration

NOMA, or Neural-Oriented Machine Architecture, is an experimental systems language and compiler designed to integrate reverse-mode automatic differentiation as a compiler pass, translating Rust to LLVM IR. Unlike traditional Python frameworks like PyTorch or TensorFlow, NOMA treats neural networks as managed memory buffers, allowing dynamic changes in network topology during training without halting the process. This is achieved through explicit language primitives for memory management, which preserve optimizer states across growth events, making it possible to modify network capacity seamlessly. The project is currently in alpha, with implemented features including native compilation, various optimizers, and tensor operations, while seeking community feedback on enhancing control flow, GPU backend, and tooling. This matters because it offers a novel approach to neural network training, potentially increasing efficiency and flexibility in machine learning systems.

NOMA, or Neural-Oriented Machine Architecture, introduces a novel approach to neural network training by integrating reverse-mode automatic differentiation as a compiler pass. This system, which compiles Rust code to LLVM IR, aims to make gradient-based training a fundamental system operation, resulting in standalone native binaries. This is a significant departure from traditional Python frameworks like PyTorch and TensorFlow, where neural networks are structured as object hierarchies. In those frameworks, altering the network topology mid-training involves stopping the loop, rebuilding objects, and resuming training—a cumbersome process. NOMA simplifies this by treating the network as a managed memory buffer, allowing dynamic changes in network capacity through language primitives like alloc, realloc, and free.

The ability to dynamically adjust network topology during training without rewriting the training loop or reconstructing model objects is a game-changer. This feature is particularly beneficial for experiments involving dynamic capacity, such as neuroevolution or growing/pruning networks. By mapping gradients to new layouts and preserving optimizer states across growth events, NOMA ensures continuity in training even as the network structure changes. This approach not only streamlines the training process but also opens up new possibilities for adaptive learning systems that can evolve over time without manual intervention.

Despite its innovative approach, NOMA is still in its alpha stage, with several limitations and areas for improvement. Currently, it supports a single numeric type, lacks a module system for multi-file programs, and has limited control flow capabilities. Additionally, while there is a micro-benchmark demonstrating faster performance than a Python baseline, the focus remains on correctness and compiler design rather than definitive speed improvements. The project is open to contributions, particularly in enhancing the LLVM backend, expanding GPU support, and developing a standard library with higher-level operations and better numerics.

The implications of NOMA’s approach are profound for the future of machine learning systems. By treating neural networks as dynamic, memory-managed entities, it aligns more closely with the principles of systems programming, potentially leading to more efficient and flexible training processes. This matters because it could pave the way for more scalable and adaptive machine learning models that can better handle complex, real-world tasks. As the project evolves, it may inspire new methodologies in compiler design and machine learning, encouraging further exploration into compiler-first autodiff and explicit memory/topology semantics. The community’s feedback and contributions will be crucial in shaping the development of this promising technology.

Read the original article here