To meet the exponential growth of AI, Neural Processing Units (NPUs) have become the cornerstone of modern computing infrastructure. The continued evolution of AI relies heavily on the advancement of NPUs, which demands collaborative efforts from both academia and industry. Currently, a gap exists between the exploration of new ideas in academia and the state-of-the-art implementation of industry. While academia continuously proposes novel systems and architectures, the lack of an open, industry-standard baseline limits the community’s ability to validate and integrate these innovations into production-grade systems. We need an open-source NPU ecosystem for the community to innovate collaboratively across the entire stack.

In this tutorial, we will present a roadmap for building an open-source ecosystem for NPU design and implementation. We first introduce the architecture and software stack of modern NPUs and highlight the missing pieces for an open NPU ecosystem. Then, we survey the open tool landscape for NPU development, including architectural design, compiler stack, system software, and hardware prototypes, discussing the capabilities and limitations of existing tools. To facilitate NPU architecture design, we will introduce NeuSim and discuss how users can utilize it to perform fast architectural design space exploration of NPU chips. Finally, we use Google TPU stack as a strong case study to demonstrate the importance of building the NPU ecosystem across the full stack. The tutorial concludes with perspectives on future research and community efforts toward an open, extensible, and production-relevant NPU ecosystem.