NPU, VPU & LPU: AI Accelerators, Compilers, and Deployment

An NPU is a domain-specific accelerator optimized for the dense linear algebra (matrix-multiply and convolution) at the heart of neural-network inference. Unlike a CPU, which is latency-optimized and general-purpose, and a GPU, which is a throughput-optimized SIMT engine built for both training and inference, an NPU trades generality for extreme energy efficiency (TOPS per watt) on quantized, low-precision math. Most consumer NPUs are IP blocks integrated into a mobile or PC SoC (Apple Neural Engine, Qualcomm Hexagon, Arm Ethos, Intel/AMD PC NPUs), sharing system DRAM and thermal budget with the CPU and GPU. The design point is inferences-per-watt at small batch sizes, not peak FLOPS or large-batch throughput.

TOPS is Tera-Operations-Per-Second, and headline figures are almost always quoted for peak INT8 throughput, scaling roughly with (number of MACs x clock x 2), since a multiply-accumulate counts as two ops. It is unreliable first because it is precision-dependent: INT8 TOPS is not comparable to INT4 or FP16 TOPS, so numbers are only meaningful at equal precision. Second, peak TOPS is a theoretical ceiling that rarely reflects effective throughput (ops actually executed divided by latency), because real workloads are usually memory-bandwidth-bound rather than compute-bound. This gap is especially large for LLM decode, where the autoregressive, one-token-at-a-time pattern is dominated by weight-fetch bandwidth, not MAC utilization.

A VPU is a class of AI accelerator purpose-built to run computer-vision and CNN inference at very low power (typically sub-1.5 W) for edge and embedded devices, exemplified by Intel's Movidius Myriad 2 and Myriad X. Architecturally it combines arrays of programmable SIMD/VLIW vector DSP cores (Movidius 'SHAVE' cores) with fixed-function ISP and vision hardware, large on-chip scratchpad, and direct camera interfaces, all tuned for inferences-per-watt rather than absolute throughput. The terms converged because Intel historically branded Movidius silicon a 'VPU,' but when it integrated a Movidius-derived inference block into Meteor Lake / Core Ultra CPUs it rebranded that block an 'NPU' with no fundamental technology change. So 'VPU' and 'NPU' are overlapping, largely marketing-driven names for the same broad low-power inference category, with 'VPU' foregrounding vision/CNN heritage and 'NPU' being the now-dominant general term.

A systolic array is a 2D grid of processing elements (multiply-accumulate units) through which operands and partial sums flow rhythmically between neighbors, so each data element is reused across many PEs as it propagates. This maximizes data reuse and minimizes traffic to the register file and SRAM, which is critical because memory movement, not the multiply itself, dominates energy in dense matrix multiply. Google's TPU Matrix Multiply Unit is the canonical example: a 128x128 (16,384 MAC) systolic array in current generations, with the Edge TPU using a smaller array on the same principle. The array is fed by a chosen stationarity (the TPU is weight-stationary, preloading and holding weights in the PEs while activations stream through) that the compiler matches to the operand-reuse pattern of the workload.

An integrated SoC NPU shares the system's LPDDR/DRAM with the CPU and GPU under a unified-memory model: there is no explicit host-to-device copy, but bandwidth is limited and contended, so the NPU leans heavily on small managed on-chip SRAM scratchpads to keep weights and tiles resident. A discrete accelerator has its own dedicated high-bandwidth memory (HBM or GDDR, tens of GB at TB/s), requires explicit host-device transfers over PCIe or NVLink, and draws hundreds of watts for orders of magnitude more compute. For a compiler this shapes tiling and memory planning: on an integrated NPU you tile aggressively to fit tiny SRAM and schedule around shared-bus contention, whereas on a discrete part you plan HBM staging, double-buffered DMA, and host-device transfer overlap. It also drives roofline reasoning, since the two tiers have very different memory-bandwidth ceilings.

Many NPU MAC arrays are integer-native, so quantization is part of the compiler rather than an optional pass; Qualcomm's Hexagon backend in QNN, for example, accepts only quantized graphs (int8/int16, uint8/uint16) expressed as Quantize-Dequantize (QDQ) graphs, and FP32 must be converted before it can execute. Post-Training Quantization (PTQ) converts a trained FP model to INT8/INT4 directly, using a small calibration dataset to pick activation ranges via min-max, entropy/KL, or percentile methods; it is fast and needs no retraining but degrades at very low bit-widths. Quantization-Aware Training (QAT) inserts fake-quant nodes and fine-tunes so weights adapt to rounding, recovering accuracy at INT4/FP4 where PTQ falls off. Granularity also matters: per-tensor is cheapest, per-channel is standard for convolutions, and per-block schemes like NVFP4/MXFP are emerging for LLMs.

Qualcomm's Snapdragon X Elite/Plus shipped first with a ~45 TOPS Hexagon NPU, followed by Intel Lunar Lake (Core Ultra 200V, NPU 4.0 at ~48 TOPS) and AMD Ryzen AI 300 'Strix Point' (XDNA 2 at ~50 TOPS), with Snapdragon X2 Elite pushing to ~80 TOPS. On Windows, apps reach the NPU through Windows ML (the recommended layer, superseding raw DirectML), which sits over ONNX Runtime and auto-selects a hardware Execution Provider such as the QNN EP for Qualcomm Hexagon or the OpenVINO EP for the Intel NPU, with graceful fallback to GPU or CPU for unsupported operators. The model is quantized (typically INT8/INT4) and compiled ahead-of-time for the specific NPU. Vendors cache the compiled graph as a serialized context/engine binary to skip recompilation at load time.

The LPU (Groq's Tensor Streaming Processor) deliberately omits the reactive, dynamically-scheduled hardware that CPUs and GPUs rely on: branch predictors, caches, arbiters, reorder buffers, and out-of-order/hardware schedulers. All scheduling is pushed into a compiler that lays out every instruction and every byte of data movement cycle-by-cycle across a functionally-sliced 2D grid of matrix, vector, switch, and memory slices, with data flowing as SIMD 'streams' like a programmable assembly line. Removing reactive machinery makes timing exactly predictable and frees die area for compute and SRAM: the first-gen 14nm chip holds up to ~230 MB of on-chip SRAM at roughly 80 TB/s as primary weight storage, with no external HBM. Because autoregressive decoding is sequential and memory-bandwidth-bound, keeping weights in SRAM at far higher bandwidth than a GPU's HBM and eliminating memory-stall variance yields high single-stream tokens/sec and flat per-token latency; the tradeoff is that only hundreds of MB fit per chip, so real models must be sharded across many chips linked by the statically-scheduled RealScale fabric.

Programmability decreases as specialization increases: GPUs are fully programmable SIMT engines (CUDA/ROCm) that run graphics, HPC, and any tensor workload; TPUs are compiler-driven systolic-array ASICs (XLA/JAX) that stay flexible across model types; NPUs and VPUs run a constrained set of quantized DNN operators through fixed-function engines; and Groq's LPU is a statically-scheduled, compiler-controlled pipeline with essentially no runtime scheduling. The memory model tracks this: GPUs and TPUs use off-chip HBM (H100 ~3.35 TB/s, TPU v5p ~2.77 TB/s), NPUs/VPUs use small on-chip SRAM plus shared system LPDDR, and the LPU is radically SRAM-only (~230 MB at ~80 TB/s, no external DRAM in the critical path). The throughput-versus-latency bias follows too: GPUs and TPUs maximize aggregate throughput with large batches and big pods, NPUs/VPUs optimize inferences-per-watt at unit batch on-device, and the LPU explicitly optimizes single-stream token latency via deterministic dataflow. Power spans roughly five orders of magnitude, from ~1-5 W mobile NPUs to hundreds of watts or ~1 kW per datacenter package.

Both are built on the same core principle: a systolic-array Matrix Multiply Unit of MAC processing elements with weight-stationary dataflow, fed by on-chip SRAM under compiler-scheduled tiling. The Cloud TPU is a discrete datacenter accelerator with 128x128 systolic MXUs, bfloat16 multiply with FP32 accumulate, multi-GB HBM (v5p ~95 GB), and liquid cooling, deployed in pods interconnected by ICI (v5p up to 8,960 chips in a 3D torus) for both training and large-scale serving. The Coral Edge TPU is its low-power sibling: an INT8-only ~4 TOPS engine at ~2 W in a USB/M.2/SoM module, running only inference on TFLite/LiteRT models compiled by the Edge TPU compiler. So what changes across the spectrum is scale and role, not the fundamental compute pattern: precision (bfloat16+FP32 versus INT8-only), memory (HBM versus tiny on-chip SRAM plus host), power (hundreds of watts versus ~2 W), multi-chip scale-out versus a single edge module, and training-plus-serving versus inference-only.

The common pipeline has five stages. (1) Train in a framework (PyTorch/TensorFlow/JAX); (2) export the model to an interchange graph IR such as ONNX, StableHLO, TorchScript, or Core ML; (3) compile and optimize for the specific target, which includes post-training INT8/INT4 quantization with calibration plus operator fusion and layout canonicalization; (4) load the compiled artifact into a runtime or execution-provider abstraction that partitions the graph and dispatches ops to the device; and (5) execute on hardware. The export-to-interchange step is what decouples the framework from the backend, and the runtime/EP layer is the portability abstraction that lets one exported model target many devices with CPU fallback for unsupported ops.

Latency-optimized serving minimizes time-to-result for a single request (small or unit batch), which is what interactive/on-device workloads and autoregressive LLM decode need; throughput-optimized serving maximizes aggregate requests processed per second, which is how datacenter GPUs and TPUs are typically run. Batching is the primary lever for throughput: grouping requests amortizes weight loads and keeps the MAC/systolic array well-fed, raising arithmetic intensity and hardware utilization. The tradeoff is that larger batches increase per-request tail latency, because a request may wait to fill a batch and then shares compute with others. Edge NPUs deliberately optimize inferences-per-watt at unit batch, while Groq's LPU is an extreme latency point that flattens per-token latency with deterministic dataflow rather than chasing batch throughput.

Qualcomm's Hexagon Tensor Processor (HTP) is integer-native, so the QNN (AI Engine Direct) backend targeting the HTP accepts quantized graphs (int8/uint8, int16/uint16); a plain FP32 graph cannot execute on the HTP. This makes the quantizer effectively part of the compiler rather than an optional pass. Models are commonly expressed as QDQ (Quantize-Dequantize) graphs, where explicit quantize/dequantize nodes carry the scales and zero-points the compiler folds into integer kernels. In practice you run PTQ with a calibration set (or QAT if accuracy degrades at low bit-width), producing per-channel weight scales, then compile and serialize to a QNN context binary. This is typically surfaced through the ONNX Runtime QNN execution provider.

ONNX Runtime treats each backend as a pluggable execution provider (EP): CUDA, TensorRT, OpenVINO, QNN, NNAPI, CoreML, DirectML, CANN, and others. At session load, the runtime partitions the model graph and assigns each supported subgraph to the highest-priority EP that can run it, so the same ONNX file retargets to many devices without re-authoring. When an EP does not implement an operator, that op or subgraph falls back to the CPU EP, which preserves correctness but can hurt performance if fallbacks are frequent or force host-device round trips. Some EPs also build and cache an optimized artifact on first run — TensorRT caches an "engine" and QNN caches a "context binary" — so subsequent loads skip recompilation. On Windows, Windows ML sits above ONNX Runtime and auto-selects the vendor EP (QNN for Qualcomm, OpenVINO for Intel, TensorRT-RTX for NVIDIA) with graceful GPU/CPU fallback.

Mobile SoC deployment uses an integrated NPU (Apple Neural Engine, Qualcomm Hexagon, MediaTek APU, Google Tensor) at roughly 1-5 W that shares unified LPDDR with the CPU/GPU and runs INT8/INT4 for camera pipelines, speech, and small on-device LLMs. Edge/embedded uses VPUs and small NPUs (Movidius, Coral Edge TPU, Jetson, Mobileye EyeQ) with SRAM scratchpads and direct sensor interfaces at single-digit watts for vision, robotics, and ADAS. AI PCs use integrated 40+ TOPS NPUs (for example Intel Core Ultra 200V / Series 2, AMD Ryzen AI 300, and Snapdragon X / X2 platforms) within laptop thermals for Copilot+ features, offloading from the CPU/iGPU for battery efficiency. Datacenter/cloud uses discrete GPUs, TPUs, and LPUs with dedicated multi-GB HBM, hundreds of watts to ~1 kW per package, and rack-scale interconnect (NVLink for GPUs, ICI for TPUs, RealScale for Groq); they alone do large-scale training plus high-throughput batched serving. The sharpest divide is memory: shared LPDDR or on-chip SRAM at the edge versus dedicated HBM requiring explicit host-device transfers in the datacenter.

Microsoft requires a 40+ TOPS integrated NPU (alongside 16 GB RAM and 256 GB storage) for Copilot+ PC certification, enabling on-device features like Recall, live captions/translation, Windows Studio Effects, and image generation; Qualcomm Snapdragon X Elite (~45 TOPS) shipped first, followed by Intel Lunar Lake / Core Ultra 200V (~48 TOPS) and AMD Ryzen AI 300 (~50 TOPS). On Windows, apps increasingly reach the NPU through Windows ML — the strategic successor to DirectML for on-device inference, layered over ONNX Runtime — which auto-loads the vendor execution provider (QNN for Qualcomm, OpenVINO for Intel, Vitis AI for AMD) with fallback to GPU/CPU; DirectML remains a supported bundled EP. On Apple platforms the Neural Engine has no direct public API and is accessed only implicitly through Core ML, which decides op placement. On Android the path was historically NNAPI, but NNAPI was deprecated in Android 15 in favor of LiteRT (TensorFlow Lite) plus vendor delegates/SDKs, with NPU coverage still fragmented.

Headline TOPS is a peak figure computed as roughly MACs × clock × 2 (a MAC counts as two ops), and it is precision-dependent, so INT8 TOPS is not comparable to INT4 or FP16 TOPS — comparing across precisions is meaningless. "Effective TOPS" (ops actually executed divided by latency) is frequently many times lower than peak because real workloads, especially LLM decode, are memory-bandwidth-bound rather than compute-bound. A roofline analysis makes this concrete: attainable throughput is capped by peak memory bandwidth until arithmetic intensity (FLOPs per byte) is high enough to hit the compute ceiling, so a kernel with low intensity never approaches peak TOPS regardless of array size. For integrated NPUs this is worsened by shared LPDDR bandwidth contended with the CPU/GPU. The practical implication: for deployment you evaluate achievable latency/throughput on the real model and shapes, and treat TOPS only as a same-precision, order-of-magnitude sanity check.

The Groq LPU stores weights in ~230 MB of on-chip SRAM at roughly 80 TB/s with no external HBM/DRAM in the critical path, versus a GPU's off-chip HBM at single-digit TB/s (e.g., an H100 at ~3.35 TB/s). That order-of-magnitude bandwidth edge, combined with fully compiler-scheduled deterministic execution (no caches, branch predictors, or reactive schedulers), gives very high tokens/sec and flat, jitter-free per-token latency with essentially no tail latency for memory-bandwidth-bound autoregressive decode. The catch is capacity: a few hundred MB per chip cannot hold a real LLM, so the model must be sharded across many LPUs, with inter-chip movement also statically scheduled over the RealScale fabric so hundreds of chips act as one deterministic machine. This raises system cost and complexity but wins the low-latency interactive/agentic LLM serving niche — deployed as the GroqCloud token API or on-prem GroqRack for regulated, air-gapped environments — rather than training or batched-throughput-only workloads.

TensorRT performs an ahead-of-time build that produces a serialized "engine": an optimized plan with layer and tensor fusion, precision calibration (INT8 PTQ/QAT, and FP8/FP4 on newer parts), kernel auto-tuning selected for the specific GPU and input shapes, and optional 2:4 structured-sparsity support on Sparse Tensor Cores. Fusion is one of the highest-impact latency optimizations because it collapses chains like conv+bias+ReLU or the attention softmax chain into single kernels, avoiding round trips to DRAM. The engine is cached so you pay the expensive build once offline instead of recompiling on every process start, which matters for autoscaling where cold starts are frequent; the analogous pattern is a QNN context binary or an XLA executable. The caveat is that an engine is tied to the GPU architecture, TensorRT version, and often the profiled shape ranges, so it must be rebuilt when any of those change — a real operational concern for fleet management.

An integrated SoC NPU shares system LPDDR and a single power/thermal budget with the CPU and GPU, so unlike a discrete accelerator with its own HBM and hundreds of watts, it has no private high-bandwidth memory and no explicit host-device copy — but it contends for bandwidth and must fit a small (~1-10 W) envelope. That makes on-chip SRAM residency a central design goal: the compiler must tile/block tensors to fit the limited scratchpad, choose a dataflow stationarity (weight/output/row-stationary) matching operand reuse, and use double-buffered (ping-pong) DMA to overlap tile loads with compute and hide memory latency. Because these arrays are integer-native, quantization to INT8/INT4 is effectively mandatory and part of the compile step, and dataflow is statically scheduled ahead-of-time rather than dynamically, so vendor compilers (QNN, OpenVINO, Core ML, RKNN, Arm Vela) do memory planning and mapping offline. The deployment artifact is a serialized, device-specific context/engine binary shipped inside the app, typically dispatched via an ONNX Runtime EP or platform runtime with CPU fallback for unsupported ops.

LLVM IR is a strongly-typed, three-address, SSA-form virtual instruction set that decouples language front-ends from hardware back-ends. Its SSA property means every value has exactly one static definition, so use-def chains are trivial and passes like constant propagation, global value numbering, and dead-code elimination become simple. It is three-address (operations name their operands and result explicitly) and strongly typed, with memory modeled explicitly through load/store on typed pointers rather than implicit register aliasing. It also exists in three isomorphic forms — in-memory C++ objects, human-readable .ll text, and compact .bc bitcode — which lets tools inspect, serialize, and re-ingest the same IR losslessly.

The pipeline is front-end, middle-end, and back-end, all centered on LLVM IR. The front-end (e.g., Clang for C/C++/Objective-C, or rustc/swiftc/flang) lowers source to LLVM IR; for Clang this is source to AST to IR. The middle-end, driven by opt, runs target-independent analyses and transforms directly on IR. The back-end, driven by llc, lowers IR to target-specific machine code through instruction selection, scheduling, register allocation, and prologue/epilogue insertion before emitting assembly or object code via the MC layer. This clean separation is why one optimizer-and-codegen investment is amortized across dozens of front-end languages and hardware targets.

Graph- and tensor-level concerns — operator fusion, tiling, layout and quantization, and dynamic shapes — are best handled in a higher-level, domain-specific IR where the semantics are still visible. Once those are resolved, stacks lower to LLVM IR specifically to inherit LLVM's mature scalar and vector optimization, register allocation, instruction scheduling, and battle-tested code emission for CPUs and GPUs. This avoids re-implementing low-level compiler engineering per accelerator and lets teams focus their effort on the parts unique to their hardware. XLA, Triton, IREE, Mojo/MAX, TorchInductor, and vendor NPU stacks all follow this 'lower-then-hand-off-to-LLVM' pattern.

Representative target-independent passes include mem2reg/SROA, function inlining, instcombine (peephole simplification), GVN (redundancy elimination), the loop passes LICM and unrolling, loop and SLP vectorization, and various interprocedural passes. mem2reg (and its more general sibling SROA, scalar replacement of aggregates) promotes stack allocas into SSA virtual registers, replacing memory traffic with pure SSA values reconciled by phi nodes at control-flow merges. This matters because most front-ends, including Clang, initially emit locals as alloca/load/store; putting them into SSA form is what unlocks the rest of the SSA-based optimizers. The new PassManager organizes all of these by scope into Module, CGSCC, Function, and Loop pass managers.

SelectionDAG is the legacy framework: it builds a dataflow DAG per basic block, legalizes types and operations, runs DAG-combines, and then greedily tree-pattern-matches nodes to target instructions — so selection is fundamentally block-local. GlobalISel is the newer replacement that operates on Generic Machine IR (gMIR) across whole functions through a pipeline of IRTranslator, Legalizer, RegBankSelect, and InstructionSelect. Working at whole-function scope enables faster and more global selection and avoids rebuilding a DAG per block. Importantly, both frameworks can share TableGen-described selection patterns, so a target's pattern investment is not wasted when migrating.

Register allocation maps an unbounded number of virtual registers onto the finite set of physical registers, inserting spill code to memory when register pressure exceeds what is available. LLVM ships four allocators selectable via -regalloc=: Fast (used at -O0, prioritizing compile speed), Basic, Greedy (the default at -O1 and above), and PBQP (Partitioned Boolean Quadratic Programming). Greedy is the production workhorse and performs global live-range splitting to reduce spills. After allocation, prologue/epilogue insertion finalizes the stack frame and resolves references to abstract stack slots.

TableGen is LLVM's declarative DSL in which target .td files describe registers, register classes, instruction encodings and operands, calling conventions, scheduling models, and instruction-selection patterns. TableGen back-ends consume these files and auto-generate large swaths of C++ — instruction info, register info, matcher tables, and GlobalISel selectors — so a new target is defined largely by data rather than hand-written code. For an accelerator vendor this dramatically lowers bring-up cost: custom RISC-V, DSP, or NPU instructions are added declaratively and can be upstreamed into LLVM before silicon tapes out. It is the mechanism that makes retargeting economical instead of a from-scratch C++ effort.

MLIR is an LLVM sub-project providing a multi-level, dialect-extensible IR for progressive (partial) lowering of abstractions that plain LLVM IR cannot represent well, such as tensors, structured loops, and dynamic shapes. A flow composes dialect conversions — for example tosa to linalg to scf/affine plus vector to memref — until everything is expressed in the llvm dialect, which maps almost 1:1 onto LLVM IR and serves as the terminal dialect. A translation step then emits real LLVM IR for final code generation through LLVM's back-ends. So MLIR handles high-level structure and multi-level optimization, and LLVM handles the low-level scalar/vector codegen it is best at; the llvm (or spirv) dialect is the stable convergence point between the two.

For GPUs, LLVM's in-tree back-ends perform the final device-code generation: the NVPTX back-end emits NVIDIA PTX (subsequently assembled by ptxas or the driver), and the AMDGPU back-end emits AMDGCN/GCN ISA (assembled into HSACO). OpenAI's Triton is built on MLIR and lowers tile-level kernels through Triton-IR (TTIR) to Triton-GPU IR (TTGIR) to LLVM IR, and finally to PTX or AMDGCN via these back-ends. NVIDIA's own NVVM is a vendor-specific instance of the same idea — an LLVM IR-based library/back-end that generates PTX for CUDA. PyTorch 2's TorchInductor emits Triton kernels for GPUs, so it inherits this exact LLVM-to-PTX/AMDGCN path.

MCJIT is the legacy JIT: it compiles an entire module eagerly through the MC layer. ORC (ORCv2) is the modern replacement — layered and concurrent, with symbol definitions living in JITDylibs and layers such as IRCompileLayer and CompileOnDemandLayer determining how they compile; by default symbols compile on first lookup, and LLLazyJIT adds true per-function lazy compilation via lazy-reexports. LLJIT is the recommended MCJIT replacement built on this infrastructure. For ML runtimes, this on-demand/lazy behavior lets a compiler specialize kernels to runtime shapes and dtypes and defer per-function codegen until first call, avoiding ahead-of-time enumeration of every possible dynamic-shape variant.

A dialect is a namespaced collection of operations, types, and attributes that together define a coherent unit of IR at some abstraction level; it is MLIR's primary extensibility mechanism, since MLIR has no fixed, built-in opcode set. The key property is that multiple dialects can coexist within the same module, so a program can hold high-level and low-level ops side by side and be lowered gradually between them. Examples span the abstraction spectrum: tosa and stablehlo represent high-level ML operators, linalg captures structured linear algebra, affine and scf express loop nests and structured control flow, vector models SIMD, memref represents buffers, and the llvm dialect mirrors LLVM IR nearly 1:1 as a terminal dialect. This open, layered design is exactly what lets hardware vendors add their own dialects without forking the whole infrastructure.

MLIR IR is a nested structure where Operations are the nodes and Values are the SSA edges connecting them. The containment hierarchy is recursive: an Operation can contain zero or more Regions, a Region contains a list of Blocks, and each Block contains a list of Operations, which can themselves contain Regions. Every Value is either an operation result or a block argument, has exactly one definition and a Type, and thus satisfies static single assignment. Blocks end in a terminator and take block arguments in place of PHI nodes, so control-flow merges pass values as successor-block arguments. This uniform, self-similar structure is what lets one framework represent everything from a whole tensor-graph op with nested regions down to a single llvm.add.

Progressive lowering means a compiler translates a program through a sequence of dialects step by step (e.g. tosa to linalg to scf/affine plus vector to memref to llvm) rather than in one monolithic frontend-to-backend jump, and partial lowering means only some ops are converted at each stage so mixed-abstraction IR is legal in between. This matters because domain semantics like convolutions, tiling structure, and tensor value semantics are preserved as long as they are useful for analysis and optimization, then discarded only when lowered away. For NPU backends specifically, it lets a vendor intercept at whatever abstraction best matches their hardware, commonly linalg or a custom vendor dialect, and inject custom micro-kernels or dataflow mappings there. Crucially, they reuse upstream tiling, fusion, and bufferization instead of rebuilding a full compiler stack, which is the core economic argument for multi-level IR.

Tiling and fusion are performed while values are still immutable, value-semantic tensors because that immutability makes the transforms provably safe and freely composable: there is no aliasing or mutable-buffer hazard to reason about, so producers and consumers can be split and merged without worrying about clobbering shared memory. The linalg dialect is the workhorse here because its named and generic structured ops carry explicit indexing maps and iterator types, which expose the loop and reuse structure needed to tile and fuse mechanically. Once tiled, loops are materialized via scf or affine and inner data movement via the vector dialect. Doing this before bufferization is deliberate; bufferization runs late, after the tiling and fusion decisions are locked in, so the resulting buffers reflect an already-optimized loop nest.

The dialect-conversion framework drives lowering from one set of dialects to another. A ConversionTarget declares legality: each operation is marked Legal, Illegal, or Dynamically Legal (legal only if a predicate holds, e.g. on operand types). ConversionPatterns are rewrite rules that transform illegal ops into legal ones, and the framework can transitively compose an A-to-C lowering from separate A-to-B and B-to-C patterns. A TypeConverter maps types across the boundary, supporting 1-to-1 and 1-to-N conversions, and the framework inserts unrealized_conversion_cast ops to bridge temporary type mismatches that later patterns reconcile. Full conversion requires every op to end up legal or it fails, whereas partial conversion tolerates remaining illegal ops, which is what enables the incremental, mixed-abstraction lowering MLIR is built around.

Transformations are expressed as pattern-based rewrites: a RewritePattern matches an op subgraph and replaces it, and these are commonly applied by the greedy pattern rewriter, which repeatedly applies patterns until it reaches a fixpoint. Patterns can be written in C++ or declaratively via DRR (table-driven declarative rewrite rules in TableGen) or PDL (the pattern-description-language dialect), which reduces boilerplate for structural matches. Passes bundle related transformations and are scheduled by the PassManager, which nests pass pipelines over the op hierarchy, for example running a pass per func.func nested inside a module pipeline. So patterns are the fine-grained rewriting mechanism, while the PassManager handles orchestration, ordering, and nesting; tools like mlir-opt compose these pipelines for development and debugging.

Bufferization converts value-semantic tensor SSA values into mutable memref buffers with explicit memory. One-Shot Bufferize performs a whole-function analysis over the tensor use-def and alias chains and prefers to reuse buffers in place, anchored by destination-passing-style out operands exposed through the DestinationStyleOpInterface, with per-op bufferization behavior driven by the BufferizableOpInterface, so a result can write into the buffer of an operand it consumes. It inserts a copy only when it detects a read-after-write conflict, i.e. a buffer would be overwritten while a still-needed value aliases it, which keeps the peak memory footprint low. It runs late, after tiling and fusion, because those transforms are far easier and safer on immutable tensors; once mutable buffers and aliasing exist, aggressive restructuring becomes much harder, so you lock in the loop structure first and materialize memory last.

TOSA is a stable, numerically specified, quantization-friendly operator set used as a portable interchange dialect, so tosa.matmul starts as a single high-level op. It first lowers to the tensor and linalg dialects, typically a tensor.empty for the result init plus a linalg.matmul (or linalg.batch_matmul) structured op carrying indexing maps. At the linalg level the compiler applies tiling and fusion to fit operands into on-chip memory and to combine adjacent ops, then materializes loops via scf or affine and vectorizes inner work through the vector dialect. Next, One-Shot Bufferize converts the remaining tensors to memref buffers, choosing in-place reuse where possible. Finally the dialect-conversion framework lowers to a target dialect such as gpu, nvvm/rocdl, spirv, or the llvm dialect, from which real LLVM IR or device code is emitted; each stage preserves semantics only as long as they remain useful, which is the multi-level payoff.

Torch-MLIR is a frontend: it introduces torch and related conversion dialects to import PyTorch (via TorchScript, FX/Dynamo, or torch.export) and lower it into backend-ready dialects like linalg, TOSA, or StableHLO, so it feeds other MLIR backends rather than generating code itself. Triton is a tile-level GPU kernel DSL built on MLIR with its own dialects (Triton/TTIR, TritonGPU/TTGIR, plus target GPU dialects), reusing arith/math/scf and lowering TTIR to TTGIR to LLVM IR for device codegen. IREE is an end-to-end compiler and runtime that ingests StableHLO/TOSA/Torch/linalg and lowers through three stages: flow forms dispatchable functions with fusion and workload partitioning, stream schedules asynchronous, potentially multi-device execution, and hal is a hardware abstraction layer managing buffers and command submission at runtime. That hal boundary is what lets one compiled artifact target CPUs, GPUs, and NPUs; NPU vendors reuse the upstream mid-level lowering and plug custom dialects/micro-kernels in below, as seen in iree-amd-aie for AMD AIE NPUs.

A single low-level IR like LLVM IR is scalar/vector SSA and cannot cleanly represent tensors, whole loop nests, dynamic shapes, or dataflow-mapping structure, so lowering to it too early destroys exactly the domain information an NPU compiler needs to reason about tiling, fusion, quantization, and mapping stationarity onto MAC/systolic arrays. MLIR keeps those high-level semantics alive across coexisting dialects and lowers progressively, so each analysis runs at the abstraction where it is natural: tensor-level fusion on linalg, loop transforms on affine/scf, SIMD on vector, and memory planning after bufferization on memref. Just as important is reuse: vendors intercept at a mid-level dialect such as linalg or a custom dialect, inject their own micro-kernels and hardware ops, and inherit upstream tiling, fusion, bufferization, and the dialect-conversion machinery instead of rebuilding a compiler from scratch. LLVM still serves as a terminal backend via the llvm dialect for final scalar/vector codegen, so MLIR complements rather than replaces it, which is why stacks like IREE, TPU-MLIR, and MLIR-AIE converge on this pattern.

Operator fusion merges a chain of adjacent operators (for example conv + bias + ReLU, or the softmax chain inside attention) into a single kernel so intermediate activations stay in on-chip registers/SRAM instead of being written out to and re-read from DRAM/HBM. The dominant cost in edge and inference workloads is off-chip memory traffic, not arithmetic, so eliminating those round-trips directly cuts both latency and energy. Fusion also raises a kernel's arithmetic intensity (FLOPs per byte moved), pushing it away from the memory-bound region of the roofline. It runs early as a mostly target-independent graph-level pass alongside constant folding, CSE, and dead-code elimination. Because it removes materialization of intermediate tensors, it is usually the highest-leverage single transform a stack applies.

PTQ takes an already-trained FP model and converts it to INT8/INT4/FP8 directly, using a small calibration set to pick activation ranges via min-max, entropy/KL, or percentile clipping; it needs no retraining and is fast, making it the default first attempt. QAT instead inserts fake-quant (simulated rounding) nodes into the graph and fine-tunes, so the weights learn to compensate for quantization error, which recovers accuracy at aggressive bit-widths like INT4/FP4 where PTQ degrades noticeably. You reach for PTQ when you need a quick deployment and INT8 accuracy loss is acceptable, and for QAT when the target precision is low or the model is accuracy-sensitive. The tradeoff is cost: QAT requires a fine-tuning pipeline, representative data, and GPU time, whereas PTQ needs only a calibration pass.

A roofline plot graphs attainable throughput (FLOP/s) against arithmetic intensity (FLOPs per byte of memory traffic), bounded by two ceilings: a horizontal peak-compute line and a diagonal peak-memory-bandwidth line. Where a kernel lands classifies it as compute-bound (under the flat ceiling) or memory-bound (under the sloped ceiling). Compilers use this to justify fusion and tiling: both raise arithmetic intensity by reusing data on-chip, moving a memory-bound kernel rightward until it escapes the bandwidth-limited region. Hardware designers 'steepen the roofline' by adding HBM or larger on-chip SRAM. The model is coarse because it ignores on-chip locality across fused ops, so finer tools like Timeloop or Stream model tiling more precisely.

NPUs have small, explicitly-managed on-chip SRAM/scratchpads rather than large caches, so large weight and activation tensors cannot fit resident; tiling splits them into blocks sized to fit on-chip, letting operands be staged once and reused heavily across the MAC array to maximize arithmetic intensity instead of being restreamed from DRAM/HBM. The difficulty is that the optimal tile shape depends jointly on SRAM capacity and banking, the array's dataflow/stationarity, DMA bandwidth, and the operator's reuse pattern, and these interact non-linearly. It is effectively a large search space over tile sizes, loop order, vectorization, and PE binding. That is why stacks like TVM (Ansor/MetaSchedule) auto-tune tiling by benchmarking many candidate schedules on the real device. A poor tile choice either overflows SRAM or leaves the array starved and underutilized.

Dataflow mapping assigns the loop nest of a tensor op onto the physical MAC/systolic array, choosing which operand stays resident in the processing elements to maximize on-chip reuse. Weight-stationary preloads and holds weights in the PEs and streams activations, best when weights are reused across many activations (Google's TPU MXU is weight-stationary); output-stationary keeps partial sums accumulating locally in each PE while both operands stream, minimizing psum writeback; input-stationary holds activations; row-stationary (Eyeriss) balances reuse of weights, activations, and partial sums simultaneously. The compiler picks the mapping whose stationary operand matches the layer's dominant reuse pattern — e.g., weight-stationary for weight-heavy convolutions, output-stationary when accumulation traffic dominates. The goal is to minimize the most expensive data movement given the layer's operand-reuse profile and the array's structure.

Bufferization converts value-semantic tensor SSA values into explicit mutable memref buffers, computing liveness and doing buffer allocation and reuse to minimize peak on-chip SRAM footprint. MLIR's One-Shot Bufferize analyzes tensor use-def chains whole-function, prefers in-place reuse anchored by destination-passing-style 'outs' operands, and inserts copies only on read-after-write conflicts. It typically runs late, after tiling and fusion, so those transforms operate on immutable tensors where they are provably safe. Double-buffering (ping-pong) allocates two buffers per pipeline stage so the DMA of the next tile writes into one buffer while the compute engine consumes the other, overlapping data movement with computation. This software-pipelining hides DMA latency and keeps the MAC array fed rather than stalling on loads.

On integer-native accelerators the MAC array only executes low-precision integer math, so an FP32 graph literally cannot run until it is quantized; Qualcomm's QNN backend for the Hexagon Tensor Processor accepts quantized graphs (uint8/uint16, int8/int16), meaning quantization is a mandatory lowering step rather than an optional accuracy-vs-speed tradeoff. Models are expressed as QDQ (Quantize-Dequantize) graphs, where explicit Quantize and Dequantize nodes are inserted around operators to encode the scale and zero-point at each tensor boundary. The compiler then folds these QDQ pairs into the integer kernels, propagating scales so real arithmetic happens in INT and requantization occurs between ops. Because deployment is impossible without it, the quantizer is embedded in the compiler pipeline. This also lets tools like ONNX Runtime's QNN EP consume standard QDQ ONNX models and map them to the HTP.

Per-tensor quantization uses a single scale/zero-point for an entire tensor — cheapest in metadata and compute but least accurate when value ranges vary widely across channels. Per-channel assigns a separate scale per output channel (standard for CNN/conv weights), capturing channel-wise range differences at negligible cost. Per-block/group quantization (e.g., NVFP4 with 16-element blocks, MXFP4 with 32-element blocks) uses a scale per small block of elements and is standard for LLMs where even per-channel is too coarse at 4-bit. Activations are harder than weights because they are data-dependent and contain large per-channel outliers that inflate the range and crush the resolution of normal values, whereas weights are static and better-behaved. This outlier problem motivates schemes like SmoothQuant (migrating outlier scale from activations into weights), GPTQ (Hessian-guided per-channel weight quantization with calibration), and AWQ (activation-aware channel scaling).

Structured 2:4 (N:M) sparsity forces exactly two of every four contiguous weights to zero, producing a regular pattern that NVIDIA Ampere+ Sparse Tensor Cores exploit for up to 2x throughput by physically skipping the zeroed multiply-accumulates. The compressed operand is stored as the non-zero values plus 2-bit-per-element metadata indicating which two positions in each group were retained, so the hardware can select the matching activations at issue time. Realizing the speedup requires the toolchain to prune to the exact 2:4 pattern, encode weights in the compressed values-plus-metadata format, and emit the sparse MMA path — exposed via cuSPARSELt and the mma.sp PTX instruction, with TensorRT supporting sparse-INT8 engines. Unlike unstructured pruning, which compresses more but yields no real speedup without hardware/format support, 2:4 is directly accelerated because its regularity matches the sparse datapath. The tradeoff is a fixed 50% density ceiling and potential accuracy loss that may require fine-tuning to recover.

MLIR hosts many coexisting dialects at different abstraction levels in one SSA IR, so a compiler can preserve domain semantics — tensors, convolutions, tiling structure — through reusable mid-level dialects (linalg, affine, scf, vector, memref, gpu) and lower them progressively rather than in one monolithic step. This matters for NPUs because vendors can intercept at exactly the right altitude, inject custom dialects or micro-kernels for their spatial/systolic array, and reuse upstream tiling, fusion, and bufferization instead of rebuilding a whole compiler; examples include MLIR-AIE/iree-amd-aie for AMD XDNA and TPU-MLIR. Tiling and fusion are performed while values are still immutable tensors (especially in linalg) because value semantics make those transforms provably safe, and bufferization runs later to materialize memrefs. Once the program reaches the llvm dialect — a near-1:1 mirror of LLVM IR — a translation step emits real LLVM IR, handing off scalar/vector optimization, register allocation, and CPU/GPU codegen (via NVPTX/AMDGPU) to LLVM's mature back-ends. Thus MLIR handles high-level structure LLVM IR cannot represent, and LLVM serves as the terminal code-generation target.