NPU and SoC Software Architecture

Converts a product request, sensor frame, or media stream into model inputs and turns model outputs into product behavior. It owns the user-visible latency and correctness contract, not only tensor I/O.

Consumes

Raw requests or sensor data, application configuration, model identity, and service-level policy.

Produces

Typed tensor bindings, shape metadata, deadlines, priorities, correlation IDs, and interpreted outputs.

Owns

Preprocessing and postprocessing versions, batching and admission policy, fallback, cancellation, and accuracy/latency tradeoffs.

Must expose

Request shape, preprocessing time, queue delay, inference time, postprocessing time, fallback reason, and final status.

Failure pattern: training used RGB normalization and NHWC tensors, while deployment silently sends BGR or NCHW; the accelerator is correct but the product result is wrong.

Represents the portable computation contract. A graph is more than operator names: opset versions, attributes, control flow, broadcasting, aliasing, layouts, quantization metadata, and shape constraints determine the result that every backend must preserve.

Consumes

Trained checkpoint or framework module, exporter configuration, example inputs, and external constant files.

Produces

A versioned graph with nodes, tensors, constants, symbolic dimensions, dtypes, quantization parameters, and named outputs.

Owns

Framework-level semantics and export fidelity; it does not own target scheduling, physical addresses, or device command order.

Must expose

Exporter revision, opset imports, source-node mapping, unresolved shape constraints, model digest, and framework golden outputs.

Failure pattern: an exporter decomposes one operation differently after an upgrade, or leaves preprocessing outside the graph, and the importer accepts a graph whose meaning no longer matches the validated model.

Separates model semantics from deployment choices. The same graph may need different precision, memory, fallback, and power decisions for a camera, battery device, automotive SoC, or throughput server.

Consumes

Model analysis, hardware capability manifest, accuracy tolerance, latency target, memory budget, thermal limits, and fleet policy.

Produces

An immutable compile profile naming target revision, shape classes, precision policy, calibration data, fallback rules, and optimization objective.

Owns

Allowed approximations, dynamic-shape bounds, SRAM/DRAM limits, power mode, digital fallback, and compatibility floor.

Must expose

Profile digest, policy source, rejected constraints, selected objective weights, calibration-set identity, and target capability version.

Failure pattern: important target flags live only in shell scripts or engineer memory, so two builds of the same model produce different artifacts with no reproducible explanation.

Turns an external graph into a canonical internal representation while preserving source identity. It should fail early on ambiguity and provide a support report before expensive target compilation.

Consumes

Framework graph, constants, deployment profile, calibration metadata, importer schema, and target support manifest.

Produces

Validated typed IR, inferred shape constraints, canonical operators, source maps, support classification, and actionable diagnostics.

Owns

Schema and opset checks, shape/type inference, constant integrity, semantic normalization, capability diagnosis, and minimal failing subgraphs.

Must expose

Unsupported node path, source location, inferred versus declared shape, fallback reason, warning severity, and normalization history.

Failure pattern: unsupported behavior is discovered after many compiler passes and reported as an internal lowering error, leaving the user unable to identify the original node or repair the model.

Converts validated IR into a legal and efficient command program. Partitioning, layout, quantization, tiling, scheduling, and memory allocation are coupled decisions because each changes traffic, lifetime, engine occupancy, and numerical behavior.

Consumes

Canonical IR, deployment profile, target ISA and capabilities, kernel library, calibration data, and measured cost models.

Produces

Partitions, lowered kernels or microcode, command DAG, DMA plan, static buffer offsets, constants, relocations, and debug maps.

Owns

Legalization, fusion, precision and layout choice, tiling, resource scheduling, memory lifetimes, spill decisions, and fallback boundaries.

Must expose

Pass pipeline, IR checkpoints, cost breakdown, rejected alternatives, peak memory, expected cycles/bytes, and every graph-to-command mapping.

Failure pattern: a locally faster fusion increases live memory, forces DRAM spills, and makes the complete graph slower because schedule and allocation were optimized independently.

Serializes compiler decisions into a bounded, versioned object that a runtime can validate and load without reconstructing compiler intent. The format is a product ABI, even when only one compiler emits it.

Consumes

Code and microcode, constants, command templates, buffer plan, relocations, entry points, ABI schemas, debug map, and build provenance.

Produces

A deterministic artifact with aligned sections, manifest, feature requirements, integrity data, signatures when required, and a stable build ID.

Owns

Section layout, bounds, compression, relocation records, target/ABI compatibility ranges, hashes, schema evolution, and reproducible serialization.

Must expose

Human-readable manifest, section inventory and hashes, compiler/pass revision, model and profile digests, required resources, and debug-symbol availability.

Failure pattern: the runtime accepts an opaque blob for the wrong firmware ABI or hardware stepping, then fails after allocation or submission with no safe compatibility diagnosis.

Provides the supported developer path through compilation, packaging, loading, execution, profiling, and diagnosis. Its job is to make correct lifetime and synchronization behavior easier than unsafe shortcuts.

Consumes

Compiler and runtime capabilities, artifact schemas, support matrix, target discovery data, and diagnostic interfaces.

Produces

Stable C ABI, C++ RAII wrappers, Python tools, CLI output, examples, headers, packages, compatibility documentation, and reports.

Owns

Public handle semantics, ownership rules, error taxonomy, thread-safety contract, binding parity, deprecation policy, and example quality.

Must expose

SDK/runtime/driver/firmware versions, target features, artifact inspection, machine-readable errors, trace export, and minimal reproducer tools.

Failure pattern: samples bypass the runtime or rely on hidden global state, so customer applications copy patterns that leak buffers, serialize asynchronous work, or break on upgrades.

Owns user-space object lifetime and asynchronous truth. It discovers devices, loads artifacts, manages sessions and buffers, submits work, resolves events, and converts lower-level failures into stable API results.

Consumes

Validated artifact, application tensor bindings, dynamic shapes, queue policy, driver capabilities, and device telemetry.

Produces

Executable/session handles, buffer and mapping objects, submissions, dependency events, completions, output bindings, and diagnostics.

Owns

Artifact cache and load policy, runtime allocation, imports/exports, queue ordering, batching, event lifetime, timeout, fallback, cancellation, and device-loss recovery.

Must expose

Session/queue/command IDs, hidden-copy count, allocation and cache statistics, queue delay, submit/complete timestamps, fallback, timeout, and terminal device state.

Failure pattern: a buffer is destroyed while commands still reference it, or a convenience call performs a hidden device-wide synchronization that destroys pipeline overlap.

Converts untrusted user requests into isolated hardware resources. It is the authority for process ownership, DMA-visible memory, queue access, interrupt delivery, firmware lifecycle, power, and reset.

Consumes

System calls or ioctls, object handles, buffer import requests, validated submission descriptors, fences, and administrator policy.

Produces

Contexts, IOMMU mappings, pinned or imported buffers, queue slots, doorbell access, completion fences, fault reports, and reset results.

Owns

Isolation, handle validation, DMA mapping lifetime, cache maintenance, interrupt routing, scheduler admission, firmware loading, power transitions, and reset escalation.

Must expose

Process/context identity, mapped bytes, submissions and interrupts, IOMMU faults, firmware health, power state, reset generation, and stable kernel error codes.

Failure pattern: a stale handle or mapping survives context teardown and lets later work DMA into released memory, creating both a correctness fault and a security boundary violation.

Receives queue work after the kernel boundary and turns it into legal engine sequences. It protects the device from malformed descriptors, arbitrates shared resources, watches progress, and owns first-stage recovery.

Consumes

Command-ring descriptors, dependency tokens, buffer mappings, queue/context metadata, deadlines, and negotiated feature bits.

Produces

Engine dispatches, DMA sequences, resource reservations, completion records, performance snapshots, and structured fault reports.

Owns

Descriptor validation, dependency scheduling, fairness, preemption points, SRAM/DMA/engine reservations, watchdogs, queue quarantine, and warm-reset policy.

Must expose

Queue and command generation, scheduler state, wait reason, resource occupancy, heartbeat, last completed unit, faulting descriptor, and recovery action.

Failure pattern: reset leaves a stale producer generation or dependency token visible, so new commands wait on work that belonged to the previous firmware epoch.

Supplies address translation, data movement, communication, time, clocks, resets, and power domains to the NPU. These services determine whether an otherwise correct accelerator can operate predictably inside the SoC.

Consumes

Device virtual addresses, transfer descriptors, NoC routes and priorities, clock requests, power votes, reset requests, and timer operations.

Produces

IOMMU translations, DMA completions, coherent or explicitly synchronized data, NoC delivery, stable clocks, powered domains, timestamps, and fault interrupts.

Owns

Initialization order, translation context, burst and alignment limits, coherency protocol, bandwidth/QoS policy, DVFS sequencing, clock gating, and reset dependencies.

Must expose

Bytes and latency per path, page faults, NoC congestion, DMA utilization, clock frequency, voltage/power state, temperature, throttling, and reset cause.

Failure pattern: a compiler schedule assumes bandwidth that the shared NoC cannot provide under camera or ISP traffic, causing deadline misses that never appear in isolated NPU tests.

Preserves identity and time across graph nodes, artifacts, queues, firmware commands, engines, memory, and host APIs. Counters without units, scope, correlation, and loss accounting are not actionable telemetry.

Consumes

Compiler debug IDs, runtime correlation IDs, firmware events, engine counters, DMA/NoC events, power changes, faults, and clock-sync samples.

Produces

A versioned bounded-overhead event stream, counter samples, host/device time conversion, layer timeline, fault bundle, and profiler database.

Owns

ID propagation, timestamp width and wrap, clock-domain alignment, event schema, units, sampling and trigger policy, buffer overflow behavior, and privacy controls.

Must expose

Lost-event count, sampling interval, counter scope, conversion uncertainty, frequency epoch, schema version, source engine, and trace buffer pressure.

Failure pattern: a layer is slow, but command, DMA, and engine records cannot be joined because each layer generated unrelated IDs and used a different clock without synchronization metadata.

Fetches and decodes validated command buffers or microcode, resolves dependencies, programs engines, and emits completion events. It is the hardware realization of the compiler and firmware command model.

Consumes

Command buffers, microcode, queue pointers, dependency tokens, tensor descriptors, addresses, engine masks, and execution modes.

Produces

Register or descriptor writes to engines, DMA launches, barriers, event updates, completion status, and precise command faults.

Owns

Fetch/decode, queue arbitration, branch or loop semantics, barrier ordering, dependency-scoreboard behavior, legal opcode handling, and forward progress.

Must expose

Current queue/opcode, commands retired, dependency wait cycles, decode faults, issue stalls, event state, and last known forward-progress point.

Failure pattern: a malformed length or unsupported opcode advances the queue incorrectly, turning one bad command into an unrecoverable device hang rather than a bounded fault.

Performs matrix, vector, scalar, activation, reduction, conversion, and optional sparse operations. The contract includes exact numeric behavior for every precision mode, not just nominal throughput.

Consumes

Input tensor tiles, weights, accumulator state, tensor views, masks, strides, precision mode, rounding mode, saturation policy, and kernel parameters.

Produces

Output tiles, partial sums or reductions, exception/status bits, dependency events, utilization counters, and numerical debug checkpoints.

Owns

Operation semantics, accumulator width, rounding and saturation, NaN/Inf policy where relevant, tail handling, sparse metadata, pipeline hazards, and deterministic modes.

Must expose

Active and issued cycles, lane utilization, operand starvation, pipeline stalls, overflow/saturation count, sparse skips, and per-mode error status.

Failure pattern: the main matrix shape passes, but tail channels use a different rounding or mask rule, creating small layer errors that accumulate only on production tensor shapes.

Delivers constants, activations, commands, and intermediate state through local SRAM, caches, DRAM, and the fabric. Capacity, bank topology, alignment, ordering, and contention shape the achievable schedule.

Consumes

Compiler buffer plan, engine load/store traffic, DMA requests, cache operations, physical or device virtual addresses, priorities, and coherency actions.

Produces

Data with defined visibility and ordering, allocation capacity, bandwidth and latency service, cache responses, fabric transactions, and memory faults.

Owns

Address spaces, SRAM bank mapping, cacheability, replacement and write policy, DRAM channels, arbitration/QoS, coherency, ECC, and memory protection.

Must expose

Bytes per level, hit/miss rate, bank conflicts, queue occupancy, DRAM and NoC stalls, peak live memory, ECC events, page faults, and spill traffic.

Failure pattern: tensors fit in total SRAM capacity but collide in the same banks during concurrent compute and DMA, so the realized throughput is far below the compiler estimate.