NPU + analog compute-in-memory
Design the whole stack. Close the analog loop.
A visual guide to lowering neural networks onto a hybrid edge NPU—where the CPU controls, digital engines cover irregular work, and ACiM arrays accelerate eligible matrix-vector operations.
- 9software layers
- 12bottlenecks
- 1closed feedback loop
01 · System boundary
Give each engine the work it handles best
An ACiM tile is one execution resource inside a heterogeneous system. The software stack must coordinate control, transfers, higher-precision or reproducible digital work when required, analog computation, and recovery.
| Concern | Example A · nonvolatile resistive current-summing crossbar | Example B · volatile SRAM charge-domain macro |
|---|---|---|
| Weight state | Nonvolatile conductance programmed toward a target value or acceptance window and verified when the target exposes that operation. | Volatile SRAM bits; reload after state loss, non-retentive wake, eviction, or model switch. Charge is redistributed or accumulated during computation but is not the retained weight. |
| Main error sources | Programming/read error, drift, endurance, stuck devices, line resistance/IR drop, and converter/periphery limits. | PVT variation, mismatch, parasitics, gain/offset, linearity, references/converters, SRAM integrity, and residency identity. |
| Profile data | Weight-to-conductance map, program/read distributions, drift time origin/model, fault state, verify/endurance state, and supported programming/readback contract. | Per-macro/lane gain, offset, nonlinearity, valid PVT corners, retained-power assumptions, weight-image identity/checksum, and disabled resources. |
| Recovery | Where supported: bounded program/verify continuation, remap, quarantine, recalibrate, recompile, or digital fallback. | Reload lost/corrupt weights, recalibrate, disable a macro/lane, remap/recompile, or digital fallback. |
Schema rule: keep technology-specific data in tagged variants and expose storage type, volatility/retention, and analog signal domain independently. At minimum, use weight_storage, retention_domain, analog_signal_domain, load_mode, readback.kind/granularity, program_verify.mode, journal.durability, reset_effects, and a versioned recovery_recipe_id as separate capability fields. The label “SRAM / charge-domain ACiM” is shorthand only for Example B, not a taxonomy that makes SRAM and charge-domain synonymous. Do not require conductance, drift, or endurance fields in a generic ACiM profile, and do not apply the resistive equation Iⱼ ≈ Σᵢ VᵢGᵢⱼ to a charge-domain SRAM macro.
Host CPU
Loads model bundles, selects profiles, submits work, and orchestrates dynamic or unsupported regions; a digital vector NPU may execute fallback subgraphs rather than the CPU itself.
- Application API
- Model lifecycle
- Fallback orchestration + recovery
Runtime + fabric
Builds command queues and overlaps transfers, computation, and synchronization.
- DMA descriptors
- SRAM allocation
- NoC routes and events
Hybrid NPU
Uses a costed partition rather than sending every nominal matrix operation to analog.
Devices + sensors
Store or stage weight states and report conditions that affect the computation.
- Load or program-and-verify
- Temperature, variation, and drift
- Fault map and calibration
02 · Model-to-device stack
Lower intent into a calibrated command stream
Every layer consumes a contract from above and produces a more concrete artifact below. Characterization data travels upward and changes the result.
- 01Framework and model import
Accept
PyTorch,ONNX, or another exchange format. Resolve shapes, constants, layouts, dynamic dimensions, model metadata, and a representative calibration dataset. - 02High-level graph IR
Represent operators, tensors, shape constraints, quantization intent, control flow, and source-level debug names without prematurely committing to a tile or command format.
- 03Legalization and fusion
Rewrite unsupported forms, fold constants, canonicalize convolution/GEMM, fuse activation or bias where legal, and expose boundaries where analog-to-digital transfers would erase the gain.
- 04Analog-aware optimization
Choose weight/activation precision, scaling and clipping; run quantization-aware or noise-aware training; model device variation, drift, nonlinearity, saturation, and finite ADC resolution.
- 05ACiM versus digital partition
Use capability, accuracy, latency, energy, conversion, transfer, and fallback costs. Keep irregular, higher-precision, small, or poorly utilized work on CPU/digital engines.
- 06Tensor mapping and placement
Tile matrices to physical arrays; bit-slice precision; encode signed weights; place tiles around faults and bandwidth; route partial sums; choose tensor quantization scales separately from physical input-drive and ADC reference/full-scale settings; size digital accumulation widths.
- 07Memory planning and scheduling
Allocate activation SRAM, arrange DMA, double-buffer stages, overlap CPU/digital/ACiM work, preserve dependencies, enforce thermal/power limits, and prevent the converters from becoming the serialized path.
- 08Code generation and model bundle
Emit command buffers, microcode, DMA descriptors, barriers, weight-loading or programming payloads, fallback code, relocation data, calibration-profile IDs, and trace metadata.
- 09Runtime, driver, firmware, and hardware
Load and authenticate bundles, load weights or program-and-verify target cells, submit queues, handle interrupts and faults, apply calibration, collect counters, reset safely, and feed measured behavior back to the toolchain.
| IR level | Makes explicit | Required proof before lowering |
|---|---|---|
| Model IR · semantic | Operators, tensors, shapes, layouts, control flow, opset/version, source metadata. | Imported semantics, shapes, types, attributes, and version conversions are valid. |
| Quantized IR · numeric | Storage and expressed types, scale, zero point, clipping, rounding, overflow, per-tensor/per-channel rules. | Dequantized behavior and task metric remain inside the allocated numeric/accuracy budget. |
| Device IR · spatial | Engine partition, tile coordinates, bit slices, physical ranges, profile constraints, buffers/lifetimes, transfers, events, fallback. | Every operation, mapping, dependency, resource, and profile requirement is legal for the selected capability manifest. |
| Command IR · executable | DMA descriptors, loads/programming, dispatches, barriers, relocations, completion/error records, recovery points. | Buffer bounds, dependency closure, ABI compatibility, reset generation, and replay behavior are verified. |
Compiler invariant: every lowering pass must prove legality, materialize an explicit conversion/fallback, or fail compilation. It must never silently change layout, clipping, rounding, overflow, or synchronization behavior.
03 · Compiler decision flow
“Matrix-heavy” is the start—not the answer
The partitioner must prove support, fit, accuracy, and whole-layer benefit before committing a region to ACiM.
- Gate 1Is the region ACiM-shaped?
Dense/convolutional matrix-vector work, stable shapes, enough reuse, and a useful problem size.
- Gate 2Can it be legalized?
Supported data layout, tensor dimensions, signed encoding, activation rules, and static parameters.
- Gate 3Can it fit?
Array rows/columns, tile count, precision slices, spare capacity, SRAM, routes, and converter capacity.
- Gate 4Does accuracy survive?
Simulate calibrated nonidealities, retrain or rescale, and compare against an explicit accuracy-loss budget.
- Gate 5Does the system win?
Include DMA, ADC/DAC, accumulation, transitions, calibration, programming, and digital fallback—not peak MACs alone.
Generate tile placement, analog parameters, transfers, schedule, commands, and a digital reference for verification.
Change clipping/scales, precision, redundancy, mapping, calibration, or noise-aware training. Do not silently relax the budget.
Keep it on CPU/digital NPU, fuse adjacent digital operations, and avoid expensive engine transitions.
04 · Mapping and scheduling
Turn one tensor operation into physical work
Array geometry, precision, converter range, faulty resources, buffers, and dataflow jointly determine the mapping.
Split dimensions that do not fit
This guide assumes input channels drive rows and column currents encode outputs; some arrays transpose that convention. Split the input dimension into digitally accumulated partial sums and the output dimension into independent result slices.
input block 0 · output block 0(K0, M1)
input block 0 · output block 1(K1, M0)
input block 1 · output block 0(K1, M1)
input block 1 · output block 1
Bit-slice what a cell cannot represent
Represent a logical weight across cells, subarrays, or time steps. The mapper must account for extra conversions, shifts, and accumulation.
high significance+middle slice+LSB slice
fine detail→wide digital accumulator
Use the converters’ useful window
Tensor quantization scales map stored integers to expressed values. Physical drive and ADC reference/full-scale settings map device signals to codes. Calibration connects these separate contracts; it does not make them the same knob.
Hide transfers behind useful work
Double-buffer activations so DMA fills one buffer while an engine consumes the other. Explicit events preserve data dependencies.
| Stage | T0 | T1 | T2 | T3 |
|---|---|---|---|---|
| DMA A | fill A | idle | fill A | idle |
| ACiM | idle | use A | use B | use A |
| DMA B | idle | fill B | idle | fill B |
| Digital | idle | idle | accumulate | activate |
05 · Hardware/software contract
Make every boundary a versioned artifact
A production stack is debuggable when the compiler, runtime, firmware, and silicon agree on explicit formats rather than hidden assumptions. The names and sample filenames below are illustrative Atlas design proposals, not ONNX, MLIR, IREE, or PJRT standards.
Capability manifest
Legal operator forms, geometry, memory/periphery limits, ABI and costs—plus explicit weight storage/retention, load mode, readback kind, program/verify mode, journal durability, reset effects, and vendor recovery recipe.
target_v3.capabilities.jsonMulti-level IR
Graph IR preserves model meaning; device IR records partitions, tiles, buffers, synchronization, analog attributes, and digital fallback.
graph → quantized → scheduled → command IRSigned model bundle
Commands, weights, relocations, buffer plan, profile requirements, compatibility range, fallback entry points, debug map, golden outputs, and security metadata.
model.acim.bundleCalibration profile
Target/stepping/ABI identity, method and provenance, validity domain, uncertainty/confidence, fault/spare state, and corrections at a practical macro/tile/row/column/converter—or selected-cell—granularity.
{target, health generation, PVT, age}Execution request
Bundle/profile ID, context and reset generation, tensor descriptors, dependency events, deadline, priority, power hint, reproducibility requirement, retry class, fallback policy, and trace ID.
enqueue(context, tensors, deadline)Telemetry schema
Per-stage latency, bytes moved, converter activity, saturation, retries, program failures, temperature, power, fault events, and output checks.
trace → profiler → cost model → recompile| Contract | Required fields | Consumer | Failure behavior |
|---|---|---|---|
| Capability manifest | Vendor/device/revision, firmware ABI, legal ops/dtypes/layouts/shapes, effective geometry and signed encoding, periphery/accumulator limits, memory/DMA/coherency, queues/reset/power, cost-model version/units, errata. | Compiler proves legality and estimates cost; runtime confirms the loaded bundle still matches device and firmware identity. | Reject, explicitly convert, partition elsewhere, or compile a declared fallback—never silently approximate an illegal form. |
| Weight-state operations | weight_storage (volatile, factory-programmed, or field-programmable), retention domain, load granularity/idempotence, readback (none, checksum, state, or conductance) with granularity/cost, program_verify (unsupported, atomic, or resumable) with safe progress unit and budgets, journal durability/commit boundary, reset effects, and versioned recovery recipe. | The compiler emits only supported load/program commands; firmware owns the target sequence; the runtime selects restart, reload, remap, quarantine, or fallback from the declared contract. | If a field or operation is absent, reject that path. Resistive storage alone does not imply field programming, conductance readback, incremental resume, or a persistent journal. |
| Calibration/health profile | Target/stepping/ABI/schema/health generation, method/provenance/time/expiry, units/sample count/uncertainty/confidence, valid temperature-voltage-frequency-load-age domain, corrections, faults/spares/effective geometry, integrity and rollback policy. | Compiler maps against effective capacity/error; runtime selects only a profile valid for this device, operating point, and bundle. | Select another profile, recalibrate, remap/recompile, or execute digital fallback; do not extrapolate outside the measured validity domain without an explicit bound. |
Runtime predicate: usable = target_match ∧ abi_match ∧ profile_schema_match ∧ health_generation_match ∧ operating_point ∈ validity_domain ∧ not_expired. Sign the identity and compatibility fields that protect this decision.
06 · Runtime, driver, and firmware
Execute safely from model load to completion
The steady-state inference path should be fast, but first load, failure, recalibration, reset, and firmware compatibility are equally part of the design.
Verify signature, versions, target compatibility, limits, and permissions.
Match device, tile health, temperature, age, and required accuracy.
Branch on the manifest: load a volatile image, invoke an atomic vendor program transaction, or continue only a target-declared resumable program/verify operation. Never infer resume support from device technology.
Use the declared proof—image checksum/ECC, vendor self-test, state readback, or conductance readback. If readback is unavailable, use the vendor restart/reload recipe or quarantine/remap/fallback; do not guess partial progress.
Bind buffers, DMA, commands, events, deadlines, and power hints.
Handle interrupts, saturation, ECC/fault events, timeouts, thermal throttling, and traces.
Return outputs only after readiness proof; otherwise run the versioned vendor recovery recipe, fall back digitally, reset a documented scope, or invalidate residency/profile state.
| Layer | Owns | Must expose across the boundary |
|---|---|---|
| Compiler + bundle | Static partition, placement, schedule, relocations, profile requirements, fallback entry points, debug/source map. | Compatibility range, resource assumptions, reset-sensitive residency, numeric behavior, and golden/reference outputs. |
| Userspace runtime | Sessions, admission, priority/deadlines, profile selection, isolation policy, fallback decision, lifecycle telemetry. | Request/context ID, bundle hash, profile, dependencies, retry class, deadline, power hint, and fallback policy. |
| Kernel driver | IOMMU/DMA mappings, synchronization, buffer lifetime, queues, interrupts, process isolation, timeout/reset interface. | Atomic completion/error record, output-valid flag, fault scope, retryability, reset generation, counters. |
| Firmware | Privileged register and power/clock sequences, supported target-specific load/program/verify, local health, scoped reset, and operation ID/journal only where declared. | State transition, partial-side-effect status, resident-weight/profile generation, self-test result, quarantined capacity, and selected recovery-recipe version. |
| Hardware | Protection checks, atomic completion/error capture, counters, fault containment, and documented reset domains. | Which writes may still arrive after abort/reset and which state is actually cleared or retained. |
| Failure class | Required response | Recovery proof |
|---|---|---|
| Bundle/version/shape rejected | Reject before any hardware side effect and explain the failed legality/compatibility predicate. | Negative corpus across malformed metadata, boundary shapes, ABI ranges, and downgrade attempts. |
| Profile mismatch or expiry | Stop ACiM dispatch; select another valid profile, recalibrate, remap/recompile, or fall back digitally. | Temperature/voltage/age boundary tests and stale health-generation injection. |
| Weight load/program failure | For volatile images, reload the whole image or only chunks the loader declares independently idempotent, then prove identity/checksum. For a field-programmable target, resume only when the manifest declares resumable program/verify and the firmware can reconcile a crash-consistent journal with target state. Without that contract—or without the required readback—mark residency unknown and run the vendor-defined restart/reload recipe, remap/quarantine, or fall back digitally. Never blindly replay programming pulses. | Inject interruption at every declared commit boundary and across reset/power loss; prove unavailable operations are rejected, budgets are preserved, and residency identity is established before dispatch. |
| DMA/IOMMU/execution timeout | Mark output invalid, stop/drain, revoke descriptors, increment reset generation, invalidate affected queues/handles/residency, then reset the smallest safe scope. | Late completion/write injection; confirm no partial result or stale handle becomes valid. |
| Thermal or accuracy sentinel excursion | Throttle/reschedule/fallback; invalidate a profile outside its domain, run golden vectors, recalibrate or remap. | Worst-valid operating corners followed by recovery and renewed health publication. |
| Persistent tile fault | Quarantine resource, change health generation, invalidate placements built for old capacity, and remap/recompile. | Fault injection plus end-to-end task metric, latency, and capacity verification on degraded geometry. |
Journal rule: only a target that declares resumable programming may use this path. Its crash-consistent record should bind operation ID, bundle/weight-image hash, device and reset generation, last verified safe progress unit, verify result, and remaining pulse/endurance budget. Firmware commits at a vendor-defined safe boundary and reconciles with device state where readback exists; “last command sent” in host memory is not recovery proof.
Reset rule: a scoped reset increments a generation and invalidates every queue, handle, residency, and profile assumption listed by reset_effects. A journal survives only when its declared durability includes that reset or power-loss class. Retry only an idempotent command or a target-declared recoverable transaction; otherwise restart/reload through the versioned vendor recipe, remap/quarantine, or fall back digitally. Never return a partial result as valid.
07 · Calibration and learning loop
Treat analog behavior as changing system state
Characterization is not a one-time lab report. The deployed device produces evidence that can select a profile, remap a model, or trigger retraining.
Sweep target-specific weight state, converters, temperature, voltage, noise, drift or retention, and load/write behavior.
Fit distributions and costs at tile granularity; version them with hardware and firmware.
Quantize, train, partition, map around faults, choose ranges, and predict accuracy/efficiency.
Apply profile corrections and capture counters, sentinel checks, temperature, and timing.
Refresh corrections, reduce range, remap tiles, select digital fallback, or retrain.
For PCM/RRAM targets, model programming/read distributions; use verify, remapping, redundancy, or training-aware distributions.
Estimate output variance; improve margins, average selectively, or adjust precision and range.
Use the target-appropriate time model, timestamp profiles, monitor sentinels, and recalibrate or reload before budget violation.
Calibrate response curves, constrain operating regions, and include the effect in simulation/training.
Where converters are present, store target-appropriate gain/offset/nonlinearity corrections and detect saturation or dead lanes.
Maintain a target-appropriate fault map; use spares, structured remapping, resource quarantine, or digital correction.
Select profiles by operating band and throttle or migrate when a bound is crossed.
Run representative golden vectors or sentinels; connect observed error to a rollback/fallback policy.
08 · Bottleneck matrix
Find what limits the full inference
Measure utilization, wait time, bytes, conversions, error, and energy at every boundary. Peak array TOPS cannot reveal these bottlenecks.
| Bottleneck | What you observe | Likely cause | Design response | Trade-off to state |
|---|---|---|---|---|
| ADC energy / latency | Converters dominate cycles or joules; arrays wait for output conversion. | High resolution, too many conversions, narrow reuse, serialized lanes. | Lower precision/range, fuse work, share or parallelize ADCs, reduce partial sums, keep more work local. | Accuracy, area, calibration, and converter contention. |
| Input encoding / activation delivery | Input path cannot feed arrays; poor tile occupancy. | Drive/conversion phases, activation bandwidth, broadcast limits, high input precision, routing conflict. | Reuse/broadcast activations, compress, double-buffer, choose dataflow and placement around fan-out. | SRAM capacity, decode cost, and scheduling constraints. |
| Array underutilization | Many empty rows/columns; theoretical TOPS far above sustained throughput. | Small/irregular shapes, depthwise/grouped ops, excess precision slices. | Pack compatible matrices, reshape/fuse, specialized mapping, or keep the operator digital. | Packing complexity, isolation, and longer compile time. |
| Weight load / programming | Cold-start or model-switch latency; for nonvolatile arrays, verify retries and endurance consumption. | Weight-transfer bandwidth or, on resistive targets, slow writes, tight tolerance, model churn, and worn cells. | Cache resident models, preload in the background, batch loads; use adaptive tolerance and wear leveling when the device requires programming. | Memory capacity, accuracy, power, endurance, and model freshness. |
| Inter-tile traffic | NoC queues and partial-sum transfers dominate. | Fragmented placement, row splits, multicast pressure, remote accumulation. | Co-locate producer/consumer tiles, minimize cuts, multicast efficiently, place accumulation near data. | Fragmentation and reduced placement flexibility. |
| Digital accumulation | ACiM finishes but outputs wait in digital reduction. | Many tile/bit-slice partials, narrow accumulator throughput. | Wider/tree accumulators, schedule overlap, reduce splits, accumulate locally, tune precision. | Area, power, and overflow/error behavior. |
| Unsupported operators | Frequent CPU/digital fallback and synchronization. | Control flow, nonlinear/reduction ops, dynamic shapes, missing kernels. | Fuse boundaries, add digital kernels, improve legalization, keep contiguous digital subgraphs. | Compiler scope and digital silicon area. |
| Accuracy recovery | Mapping passes performance but violates task accuracy. | Noise, range saturation, quantization, drift, weak cells. | Noise-aware/QAT retraining, remap, recalibrate, increase precision selectively, fall back critical layers. | Training cost, latency, energy, and model portability. |
| Calibration overhead | Too much downtime or foreground latency; profiles expire often. | Fine-grained calibration, unstable conditions, excessive sweeps. | Hierarchical/targeted calibration, sentinels, background work, profile bands, confidence thresholds. | Residual error and fleet-management complexity. |
| Host ↔ NPU transfer | End-to-end latency tracks copy time, not compute. | Small partitions, redundant layout conversion, off-chip tensors. | Keep larger subgraphs resident, share mapped buffers only where ownership/coherency permits, fuse transforms, batch requests, and reuse buffers. | DMA setup/synchronization, cache maintenance, memory pressure, batching latency, and isolation. |
| Thermal / power limit | Frequency drops, queue latency spikes, accuracy profile mismatch. | Concurrent tiles/converters, sustained duty cycle, weak cooling. | Power-aware scheduling, stagger tiles, DVFS, thermal profiles, admission control. | Throughput, tail latency, and scheduling predictability. |
| Fault / drift churn | Increasing remaps, verify failures, accuracy alarms. | Aging, device defects, environmental changes. | Spare capacity, fault-aware placement, health scoring, predictive maintenance, digital escape path. | Usable capacity and long-term fleet variance. |
09 · Compiler + runtime quantitative workbench
Prove fit before quoting TOPS
Carry one matrix through effective geometry, signed encoding, precision slices, converter phases, residency waves, movement, critical-path latency, energy, and composed task accuracy.
Map a 512 × 512 matrix onto 16 one-array tiles
Assume y = Wx, W ∈ R^(M×K), K maps to rows, M maps to sensed columns, each array is 256 × 128, differential signed encoding costs two physical columns per logical value, weights are 4-bit in 2-bit cells, and 8-bit activations use a 2-bit drive path.
- Compute effective geometry
R_eff = 256andC_logical = floor(128 ÷ 2) = 64. Faults, spares, reserved columns, or converter topology can reduce these values further. - Split K and M
N_K = ceil(512 ÷ 256) = 2input blocks;N_M = ceil(512 ÷ 64) = 8output blocks. K splits create partial sums to reduce; M splits create independent outputs to concatenate. - Pay for weight precision
S_w = ceil(4 ÷ 2) = 2, so a fully spatial mapping needs2 × 8 × 2 = 32array placements. Sixteen one-array tiles requireW = ceil(32 ÷ 16) = 2schedule/residency waves. On volatile SRAM this means loading the next resident weight image when it is not already retained; on a field-programmable nonvolatile target it means reprogramming only if program time, energy, and endurance make that policy viable. - Pay for activation precision
P_a = ceil(8 ÷ 2) = 4digit-serial input phases. With all 16 arrays and the required converter lanes available concurrently, the mapping therefore needs at leastW × P_a = 2 × 4 = 8array-evaluation slots per input vector, before any extra converter serialization, setup, sensing, or reduction cycles. Spatial/temporal choices can move cost between arrays and cycles but cannot erase it. - Close the fit decisionOne vector performs
512 × 512 = 262,144MACs under the stated convention. Ask whether reuse or resident-wave batching amortizes SRAM reload—or, only where supported and viable, nonvolatile programming—without unacceptable activation/partial-result storage or latency. Then include at least eight evaluation slots, conversion, reductions, and engine transitions; otherwise keep it digital.
| Budget | Include | Correct reasoning | Evidence |
|---|---|---|---|
| Latency | DMA, layout conversion, weight load/program, input encoding, array evaluation, sensing/ADC, digital accumulation/activation, synchronization, fallback. | T_region = critical_path(scheduled event graph). Overlapped transfer and compute contribute through dependencies, not a naïve sum. | Warm and cold p50/p95/p99 by shape, arrival rate, batch, concurrency, profile, and operating corner. |
| Energy | Movement, amortized load/program, input drive, array, sensing/conversion, accumulation, digital work, leakage/idle, retries, calibration, fallback. | E_region,expected = Σ_p P(p) × E_complete_path(p) + E_calibration,window ÷ N_inferences,window, where paths distinguish nominal success, retry then success, fallback after partial ACiM work, early digital fallback, and rejection. Put each load/program/recovery action only in paths that execute it, amortize residency inside those path costs, and report nominal, expected, and worst-valid energy separately. | Per-rail or modeled component energy with synchronized counters and a declared SoC/accelerator boundary. |
| Accuracy | Quantization/rounding/overflow, clipping, mapping, analog noise/nonlinearity/saturation, PVT, drift/retention, faults, calibration, fallback boundaries. | Validate the composed system on representative data; use layer sensitivity to allocate an error budget. One percentage point and one percent relative loss are different. | Named metric/dataset from FP baseline through worst valid temperature, age, profile, and degraded geometry. |
| Sustained benefit | Useful operations, wall time, occupancy, queue gaps, converter duty, bytes moved, transitions, fallback, throttling, health maintenance. | Compare end-to-end against the best digital baseline with equal accuracy and power boundary—not against isolated array peak. | Representative models and request mixes after thermal equilibrium, including model switches and recovery. |
- FP baselineName model, dataset, metric, preprocessing, and reproducibility settings.
- Quantized idealIsolate storage type, scale, zero point, clipping, rounding, and overflow loss.
- Mapped idealAdd tiling, bit slices, accumulator behavior, and digital/analog boundaries without device noise.
- Calibrated nominalApply measured target profile and compare prediction with device output.
- Worst valid stateVerify temperature, voltage, age, fault map, converter saturation, and declared fallback.
Ask what one “tile” contains, signed-column cost, whether slices are spatial or temporal, effective geometry after faults/spares, and simultaneous converter lanes.
State whether weights load once per model residency, wake, switch, or inference; include interrupted load, checksum/readback, eviction, and health-generation changes.
Carry request/context/reset generation, bundle/profile identity, buffer descriptors, dependencies, deadline, retry class, and fallback policy into an atomic completion record.
Differential-test each IR against a digital reference, simulate calibrated distributions, then measure many devices/corners so profiles and cost/error models can be updated.
One-minute follow-up: Would you still choose ACiM if only eight converter lanes operate simultaneously, weights change every request, the worst-temperature profile loses two percentage points, or a reset invalidates resident weights? Recalculate waves, conversions, bytes, critical path, and fallback—not only placements.
10 · Full interview practice
Design an edge hybrid NPU software stack
Work each checkpoint aloud before opening the coaching note. The goal is a defensible system—not a memorized compiler diagram.
Deploy vision and audio models on a 5 W edge SoC
The SoC contains a quad-core CPU, digital vector NPU, 16 one-array ACiM tiles, shared SRAM, DMA/NoC, and analog converter periphery. Assume warm batch-1 requests from up to four concurrent contexts: vision arrives at 30 requests/s steady and bursts to 60 requests/s for 2 s, while audio arrives at 50 windows/s steady. Require p95 < 20 ms separately for vision and audio at the 80-request/s steady mix after thermal equilibrium, bounded backlog with an explicit per-class latency/drop policy during the 110-request/s burst, 5 W for the complete SoC, and less than one percentage-point task-metric loss on a named dataset per model unless you explicitly revise those assumptions.
- PyTorch + ONNX
- 80 req/s steady
- 110 req/s · 2 s burst
- 4 contexts
- per-class warm p95 < 20 ms at thermal steady state
- 5 W complete SoC
- < 1 percentage-point metric loss/model
- field drift
- secure updates
- digital fallback
| Thermally sustainable service μ | Peak backlog after 2 s at 110/s | Drain after return to 80/s | Interview conclusion |
|---|---|---|---|
| 80 requests/s | B_peak = (110 − 80) × 2 = 60 requests. | No drain: μ − λ_steady = 0. The 60-request queue remains. | Invalid no-drop design. A finite burst bounds memory, but not recovery time or latency; add capacity or reject/drop/coalesce work. |
| 100 requests/s | B_peak = (110 − 100) × 2 = 20 requests. | T_drain = 20 ÷ (100 − 80) = 1 s. | The last queued work sees roughly 20 ÷ 100 = 200 ms of fluid queueing before service, so the 20 ms objective fails even though the queue eventually drains. |
| 110 requests/s | Zero deterministic fluid backlog during the specified burst. | Nothing to drain. | This is only the arithmetic floor. At 100% utilization there is no margin for service variance, retries, calibration, or class interference, so it does not prove p95. |
| 120 requests/s | Zero deterministic fluid backlog with 10 requests/s burst headroom. | Nothing to drain in the fluid model. | A plausible no-drop candidate only after measured per-class queueing + service p95 stays below 20 ms at thermal equilibrium and complete-SoC power remains within 5 W. |
-
00–05 minClarify workload and success Separate model, product, silicon, lifecycle, and safety constraints.
Your turnWhich questions decide the architecture before you draw the stack?Reveal coaching
Confirm whether the supplied vision/audio rates, four-context concurrency, two-second burst, percentile boundary, and drop policy reflect product intent; then state the queue boundary and require steady arrival to remain below sustained thermally limited service. Ask about model families/operators and shapes; per-model deadlines and priority; energy and accuracy-loss budgets; determinism and cold start; model size/update frequency; ACiM array geometry/device precision and fault rate; converter/SRAM/NoC limits; target temperatures and lifetime; security/isolation; and whether the stack owns training. State exclusions, such as not designing the memory device itself.
Close the stated load: use
B_peak = max(0, λ_burst − μ) × 2 sand, only whenμ > 80/s,T_drain = B_peak ÷ (μ − 80/s). At μ = 80/s the backlog is 60 and never drains; at μ = 100/s it is 20, drains in one second, and can impose about 200 ms of queueing. Therefore define per-class admission/reservation and deadline behavior before claiming p95. -
05–10 minDefine the hardware/software contract Turn the accelerator into a target that a compiler can reason about.
Your turnWhat must the target expose beyond “16 tiles”?Reveal coaching
- Supported operator forms, data types, accumulator behavior, array rows/columns, cell precision and signed encoding.
- Tile topology, SRAM banks, alignment, DMA and NoC bandwidth, converter modes, barriers, queue semantics, interrupts, reset scope, and power states.
- Weight storage and retention domain; load granularity/idempotence; readback kind and granularity; program/verify mode; journal durability and safe commit unit; reset effects; recovery recipe/version.
- Versioned fault/calibration profile and latency/energy/error cost models. The compiler consumes capabilities; the runtime checks compatibility.
-
10–17 minDesign import, IR, and transforms Preserve model intent while progressively making hardware choices explicit.
Your turnHow does PyTorch or ONNX become an analog-aware executable?Reveal coaching
Import to a high-level graph IR; infer shapes and fold constants; canonicalize GEMM/convolution; legalize and fuse; quantize with representative data; annotate candidate ACiM regions; run noise-aware optimization; lower selected regions into a device IR with tiles, precision slices, buffers, transfers, synchronization, fallback, calibration/profile references; then emit commands and a debug map.
-
17–25 minPartition by whole-system cost Show that “supported” does not automatically mean “profitable.”
Your turnWhen should an eligible matrix operation remain digital?Reveal coaching
Keep it digital when the shape underfills arrays, transfers/conversions dominate, precision or accuracy recovery is too costly, weights change too often, program time exceeds reuse, dynamic shape/control prevents stable placement, an adjacent digital fusion is cheaper, the device profile is unhealthy, or the latency/energy model predicts no end-to-end win. Minimize engine boundaries, not just operator costs.
-
25–32 minMap, place, and schedule Translate selected regions into arrays, buffers, routes, and time.
Your turnHow do precision, faults, and converters change placement?Reveal coaching
- Tile input dimension K onto effective rows and output dimension M onto logical columns; bit-slice logical precision; encode signed weights; reserve target-specific spare capacity.
- Use the fault map to avoid bad resources and co-locate producers/consumers to reduce partial-sum traffic.
- Keep tensor quantization scales distinct from physical input-drive and sensing/ADC ranges; size digital accumulators for all slices and tiles.
- Double-buffer SRAM and overlap DMA, ACiM, accumulation, activation, and digital-engine work with explicit events and deadlines.
-
32–37 minDesign load and execution Cover the path that starts before the first inference.
Your turnWhat happens from signed bundle to a completed result?Reveal coaching
The runtime authenticates the bundle, checks ABI/firmware/capability compatibility, selects a calibration profile, and reserves resources. It asks firmware to load a volatile image, run an atomic vendor program operation, or resume program/verify only when the target explicitly supports that mode and its journal/reconciliation proof. The driver maps buffers, queues DMA/commands, and handles interrupts. Firmware owns the safe target-specific sequence, health, reset, and power state. On interruption, use declared readback and journaling where available; otherwise invalidate residency and run the vendor restart/reload recipe, remap/quarantine, or execute digital fallback.
-
37–41 minClose the calibration loop Make device variation observable and actionable.
Your turnHow does deployed evidence reach the next compilation?Reveal coaching
Version profiles by device/tile, firmware, temperature band, age, and test method. Collect saturation, converter statistics, program retries, faults, temperature, latency/energy, and periodic golden/sentinel outputs. Policy selects an existing profile, triggers targeted recalibration, remaps around degradation, chooses fallback, or exports a profile for fleet-level cost/error model updates and model recompilation. Protect telemetry privacy and authenticate profile updates.
-
41–45 minVerify and defend trade-offs Prove correctness, performance, resilience, and maintainability.
Your turnWhat test pyramid and production metrics make the stack trustworthy?Reveal coaching
- Unit-test graph transforms and command encoding; differential-test IR and digital reference; simulate calibrated noise/fault distributions; test firmware on emulation/silicon; run end-to-end representative datasets across devices, temperatures, and ages.
- Track task accuracy delta, output error distribution, p50/p95/p99 latency, sustained throughput, energy/inference, ACiM and converter utilization, bytes moved, fallback rate, calibration age, verify retries, and fault growth.
- State the central trade-off: analog efficiency is useful only when accuracy recovery, conversion, transfer, calibration, and lifecycle cost remain inside the product budget.
Did product targets, models, device limits, and lifecycle constraints shape the design?
Are capabilities, bundle, calibration, commands, telemetry, and versions explicit?
Did the cost include transfers, converters, programming, errors, and fallback?
Can the system detect drift/faults, degrade digitally, reset safely, and recover?
Can measured silicon behavior change profiles, mapping, models, and future compiles?
11 · Primary references
Study the stack from model to device
Start with portable graph and compiler contracts, then runtime/DMA ownership, architecture-level cost modeling, and device-specific accuracy evidence. Each source below anchors a different claim; no one simulator covers semantics, accuracy, latency, energy, area, and lifecycle.
IBM Analog Hardware Acceleration Kit
Open-source Python tooling with PyTorch integration for exploring in-memory computing devices and analog neural-network behavior.
IBM AIHWKit on GitHub ↗ONNX IR Specification
Graph, tensor, type, shape, version, metadata, and operator-set contracts that an importer must preserve before target lowering.
Read the ONNX IR specification ↗MLIR Dialect Conversion
Conversion targets, legal/illegal operations, rewrite patterns, and type conversion—the core vocabulary for “lower, explicitly convert, or fail.”
Read the MLIR guide ↗MLIR Quant Dialect
Stored and expressed types, scales, zero points, per-axis quantization, and the separation between tensor semantics and target converter settings.
Read the Quant dialect docs ↗IREE Stream and HAL
Asynchronous resource lifetimes, scheduling, devices, buffers, executables, command submission, and synchronization patterns for heterogeneous execution.
Study IREE Stream ↗ONNX Runtime architecture
Execution-provider capability discovery, graph partitioning, compilation, kernel execution, and fallback as a concrete heterogeneous-runtime reference.
Read the architecture guide ↗Linux DMA mapping guide
DMA mappings, direction, coherency, synchronization, barriers, descriptor ordering, and why a mapped/shared buffer is not automatically zero-copy-safe.
Read the kernel guide ↗Compiling Neural Networks for a Computational Memory Accelerator
A research software stack centered on the hardware/software interface and mapping neural-network models onto a multi-core computational-memory accelerator.
Read the paper on arXiv ↗ALPINE
A system architecture for tight integration of analog in-memory acceleration with multi-core CPUs, including an ISA extension and software library.
Read the ALPINE paper ↗Sandia CrossSim
An open-source resistive-crossbar accuracy/co-design simulator for analog hardware effects and neural-network inference. Its own scope excludes energy, area, and speed modeling.
Sandia CrossSim on GitHub ↗Accelergy
Architecture-level component action/count energy and area modeling; use measured target data to calibrate estimates and pair it with a dedicated compute-in-memory mapper/model.
Explore Accelergy on GitHub ↗CiMLoop
A compute-in-memory modeling framework that connects architecture, mapping, technology, and component models for energy, area, and performance exploration.
Explore CiMLoop on GitHub ↗Accurate DNN inference using PCM
Differential conductance mapping, programming/read error, drift, compensation, and experiment-backed analog-inference accuracy.
Read the Nature Communications paper ↗64-core PCM inference chip
Evidence that digital operations and on-chip communication remain necessary for useful end-to-end inference—not merely isolated array efficiency.
Read the Nature Electronics paper ↗A Charge Domain SRAM Compute-in-Memory Macro
An IEEE Journal of Solid-State Circuits design using C-2C ladder-based 8-bit MAC units, with analysis of PVT, parasitics, mismatch, noise, and multibit linearity.
Read the IEEE paper ↗