NPU + analog compute-in-memory

Design the whole stack. Close the analog loop.

A visual guide to lowering neural networks onto a hybrid edge NPU—where the CPU controls, digital engines cover irregular work, and ACiM arrays accelerate eligible matrix-vector operations.

  • 9software layers
  • 12bottlenecks
  • 1closed feedback loop
Hybrid edge NPUcontrol + higher-precision/reproducible digital paths + efficient analog
HostCPUModel lifecycle, control flow, orchestration, recovery
FabricDMA + NoC + SRAMMove and reuse activations without starving engines
Digital pathDigital vector coresNonlinear, reduction, fallback, and reproducible execution when the implementation guarantees it
Analog pathACiM tilesWeight-stationary matrix-vector multiply
PeripheryInput encoding / DAC → compute array → sensing / ADC → digital accumulationThe exact physical path is technology-specific; conversion, precision, noise, and movement decide the real benefit
Design rule: accelerate the end-to-end layer, not an isolated analog MAC.

01 · System boundary

Give each engine the work it handles best

An ACiM tile is one execution resource inside a heterogeneous system. The software stack must coordinate control, transfers, higher-precision or reproducible digital work when required, analog computation, and recovery.

Technology scopeThe two columns below are example target families: a nonvolatile resistive current-summing crossbar and a volatile SRAM charge-domain macro. Storage technology and analog signal domain are separate axes: SRAM designs may compute in charge, current, or time, while nonvolatile devices can also support charge-domain CiM. Record both axes before choosing lifecycle, error, and recovery behavior.
These example families can share graph partitioning, tiling, scheduling, and fallback passes—but their storage and compute-circuit contracts create different device lifecycles.
ConcernExample A · nonvolatile resistive current-summing crossbarExample B · volatile SRAM charge-domain macro
Weight stateNonvolatile conductance programmed toward a target value or acceptance window and verified when the target exposes that operation.Volatile SRAM bits; reload after state loss, non-retentive wake, eviction, or model switch. Charge is redistributed or accumulated during computation but is not the retained weight.
Main error sourcesProgramming/read error, drift, endurance, stuck devices, line resistance/IR drop, and converter/periphery limits.PVT variation, mismatch, parasitics, gain/offset, linearity, references/converters, SRAM integrity, and residency identity.
Profile dataWeight-to-conductance map, program/read distributions, drift time origin/model, fault state, verify/endurance state, and supported programming/readback contract.Per-macro/lane gain, offset, nonlinearity, valid PVT corners, retained-power assumptions, weight-image identity/checksum, and disabled resources.
RecoveryWhere supported: bounded program/verify continuation, remap, quarantine, recalibrate, recompile, or digital fallback.Reload lost/corrupt weights, recalibrate, disable a macro/lane, remap/recompile, or digital fallback.

Schema rule: keep technology-specific data in tagged variants and expose storage type, volatility/retention, and analog signal domain independently. At minimum, use weight_storage, retention_domain, analog_signal_domain, load_mode, readback.kind/granularity, program_verify.mode, journal.durability, reset_effects, and a versioned recovery_recipe_id as separate capability fields. The label “SRAM / charge-domain ACiM” is shorthand only for Example B, not a taxonomy that makes SRAM and charge-domain synonymous. Do not require conductance, drift, or endurance fields in a generic ACiM profile, and do not apply the resistive equation Iⱼ ≈ Σᵢ VᵢGᵢⱼ to a charge-domain SRAM macro.

Control plane

Host CPU

Loads model bundles, selects profiles, submits work, and orchestrates dynamic or unsupported regions; a digital vector NPU may execute fallback subgraphs rather than the CPU itself.

  • Application API
  • Model lifecycle
  • Fallback orchestration + recovery
Data plane

Runtime + fabric

Builds command queues and overlaps transfers, computation, and synchronization.

  • DMA descriptors
  • SRAM allocation
  • NoC routes and events
Execution plane

Hybrid NPU

Uses a costed partition rather than sending every nominal matrix operation to analog.

Digital engineActivation, pooling, reduction, and higher-precision/reproducible fallback when specified
ACiM engineDense or convolutional matrix-vector kernels with reusable weights
Physical plane

Devices + sensors

Store or stage weight states and report conditions that affect the computation.

  • Load or program-and-verify
  • Temperature, variation, and drift
  • Fault map and calibration
Memory hookCPU orchestrates → fabric feeds → digital completes → ACiM multiplies → telemetry corrects.

02 · Model-to-device stack

Lower intent into a calibrated command stream

Every layer consumes a contract from above and produces a more concrete artifact below. Characterization data travels upward and changes the result.

  1. 01Framework and model import

    Accept PyTorch, ONNX, or another exchange format. Resolve shapes, constants, layouts, dynamic dimensions, model metadata, and a representative calibration dataset.

  2. 02High-level graph IR

    Represent operators, tensors, shape constraints, quantization intent, control flow, and source-level debug names without prematurely committing to a tile or command format.

  3. 03Legalization and fusion

    Rewrite unsupported forms, fold constants, canonicalize convolution/GEMM, fuse activation or bias where legal, and expose boundaries where analog-to-digital transfers would erase the gain.

  4. 04Analog-aware optimization

    Choose weight/activation precision, scaling and clipping; run quantization-aware or noise-aware training; model device variation, drift, nonlinearity, saturation, and finite ADC resolution.

  5. 05ACiM versus digital partition

    Use capability, accuracy, latency, energy, conversion, transfer, and fallback costs. Keep irregular, higher-precision, small, or poorly utilized work on CPU/digital engines.

  6. 06Tensor mapping and placement

    Tile matrices to physical arrays; bit-slice precision; encode signed weights; place tiles around faults and bandwidth; route partial sums; choose tensor quantization scales separately from physical input-drive and ADC reference/full-scale settings; size digital accumulation widths.

  7. 07Memory planning and scheduling

    Allocate activation SRAM, arrange DMA, double-buffer stages, overlap CPU/digital/ACiM work, preserve dependencies, enforce thermal/power limits, and prevent the converters from becoming the serialized path.

  8. 08Code generation and model bundle

    Emit command buffers, microcode, DMA descriptors, barriers, weight-loading or programming payloads, fallback code, relocation data, calibration-profile IDs, and trace metadata.

  9. 09Runtime, driver, firmware, and hardware

    Load and authenticate bundles, load weights or program-and-verify target cells, submit queues, handle interrupts and faults, apply calibration, collect counters, reset safely, and feed measured behavior back to the toolchain.

Four IR levels, four proof obligations: lowering is not merely changing syntax. Each IR preserves earlier promises while making one new class of decisions explicit.
IR levelMakes explicitRequired proof before lowering
Model IR · semanticOperators, tensors, shapes, layouts, control flow, opset/version, source metadata.Imported semantics, shapes, types, attributes, and version conversions are valid.
Quantized IR · numericStorage and expressed types, scale, zero point, clipping, rounding, overflow, per-tensor/per-channel rules.Dequantized behavior and task metric remain inside the allocated numeric/accuracy budget.
Device IR · spatialEngine partition, tile coordinates, bit slices, physical ranges, profile constraints, buffers/lifetimes, transfers, events, fallback.Every operation, mapping, dependency, resource, and profile requirement is legal for the selected capability manifest.
Command IR · executableDMA descriptors, loads/programming, dispatches, barriers, relocations, completion/error records, recovery points.Buffer bounds, dependency closure, ABI compatibility, reset generation, and replay behavior are verified.

Compiler invariant: every lowering pass must prove legality, materialize an explicit conversion/fallback, or fail compilation. It must never silently change layout, clipping, rounding, overflow, or synchronization behavior.

03 · Compiler decision flow

“Matrix-heavy” is the start—not the answer

The partitioner must prove support, fit, accuracy, and whole-layer benefit before committing a region to ACiM.

  1. Gate 1Is the region ACiM-shaped?

    Dense/convolutional matrix-vector work, stable shapes, enough reuse, and a useful problem size.

  2. Gate 2Can it be legalized?

    Supported data layout, tensor dimensions, signed encoding, activation rules, and static parameters.

  3. Gate 3Can it fit?

    Array rows/columns, tile count, precision slices, spare capacity, SRAM, routes, and converter capacity.

  4. Gate 4Does accuracy survive?

    Simulate calibrated nonidealities, retrain or rescale, and compare against an explicit accuracy-loss budget.

  5. Gate 5Does the system win?

    Include DMA, ADC/DAC, accumulation, transitions, calibration, programming, and digital fallback—not peak MACs alone.

All gates pass → ACiM partition

Generate tile placement, analog parameters, transfers, schedule, commands, and a digital reference for verification.

Accuracy fails → repair and reevaluate

Change clipping/scales, precision, redundancy, mapping, calibration, or noise-aware training. Do not silently relax the budget.

Benefit fails → digital fallback

Keep it on CPU/digital NPU, fuse adjacent digital operations, and avoid expensive engine transitions.

Decision sentenceMap region R to ACiM because its measured end-to-end latency/energy improves within accuracy budget A under profile P; otherwise execute digitally.

04 · Mapping and scheduling

Turn one tensor operation into physical work

Array geometry, precision, converter range, faulty resources, buffers, and dataflow jointly determine the mapping.

Array-aware tiling

Split dimensions that do not fit

This guide assumes input channels drive rows and column currents encode outputs; some arrays transpose that convention. Split the input dimension into digitally accumulated partial sums and the output dimension into independent result slices.

(K0, M0)
input block 0 · output block 0
(K0, M1)
input block 0 · output block 1
(K1, M0)
input block 1 · output block 0
(K1, M1)
input block 1 · output block 1
Precision lowering

Bit-slice what a cell cannot represent

Represent a logical weight across cells, subarrays, or time steps. The mapper must account for extra conversions, shifts, and accumulation.

MSB slice
high significance
+middle slice+LSB slice
fine detail
wide digital accumulator
Signed weightsDifferential pair, offset encoding, or separate positive/negative arrays—each changes utilization and error.
Analog range

Use the converters’ useful window

Tensor quantization scales map stored integers to expressed values. Physical drive and ADC reference/full-scale settings map device signals to codes. Calibration connects these separate contracts; it does not make them the same knob.

Quantized activation + tensor scale
Input encoding / DAC setting
Technology-specific array response
Sensing / ADC setting + code
Dequantize + accumulate
WatchSaturation, low signal-to-noise ratio, periphery energy, calibration mismatch, and range changes across batches.
Pipeline schedule

Hide transfers behind useful work

Double-buffer activations so DMA fills one buffer while an engine consumes the other. Explicit events preserve data dependencies.

Illustrative event schedule: a buffer may be consumed only after its fill-complete event; reuse waits for the consumer-complete event.
StageT0T1T2T3
DMA Afill Aidlefill Aidle
ACiMidleuse Ause Buse A
DMA Bidlefill Bidlefill B
Digitalidleidleaccumulateactivate

05 · Hardware/software contract

Make every boundary a versioned artifact

A production stack is debuggable when the compiler, runtime, firmware, and silicon agree on explicit formats rather than hidden assumptions. The names and sample filenames below are illustrative Atlas design proposals, not ONNX, MLIR, IREE, or PJRT standards.

Compiler input · illustrative

Capability manifest

Legal operator forms, geometry, memory/periphery limits, ABI and costs—plus explicit weight storage/retention, load mode, readback kind, program/verify mode, journal durability, reset effects, and vendor recovery recipe.

target_v3.capabilities.json
Compiler middle

Multi-level IR

Graph IR preserves model meaning; device IR records partitions, tiles, buffers, synchronization, analog attributes, and digital fallback.

graph → quantized → scheduled → command IR
Compiler output · illustrative

Signed model bundle

Commands, weights, relocations, buffer plan, profile requirements, compatibility range, fallback entry points, debug map, golden outputs, and security metadata.

model.acim.bundle
Device state · target-specific

Calibration profile

Target/stepping/ABI identity, method and provenance, validity domain, uncertainty/confidence, fault/spare state, and corrections at a practical macro/tile/row/column/converter—or selected-cell—granularity.

{target, health generation, PVT, age}
Runtime input · illustrative

Execution request

Bundle/profile ID, context and reset generation, tensor descriptors, dependency events, deadline, priority, power hint, reproducibility requirement, retry class, fallback policy, and trace ID.

enqueue(context, tensors, deadline)
Runtime output

Telemetry schema

Per-stage latency, bytes moved, converter activity, saturation, retries, program failures, temperature, power, fault events, and output checks.

trace → profiler → cost model → recompile
Capabilities describe what a hardware design can legally do; a calibration profile describes how one physical device behaves now. Do not hide both behind one “target config.”
ContractRequired fieldsConsumerFailure behavior
Capability manifestVendor/device/revision, firmware ABI, legal ops/dtypes/layouts/shapes, effective geometry and signed encoding, periphery/accumulator limits, memory/DMA/coherency, queues/reset/power, cost-model version/units, errata.Compiler proves legality and estimates cost; runtime confirms the loaded bundle still matches device and firmware identity.Reject, explicitly convert, partition elsewhere, or compile a declared fallback—never silently approximate an illegal form.
Weight-state operationsweight_storage (volatile, factory-programmed, or field-programmable), retention domain, load granularity/idempotence, readback (none, checksum, state, or conductance) with granularity/cost, program_verify (unsupported, atomic, or resumable) with safe progress unit and budgets, journal durability/commit boundary, reset effects, and versioned recovery recipe.The compiler emits only supported load/program commands; firmware owns the target sequence; the runtime selects restart, reload, remap, quarantine, or fallback from the declared contract.If a field or operation is absent, reject that path. Resistive storage alone does not imply field programming, conductance readback, incremental resume, or a persistent journal.
Calibration/health profileTarget/stepping/ABI/schema/health generation, method/provenance/time/expiry, units/sample count/uncertainty/confidence, valid temperature-voltage-frequency-load-age domain, corrections, faults/spares/effective geometry, integrity and rollback policy.Compiler maps against effective capacity/error; runtime selects only a profile valid for this device, operating point, and bundle.Select another profile, recalibrate, remap/recompile, or execute digital fallback; do not extrapolate outside the measured validity domain without an explicit bound.

Runtime predicate: usable = target_match ∧ abi_match ∧ profile_schema_match ∧ health_generation_match ∧ operating_point ∈ validity_domain ∧ not_expired. Sign the identity and compatibility fields that protect this decision.

06 · Runtime, driver, and firmware

Execute safely from model load to completion

The steady-state inference path should be fast, but first load, failure, recalibration, reset, and firmware compatibility are equally part of the design.

01Authenticate bundle

Verify signature, versions, target compatibility, limits, and permissions.

02Select profile

Match device, tile health, temperature, age, and required accuracy.

03Load or program weights

Branch on the manifest: load a volatile image, invoke an atomic vendor program transaction, or continue only a target-declared resumable program/verify operation. Never infer resume support from device technology.

04Verify readiness

Use the declared proof—image checksum/ECC, vendor self-test, state readback, or conductance readback. If readback is unavailable, use the vendor restart/reload recipe or quarantine/remap/fallback; do not guess partial progress.

05Submit queues

Bind buffers, DMA, commands, events, deadlines, and power hints.

06Run + observe

Handle interrupts, saturation, ECC/fault events, timeouts, thermal throttling, and traces.

07Complete or recover

Return outputs only after readiness proof; otherwise run the versioned vendor recovery recipe, fall back digitally, reset a documented scope, or invalidate residency/profile state.

Runtime invariantA timeout or reset must never leave unknown weights, aliased buffers, or a stale ownership token visible to the next request.
Draw who owns every unsafe transition. Shared responsibility without one decision owner becomes a recovery gap.
LayerOwnsMust expose across the boundary
Compiler + bundleStatic partition, placement, schedule, relocations, profile requirements, fallback entry points, debug/source map.Compatibility range, resource assumptions, reset-sensitive residency, numeric behavior, and golden/reference outputs.
Userspace runtimeSessions, admission, priority/deadlines, profile selection, isolation policy, fallback decision, lifecycle telemetry.Request/context ID, bundle hash, profile, dependencies, retry class, deadline, power hint, and fallback policy.
Kernel driverIOMMU/DMA mappings, synchronization, buffer lifetime, queues, interrupts, process isolation, timeout/reset interface.Atomic completion/error record, output-valid flag, fault scope, retryability, reset generation, counters.
FirmwarePrivileged register and power/clock sequences, supported target-specific load/program/verify, local health, scoped reset, and operation ID/journal only where declared.State transition, partial-side-effect status, resident-weight/profile generation, self-test result, quarantined capacity, and selected recovery-recipe version.
HardwareProtection checks, atomic completion/error capture, counters, fault containment, and documented reset domains.Which writes may still arrive after abort/reset and which state is actually cleared or retained.
Failure is a state machine, not a footer. Recovery completes only after ownership, state, profile validity, and self-test are re-established.
Failure classRequired responseRecovery proof
Bundle/version/shape rejectedReject before any hardware side effect and explain the failed legality/compatibility predicate.Negative corpus across malformed metadata, boundary shapes, ABI ranges, and downgrade attempts.
Profile mismatch or expiryStop ACiM dispatch; select another valid profile, recalibrate, remap/recompile, or fall back digitally.Temperature/voltage/age boundary tests and stale health-generation injection.
Weight load/program failureFor volatile images, reload the whole image or only chunks the loader declares independently idempotent, then prove identity/checksum. For a field-programmable target, resume only when the manifest declares resumable program/verify and the firmware can reconcile a crash-consistent journal with target state. Without that contract—or without the required readback—mark residency unknown and run the vendor-defined restart/reload recipe, remap/quarantine, or fall back digitally. Never blindly replay programming pulses.Inject interruption at every declared commit boundary and across reset/power loss; prove unavailable operations are rejected, budgets are preserved, and residency identity is established before dispatch.
DMA/IOMMU/execution timeoutMark output invalid, stop/drain, revoke descriptors, increment reset generation, invalidate affected queues/handles/residency, then reset the smallest safe scope.Late completion/write injection; confirm no partial result or stale handle becomes valid.
Thermal or accuracy sentinel excursionThrottle/reschedule/fallback; invalidate a profile outside its domain, run golden vectors, recalibrate or remap.Worst-valid operating corners followed by recovery and renewed health publication.
Persistent tile faultQuarantine resource, change health generation, invalidate placements built for old capacity, and remap/recompile.Fault injection plus end-to-end task metric, latency, and capacity verification on degraded geometry.

Journal rule: only a target that declares resumable programming may use this path. Its crash-consistent record should bind operation ID, bundle/weight-image hash, device and reset generation, last verified safe progress unit, verify result, and remaining pulse/endurance budget. Firmware commits at a vendor-defined safe boundary and reconciles with device state where readback exists; “last command sent” in host memory is not recovery proof.

Reset rule: a scoped reset increments a generation and invalidates every queue, handle, residency, and profile assumption listed by reset_effects. A journal survives only when its declared durability includes that reset or power-loss class. Retry only an idempotent command or a target-declared recoverable transaction; otherwise restart/reload through the versioned vendor recipe, remap/quarantine, or fall back digitally. Never return a partial result as valid.

07 · Calibration and learning loop

Treat analog behavior as changing system state

Characterization is not a one-time lab report. The deployed device produces evidence that can select a profile, remap a model, or trigger retraining.

CharacterizeMeasure the device

Sweep target-specific weight state, converters, temperature, voltage, noise, drift or retention, and load/write behavior.

ModelBuild error + cost profiles

Fit distributions and costs at tile granularity; version them with hardware and firmware.

CompileOptimize and place

Quantize, train, partition, map around faults, choose ranges, and predict accuracy/efficiency.

ExecuteRun the workload

Apply profile corrections and capture counters, sentinel checks, temperature, and timing.

AdaptRecalibrate or recompile

Refresh corrections, reduce range, remap tiles, select digital fallback, or retrain.

Resistive programming variation

For PCM/RRAM targets, model programming/read distributions; use verify, remapping, redundancy, or training-aware distributions.

Temporal/read noise

Estimate output variance; improve margins, average selectively, or adjust precision and range.

Drift, retention, and aging

Use the target-appropriate time model, timestamp profiles, monitor sentinels, and recalibrate or reload before budget violation.

Nonlinearity

Calibrate response curves, constrain operating regions, and include the effect in simulation/training.

Converter/reference error

Where converters are present, store target-appropriate gain/offset/nonlinearity corrections and detect saturation or dead lanes.

Faulty cells or macros

Maintain a target-appropriate fault map; use spares, structured remapping, resource quarantine, or digital correction.

Temperature/voltage

Select profiles by operating band and throttle or migrate when a bound is crossed.

Model accuracy drift

Run representative golden vectors or sentinels; connect observed error to a rollback/fallback policy.

08 · Bottleneck matrix

Find what limits the full inference

Measure utilization, wait time, bytes, conversions, error, and energy at every boundary. Peak array TOPS cannot reveal these bottlenecks.

Common full-inference limits, their evidence, likely cause, first response, and the trade-off an interview answer must state.
BottleneckWhat you observeLikely causeDesign responseTrade-off to state
ADC energy / latencyConverters dominate cycles or joules; arrays wait for output conversion.High resolution, too many conversions, narrow reuse, serialized lanes.Lower precision/range, fuse work, share or parallelize ADCs, reduce partial sums, keep more work local.Accuracy, area, calibration, and converter contention.
Input encoding / activation deliveryInput path cannot feed arrays; poor tile occupancy.Drive/conversion phases, activation bandwidth, broadcast limits, high input precision, routing conflict.Reuse/broadcast activations, compress, double-buffer, choose dataflow and placement around fan-out.SRAM capacity, decode cost, and scheduling constraints.
Array underutilizationMany empty rows/columns; theoretical TOPS far above sustained throughput.Small/irregular shapes, depthwise/grouped ops, excess precision slices.Pack compatible matrices, reshape/fuse, specialized mapping, or keep the operator digital.Packing complexity, isolation, and longer compile time.
Weight load / programmingCold-start or model-switch latency; for nonvolatile arrays, verify retries and endurance consumption.Weight-transfer bandwidth or, on resistive targets, slow writes, tight tolerance, model churn, and worn cells.Cache resident models, preload in the background, batch loads; use adaptive tolerance and wear leveling when the device requires programming.Memory capacity, accuracy, power, endurance, and model freshness.
Inter-tile trafficNoC queues and partial-sum transfers dominate.Fragmented placement, row splits, multicast pressure, remote accumulation.Co-locate producer/consumer tiles, minimize cuts, multicast efficiently, place accumulation near data.Fragmentation and reduced placement flexibility.
Digital accumulationACiM finishes but outputs wait in digital reduction.Many tile/bit-slice partials, narrow accumulator throughput.Wider/tree accumulators, schedule overlap, reduce splits, accumulate locally, tune precision.Area, power, and overflow/error behavior.
Unsupported operatorsFrequent CPU/digital fallback and synchronization.Control flow, nonlinear/reduction ops, dynamic shapes, missing kernels.Fuse boundaries, add digital kernels, improve legalization, keep contiguous digital subgraphs.Compiler scope and digital silicon area.
Accuracy recoveryMapping passes performance but violates task accuracy.Noise, range saturation, quantization, drift, weak cells.Noise-aware/QAT retraining, remap, recalibrate, increase precision selectively, fall back critical layers.Training cost, latency, energy, and model portability.
Calibration overheadToo much downtime or foreground latency; profiles expire often.Fine-grained calibration, unstable conditions, excessive sweeps.Hierarchical/targeted calibration, sentinels, background work, profile bands, confidence thresholds.Residual error and fleet-management complexity.
Host ↔ NPU transferEnd-to-end latency tracks copy time, not compute.Small partitions, redundant layout conversion, off-chip tensors.Keep larger subgraphs resident, share mapped buffers only where ownership/coherency permits, fuse transforms, batch requests, and reuse buffers.DMA setup/synchronization, cache maintenance, memory pressure, batching latency, and isolation.
Thermal / power limitFrequency drops, queue latency spikes, accuracy profile mismatch.Concurrent tiles/converters, sustained duty cycle, weak cooling.Power-aware scheduling, stagger tiles, DVFS, thermal profiles, admission control.Throughput, tail latency, and scheduling predictability.
Fault / drift churnIncreasing remaps, verify failures, accuracy alarms.Aging, device defects, environmental changes.Spare capacity, fault-aware placement, health scoring, predictive maintenance, digital escape path.Usable capacity and long-term fleet variance.

09 · Compiler + runtime quantitative workbench

Prove fit before quoting TOPS

Carry one matrix through effective geometry, signed encoding, precision slices, converter phases, residency waves, movement, critical-path latency, energy, and composed task accuracy.

Worked interview fit test · illustrative target

Map a 512 × 512 matrix onto 16 one-array tiles

Assume y = Wx, W ∈ R^(M×K), K maps to rows, M maps to sensed columns, each array is 256 × 128, differential signed encoding costs two physical columns per logical value, weights are 4-bit in 2-bit cells, and 8-bit activations use a 2-bit drive path.

  1. Compute effective geometryR_eff = 256 and C_logical = floor(128 ÷ 2) = 64. Faults, spares, reserved columns, or converter topology can reduce these values further.
  2. Split K and MN_K = ceil(512 ÷ 256) = 2 input blocks; N_M = ceil(512 ÷ 64) = 8 output blocks. K splits create partial sums to reduce; M splits create independent outputs to concatenate.
  3. Pay for weight precisionS_w = ceil(4 ÷ 2) = 2, so a fully spatial mapping needs 2 × 8 × 2 = 32 array placements. Sixteen one-array tiles require W = ceil(32 ÷ 16) = 2 schedule/residency waves. On volatile SRAM this means loading the next resident weight image when it is not already retained; on a field-programmable nonvolatile target it means reprogramming only if program time, energy, and endurance make that policy viable.
  4. Pay for activation precisionP_a = ceil(8 ÷ 2) = 4 digit-serial input phases. With all 16 arrays and the required converter lanes available concurrently, the mapping therefore needs at least W × P_a = 2 × 4 = 8 array-evaluation slots per input vector, before any extra converter serialization, setup, sensing, or reduction cycles. Spatial/temporal choices can move cost between arrays and cycles but cannot erase it.
  5. Close the fit decisionOne vector performs 512 × 512 = 262,144 MACs under the stated convention. Ask whether reuse or resident-wave batching amortizes SRAM reload—or, only where supported and viable, nonvolatile programming—without unacceptable activation/partial-result storage or latency. Then include at least eight evaluation slots, conversion, reductions, and engine transitions; otherwise keep it digital.
Budget the scheduled critical path and composed accuracy envelope. Do not sum stages that overlap, and do not add interacting task-metric losses blindly.
BudgetIncludeCorrect reasoningEvidence
LatencyDMA, layout conversion, weight load/program, input encoding, array evaluation, sensing/ADC, digital accumulation/activation, synchronization, fallback.T_region = critical_path(scheduled event graph). Overlapped transfer and compute contribute through dependencies, not a naïve sum.Warm and cold p50/p95/p99 by shape, arrival rate, batch, concurrency, profile, and operating corner.
EnergyMovement, amortized load/program, input drive, array, sensing/conversion, accumulation, digital work, leakage/idle, retries, calibration, fallback.E_region,expected = Σ_p P(p) × E_complete_path(p) + E_calibration,window ÷ N_inferences,window, where paths distinguish nominal success, retry then success, fallback after partial ACiM work, early digital fallback, and rejection. Put each load/program/recovery action only in paths that execute it, amortize residency inside those path costs, and report nominal, expected, and worst-valid energy separately.Per-rail or modeled component energy with synchronized counters and a declared SoC/accelerator boundary.
AccuracyQuantization/rounding/overflow, clipping, mapping, analog noise/nonlinearity/saturation, PVT, drift/retention, faults, calibration, fallback boundaries.Validate the composed system on representative data; use layer sensitivity to allocate an error budget. One percentage point and one percent relative loss are different.Named metric/dataset from FP baseline through worst valid temperature, age, profile, and degraded geometry.
Sustained benefitUseful operations, wall time, occupancy, queue gaps, converter duty, bytes moved, transitions, fallback, throttling, health maintenance.Compare end-to-end against the best digital baseline with equal accuracy and power boundary—not against isolated array peak.Representative models and request mixes after thermal equilibrium, including model switches and recovery.
  1. FP baselineName model, dataset, metric, preprocessing, and reproducibility settings.
  2. Quantized idealIsolate storage type, scale, zero point, clipping, rounding, and overflow loss.
  3. Mapped idealAdd tiling, bit slices, accumulator behavior, and digital/analog boundaries without device noise.
  4. Calibrated nominalApply measured target profile and compare prediction with device output.
  5. Worst valid stateVerify temperature, voltage, age, fault map, converter saturation, and declared fallback.
FitGeometry is only the first gate

Ask what one “tile” contains, signed-column cost, whether slices are spatial or temporal, effective geometry after faults/spares, and simultaneous converter lanes.

ResidencyProgram/load cost needs reuse

State whether weights load once per model residency, wake, switch, or inference; include interrupted load, checksum/readback, eviction, and health-generation changes.

RuntimeOutput validity is explicit

Carry request/context/reset generation, bundle/profile identity, buffer descriptors, dependencies, deadline, retry class, and fallback policy into an atomic completion record.

EvidenceModels and silicon close the loop

Differential-test each IR against a digital reference, simulate calibrated distributions, then measure many devices/corners so profiles and cost/error models can be updated.

One-minute follow-up: Would you still choose ACiM if only eight converter lanes operate simultaneously, weights change every request, the worst-temperature profile loses two percentage points, or a reset invalidates resident weights? Recalculate waves, conversions, bytes, critical path, and fallback—not only placements.

10 · Full interview practice

Design an edge hybrid NPU software stack

Work each checkpoint aloud before opening the coaching note. The goal is a defensible system—not a memorized compiler diagram.

45-minute prompt

Deploy vision and audio models on a 5 W edge SoC

The SoC contains a quad-core CPU, digital vector NPU, 16 one-array ACiM tiles, shared SRAM, DMA/NoC, and analog converter periphery. Assume warm batch-1 requests from up to four concurrent contexts: vision arrives at 30 requests/s steady and bursts to 60 requests/s for 2 s, while audio arrives at 50 windows/s steady. Require p95 < 20 ms separately for vision and audio at the 80-request/s steady mix after thermal equilibrium, bounded backlog with an explicit per-class latency/drop policy during the 110-request/s burst, 5 W for the complete SoC, and less than one percentage-point task-metric loss on a named dataset per model unless you explicitly revise those assumptions.

  • PyTorch + ONNX
  • 80 req/s steady
  • 110 req/s · 2 s burst
  • 4 contexts
  • per-class warm p95 < 20 ms at thermal steady state
  • 5 W complete SoC
  • < 1 percentage-point metric loss/model
  • field drift
  • secure updates
  • digital fallback
Close the burst with numbers before drawing queues. This illustrative fluid model treats requests as equal-cost work; a real design repeats the calculation in measured service-time or compute-work units for vision and audio separately.
Thermally sustainable service μPeak backlog after 2 s at 110/sDrain after return to 80/sInterview conclusion
80 requests/sB_peak = (110 − 80) × 2 = 60 requests.No drain: μ − λ_steady = 0. The 60-request queue remains.Invalid no-drop design. A finite burst bounds memory, but not recovery time or latency; add capacity or reject/drop/coalesce work.
100 requests/sB_peak = (110 − 100) × 2 = 20 requests.T_drain = 20 ÷ (100 − 80) = 1 s.The last queued work sees roughly 20 ÷ 100 = 200 ms of fluid queueing before service, so the 20 ms objective fails even though the queue eventually drains.
110 requests/sZero deterministic fluid backlog during the specified burst.Nothing to drain.This is only the arithmetic floor. At 100% utilization there is no margin for service variance, retries, calibration, or class interference, so it does not prove p95.
120 requests/sZero deterministic fluid backlog with 10 requests/s burst headroom.Nothing to drain in the fluid model.A plausible no-drop candidate only after measured per-class queueing + service p95 stays below 20 ms at thermal equilibrium and complete-SoC power remains within 5 W.
One defensible overload policy at μ = 100/sReserve service for all 50 audio windows/s, then cap accepted vision at no more than 50/s. During the two-second 60/s vision burst, accept at most 100 of 120 frames and drop or coalesce at least 20 according to a declared freshness policy. That stops deterministic growth, but 100/s is still a saturation ceiling—not p95 proof—so lower the admission cap or provision more capacity for queueing and service-time headroom. Measure each class’s total queueing-plus-service latency, not a mixed average. If neither class may drop, the design must sustain the 110/s mix with additional margin.
Model + calibration data
Graph compiler + analog optimizer
Partition + mapper + scheduler
Signed model bundle
Runtime + driver + firmware
Hybrid NPU + telemetry
  1. 00–05 minClarify workload and success

    Separate model, product, silicon, lifecycle, and safety constraints.

    Your turnWhich questions decide the architecture before you draw the stack?
    Reveal coaching

    Confirm whether the supplied vision/audio rates, four-context concurrency, two-second burst, percentile boundary, and drop policy reflect product intent; then state the queue boundary and require steady arrival to remain below sustained thermally limited service. Ask about model families/operators and shapes; per-model deadlines and priority; energy and accuracy-loss budgets; determinism and cold start; model size/update frequency; ACiM array geometry/device precision and fault rate; converter/SRAM/NoC limits; target temperatures and lifetime; security/isolation; and whether the stack owns training. State exclusions, such as not designing the memory device itself.

    Close the stated load: use B_peak = max(0, λ_burst − μ) × 2 s and, only when μ > 80/s, T_drain = B_peak ÷ (μ − 80/s). At μ = 80/s the backlog is 60 and never drains; at μ = 100/s it is 20, drains in one second, and can impose about 200 ms of queueing. Therefore define per-class admission/reservation and deadline behavior before claiming p95.

  2. 05–10 minDefine the hardware/software contract

    Turn the accelerator into a target that a compiler can reason about.

    Your turnWhat must the target expose beyond “16 tiles”?
    Reveal coaching
    • Supported operator forms, data types, accumulator behavior, array rows/columns, cell precision and signed encoding.
    • Tile topology, SRAM banks, alignment, DMA and NoC bandwidth, converter modes, barriers, queue semantics, interrupts, reset scope, and power states.
    • Weight storage and retention domain; load granularity/idempotence; readback kind and granularity; program/verify mode; journal durability and safe commit unit; reset effects; recovery recipe/version.
    • Versioned fault/calibration profile and latency/energy/error cost models. The compiler consumes capabilities; the runtime checks compatibility.
  3. 10–17 minDesign import, IR, and transforms

    Preserve model intent while progressively making hardware choices explicit.

    Your turnHow does PyTorch or ONNX become an analog-aware executable?
    Reveal coaching

    Import to a high-level graph IR; infer shapes and fold constants; canonicalize GEMM/convolution; legalize and fuse; quantize with representative data; annotate candidate ACiM regions; run noise-aware optimization; lower selected regions into a device IR with tiles, precision slices, buffers, transfers, synchronization, fallback, calibration/profile references; then emit commands and a debug map.

  4. 17–25 minPartition by whole-system cost

    Show that “supported” does not automatically mean “profitable.”

    Your turnWhen should an eligible matrix operation remain digital?
    Reveal coaching

    Keep it digital when the shape underfills arrays, transfers/conversions dominate, precision or accuracy recovery is too costly, weights change too often, program time exceeds reuse, dynamic shape/control prevents stable placement, an adjacent digital fusion is cheaper, the device profile is unhealthy, or the latency/energy model predicts no end-to-end win. Minimize engine boundaries, not just operator costs.

  5. 25–32 minMap, place, and schedule

    Translate selected regions into arrays, buffers, routes, and time.

    Your turnHow do precision, faults, and converters change placement?
    Reveal coaching
    • Tile input dimension K onto effective rows and output dimension M onto logical columns; bit-slice logical precision; encode signed weights; reserve target-specific spare capacity.
    • Use the fault map to avoid bad resources and co-locate producers/consumers to reduce partial-sum traffic.
    • Keep tensor quantization scales distinct from physical input-drive and sensing/ADC ranges; size digital accumulators for all slices and tiles.
    • Double-buffer SRAM and overlap DMA, ACiM, accumulation, activation, and digital-engine work with explicit events and deadlines.
  6. 32–37 minDesign load and execution

    Cover the path that starts before the first inference.

    Your turnWhat happens from signed bundle to a completed result?
    Reveal coaching

    The runtime authenticates the bundle, checks ABI/firmware/capability compatibility, selects a calibration profile, and reserves resources. It asks firmware to load a volatile image, run an atomic vendor program operation, or resume program/verify only when the target explicitly supports that mode and its journal/reconciliation proof. The driver maps buffers, queues DMA/commands, and handles interrupts. Firmware owns the safe target-specific sequence, health, reset, and power state. On interruption, use declared readback and journaling where available; otherwise invalidate residency and run the vendor restart/reload recipe, remap/quarantine, or execute digital fallback.

  7. 37–41 minClose the calibration loop

    Make device variation observable and actionable.

    Your turnHow does deployed evidence reach the next compilation?
    Reveal coaching

    Version profiles by device/tile, firmware, temperature band, age, and test method. Collect saturation, converter statistics, program retries, faults, temperature, latency/energy, and periodic golden/sentinel outputs. Policy selects an existing profile, triggers targeted recalibration, remaps around degradation, chooses fallback, or exports a profile for fleet-level cost/error model updates and model recompilation. Protect telemetry privacy and authenticate profile updates.

  8. 41–45 minVerify and defend trade-offs

    Prove correctness, performance, resilience, and maintainability.

    Your turnWhat test pyramid and production metrics make the stack trustworthy?
    Reveal coaching
    • Unit-test graph transforms and command encoding; differential-test IR and digital reference; simulate calibrated noise/fault distributions; test firmware on emulation/silicon; run end-to-end representative datasets across devices, temperatures, and ages.
    • Track task accuracy delta, output error distribution, p50/p95/p99 latency, sustained throughput, energy/inference, ACiM and converter utilization, bytes moved, fallback rate, calibration age, verify retries, and fault growth.
    • State the central trade-off: analog efficiency is useful only when accuracy recovery, conversion, transfer, calibration, and lifecycle cost remain inside the product budget.
Requirements

Did product targets, models, device limits, and lifecycle constraints shape the design?

Contracts

Are capabilities, bundle, calibration, commands, telemetry, and versions explicit?

Partition

Did the cost include transfers, converters, programming, errors, and fallback?

Failure

Can the system detect drift/faults, degrade digitally, reset safely, and recover?

Evidence

Can measured silicon behavior change profiles, mapping, models, and future compiles?

11 · Primary references

Study the stack from model to device

Start with portable graph and compiler contracts, then runtime/DMA ownership, architecture-level cost modeling, and device-specific accuracy evidence. Each source below anchors a different claim; no one simulator covers semantics, accuracy, latency, energy, area, and lifecycle.

Framework + simulation

IBM Analog Hardware Acceleration Kit

Open-source Python tooling with PyTorch integration for exploring in-memory computing devices and analog neural-network behavior.

IBM AIHWKit on GitHub ↗
Portable model semantics

ONNX IR Specification

Graph, tensor, type, shape, version, metadata, and operator-set contracts that an importer must preserve before target lowering.

Read the ONNX IR specification ↗
Compiler legality

MLIR Dialect Conversion

Conversion targets, legal/illegal operations, rewrite patterns, and type conversion—the core vocabulary for “lower, explicitly convert, or fail.”

Read the MLIR guide ↗
Numeric contract

MLIR Quant Dialect

Stored and expressed types, scales, zero points, per-axis quantization, and the separation between tensor semantics and target converter settings.

Read the Quant dialect docs ↗
Scheduling + lifetimes

IREE Stream and HAL

Asynchronous resource lifetimes, scheduling, devices, buffers, executables, command submission, and synchronization patterns for heterogeneous execution.

Study IREE Stream ↗
Heterogeneous partitioning

ONNX Runtime architecture

Execution-provider capability discovery, graph partitioning, compilation, kernel execution, and fallback as a concrete heterogeneous-runtime reference.

Read the architecture guide ↗
Buffer ownership

Linux DMA mapping guide

DMA mappings, direction, coherency, synchronization, barriers, descriptor ordering, and why a mapped/shared buffer is not automatically zero-copy-safe.

Read the kernel guide ↗
Compiler architecture

Compiling Neural Networks for a Computational Memory Accelerator

A research software stack centered on the hardware/software interface and mapping neural-network models onto a multi-core computational-memory accelerator.

Read the paper on arXiv ↗
CPU + accelerator integration

ALPINE

A system architecture for tight integration of analog in-memory acceleration with multi-core CPUs, including an ISA extension and software library.

Read the ALPINE paper ↗
Crossbar accuracy + nonidealities

Sandia CrossSim

An open-source resistive-crossbar accuracy/co-design simulator for analog hardware effects and neural-network inference. Its own scope excludes energy, area, and speed modeling.

Sandia CrossSim on GitHub ↗
Architecture cost modeling

Accelergy

Architecture-level component action/count energy and area modeling; use measured target data to calibrate estimates and pair it with a dedicated compute-in-memory mapper/model.

Explore Accelergy on GitHub ↗
Compute-in-memory modeling

CiMLoop

A compute-in-memory modeling framework that connects architecture, mapping, technology, and component models for energy, area, and performance exploration.

Explore CiMLoop on GitHub ↗
Resistive-device evidence

Accurate DNN inference using PCM

Differential conductance mapping, programming/read error, drift, compensation, and experiment-backed analog-inference accuracy.

Read the Nature Communications paper ↗
End-to-end mixed-signal system

64-core PCM inference chip

Evidence that digital operations and on-chip communication remain necessary for useful end-to-end inference—not merely isolated array efficiency.

Read the Nature Electronics paper ↗
SRAM ACiM

A Charge Domain SRAM Compute-in-Memory Macro

An IEEE Journal of Solid-State Circuits design using C-2C ladder-based 8-bit MAC units, with analysis of PVT, parasitics, mismatch, noise, and multibit linearity.

Read the IEEE paper ↗
Practice loopChoose one model → establish a digital baseline → add calibrated nonidealities → partition and map → measure end-to-end → explain the trade-off.