NPU Architecture, Compiler & Performance Study Lab

Read the role correctly

Each experience statement asks for inspectable engineering evidence

Memorizing accelerator vocabulary is not enough. For each requirement, be ready to show the decision you made, the contract you wrote, the experiment you ran, and the evidence that changed or confirmed the design.

Architecture, ISA, verification Derive compute, memory, interconnect, control, precision, and observability from a workload envelope. Produce an architecture and ISA review

Conv, GEMM, Attention Derive work and traffic, classify the bottleneck, select a mapping, and verify numerical parity. Produce operator optimization reports

Compiler, RTL, driver co-design Trace one feature through operator legality, lowering, commands, hardware behavior, submission, and tests. Produce a cross-team contract

Product and commercialization Balance end-to-end targets, PPA and system cost with compatibility, security, recovery, and field support. Produce a product-readiness review

SystemC/TLM and simulators Choose explicit fidelity, model transactions and timing, expose counters, and measure correlation error. Produce a validated virtual platform

Fabless accelerator delivery Own the product, architecture, RTL, software, verification, and customer outcome while coordinating foundry, IP, package, and test partners. Produce a stage-gate and bring-up plan

Open-source compiler customization Select the correct TVM, MLIR, LLVM, or OpenXLA extension seam and sustain it with pinned builds, tests, and an upstream policy. Produce an extension decision record

Frontend compiler, backend interface, and runtime Develop graph-level optimization passes and operator-level lowering passes, schedule dataflow and memory, emit a versioned backend contract, then execute, manage resources, and profile compiled code. Produce a boundary contract and performance trace

Modern models and low precision Classify model workload signatures and negotiate format, scaling, accumulation, accuracy, conversion, and fallback across the stack. Produce a workload and precision report

Accelerator DSL and integration Define precise operations, types, effects, legality, diagnostics, staged lowering, versioning, and a controlled escape hatch. Produce a DSL-to-device lowering trace

Companion foundations Reuse the existing compiler, runtime, driver, firmware, SDK, profiling, and debugging references. Open the SoC software path

The logic flow

Architecture starts with a scenario and ends with reproducible proof

Do not begin by choosing an array size or inventing opcodes. Each step consumes a reviewed artifact from the previous step and adds measurable obligations for the next one.

01 · CharacterizeModel workloadFamilies, operators, shapes, state, arrivals, precision, quality

02 · BudgetProduct targetsp50/p99 latency, throughput, energy, memory, cost, thermal

03 · ModelBounds and sweepsOperations, bytes, reuse, Roofline, sensitivity, bottleneck

04 · DesignArchitecture and ISAEngines, dataflow, hierarchy, commands, numerical rules

05 · RealizeCompiler and runtimeDSL, lowering, artifact, firmware, driver, execution, telemetry

06 · QualifyCorrelated product proofGolden, simulator, RTL, tape-out, silicon, SDK, sustaining

Keep three claims separate. Functional verification asks whether results and faults match the specification. Performance validation asks whether a fixed implementation meets its targets. Model correlation asks how closely one abstraction predicts another. Passing one does not imply the other two.

Architecture and ISA definition

Translate workload reuse and product constraints into explicit resources

A useful architecture document connects every resource to a workload need and every software-visible behavior to a test. The ISA is the stable contract; pipelines, arbitration, and implementation scheduling remain microarchitectural unless software can observe them.

Compute budgetrequired ops/s ≥ work ÷ deadlineInclude tail utilization, unsupported work, sparsity rules, and precision-specific peak rates.

Bandwidth budgetrequired B/s ≥ transferred bytes ÷ deadlineCount traffic at a named boundary: DRAM, NoC, SRAM bank, or register file.

Capacity budgetlive bytes = tensors + tiles + queues + stateInclude alignment, double buffers, fragmentation, firmware, and concurrent contexts.

Roofline boundrate ≤ min(peak, bandwidth × intensity)Operational intensity is work divided by bytes observed at the same memory boundary.

Architecture plane	Decisions to make	Evidence to demand	Common failure
Compute	Matrix dimensions, vector/scalar lanes, accumulator width, special functions, sparsity, supported precisions.	Shape-utilization sweep, dependency schedule, numerical tests, area and timing estimates.	Quoting peak TOPS while awkward shapes, reductions, or elementwise work leave engines idle.
Memory	Register files, banked SRAM, cache or scratchpad policy, DMA, compression, address translation, coherency.	Per-level traffic, bank conflicts, burst efficiency, residency plan, worst-case live set.	Counting ideal reuse that the actual tile capacity or layout cannot retain.
Interconnect	Topology, multicast, reduction, routing, virtual channels, arbitration, quality of service, backpressure.	Contention tests, saturation curves, deadlock proof, fairness and tail-latency distributions.	Sizing links from average traffic while synchronized bursts overload one cut of the network.
Execution	Static schedule or dynamic issue, queues, dependencies, preemption, context state, firmware ownership.	Queue occupancy, overlap timeline, progress guarantees, timeout/reset and multi-context tests.	Optimizing engine cycles while host launch, DMA, queue wait, or firmware dominates latency.
Observability	Stable IDs, counters, trace events, error records, snapshots, clock correlation, replay data.	Counter self-tests, trace schema, overflow behavior, timestamp uncertainty, first-divergence workflow.	Discovering after RTL freeze that no counter can distinguish compute stalls from memory stalls.

Normative contract

ISA and command definition checklist

Encoding, reserved values, decode precedence, capability discovery, and version negotiation.
Operand shapes, ranges, strides, layouts, alignment, address spaces, and alias rules.
Accumulation width, rounding, saturation, overflow, NaN behavior, determinism, and quantization.
Dependencies, barriers, fences, completion visibility, coherency, and memory ordering.
Privilege, isolation, IOMMU behavior, context state, cancellation, timeout, reset, and watchdog.
Illegal commands, page faults, engine faults, partial writes, recovery, and error propagation.
Debug state, counters, trace identity, assembler/disassembler, simulator, and conformance tests.

Example: one command as a testable contractIllustrative YAML

command: GEMM_V1
encoding:
  opcode: 0x18
  descriptor_bytes: 64
operands:
  A: {dtype: int8, layout: row_major, align: 64}
  B: {dtype: int8, layout: blocked_k32, align: 64}
  C: {dtype: int32, layout: row_major, align: 64}
semantics:
  value: C[m,n] = sum_k(A[m,k] * B[k,n])
  accumulator: signed_int32
  overflow: wrap_twos_complement
ordering:
  reads_after: dependency_tokens
  completion:
    order: release
    scope: system
    covers: [C_writes, status_record]
faults:
  bad_alignment: INVALID_DESCRIPTOR
  unmapped_iova: PAGE_FAULT
  unsupported_shape: UNSUPPORTED
observability:
  counters: [active_cycles, macs, sram_stall_cycles]
  trace: [command_id, start_tick, end_tick, status]

Convention warning: this page uses two arithmetic operations per multiply-accumulate when presenting GEMM/Conv operation counts. Hardware data sheets sometimes count one MAC as one operation and sometimes as two. State the convention before comparing throughput claims.

Deep dive: ISA and device integration Deep dive: interface contracts Deep dive: measurement formulas

Modern model architectures and low-precision inference

Treat each model as a workload signature and each datatype as a full-stack contract

Model names change faster than silicon. Study the stable architectural signatures: dominant operators, tensor shapes, arithmetic intensity, persistent state, dynamic control, communication, and the end-to-end service goal. Then decide precision with measured task quality and real kernels—not bit width alone.

Model family	Workload signature to study	Accelerator consequence	Required experiment
Dense decoder Transformer	Causal attention, RoPE, GQA/MQA, gated FFN, normalization, KV cache, continuous batching; separate prompt prefill from token decode.	Prefill exposes large GEMMs; low-batch decode is often bandwidth- or launch-bound as it repeatedly moves weights and KV state through small GEMM/GEMV work. Batching, cache layout, and inter-token tail latency matter.	Sweep batch, prompt, generated length, KV precision, and memory bandwidth; report time-to-first-token and inter-token latency separately.
Vision and multimodal Transformer	Patchification, encoders, projectors, cross-attention, variable image/video token counts, ragged inputs, and modality preprocessing.	Resolution and patch size sharply change sequence length. Layout conversion, uncached encoder work, cache/reuse policy, host preprocessing, and variable-shape batching can dominate.	Measure complete request latency across resolutions and modalities, including preprocessing, transfers, and reused versus uncached encoders.
Diffusion models / DiT	A diffusion system may use a Conv or Transformer denoiser; DiT specifically uses latent patch tokens and Transformer blocks. Study conditioning, guidance, scheduler, repeated model evaluations, and fixed text/VAE stages.	Denoiser work grows roughly with the number of model evaluations, while fixed encode/decode and scheduler work remains. High-resolution activations, capacity, repeated weight reuse, and conditioning matter more than one block benchmark.	Profile one denoiser evaluation, fixed stages, and the full generation loop; sweep resolution, evaluations, guidance, batch, memory high-water mark, and output quality.
Sparse mixture of experts	Router, top-k selection, token permutation, capacity, expert imbalance, expert GEMMs, and all-to-all communication when experts are sharded across devices.	Nominal active FLOPs omit gather/scatter, padding, irregular tiles, routing latency, communication, and stragglers. Placement and load-aware scheduling become architectural.	Simulate skewed routing; report expert load, padding, tile utilization, bytes, communication time, and p99—not only average active parameters.
State-space / hybrid sequence model	Selective state update, scan, short causal Conv, recurrent state, fusion boundaries, plus any interleaved attention layers.	Linear sequence complexity does not guarantee utilization. Dependencies, elementwise work, state traffic, fusion, and recurrent versus parallel execution determine speed.	Compare full-sequence and one-token modes; attribute time to projection GEMMs, scan/state movement, elementwise work, synchronization, and fallback.
Recommendation / sparse features	Large embedding tables, random gathers, pooling, feature interaction, dense MLPs, sharding, and model-parallel traffic.	Capacity, locality, memory latency, and interconnect can outweigh tensor TOPS. Sparse and dense phases need different resource paths and scheduling policies.	Sweep table size and access distribution; measure hit rate, effective bandwidth, gather latency, sharding traffic, and transition cost into dense compute.

Architecture rule: “state of the art” is not a list of model names. Maintain a versioned workload census and trace each family to operator coverage, shapes, state, precision, memory traffic, control flow, system objective, and a reproducible model artifact.

BF16 / FP16range versus precisionBF16 has an FP32-like exponent range with fewer fraction bits; FP16 has a narrower range with more fraction bits. Specify conversion, subnormals, special values, accumulation, and fallback.

FP8 / microscalingtensor scale versus block scaleConventional FP8 may use current or delayed per-tensor scaling; amax history belongs to delayed recipes. MX formats share compact scales across fixed blocks, constraining metadata, layout, and tiles.

INT8 activation + weightq = clamp(round(x / s) + z)Define symmetric/asymmetric mode, per-tensor/channel scales, zero point, INT32 accumulation, bias, requantization, rounding, and saturation.

INT4 weights and activationspacked values + group scalesSeparate weight-only W4A16/BF16 from W4A8 and fully low-bit W4A4/FP4. Count packing, scale metadata, alignment, dequantization, mixed fallback, and native kernel coverage.

Precision stage	Engineering work	Evidence and gate
1 · Baseline	Freeze model revision, tokenizer/preprocessing, dataset, seeds, FP32/BF16 quality, shapes, latency, memory, and power boundary.	Reproducible reference metrics and tensor probes; no precision claim without the same evaluation contract.
2 · Calibrate	Select representative data; sweep clipping, symmetric/asymmetric and tensor/channel/group/block granularity; inspect outliers, saturation, SQNR, and layer error.	Calibration is separate from evaluation; rare and adversarial cases are retained. Use quantization-aware training (QAT) when post-training quantization (PTQ) cannot meet the quality target.
3 · Transform	Apply PTQ or QAT, equalization or outlier handling such as SmoothQuant, choose weight-only versus W8A8, and mark higher-precision islands.	Task quality, not only tensor MSE, passes across the required data distribution with a documented fallback policy.
4 · Lower	Insert or fold conversions; propagate scales; pack weights and metadata; select aligned fused kernels; plan layouts, accumulation, and epilogues.	Compiler dumps show every conversion, fallback, and kernel choice; unsupported shapes fail clearly or follow the declared path.
5 · Verify	Compare a bit-accurate reference, simulator, RTL, and silicon on ties, overflow, minimum values, special floats, partial blocks, and mixed accumulation.	Integer conversion and packing conformance is exact; reordered floating reductions follow a declared tolerance; end-task quality passes its separate threshold.
6 · Measure	Measure end-to-end latency, throughput, bytes, capacity, energy, conversion/fallback time, scale traffic, and quality.	Fewer bits count as an optimization only when the deployed kernels and boundaries improve the product scenario.

Normative contract

Write precision semantics before selecting peak TOPS

Encoding, exponent/mantissa or integer range, scale and zero-point ownership, and metadata layout.
Per-tensor/channel/group/block granularity, block shape, packing order, alignment, and tail rules.
Rounding mode, saturation or wrap, accumulator width, bias precision, reduction order, and determinism.
NaN, Inf, signed zero, subnormal, divide-by-zero, overflow, underflow, and conversion behavior.
Supported operator/shape/layout combinations, higher-precision islands, fallback, capability discovery, and revisioning.

Quality + systems proof

Avoid the low-bit performance illusion

Count scale, zero-point, packed-weight, padding, conversion, and temporary-buffer traffic.
Confirm that the target shape uses a native kernel rather than unpacking into a wider fallback.
Measure end-task perplexity, accuracy, generation or perceptual quality and worst cases—not layer MSE alone.
Separate theoretical footprint from allocated memory and theoretical operations/s from full-request latency.
Preserve one bit-accurate executable reference shared by compiler, simulator, RTL, firmware, and validation.

Practice: Transformer foundations Practice: quantization Next: map the dominant operators

Conv, GEMM, and Attention

Use the same operator loop: semantics → work → traffic → mapping → proof

Operator names hide the cases that determine a product: depthwise and grouped Conv, skinny and batched GEMM, attention prefill versus decode, odd tails, dynamic shapes, masks, layouts, and low-precision edge behavior.

NCHW notation

Convolution is a family of reuse patterns, not one kernel

For output N×K×P×Q, filter K×(C/G)×R×S, and G groups, the conventional count is 2×N×P×Q×K×(C/G)×R×S operations. First derive P and Q from padding, stride, and dilation; then inspect reuse and tail behavior.

ClassifyRegular, 1×1, depthwise, grouped, dilated, or transposed; each changes reuse, reduction width, and array utilization.

MapChoose direct, implicit GEMM, explicit im2col, or Winograd; tile N/P/Q/K/C and account for halo fetches.

OptimizeRetain activations/weights/partial sums, align channels, double-buffer, compress zeros, and fuse bias/activation/pooling.

ProveRandomized shape parity, padding/dilation/groups, quantized rounding, edge tiles, traffic counters, and end-to-end layout cost.

Key trap: explicit im2col simplifies GEMM reuse but can replicate activations and inflate bandwidth. Implicit GEMM forms needed tiles during load, trading address-generation complexity for lower external traffic.

Case	Likely limiter	First counter or experiment	High-value lever
3×3 regular Conv	SRAM/NoC traffic or MAC occupancy, depending on retained reuse.	Bytes and reuse per level; halo overhead; active MAC lanes.	Spatial/channel tiling, row/output stationary mapping, double buffering.
Depthwise Conv	Bandwidth and launch overhead; little cross-channel reuse.	Operations per transferred byte and vector-lane utilization.	Fuse adjacent pointwise/activation work and specialize layout.
Large square GEMM	Compute when blocking retains operands effectively.	Array active cycles, partial-sum spills, SRAM bank conflicts.	Hierarchical blocking, pipelined copies, aligned vector access.
Skinny GEMM	Array underfill, reduction/launch overhead, or bandwidth.	Per-dimension lane occupancy and fill/drain cycles.	Batch fusion, alternate array mapping, vector fallback, persistent kernel.
Attention prefill	Matrix compute plus score-matrix IO for naïve algorithms.	Score bytes, Q/K/V reuse, softmax/vector overlap, tensor utilization.	IO-aware tiled attention with online softmax and fusion.
Attention decode	KV-cache capacity/bandwidth and small GEMM utilization.	KV bytes per output token, cache hit/page behavior, and inter-token latency; track time-to-first-token separately with prefill.	KV layout/paging, GQA/MQA when model permits, request batching.

Frontend compilers, backend interfaces, accelerator runtimes, and domain-specific languages

Choose the smallest durable extension seam, then preserve semantics to the device

The frontend owns a verified IR handoff contract for the backend: graph optimization, operator lowering, dataflow and memory scheduling, IR invariants, and provenance. The backend turns that contract into target code; the runtime owns concrete execution and resource lifetimes. Correlated evidence must flow back across all three.

01 · ImportValidate semanticsIR/opset version, types, possible shapes, effects, source IDs

02 · OptimizeTransform the graphFold, CSE/DCE, canonicalize, graph/operator fusion, propagate layout/precision

03 · DecideSelect ownershipContract-based capability query, cost selection, partition, specialization, fallback

04 · Lower & scheduleCommit the implementationOperator/type conversion, tiles, layouts, memory spaces, liveness, DMA overlap

05 · Handoff & generateIR contract → target artifactBackend physical/resource legality, kernels/instructions, code object, executable ABI

06 · RunExecute and correlateValidate/cache, allocate, queue, synchronize, profile, recover, feed back

OptimizePreserve defined observable semantics while improving a cost, usually without the deliberate target commitment of lowering; prove equivalence and measure the change.

PartitionChoose which compiler or device owns each legal region; keep unsupported and legal-but-not-selected reasons distinct.

LowerMove to an IR with fewer abstractions and more target commitments while preserving padding, broadcasting, numerics, effects, and errors.

ScheduleChoose when and where computation and movement occur: producer-consumer or loop fusion, tiles, layouts, memory spaces, lifetimes, overlap, barriers, and tails.

Extension seam	Use it when	Implementation evidence	Boundary to respect
TVM BYOC / external codegen	A supported Relax subgraph can be partitioned and handed to an external compiler or library while TVM retains graph execution.	Registered patterns and legality predicates; before/after partitioned IR; separate selection/cost decision; `RunCodegen` output; runtime module; fallback and copy trace.	First determine legal support from shape, dtype, layout, quantization, alignment, and revision. Then let a separate cost/partition policy decide whether supported work is selected; report “unsupported” versus “supported, not selected.”
Native TVM target/runtime	The accelerator needs target properties, native code generation, runtime modules, and a device API rather than a bounded library offload.	Target definition; TIR schedules/tensorization; artifact metadata; module load; allocation/copy/stream behavior; compile-host versus run-device tests.	Compile-time codegen consumes declared target properties; it must not assume it can query the eventual device.
MLIR dialect and conversion	A reusable domain or accelerator IR needs custom operations, types, attributes, interfaces, verification, transformations, and progressive lowering.	ODS/TableGen definitions; parser/printer; valid and invalid tests; canonicalization; conversion legality; type conversion; complete pass-pipeline snapshots.	Publish precise semantics and require full conversion at boundaries; reject leftover illegal operations and unresolved casts.
LLVM target/backend	A real custom ISA needs ABI, instruction selection, scheduling, register allocation, object emission, assembly/disassembly, and relocation support.	Data layout and calling convention; instruction/register descriptions; legalization; scheduling model; MC encoder/decoder; object and round-trip tests.	Do not add a core LLVM IR instruction just to represent one accelerator feature. Prefer intrinsics, target-specific nodes, or a higher-level dialect unless universal semantics justify the change.
StableHLO + XLA FFI	A portable HLO graph needs a bounded target-specific escape hatch with typed runtime binding.	Versioned StableHLO; namespaced `custom_call`; typed handler; shape/rank, dtype, backend attributes, aliasing, side-effect, and ABI metadata; compatibility tests.	Custom calls weaken portability. The current XLA FFI API/ABI is documented as experimental, so pin versions and qualify supported combinations.
PJRT plugin	The whole accelerator should appear to frameworks through a uniform device interface rather than a single external operation.	Client and device discovery; buffers/transfers; compile and loaded executable; execution/events/errors; topology; packaging; framework smoke test.	Version the framework binding, PJRT API, portable artifact, compiler, runtime, driver, and hardware capability as one tested matrix.

Internal customization policy

Make the fork burden visible

Write an extension-seam decision record covering ownership, fallback, performance, portability, integration cost, and expected lifetime.
Pin the upstream commit and toolchain; make builds reproducible; record patches, licenses, security response, and supported host platforms.
Isolate vendor dialects, targets, runtime modules, and feature flags. Keep unavoidable core patches small and review whether generic fixes can go upstream.
Budget rebases; test both the pinned baseline and candidate upgrades; retain IR reproducers for pass crashes and changed optimization decisions.
Qualify old and new model artifacts, compiler/runtime/driver versions, devices, and cached executables before an internal release.

Acceptance suite

Test decisions, not only generated answers

Golden import and IR snapshots, round trips, verifier negatives, pass determinism, and source-location preservation.
Differential operator/model results across boundary shapes, layouts, dtypes, quantization, dynamic dimensions, and hardware revisions.
Visible partition/fallback reasons, cost-model decisions, host/device copies, cache keys, compile time, and unsupported-input diagnostics.
Artifact load/reject behavior, buffer lifetime, stream ordering, cancellation, asynchronous errors, faults, and recovery.
Stable profiling IDs connecting framework node → IR operation → command → counter/trace event → device result.

Owner	Core responsibilities	Inspectable artifact	Typical failure
Frontend graph optimizer	Import, type and shape inference; canonicalization, folding, CSE/DCE; fusion; layout and precision propagation; capability-aware partitioning.	Versioned input, before/after IR, pass statistics, semantic differential, and support, rejection, or fallback log.	A topology-only pattern ignores extra consumers, effects, dtypes, or dynamic cases; over-fusion increases live ranges, spills, and transfers.
Frontend lowering and scheduler	Decompose, legalize, and convert types; lower operators to target or kernel IR; choose tiles, layouts, memory spaces, liveness/reuse, bufferization, and DMA overlap.	Legality matrix, lowering snapshots, memory plan and live ranges, traffic/capacity estimate, and edge/tail tests.	Lowering loses broadcast, padding, reduction, or numerical semantics; early bufferization can freeze alias/reuse decisions, introduce allocations or copies, and prevent later tensor-level fusion.
Frontend ↔ backend IR contract	Version operations, types, possible shapes, layouts, precision, effects, memory-space intent, target capabilities, rejection reasons, and source/profile provenance.	Machine-readable capability and IR schemas, verifier and negative vectors, supported/unsupported decision logs, and two-revision consumer tests.	An implicit semantic or legality rule causes a code-generation mismatch, silent fallback, lost provenance, or wrong result.
Backend compiler and code generation	Enforce target legality; map and schedule target operations; select kernels or instructions; allocate registers and scratch resources; emit, link, and relocate code.	Code object or disassembly, resource report, symbol/relocation map, backend diagnostics, and source-to-kernel profile map.	Frontend-legal work is physically infeasible, or backend layout, buffer-order, numerical, or resource decisions erase the predicted gain.
Backend ↔ runtime executable contract	Version code objects, entry points, argument bindings, constants, layouts, alignment, ownership/lifetimes, workspace, dependencies/events, target/ISA/firmware/driver compatibility, cache identity, and debug maps.	Executable manifest and ABI schema, loader conformance and negative vectors, compatibility matrix, cache-key tests, and two-revision load/execute tests.	A changed binding, lifetime, workspace, event, compatibility, or cache rule causes load rejection, stale execution, corruption, a leak, or a race.
Runtime	Discover devices/topology; validate, load, and cache artifacts; manage quotas, pools, buffers, and transfers; submit queues/events; synchronize; dispatch; recover; profile.	Lifecycle trace, cache key, allocation high-water mark and fragmentation, queue timeline, structured errors, and device capability record.	Hidden copies, premature reclamation, asynchronous timing without a fence, stale cache, fragmentation, or command overhead dominates execution.
Joint performance loop	Separate compile, load, allocate, copy, queue, device, and synchronization time; correlate stable IDs; classify the bottleneck; assign an owner and test one change.	Aligned graph/IR/kernel/runtime/counter trace, before/after distributions and counters, correctness/quality result, and cost-model update.	A microkernel improves while end-to-end or p99 worsens; an estimate is reported as measured data; learning never reaches the compiler or architecture model.

Boundary diagnosis rule: preserve stable identity from framework node → graph IR → lowered operation → backend kernel/command → runtime event → device counter. First label a case unsupported, legal but not selected, or selected and failed. Then split latency into compile, load, allocation, copy, queue, device, and synchronization time; inspect reformats, spills, fragmentation, launches, fallback, and utilization. Change one factor, recheck correctness or task quality plus p50/p99, throughput, and peak memory, then feed the result into capability, cost, and architecture models.

Example accelerator DSL contractIllustrative, not executable syntax

kernel qgemm(A: i8[M,K], B: i8[K,N],
             a_scale: f32, b_scale: f32[N],
             c_scale: f32, C: i8[M,N])
  requires K % 32 == 0 and K <= 65535
    and a_scale > 0 and all(b_scale > 0) and c_scale > 0
    and aligned(A, 64) and aligned(B, 64)

  numerics {
    zero_points: A = 0, B = 0, C = 0
    accumulator: acc[m,n] = sum_k(i32(A[m,k]) * i32(B[k,n]))
    real_scale:  r[n] = a_scale * b_scale[n] / c_scale
    output:      C[m,n] = saturate_i8(round_nearest_even(acc[m,n] * r[n]))
  }

  schedule non_semantic {
    tile M by 64, N by 64, K by 32
    double_buffer A, B in scratchpad
    map [M, N] to matrix_engine
  }

  lowers_to @npu.gemm_v1
  fallback  @reference.qgemm

DSL design review

Specify meaning before syntax

Purpose: name the user and abstraction boundary. Separate algorithmic semantics from a retargetable schedule.
Types: define shapes, dynamic dimensions, layouts, precision/scales, memory spaces, ownership, aliases, effects, and synchronization.
Numerics: define accumulation, reduction order, overflow, saturation, rounding, NaNs, determinism, and approximation tolerance.
Legality: implement verifiers with source-located diagnostics and invalid-program tests; do not defer every error to code generation.
Lowering: retain before/after IR for semantic NPU → tiled accelerator → buffer/command → LLVM IR or custom ISA stages.
Evolution: version operations and artifacts, preserve profiling identity, provide a bounded escape hatch, and document deprecation.

Capstone proof: compile one fused graph through StableHLO or Relax → graph optimization → semantic NPU DSL → scheduled/bufferized IR → backend target code → executable + metadata → runtime → mock driver/device. Retain partition and lowering decisions, every IR stage, reference diffs, interface and artifact compatibility, buffer/event traces, unsupported cases, and a correlated latency breakdown of compile, load, allocation, transfer, queue, device, and synchronization time.

Companion: TVM path Companion: scheduling and memory planning Companion: runtime and queues Foundation: LLVM and MLIR

Performance verification

Build a ladder of models and measure the error between adjacent rungs

A fast analytical model explores a large design space; a detailed timing model explains contention; RTL proves implementation behavior; FPGA and silicon reveal physical effects. None is a universal oracle.

Level 0Golden semanticsReference math, legal ranges, numerical and fault behavior

Level 1Analytical boundsOperations, compulsory bytes, Roofline, capacity, sensitivity

Level 2Functional / TLMCommands, resources, latency annotations, queues, contention

Level 3RTL / emulationCycle behavior, protocol assertions, coverage, long regressions

Level 4FPGA / siliconClock, power, thermal, DRAM, OS, firmware, real counters

Verification layer	Required tests	Required outputs	Gate
Operator	Directed corners, seeded random shapes, dtype/layout/quantization/mask cases, negative constraints.	Reference diff, tolerance policy, first mismatch, reproducible seed.	Every legal case matches; every illegal case returns the specified error without unsafe side effects.
ISA / commands	Encode/decode round trips, legal and illegal streams, dependency cycles, misalignment, reset and fault injection.	Architectural state diff, command trace, coverage of fields and faults.	Functional simulator and RTL retire the same visible state or a documented exception.
Microarchitecture	Compute, memory, NoC, queue, arbitration and backpressure saturation sweeps.	Cycles, bytes, utilization, stalls, occupancy, service distributions.	Counters reconcile with events and modeled conservation rules; no unexplained cycles or bytes.
Networks	Representative models/shapes, concurrency, cold/warm start, fallback, sustained thermal load.	p50/p90/p99, throughput, energy/inference, accuracy, memory high-water mark.	Scenario-level product targets pass, including pre/post-processing and boundary transfers.
Correlation	Same artifact, input, clock, memory parameters, software versions, and trace IDs on adjacent models.	Absolute/relative error, bias by workload class, confidence interval, root-cause tags.	Error stays within an agreed envelope; calibration never hides an unmodeled mechanism.

Benchmark hygiene

Fix the measurement boundary

State warm-up, repetitions, clocks, temperature, batch and arrival process, included transfers, compile/load time, and percentile calculation.

Counter hygiene

Reconcile rather than decorate

Check active + stall + idle against elapsed cycles; requested versus delivered bytes; issued versus retired work; overflow and multiplexing behavior.

Claim hygiene

Carry correctness with speed

A performance result is invalid if operator semantics, model accuracy, precision, fallback, preprocessing, or concurrency changed outside the declared experiment.

Cross-functional architecture requirements

One feature must remain traceable from product intent to a passing test

Compiler, RTL, firmware, driver, runtime, validation, and product teams should review the same behavioral contract. A requirement is ready only when it is measurable, owned, observable, and testable.

Use caseCustomer outcomeScenario, workload, accuracy, latency, lifecycle

CompilerLegal loweringShapes, layout, precision, tiling, fallback, artifact

ISA / ABIVisible commandEncoding, ordering, faults, version, telemetry

RTL / firmwareExecution behaviorResources, scheduling, progress, counters, reset

Driver / runtimeSystem behaviorBuffers, IOVA, queues, fences, timeout, recovery

VerificationAcceptance evidenceGolden, negative, stress, performance, correlation

Shared artifact	Compiler question	RTL question	Driver/runtime question
Capability manifest	Which op/shape/dtype/layout combinations are legal and profitable?	Which target revision and resource limits implement each capability?	How are revision, firmware, memory, and concurrency limits queried?
ISA and descriptor schema	How are tiles, addresses, dependencies, relocations, and debug IDs emitted?	What is normative at decode/retire, and what malformed state is rejected?	What must be validated before submit, and how are errors returned?
Memory contract	Which tensors alias, spill, persist, or require layout conversion?	What alignment, burst, bank, coherency, and page-fault behavior exists?	Who allocates/maps/unmaps, owns fences, synchronizes CPU access, and recovers faults?
Trace and counters	Which graph/source ID maps to each command and estimated cost?	Which events and stalls are counted, at what clock and overflow rule?	How are sessions, buffers, timestamps, permissions, and schema versions managed?
Compatibility matrix	Which compiler emits each artifact/ISA version?	Which hardware/firmware revisions accept it?	Which driver/runtime combinations negotiate, reject, or translate it?

Feature requirement templateReview before implementation

ID: PERF-ATTN-017
Scenario: decoder, batch 1..8, context 1k..16k
Target: p99 next-token latency <= X ms at Y W
Semantics: ONNX Attention rules + stated cache extension
Supported: dtype/layout/head/mask/sequence predicates
Compiler: fusion, tiling, artifact fields, fallback rule
Hardware: engines, SRAM, DMA, queue and progress behavior
Driver: buffers, IOVA, fences, timeout, reset, telemetry
Errors: unsupported, OOM, page fault, timeout, engine fault
Evidence: golden vectors, stress, counters, model correlation
Owners: product / compiler / RTL / firmware / driver / DV
Compatibility: capability bit, ABI version, rollback behavior

Review questions that prevent late surprises

What exact shapes and arrival patterns represent the product, including the tails?
Which behavior is normative, which is a performance hint, and which may change by revision?
Who owns every buffer, state transition, timeout, error, counter, and compatibility decision?
How does unsupported work fail or fall back, and what is its end-to-end cost?
What can be observed before silicon, and what evidence closes the requirement after silicon?
Which version combinations are supported, rejected, translated, rolled back, and tested in CI?

Product development and commercialization

Design the hardware and software interface for years of use, not one demo

A commercially useful NPU meets customer workloads within system cost and power while remaining integrable, secure, diagnosable, recoverable, upgradeable, and supportable across hardware and software revisions.

Value

End-to-end workload fit

Define customer models, quality, latency distributions, throughput, concurrency, cold start, memory, power, thermal, and fallback—not a single peak metric.

Economics

PPA and system cost

Trade compute, SRAM, NoC, DRAM bandwidth, die/package area, IP, yield, board memory, cooling, and engineering schedule against the actual workload mix.

Adoption

Compiler and SDK coverage

Make supported semantics discoverable, failures actionable, examples executable, results reproducible, and migration possible across target revisions.

Compatibility

Version every boundary

Negotiate capabilities and size-versioned messages; test old/new compiler, artifact, runtime, driver, firmware, and hardware combinations.

Trust

Isolation and safe failure

Validate descriptors, enforce IOMMU/context ownership, prevent cross-process leakage, bound execution, recover from hangs, and report partial effects.

Operations

Field diagnostics and lifecycle

Provide health counters, trace/crash capture, firmware update and rollback, manufacturing tests, failure triage, deprecation policy, and support documentation.

Commercialization perspective: a fabless accelerator team typically owns product definition, architecture, RTL, software, verification, validation, and the customer result while coordinating external manufacturing and ecosystem partners, including outsourced semiconductor assembly and test (OSAT). Outsourcing wafer fabrication does not outsource technical accountability: process, memories and PHY IP, package, thermal limits, test access and automated test equipment (ATE), yield, board design, compiler coverage, bring-up, and customer qualification must enter architecture decisions early.

01 · DiscoverProduct and customerWorkloads, scenarios, quality, PPA, cost, SKU, schedule, acceptance

02 · Co-designArchitecture and softwareModel, ISA/ABI, node, memory/IP, package, compiler, runtime, debug

03 · CloseRTL to tape-outDV/formal, emulation, DFT/ATE readiness, P&R, timing, power, IR/EM, DRC/LVS

04 · ProduceWafer, package, and testMask release, fabrication, wafer probe, package assembly, final test, yield

05 · CorrelateFirst siliconBoot, SRAM/DMA/DRAM, PVT, counters, ATE, model/RTL delta, errata

06 · SustainSDK and customer releaseQualification, yield/binning, model matrix, support, RMA, updates, respin

Fabless stage	Required outputs	Critical interfaces	Review question
Product discovery	Target customers and workload envelope; latency/throughput/quality/power/cost limits; process/package assumptions; SKU, schedule, benchmark, and acceptance criteria.	Product, customers, ML applications, architecture, finance, sales, supply chain.	Is the product objective measurable end to end, and does its volume/value justify the silicon and software schedule?
Architecture co-design	Executable golden model; performance/power model; architecture; ISA/ABI; memory map; operator and precision matrix; compiler/runtime/driver, debug, and profiling plan.	Architecture, compiler, runtime, driver, firmware, RTL/DV, physical design, foundry, EDA and IP vendors, package team.	Which external assumption—process, SRAM, PHY, package, bandwidth, IP, license, or tool—can invalidate the architecture?
RTL to tape-out	Requirements-to-tests trace; formal/DV coverage; emulation; DFT/scan/BIST; synthesis/P&R; STA; power; IR/EM; CDC/RDC; DRC/LVS; package SI/PI; firmware smoke image.	RTL, DV, DFT, physical design, firmware, EDA support, foundry, package and OSAT/test partners.	Can every sign-off exception be traced to an owner, risk, containment, first-silicon test, and schedule decision?
Fabrication, package, and test	Mask release and manufacturing data; front-end wafer fabrication; wafer sort/probe; package assembly; final test; lot, bin, test, and yield-learning records; reviewed handoffs.	Foundry, mask operations, OSAT, package, product/test engineering, quality, physical design, DFT, supply chain.	Are foundry-to-probe-to-package-to-final-test data, ownership, excursion response, sample traceability, and yield feedback unambiguous?
First silicon	Power/clock/reset and boot plan; register/ISA known-answer tests; SRAM/DMA/DRAM; interrupts; PVT characterization; counter/model/RTL correlation; ATE correlation; errata/workarounds.	Lab, firmware, driver, compiler/runtime, DV, board, test/product engineering, foundry and package/test partners.	Can the team isolate power, signal/package, RTL, firmware, driver, compiler, model, test, and workload causes with existing observability?
Qualification, volume, and sustaining	Reliability/qualification; yield and binning; cost; security; SDK/model support matrix; customer enablement; telemetry/RMA diagnostics; revision, deprecation, and respin criteria.	Quality, manufacturing/test, software release, security, field applications, customers, sales and supply chain.	Can shipped systems be supported safely across device, board, firmware, driver, compiler, framework, model, and OS revisions?

Artifacts that show fabless accelerator experience

Measurable PRD and workload, precision, and operator support matrix tied to architecture and software milestones.
Versioned ISA, ABI, firmware, and driver contracts plus requirements-to-tests traceability and a responsible, accountable, consulted, and informed (RACI) ownership matrix.
PPA and system-cost model with foundry, IP, package, memory, EDA, license, supply, software, and schedule risks.
Tape-out, bring-up, ATE correlation, qualification, SDK and customer-release, errata, and sustaining checklists.

First-silicon debugging order

Prove safe power, clocks, reset, scan/JTAG access, boot, register reads, and controlled shutdown.
Run SRAM, DMA, external-memory, interrupt, address-translation, and fault-path known-answer tests.
Execute one command from a fixed artifact; compare golden, simulator, RTL, firmware, driver, and silicon IDs.
Reconcile cycles, bytes, clocks, voltage, temperature, and ATE/lab measurements before tuning the model.
Record erratum, impact, containment, software workaround, qualification evidence, and respin criterion.

Release gate	Minimum evidence	Question for sign-off
Architecture freeze	Workload census, sensitivity sweeps, PPA estimate, memory/NoC proof, software contract draft, observable counters.	Which target fails first if workload mix, bandwidth, or SRAM assumption moves by 25%?
RTL freeze	ISA/ABI conformance, performance-model correlation, fault/recovery coverage, compiler schedule proof, trace/counter validation.	Can software diagnose every target miss without a new signal or register?
Tape-out release	DV/formal and coverage closure; DFT/ATE readiness; physical and package sign-off; approved waivers; mask/foundry/OSAT handoffs; bring-up vectors, firmware, boards, owners, and containment plans.	Is every residual risk accepted with a detectable first-silicon symptom, owner, containment, and schedule impact?
First-silicon / qualification release	Power/boot and subsystem bring-up; model/RTL/ATE correlation; PVT, reliability, yield/binning, security and recovery evidence; errata/workarounds; SDK and board readiness.	Does measured silicon meet the qualified operating envelope, and are every deviation and customer-visible workaround controlled?
SDK beta	Representative model coverage, deterministic packaging, examples, profiler, negative diagnostics, compatibility matrix.	Can an external team reproduce success and distinguish unsupported input from a product defect?
Production	Scenario benchmarks, accuracy, sustained power/thermal, security, reset/recovery, update/rollback, manufacturing and field diagnostics.	Can the device fail safely, produce a supportable record, and return to service?
Lifecycle	Telemetry policy, regression corpus, vulnerability response, revision migration, deprecation dates, fleet rollback and data retention.	How will a five-year-old application behave after a compiler, firmware, OS, or silicon revision changes?

SystemC/TLM and simulator development

Choose fidelity explicitly, then validate every timing claim

SystemC supplies an event-driven C++ simulation kernel. TLM-2.0 standardizes transaction interfaces. Neither makes a model cycle-accurate automatically: timing points, contention, arbitration, pipelines, and calibration are deliberate modeling choices.

UntimedFunctional modelSemantics, software enablement, golden state; highest speed

Register-accurateProgrammer viewMMIO, descriptors, interrupts, faults, firmware and driver flow

Loosely timedTLM b_transportCoarse annotated delays, temporal decoupling, architecture sweeps

Approximately timedTLM nb_transportMultiple phases, overlap, backpressure, arbitration and contention

Cycle-awareDetailed model / RTLPipelines, cycle events and validated timing; lowest speed

Blocking transport

Use LT to boot software early and sweep architecture parameters quickly

An initiator calls b_transport(payload, delay). The target executes the transaction in one blocking call and annotates coarse delay. A quantum keeper can let processes run ahead locally before synchronizing, improving simulation speed.

ModelRegisters, memories, commands, functional engines, interrupts, coarse DMA/compute latency, counters, errors.

Do not claimCycle-level arbitration, exact queue overlap, pipeline bubbles, or tail latency unless those mechanisms are explicitly represented and correlated.

Non-blocking transport

Use AT where request/response phases and contention change the answer

The base protocol uses nb_transport_fw/nb_transport_bw and four timing points: BEGIN_REQ, END_REQ, BEGIN_RESP, and END_RESP. Multiple transactions may be in flight, so payload lifetime, phase transitions, exclusion rules, and backpressure must be correct.

Add detailQueue depth, accept interval, service latency, arbitration, NoC/DRAM contention, DMA overlap, response ordering.

Pay the costMore scheduler interaction, protocol state, validation effort, slower simulation, and harder debugging.

Minimal LT target shapeSystemC / TLM-2.0 sketch

struct npu_target : sc_core::sc_module {
  tlm_utils::simple_target_socket<npu_target> socket{"socket"};

  SC_CTOR(npu_target) {
    socket.register_b_transport(this, &npu_target::b_transport);
  }

  void b_transport(tlm::tlm_generic_payload& tx,
                   sc_core::sc_time& delay) {
    const auto address = tx.get_address();
    auto* data = tx.get_data_ptr();

    if (!data || tx.get_data_length() != 4) {
      tx.set_response_status(tlm::TLM_BURST_ERROR_RESPONSE);
      return;
    }

    if (tx.is_write() && address == kDoorbell) {
      const auto command = read_u32(data);
      const auto arrival_delay =
          delay + sc_core::sc_time(20, sc_core::SC_NS);

      // A PEQ/event accepts work at the annotated arrival time;
      // it must not mutate visible state early.
      schedule_command(command, arrival_delay);
      delay = arrival_delay;
      tx.set_response_status(tlm::TLM_OK_RESPONSE);
      return;
    }

    tx.set_response_status(tlm::TLM_ADDRESS_ERROR_RESPONSE);
  }
};

Virtual-platform build order

Implement command semantics and deterministic golden state without time.
Add register map, memory targets, descriptors, interrupts, errors, and a driver-like test.
Annotate compute and DMA latency; expose cycles, bytes, queue occupancy, and stalls.
Add contention/backpressure only where sensitivity analysis shows it can change a decision.
Replay seeded workloads against functional model and RTL; retain first-divergence traces.
Calibrate with documented parameters, report error envelopes, and lock correlation regressions.

Also understand generic payload ownership, DMI and invalidation, debug transport, byte enables, streaming width, endianness, delta cycles, SC_THREAD/SC_METHOD behavior, and deterministic event ordering.

Companion: C++/SystemC/RTL model correlation Companion: runtime, queue, and driver flow

Eight-week build path

Finish with a mini-NPU full-stack evidence package

Use INT8 GEMM plus one mapped Conv or attention case. Each week produces a reviewable artifact; the final result should let another engineer reproduce correctness, performance, compatibility, and model limitations.

0 / 8 complete

Week 1 · Workload censusTwelve representative Conv/GEMM/Attention cases with semantics, shapes, precision, operations, bytes, targets, and accuracy tolerance. Week 2 · Roofline and sensitivity modelCompute/bandwidth/capacity bounds, three candidate configurations, bottleneck predictions, and ±25% parameter sweeps. Week 3 · Operator mappingBlocked GEMM and one Conv or attention implementation with reference parity, traffic estimates, tile sweeps, and edge cases. Week 4 · Architecture and ISACompute/memory/NoC/control design, versioned command schema, numerical semantics, faults, counters, and conformance vectors. Week 5 · SystemC/TLM platformCPU initiator, interconnect, memories, NPU target, driver-like sequence, annotated timing, queues, interrupts, and counters. Week 6 · Cross-team feature RFCTrace one feature through product, compiler, ISA, RTL, firmware, driver/runtime, telemetry, tests, owners, and compatibility. Week 7 · Verification and correlationSeeded random streams, functional/timing/RTL comparison, fault injection, counter reconciliation, error envelope, and regressions. Week 8 · Product-readiness reviewScenario results, PPA/system cost, security, recovery, version matrix, diagnostics, update/rollback, limits, and launch decision.

Five-lab compiler and product extension

Turn the added requirements into one portfolio-grade accelerator path

Complete this after, or alongside, the core eight-week lab. Reuse one model slice and one target contract so the evidence connects from workload and precision through DSL/compiler/runtime to a fabless release decision.

0 / 5 complete

Lab 1 · Model signature and precision sweepTrace one dense LLM plus one DiT, MoE, SSM, multimodal, or recommendation case. Inventory shapes, work, bytes, state, dynamic behavior, and service goals; compare BF16, FP8-emulated, INT8, and groupwise INT4 for task quality, footprint, metadata, transfers, and fallback. Emulation validates numerics only; measure latency on native supported kernels and never infer speed from emulation. Lab 2 · Frontend compiler and backend interfaceFirst justify the extension seam, pin a reproducible build, and record patch surface, rebase cost, and upstream-versus-fork policy. Then implement one capability-aware graph optimization and one operator-level lowering in TVM Relax, MLIR/StableHLO, or XLA HLO; use LLVM only for the backend/code-generation portion. Add a dataflow and memory-hierarchy schedule, emit a versioned backend interface schema, and retain before/after IR, legality and rejection decisions, negative/fallback tests, kernel count, transfers, peak live bytes, and end-to-end latency. Lab 3 · Accelerator DSL and loweringDefine two semantic operations with types, shapes, layouts, precision, memory/effects, rounding and invalid cases. Add parser/printer, verifier, canonicalization, source diagnostics, and a staged semantic → scheduled → buffer/command → target lowering trace. Lab 4 · Runtime execution and correlated profilingLoad, validate, and cache a versioned executable; discover capabilities; manage memory pools, quotas, buffers, queues, events, and asynchronous lifetimes; add structured errors and recovery; preserve stable profile IDs. Report cold/warm and p99 results with load, allocation, copy, queue, device, and synchronization time separated. Lab 5 · Fabless tape-out and first-silicon capstoneRun a go/no-go tabletop covering PRD traceability, PPA/system cost, DV/formal, DFT, physical sign-off, foundry/IP/package/test dependencies, software readiness, bring-up known-answer tests, ATE/model correlation, errata, qualification, SDK/customer release, and respin criteria.

Final evidence chain: reviewed model artifact → workload/precision report → extension decision → DSL and IR snapshots → executable manifest → runtime/driver trace → reference/simulator/RTL comparison → tape-out and first-silicon review. A reviewer should be able to reproduce the result and identify every unsupported or unproven claim.

Interview and portfolio evidence

Explain decisions with a compact evidence chain

Architecture

Workload → resource

Show how shape/utilization, operations, bytes, latency, power, and cost selected array, SRAM, NoC, precision, and control parameters.

Evidence: assumptions, sweeps, rejected alternatives, and the trigger that would reopen the decision.

ISA

Contract → implementation freedom

Separate normative operands, numerical behavior, ordering, faults, state, and versioning from an internal pipeline or schedule.

Evidence: schema, pseudocode, encoder/decoder, simulator, illegal cases, and differential tests.

Operators

Bottleneck → optimization

Derive work/traffic, name the hierarchy, classify the limit, predict a counter change, implement the lever, and retain correctness.

Evidence: before/after traces, counters, distributions, reference diff, and regression guard.

Co-design

Feature → traceability

Connect product scenario to compiler legality, command, RTL resource, firmware policy, driver lifecycle, observable event, and test.

Evidence: reviewed RFC, owner matrix, compatibility impact, adversarial cases, and sign-off result.

Product

Demo → lifecycle

Discuss coverage, system cost, security, multi-process behavior, recovery, field diagnostics, updates, rollback, and support boundaries.

Evidence: readiness gates, failure drills, old/new version matrix, known limits, and rollout plan.

Modeling

Fidelity → correlation

Explain why LT or AT was sufficient, which mechanisms were omitted, how timing was calibrated, and where prediction error remains.

Evidence: shared traces, per-class error, raw configuration, first-divergence report, and correlation CI.

Fabless delivery

Requirement → shipped silicon

Connect customer goals to architecture/software milestones, partner dependencies, tape-out sign-off, first-silicon correlation, qualification, SDK release, and sustaining.

Evidence: stage-gate RACI, risk register, bring-up plan, errata/workaround record, and customer acceptance result.

Frontend compiler stack

Extension seam → verified IR handoff

Justify the maintained TVM, MLIR, XLA, or LLVM/backend extension boundary, then explain the graph transformation, operator lowering, dataflow/memory schedule, IR invariant, and versioned interface that let the backend generate correct target code.

Evidence: pinned build, patch/rebase/upstream policy, before/after IR, semantic and legality checks, capability decisions, memory plan, interface schema, target artifact, source map, and end-to-end result.

Runtime

Artifact → correlated execution

Trace validation and caching, device capabilities, resource ownership and lifetimes, transfers, queues/events, driver calls, profiling, asynchronous errors, and recovery.

Evidence: versioned manifest, allocation and buffer/event trace, stable profile IDs, error injection, compatibility matrix, and full latency breakdown.

Models & precision

Workload signature → numerical contract

Relate current model families to operations, state, control, traffic, service goals, and mixed BF16/FP8/INT8/INT4 choices.

Evidence: workload census, calibration record, bit-accurate tests, task-quality results, kernel coverage, bytes, and end-to-end speed.

Accelerator DSL

Semantics → staged lowering

Separate domain meaning from target schedule while specifying types, shapes, layout, precision, memory, effects, legality, diagnostics, versioning, and escape hatches.

Evidence: language spec, valid/invalid corpus, IR snapshots, source maps, complete conversion, and target execution.

Primary specifications and original work

Source-backed reading path

Specifications define behavior; original architecture and algorithm papers explain the tradeoffs behind the design.

Google: In-Datacenter Performance Analysis of a TPUProduction workload, systolic matrix unit, software-managed memory, latency, utilization, and energy analysis

MIT Eyeriss project and papersDataflow, spatial array, memory hierarchy, on-chip network, reuse, and energy-efficient convolution

Berkeley: Roofline performance modelOperational intensity, bandwidth and compute ceilings, bottleneck-oriented analysis

NVIDIA Research: TimeloopSystematic evaluation of mappings and architecture configurations for DNN accelerators

NVIDIA CUTLASS: Efficient GEMMHierarchical tiling, data movement, pipelining, and high-performance GEMM structure

NVIDIA CUTLASS: Implicit GEMM convolutionConvolution-to-GEMM mapping without materializing the full im2col matrix

FlashAttentionExact IO-aware attention, on-chip tiling, online softmax, and reduced high-bandwidth-memory traffic

ONNX Conv operatorPortable convolution semantics, shapes, attributes, grouping, padding, stride, and dilation

ONNX Gemm operatorPortable general matrix multiplication semantics and attributes

ONNX Attention operatorScaled dot-product attention, self/cross variants, masks, shapes, and cache-related behavior

ONNX IR specificationVersioned, runtime-agnostic SSA graphs, types, shapes, operator sets, metadata, and compatibility rules for frontend import

RISC-V unprivileged ISA specificationSoftware-visible ISA design, extension structure, memory model, and current ratified specifications

RISC-V ISA overview and custom encoding spacesStandard, reserved, and vendor-specific custom extension categories

MLIR Linalg dialectStructured operators, tiling, promotion, fusion, vectorization, and lowering toward hardware

MLIR dialect conversionLegal targets, rewrite patterns, and type conversion for target-specific lowering contracts

MLIR pattern rewritingRewrite benefits, constraints, drivers, bounded recursion, diagnostics, and safe IR mutation for graph and lowering passes

MLIR canonicalizationBest-effort canonical forms, greedy rewriting, folding, convergence, and the limits of treating canonicalization as a guaranteed normal form

MLIR bufferizationTensor-to-memref conversion, in-place alias analysis, operation interfaces, late bufferization, and separate ownership deallocation

MLIR remarksStructured optimization diagnostics and location-aware explanations for missed, applied, and analysis decisions

Linux dma-buf documentationCross-device buffer sharing, ownership, CPU access, asynchronous access, fences, and synchronization

Accellera TLM-2.0 Language Reference ManualBlocking/non-blocking transport, LT/AT styles, phases, temporal decoupling, DMI, and generic payload

SystemC transaction-level modeling overviewStandard interfaces and use cases for software development, architecture analysis, performance, and verification

gem5 documentationEvent-driven computer-system modeling, architectural experimentation, statistics, and simulation workflow

MLPerf Inference benchmark suiteScenario-based latency, throughput, accuracy, system and power measurement methodology

NVDLA hardware architectureOpen convolution accelerator, memory interfaces, execution pipeline, precision, and integration reference

Attention Is All You NeedOriginal Transformer architecture, self-attention, multi-head attention, feed-forward blocks, and sequence modeling

The Llama 3 Herd of ModelsModern dense decoder model design, grouped-query attention, training, evaluation, safety, and deployment considerations

An Image Is Worth 16×16 WordsVision Transformer patch tokens and the relationship among resolution, patch size, sequence length, and attention

Learning Transferable Visual Models From Natural Language SupervisionCLIP dual encoders, contrastive image-text training, modality-specific preprocessing, projection, and zero-shot transfer

Scalable Diffusion Models with TransformersDiffusion Transformer architecture, latent patches, conditioning, scaling, and full denoising-loop workload

Switch TransformersSparse expert routing, capacity, load balancing, communication, scale, and low-precision training challenges

Mamba: Linear-Time Sequence ModelingSelective state-space models, recurrent state, hardware-aware scan, and sequence-length scaling

Deep Learning Recommendation ModelSparse embeddings, feature interactions, dense compute, memory capacity, bandwidth, and recommendation benchmarking

NVIDIA Transformer Engine FP8 primerE4M3/E5M2 formats, scaling recipes, accumulation, alignment, FP8 applicability, and current FP4/microscaling guidance

Google Cloud TPU bfloat16 guideBF16 exponent/significand tradeoff, conversion, accumulation, range, precision, and platform-specific numerical behavior

OCP Microscaling Formats specificationMX block-scale encodings, element formats, scale semantics, block sizes, conversion, and numerical behavior

SmoothQuantPost-training W8A8 LLM quantization through mathematically equivalent channel scaling of activation outliers

AWQ: Activation-aware Weight QuantizationLow-bit weight-only LLM quantization that protects salient channels using activation observations

Apache TVM design and architectureRelax graph IR, TensorIR kernels, cross-level transformations, scheduling, target translation, and runtime architecture

TVM external library dispatch / BYOCPattern registration, fusion and partitioning, external code generation, runtime modules, and deployable execution

TVM device and target interactionsCompile-time target properties versus runtime device discovery and execution responsibilities

TVM Relax virtual machineGraph optimization and buffer planning, VM bytecode plus native kernels, executable packaging, devices, and allocator initialization

MLIR defining dialectsDialect operations, types, attributes, interfaces, services, and declarative definitions

MLIR operation definition specificationODS/TableGen operation syntax, constraints, builders, verification, assembly format, and generated code

MLIR pass infrastructurePass construction, nesting, dependent dialects, reproducible pipelines, crash reproducers, threading, statistics, and instrumentation

MLIR bytecode formatBytecode encoding, dialect-specific resources, versioning hooks, compatibility, and safe evolution of serialized IR

LLVM: Writing an LLVM backendTarget machine, registers, instructions, instruction selection, scheduling, assembly printing, and backend structure

LLVM target-independent code generatorCode-generation pipeline, instruction selection, scheduling, register allocation, prolog/epilog, late passes, and MC emission

StableHLO specificationPortable high-level operation semantics, types, quantization, custom calls, effects, alias metadata, and extensibility surface

StableHLO compatibilityVersion policy, backward and forward compatibility, artifact upgrades/downgrades, and compatibility testing

XLA architectureStableHLO input, target-independent HLO optimization, backend-specific passes, target code generation, and executable production

OpenXLA custom calls and typed FFITarget-specific HLO escape hatches, typed runtime handlers, attributes, buffers, registration, and API/ABI status

OpenXLA PJRT device APIUniform framework/device API and plugin model with C API, versioning, integration, and implementation resources

PJRT C++ API overviewClient, device, memory-space, buffer, compiler, executable, ownership, and asynchronous execution interfaces

OpenXLA PJRT plugin integrationPlugin initialization, framework discovery, API version checks, packaging, dynamic loading, and integration testing

OpenXLA XProfTrace, graph, memory, operation, and Roofline views that correlate compiler metadata with hardware events and counters

SIA semiconductor production and business-model primerDesign, front-end wafer fabrication, back-end assembly/test, and fabless/foundry/IDM production roles

TSMC Open Innovation PlatformFoundry process and design enablement ecosystem spanning EDA, silicon-proven IP, design services, cloud, and packaging

NPU Architecture, Compiler Systems, and Performance Verification

Each experience statement asks for inspectable engineering evidence

Architecture starts with a scenario and ends with reproducible proof

Translate workload reuse and product constraints into explicit resources

ISA and command definition checklist

Treat each model as a workload signature and each datatype as a full-stack contract

Write precision semantics before selecting peak TOPS

Avoid the low-bit performance illusion

Use the same operator loop: semantics → work → traffic → mapping → proof

Convolution is a family of reuse patterns, not one kernel

GEMM performance follows the entire blocking hierarchy

Prefill is compute-rich; autoregressive decode is often KV-traffic dominated

Choose the smallest durable extension seam, then preserve semantics to the device

Make the fork burden visible

Test decisions, not only generated answers

Specify meaning before syntax

Build a ladder of models and measure the error between adjacent rungs

Fix the measurement boundary

Reconcile rather than decorate

Carry correctness with speed

One feature must remain traceable from product intent to a passing test

Review questions that prevent late surprises

Design the hardware and software interface for years of use, not one demo

End-to-end workload fit

PPA and system cost

Compiler and SDK coverage

Version every boundary

Isolation and safe failure

Field diagnostics and lifecycle

Artifacts that show fabless accelerator experience

First-silicon debugging order

Choose fidelity explicitly, then validate every timing claim

Use LT to boot software early and sweep architecture parameters quickly

Use AT where request/response phases and contention change the answer

Correlate shared events, not unrelated summary numbers

Virtual-platform build order

Finish with a mini-NPU full-stack evidence package

Turn the added requirements into one portfolio-grade accelerator path

Explain decisions with a compact evidence chain

Workload → resource

Contract → implementation freedom

Bottleneck → optimization

Feature → traceability

Demo → lifecycle

Fidelity → correlation

Requirement → shipped silicon

Extension seam → verified IR handoff

Artifact → correlated execution

Workload signature → numerical contract

Semantics → staged lowering

Source-backed reading path