Accelerator Repository Blueprint

Observed in TT-Metal

The repository is an engineering system, not a large source folder

The current TT-Metal repository separates the Metalium programming layer, TT-NN, models, tests, documentation, tools, scripts, build infrastructure, and GitHub automation. The exact names and scale are Tenstorrent-specific; the separation of ownership is the useful pattern.

Product codett_metal/Host API, hardware kernels, low-level runtime, device implementation, examples, tests, and architecture descriptors.

Operator and Python layerttnn/C++ implementation, Python package, public API, examples, tutorials, and component tests.

Workload evidencemodels/Demos, reusable model utilities, reference paths, performance support, datasets, and model CI tiers.

Cross-stack teststests/Metalium, TT-NN, end-to-end, sweeps, performance, nightly, stress, distributed, and shared test infrastructure.

Debug and profilingtools/Hardware debug, triage, tracing, profiling, power support, inspection, and test utilities.

Automationscripts/Build, test, API checks, profiling, CI helpers, code health, and generated-data maintenance.

Build and configurationcmake/Toolchains, dependencies, sanitizers, packaging, compiler checks, caching, and version construction.

Project contractdocs/ .github/ contributing/Documentation, workflows, time budgets, ownership, pull-request rules, and engineering practices.

The current test tree is purpose-oriented: tests/tt_metal/ covers low-level subsystems and microbenchmarks; tests/ttnn/ includes unit, integration, performance, profiling, statistical, stress, and nightly paths; tests/end_to_end_tests/ crosses stack boundaries; tests/sweep_framework/ explores parameter spaces; and tests/nightly/ groups deeper hardware configurations. This is the model for the proposed test ladder later on this page.

Copy the architecture of responsibility, not the repository size. A new ACiM stack should begin with one contract path, one simulator backend, one runtime path, and one operator. Create a directory only when it has a clear owner, dependency direction, test lane, and generated artifact.

TT-Metal pattern	General lesson	ACiM translation
C++ core plus Python product layer	Keep performance-sensitive and ABI-sensitive code below a usable high-level API.	C++ compiler/runtime/HAL with Python bindings, model tools, calibration tools, and framework integration.
Tests at root and inside components	Local tests protect implementation details; root tests protect cross-component behavior.	Place private compiler/runtime tests beside their components and complete system lanes under `tests/`.
GoogleTest and pytest	Use each framework where its language, fixtures, and failure diagnostics are strongest.	GoogleTest for C++ contracts and state machines; pytest for Python APIs, parametrized operators, models, and lab scheduling.
Dedicated tools tree	Debugging and profiling are products with stable inputs and outputs.	Ship artifact inspection, IR dumps, command traces, calibration inspection, first-divergence analysis, triage bundles, and profilers.
Fast and deep CI lanes	A pull request cannot run every expensive accelerator test, so routing and post-commit coverage must be explicit.	Use host-only PR gates, selected hardware smoke, merge integration, nightly sweeps/performance, and release qualification.

Repository terminology

Terms used throughout the blueprint

ABI: Application Binary InterfaceThe binary-level contract for calling functions or exchanging structures across independently built components.

HAL: Hardware Abstraction LayerAn internal device-neutral interface implemented by simulator, fake-device, and real-hardware backends.

SDK: Software Development KitThe supported APIs, libraries, tools, examples, documentation, packages, and diagnostics used by application developers.

UMD / KMDUser-Mode Driver and Kernel-Mode Driver. UMD integrates the runtime; KMD owns privileged OS resources such as DMA, interrupts, isolation and reset.

CI: Continuous IntegrationAutomated build, static analysis, test, packaging, and evidence collection triggered by code changes or schedules.

CTestCMake's test runner and test registry. It can launch GoogleTest executables and label or group native test targets.

Smoke testThe shortest complete health path: discover, open, execute one known operation, compare, close, and leave the device usable.

Sweep testA parameterized exploration across shapes, dtypes, layouts, precision, faults, calibration, or other supported dimensions.

Soak testA long-running stress test intended to reveal leaks, state accumulation, thermal behavior, rare races, and recovery failures.

Golden testA deterministic comparison against an approved IR, artifact, command sequence, tensor, or report. Updates require an explained semantic change.

PVT: Process, Voltage, TemperatureThe manufacturing and operating corners used to characterize timing, power, numerical variation, and reliability.

ADR: Architecture Decision RecordA short document preserving the context, alternatives, chosen design, consequences, migration, tests, and owner of a durable decision.

SKU: Stock Keeping UnitA specific product or board configuration that may differ in devices, memory, topology, clocks, power, firmware, or supported features.

SanitizerCompiler instrumentation such as AddressSanitizer or UndefinedBehaviorSanitizer that detects classes of memory and language errors at runtime.

TracyA real-time frame profiler used here for host C++ zones, frames, messages, plots, threads, locks, memory, and operating-system activity.

ZoneA named interval with a begin and end timestamp. Nested zones divide a host request or device command into attributable phases.

Device cycle counterA monotonically increasing hardware timer read by firmware. Its clock domain and frequency must accompany the captured value.

Clock domainLogic driven by one clock source. Host, controller, array sequencer, converter, DSP, and multiple devices may not share a timestamp origin or rate.

Correlation IDA stable identifier carried through API call, queue command, firmware descriptor, device trace, interrupt, and result so records from different planes can be joined.

Probe effectThe latency, memory, bandwidth, scheduling, or power change caused by observation itself. Profiled and official uninstrumented measurements must be compared.

Recommended repository

A smaller tree with production boundaries from the beginning

The tree below is a proposed starting architecture for an accelerator stack named acim-stack. Rename it for the product, but keep the ownership and dependency rules. Blue paths are executable contracts, red paths are verification, and amber paths are developer tools.

Proposed repository treeStart narrow; expand each branch only with a working owner and test

acim-stack/
|-- README.md
|-- CONTRIBUTING.md
|-- CODE_OF_CONDUCT.md
|-- SECURITY.md
|-- CHANGELOG.md
|-- LICENSE
|-- CMakeLists.txt
|-- CMakePresets.json
|-- pyproject.toml
|-- pytest.ini
|-- .clang-format
|-- .clang-tidy
|-- .pre-commit-config.yaml
|-- .github/
|   |-- CODEOWNERS
|   |-- pull_request_template.md
|   `-- workflows/
|       |-- pr-fast.yml
|       |-- pr-hardware-smoke.yml
|       |-- post-commit.yml
|       |-- nightly-sweeps.yml
|       |-- nightly-performance.yml
|       |-- release-qualification.yml
|       `-- acim-minimal.yml
|-- cmake/
|   |-- dependencies.cmake
|   |-- sanitizers.cmake
|   |-- warnings.cmake
|   |-- packaging.cmake
|   `-- toolchains/
|-- configs/
|   |-- targets/
|   |-- test-matrices/
|   |-- performance-thresholds/
|   |-- model-manifests/
|   `-- schemas/
|-- docs/
|   |-- architecture/
|   |-- adr/
|   |-- api/
|   |-- bringup/
|   |-- debugging/
|   |-- testing/
|   |-- tutorials/
|   `-- generated/
|-- schemas/
|   |-- target.fbs
|   |-- artifact.fbs
|   |-- command.fbs
|   |-- calibration.fbs
|   `-- trace.fbs
|-- generated/
|   |-- cpp/
|   |-- python/
|   `-- firmware/
|-- include/acim/
|   |-- c_api/
|   |-- compiler/
|   |-- runtime/
|   `-- hal/
|-- src/
|   |-- compiler/
|   |   |-- import/
|   |   |-- ir/
|   |   |-- passes/
|   |   |-- mapping/
|   |   |-- codegen/
|   |   `-- artifact/
|   |-- runtime/
|   |-- hal/
|   |   |-- simulator/
|   |   |-- fake_device/
|   |   `-- hardware/
|   |-- driver/
|   |-- firmware/
|   |-- simulator/
|   `-- common/
|-- kernels/
|   |-- analog/
|   |-- digital/
|   `-- common/
|-- bindings/
|   `-- python/
|-- python/acim/
|   |-- compiler/
|   |-- runtime/
|   |-- framework/
|   |-- calibration/
|   `-- tools/
|-- models/
|   |-- demos/
|   |-- reference/
|   |-- manifests/
|   |-- accuracy/
|   `-- performance/
|-- examples/
|   |-- acim-minimal/
|   |-- cpp/
|   |-- python/
|   `-- notebooks/
|-- tools/
|   |-- artifact-inspect/
|   |-- ir-dump/
|   |-- trace/
|   |-- profiler/
|   |-- triage/
|   |-- calibration-inspect/
|   `-- codegen/
|-- tests/
|   |-- cpp/unit/
|   |-- cpp/integration/
|   |-- python/unit/
|   |-- python/integration/
|   |-- smoke/
|   |-- sweeps/
|   |-- end_to_end/
|   |-- performance/
|   |-- stress/
|   |-- hardware/
|   |-- qualification/
|   |-- data/golden/
|   `-- common/
|-- scripts/
|   |-- build/
|   |-- ci/
|   |-- test/
|   |-- release/
|   `-- trace/
|-- docker/
|-- third_party/
`-- packaging/

Rule 1One contract sourceschemas/ generates C++, Python, and firmware readers. Never maintain three handwritten definitions of a command or artifact.

Rule 2Public API is narrowinclude/acim/ exposes supported interfaces. Private implementation headers remain below src/.

Rule 3Hardware is a backendThe runtime targets a HAL implemented by functional simulation, queued fake device, and hardware. Compiler tests do not require a board.

Rule 4Tests state their scopeA developer can tell whether a test needs no device, one device, a topology, calibration data, or a qualification lab from its path and markers.

Rule 5Models are consumersModel demos may use the public SDK; they must not reach into compiler internals, physical addresses, or private driver APIs.

Rule 6Tools consume stable evidenceInspectors and profilers read versioned artifacts and traces. They do not scrape human log text as their primary interface.

Rule 7Configuration is validated dataTarget, test-matrix, model, and threshold files have schemas, units, version fields, examples, and validation tests.

Rule 8Generated output is reproducibleGenerated files record generator version and source hash; CI fails when committed output differs from regeneration.

Dependency direction

The directory tree matters only when dependencies follow it

Keep stable data contracts at the bottom, device-neutral behavior in the middle, and user-facing integrations at the top. Hardware implementations satisfy interfaces; they do not redefine them.

FoundationSchemas and typesStatus codes, target, artifact, command, calibration, and trace contracts

MechanismCompiler and HALLowering, mapping, code generation, buffers, queues, events, and device interface

ExecutionRuntimeCompatibility, allocation, binding, submission, synchronization, recovery, telemetry

LanguageC API and bindingsStable ABI plus C++ and Python convenience layers

IntegrationFramework and SDKONNX Runtime, PyTorch tools, CLI, packaging, examples

Product proofModels and applicationsReal preprocessing, fallback, execution, postprocessing, accuracy, and performance

Boundary	May depend on	Must not depend on	Enforcement
`schemas/`	Schema compiler and primitive types	Runtime, framework, board, or model code	Standalone generation target and compatibility tests
`src/compiler/`	IR, target descriptors, operation/cost models, artifact writer	Live device handles, kernel-driver calls, serial-specific state	Link boundaries plus host-only compiler tests
`src/runtime/`	Public contracts, HAL, artifact reader, telemetry	Framework graph classes or compiler-private passes	Runtime target links only approved libraries
`src/hal/`	Device-neutral interfaces and backend-specific implementation	Model semantics, graph partition policy, Python objects	Backend conformance suite
`python/acim/`	Public C ABI or supported bindings	Private C++ symbols and raw MMIO	Wheel test against installed headers and libraries
`models/`	Installed SDK, public framework plugin, test utilities	Source-tree implementation imports	Run selected demos from a packaged installation

Native build toolchain

Turn one ACiM source file into a repeatable binary and test result

A build stack is a chain of different tools. CMake describes targets and generates a build graph; Ninja or Make executes that graph; GCC, Clang, or MSVC compiles C/C++; an archiver and linker create libraries and executables; CTest runs the native tests. Treating all of this as "the compiler" makes failures difficult to locate.

1. SelectPreset and toolchainHost OS, compiler, build type, features, target/sysroot

2. ConfigureCMakeDetect capabilities, resolve packages, create explicit targets

3. GenerateNinja build graphCommands, dependencies, generated files and output paths

4. CompileGCC / Clang / MSVCPreprocess, parse, optimize, generate objects and diagnostics

5. LinkArchiver and linkerStatic/shared libraries, executables, symbols, ABI and runtime paths

6. ProveCTest and installUnit, smoke, labels, package layout and saved build manifest

Know which tool owns each failure

Tool	Responsibility	Recommended ACiM use	Typical failure or evidence
CMake	Meta-build system: declares targets, requirements, generated files, installation, tests and package discovery.	One target per compiler/runtime/HAL boundary; checked-in presets; target-scoped options instead of global flags.	Configure log, compiler ID/version, cache, target graph, package resolution, generated Ninja files.
Ninja	Executes the generated dependency graph with fast incremental and parallel builds.	Default local and CI generator. Let CMake generate `build.ninja`; do not hand-author it.	Exact failed command, missing generated dependency, stale output, unexpected rebuild or parallel race.
GNU Make	Alternative build executor and the basis of Linux Kbuild.	Accept as a CMake generator when needed; use the kernel's Make/Kbuild flow for the Linux KMD.	Make target, dependency/rule expansion, kernel build output.
GCC / G++	GNU C/C++ compiler driver, frontend, optimizer and code generator; invokes assembler and linker.	Linux baseline for host compiler, runtime, simulator and SDK release builds with `libstdc++`.	Standards/warning diagnostics, generated objects, optimization report, GNU ABI and runtime dependencies.
Clang / Clang++	LLVM C/C++ compiler driver with independent diagnostics and sanitizer integration.	Second compiler in CI, developer diagnostics, AddressSanitizer/UndefinedBehaviorSanitizer and future LLVM/MLIR integration.	Independent compile result, sanitizer trace, optimization remarks, LLVM IR when explicitly requested.
MSVC / clang-cl	Windows-native C/C++ toolchains using the Microsoft command-line and ABI conventions.	Host-only Windows simulator/SDK confidence. Keep the production Linux PCIe/KMD lane separate.	MSVC ABI, CRT selection, PDB debug symbols, DLL import/export and Windows packaging behavior.
Assembler	Turns compiler-generated assembly into architecture-specific object files.	Normally driven by the compiler; direct use appears in firmware startup, interrupt stubs and ISA kernels.	Object sections, relocation records, instruction encoding and target ISA mismatch.
Archiver	Packages object files into static libraries such as `.a` or `.lib`.	Create narrow compiler, artifact, runtime and simulator libraries; inspect symbol ownership when links fail.	Archive member list, duplicate/missing symbols and unexpected private implementation exposure.
GNU ld / LLD / link.exe	Resolves symbols and relocations into shared libraries and executables.	Start with the compiler's default linker; add LLD only as a measured build-time optimization or target requirement.	Link map, unresolved/duplicate symbol, section size, RPATH, SONAME, import library and ABI mismatch.
CTest	CMake's native test registry and runner with labels, fixtures, timeouts and presets.	Common entry point for dependency-free unit tests, GoogleTest discovery, smoke tests and install checks.	Test name, label, command, timeout, exit code, output and per-lane summary.
GoogleTest	C++ test framework with typed assertions, fixtures, parameterized tests and CMake discovery.	Use for compiler passes, schemas, allocators, runtime state machines and backend contracts after the minimal offline tests work.	Focused assertion, parameter, fixture lifecycle and `--gtest_filter` reproducer.
clang-format / clang-tidy	Mechanical formatting plus AST-based diagnostics and refactoring checks.	Format automatically; run selected analyzer, bug-prone, performance and portability checks on owned targets.	Formatting diff or source-located diagnostic. Do not apply project warnings to third-party code.
Sanitizers	Compiler instrumentation for memory errors, undefined behavior, races or uninitialized reads.	Clang ASan+UBSan PR lane; separate TSan lane for queue/runtime concurrency; never use sanitizer timing as performance data.	Symbolized stack, allocation/free history, undefined operation or race report.
gcov / llvm-cov	Compiler instrumentation and reporting for function, line, region and branch coverage.	Measure owned host code after tests are stable; keep GCC and Clang coverage artifacts in separate lanes.	Instrumented binary, raw/indexed profile, source report and uncovered branch list. Coverage percentage is not correctness.
ccache / sccache	Caches compiler outputs keyed by source, compiler and command inputs.	Add after the uncached build is correct; report hit rate and include compiler/config hashes in CI cache keys.	Hit/miss statistics, cache-key mismatch and stale/non-reproducible input diagnosis.
Package/dependency tools	`find_package`, package managers, dependency providers, `pkg-config` and optional `FetchContent`.	Prefer installed/pinned packages for release builds. Keep network fetching opt-in and hash-pinned.	Resolved package version, include/library path, dependency lock and provenance manifest.
Schema/code generators	Tools such as `flatc`, `protoc` or TableGen turn one contract source into language-specific code.	Generate C++, Python and firmware readers into the build tree; verify committed generated output only when distribution requires it.	Generator version, schema hash, generated diff, compatibility test and dependency edge in Ninja.
Python wheel and bindings	`pyproject.toml` build backend plus pybind11/nanobind or a C ABI wrapper creates an installable Python extension.	Add after the C++ install tree is stable; build wheels against the supported Python/OS/architecture matrix.	Wheel contents, extension ABI, imported shared-library paths, Python tag and installed-package smoke test.
Doxygen / Sphinx	Generates C/C++ API documentation and narrative SDK documentation as explicit build targets.	Run warnings-as-errors for public API comments, examples and cross-links without putting docs in the default native compile target.	Generated HTML, missing-symbol/comment warnings, broken links and versioned API reference.
Cross compiler and sysroot	Builds host runtime or firmware for a target ISA/OS using target headers, libraries and ABI.	CMake toolchain file for AArch64 host runtime; separate freestanding project for the ACiM controller firmware.	Target triple, sysroot hash, compiler runtime, linker script, ELF attributes and emulation/hardware smoke result.
Kbuild	Linux kernel's supported build system for external and in-tree modules.	Build the kernel-mode driver with Kbuild. Top-level CMake may invoke and package it, but should not replace its rules.	Kernel version/config, module symbols, modpost output, signing and load/unload test.

Recommended choices for the first repository

Decision	Start with	Add when needed	Avoid in the first loop
Language	ISO C++20 through `target_compile_features`; C ABI only at the supported SDK boundary	C17 for firmware/KMD boundaries and target-specific assembly where the ABI requires it	Compiler-specific language extensions in public headers
Linux compiler	GCC baseline plus Clang as an independent CI compiler	Versioned release matrix and target cross-compilers	Supporting many old compilers before a product/toolchain requirement exists
Windows compiler	MSVC for host simulator and SDK portability	`clang-cl` if Windows LLVM diagnostics or compatibility is valuable	Claiming Windows device support from a host-only simulator build
Generator	CMake plus Ninja	Unix Makefiles or IDE generators as secondary paths	Handwritten per-platform build scripts that duplicate the target graph
Linker	Compiler default	LLD after measuring link time, memory or cross-target requirements	Mixing C++ standard libraries or CRT modes inside one ABI without an explicit boundary
Dependencies	No dependency for the first smoke/unit path; `find_package` for installed packages	Pinned provider/package manager; hash-pinned `FetchContent` for developer convenience	Downloading LLVM, MLIR or GoogleTest on every default configure
LLVM / MLIR	Disabled in the minimal build	Prebuilt, version-pinned package behind `ACIM_ENABLE_MLIR`	Making a full LLVM source build a prerequisite for the first MVM smoke test
Device firmware	Host model of descriptors and state machines	Separate cross-toolchain, linker script, startup and hardware/RTL tests after ISA/ABI freeze	Linking host C++ runtime libraries into freestanding firmware by accident
Linux driver	Fake HAL and userspace tests	Kbuild module, kernel-version matrix and PCIe lab after UAPI/resource ownership is defined	Hiding driver commands inside general CMake targets or unit tests

Compiler and target matrix

Lane	Builds	Runs	Primary question	Suggested trigger
GCC Debug	All host libraries, tools and dependency-free tests	Unit, smoke, simulator integration	Does the Linux baseline compile and behave correctly?	Every PR
Clang Debug	Same public targets with independent diagnostics	Unit and smoke	Did GCC accept a non-portable or suspicious construct?	Every PR
Clang ASan+UBSan	Owned host code with instrumentation	Unit and simulator integration	Are memory lifetime and language operations valid?	Every PR or post-commit
Clang TSan	Runtime/queue subset with race instrumentation	Deterministic concurrency tests	Are session, queue, event and callback accesses synchronized?	Post-commit/nightly
GCC Release	Install tree, shared/static policy, CLI and package	Packaged smoke plus controlled benchmarks	Does the shipped optimization/ABI configuration work?	Post-commit/release
MSVC Debug	Host simulator, public headers and SDK utilities	Unit and smoke	Are public C++ and data contracts portable to Windows?	Post-commit when Windows is supported
AArch64 Linux cross	Runtime, HAL interfaces and CLI against target sysroot	QEMU/package check, then target smoke	Are target dependencies, ABI and package paths correct?	Post-commit/release
Controller firmware	Freestanding image, map and size report	Host model, ISA simulator, RTL/FPGA, silicon in stages	Does firmware fit and implement the command ABI safely?	Every firmware PR plus hardware lanes
Linux KMD	Module through Kbuild for supported kernels	Static checks, load/unload, DMA/interrupt/reset smoke	Does privileged resource ownership work for this kernel/device?	Driver PR, nightly matrix, release

A working minimal build is included

examples/acim-minimal/ contains a dependency-free C++20 functional MVM simulator, ADC clamp, CMake target policy, Ninja presets for GCC/Clang/MSVC, CTest unit and smoke lanes, optional GoogleTest discovery, clang-format/clang-tidy configuration, installation rules, and an AArch64 Linux toolchain example. It is a software reference model, not an analog circuit model.

First configure, build and testRun inside examples/acim-minimal

# Linux baseline
cmake --preset gcc
cmake --build --preset gcc
ctest --preset gcc

# Independent compiler and sanitizer lane
cmake --preset clang
cmake --build --preset clang
ctest --preset clang
cmake --preset asan-clang
cmake --build --preset asan-clang
ctest --preset asan-clang

# Windows host path from a Visual Studio Developer PowerShell
cmake --preset msvc
cmake --build --preset msvc
ctest --preset msvc

ABI rule: record compiler ID/version, target triple, C++ standard library, build type, sanitizer mode and public ABI version. Do not pass C++ standard-library objects across a plugin boundary compiled with incompatible libstdc++, libc++ or MSVC CRT/ABI settings; keep the stable external boundary in C handles, fixed-width fields and explicitly versioned structures.

Build graph and configuration

CMake owns native targets; pyproject owns the Python product

Keep target relationships explicit in CMake, package and tool entry points explicit in pyproject.toml, and runtime/test choices in validated configuration files. Environment variables may override a declared option, but the resolved value must appear in the run manifest.

GenerateContract codeacim_schema_codegen emits C++, Python and firmware definitions

FoundationArtifact and commonVersioning, hashing, status, logging, target and executable readers

CoreCompiler and HAL APIHost-only compiler plus backend-neutral device interfaces

ExecuteRuntime and backendsFunctional, queued simulation and hardware implementations

ExposeC API and PythonInstalled ABI, bindings, wheel, CLI and framework adapters

VerifyTests and packagesLink public targets, install tree and selected private test helpers

Recommended native target graphOne target per ownership boundary; private dependencies stay private

acim_schema_codegen
  -> acim_common
  -> acim_artifact
  -> acim_compiler
  -> acim_hal
       -> acim_hal_functional
       -> acim_hal_queued_sim
       -> acim_hal_hardware
  -> acim_runtime
  -> acim_c_api
  -> acim_python_binding
  -> acim_tools

acim_gtest_contracts        # schema, artifact and ABI layout
acim_gtest_compiler         # IR, passes, mapping and encoding
acim_gtest_runtime          # lifecycle, queues, memory and recovery
acim_gtest_hal_conformance  # same cases for every backend
acim_gtest_firmware_model   # descriptor validation and state machines

Configuration	Owns	Required fields or profiles	Validation rule
`CMakePresets.json`	Repeatable native configure/build/test modes	`dev`, `release`, `asan`, `ubsan`, `coverage`, cross-compile presets	CI invokes the same presets documented for developers.
`pyproject.toml`	Python package, build backend, dependencies, optional groups and CLI entry points	Supported Python versions, core/runtime extras, test/docs tools, wheel contents	Build and test a wheel in a clean environment; model tests use the installed package.
`pytest.ini`	Test discovery, markers and default diagnostics	`unit`, `integration`, `hardware`, `smoke`, `nightly`, `performance`, model tiers	Unknown markers fail; hardware tests cannot enter the default host-only lane.
`configs/targets/*.yaml`	Target class and capabilities	Schema/ABI version, revision, arrays, converters, memory, engines, commands, limits and units	Schema validation plus compiler/runtime compatibility tests; every compile records the target hash.
`configs/test-matrices/*.yaml`	Where each test lane runs	SKU, topology, marker, runner label, timeout, exclusivity, firmware, cleanup and retry policy	CI validates that each required lane has a compatible runner and bounded timeout.
`configs/performance-thresholds/*.yaml`	Versioned regression policy	Workload ID, metric, baseline, absolute limit, allowed change, owner and review date	A threshold change is reviewed separately from the implementation it would excuse.
`configs/model-manifests/*.yaml`	Reproducible model workload	Model/input hashes, preprocessing, shapes, precision, backend, accuracy metric, fallback, performance scenario	E2E jobs reject implicit downloads, mutable model names and missing reference thresholds.

Configuration rule: defaults belong beside their schema; overrides are typed, bounded and logged; units are part of the field name or type; deprecated fields have migration tests; and the fully resolved configuration hash is attached to every artifact, benchmark and failure bundle.

Implementation chain

Build one executable path before expanding the surface area

Each milestone adds exactly one new responsibility and must leave a testable artifact behind. This sequence prevents the model zoo, framework plugin, and hardware backend from hiding an unstable compiler/runtime contract.

1. BootstrapBuild and lintCMake presets, Python package, warnings, format, sanitizers

2. ContractSchema and ABITarget, artifact, command, status, trace, compatibility

3. ReferenceFunctional modelOne MVM, digital epilogue, deterministic golden vectors

4. CompileOne vertical sliceImport, legalize, map, emit artifact, inspect

5. ExecuteRuntime and fake HALLoad, allocate, bind, submit, wait, trace, reset

6. CorrelateHardware and modelBring-up backend, framework path, accuracy, performance

Milestone	Code added	Required proof	Do not add yet
M0: reproducible checkout	Build presets, dependency lock, package metadata, pre-commit, minimal docs	Clean machine configures, builds, installs, imports Python, and runs an empty CTest/pytest lane	Board support, model integrations, optimization
M1: machine contract	Target schema, artifact envelope, commands, status codes, generated readers	Round-trip, malformed-input, old/new schema, and cross-language compatibility tests	Compiler policy hidden in ad hoc YAML
M2: numerical reference	FP32, quantized, ideal-array, and configurable nonideality implementations	Golden MVM vectors and property tests across shapes, signs, saturation, and rounding	Cycle claims or silicon accuracy claims
M3: one-op compiler	MatMul import, validation, mapping, command emission, artifact inspector	Golden IR, illegal-target diagnostics, deterministic artifact hash, reference equivalence	General fusion and broad operator coverage
M4: executable runtime	C API, sessions, buffers, queues, events, fake/functional HALs, trace	Backend conformance, timeout/reset, resource cleanup, multithreaded lifecycle tests	Framework-specific state in the runtime
M5: first hardware path	KMD/UMD or platform driver, firmware transport, DMA, interrupt, health, recovery	Discovery, loopback, one-command smoke, reset after fault, simulator/hardware trace correlation	Performance optimization before correctness and recovery
M6: first model path	Framework partition, package, one representative model, preprocessing/postprocessing	End-to-end accuracy, fallback accounting, reproducible latency and power manifest	Large model catalog with duplicate utilities
M7: optimization and release	Cost model, scheduling, fusion, caching, profiler, package and compatibility policy	Measured regression thresholds, stress, multi-device if required, qualification report	Optimization accepted without an attributed measurement

Verification architecture

The test chain grows in cost, realism, and diagnostic distance

Lower levels answer narrow questions quickly. Higher levels validate integration and product behavior, but they should never be the first place a compiler rule, allocator invariant, or command encoding is tested.

QualificationChip populations, PVT, aging, drift, defects, power and thermal limitsRelease or silicon gate

Stress and soakLong duration, concurrency, repeated reset, memory pressure, fault injectionNightly or weekly

End to endFramework through preprocessing, compile, runtime, device, fallback, postprocessingMerge and nightly tiers

PerformanceKernel, operator and model latency, throughput, bytes, utilization, energyControlled runners

Sweep and differentialShapes, dtypes, layouts, precision, faults and calibration against a referencePost-commit and nightly

IntegrationCompiler-artifact-runtime-simulator contracts and selected real-device pathsPR routes and merge

SmokeDiscover, open, allocate, execute one operation, compare, close and resetFast hardware confidence

UnitOne class, pass, encoding, state transition or Python API with deterministic inputsEvery relevant PR

Static checksFormat, lint, schema, docs, dependency rules, generated files and compilationLocal and every PR

Give every test lane one precise contract

Lane	Question answered	Environment	Typical trigger	Acceptance and saved evidence
Static	Is the change structurally valid and reproducible?	No hardware; clean source checkout	Pre-commit and every PR	Formatting, warnings-as-errors, lint, schema validation, docs links, dependency checks, generated-diff check all pass.
C++ unit	Does one low-level contract or state machine behave correctly?	Host process with fakes; ASan/UBSan variants	Every relevant PR	GoogleTest assertions identify the exact invariant; no board, network, sleeps, or shared mutable lab state.
Python unit	Does one public Python API, importer rule, or analysis function behave correctly?	Installed wheel or source package with mocks	Every relevant PR	pytest result, stable exception/status, parametrized shapes and dtypes, no hidden hardware requirement.
Smoke	Can the minimum supported execution path complete now?	One simulator or reserved device	Selected PRs, deployment, lab health	Discover, open, allocate, execute one known vector, compare, close; save serial, firmware, artifact hash, trace, and reset result.
Integration	Do two or more owned components honor the same contract?	Functional/queued simulator; selected hardware	Changed-path PR route and post-commit	Artifact loads across public C API; compiler/runtime/backend traces correlate; fault and cleanup paths are checked.
Sweep	Where does behavior fail across the supported state space?	Parallel host simulation, then sampled hardware	Post-commit and nightly	Vector source, parameter set, random seed, expected result, actual result, first divergence, and minimized reproducer are stored.
Performance	Did a measured metric regress under a controlled configuration?	Dedicated, thermally stable runner	Kernel/model changes, nightly, release	Warmups, sample distribution, p50/p95, cycles, bytes, energy, clocks, temperature, artifact and calibration IDs; compare with versioned threshold.
End to end	Does a real user workflow meet correctness and service-level goals?	Packaged SDK and representative hardware	Model tier policy, merge, nightly	Real preprocessing/postprocessing, accuracy metric, fallback share, compile/load/execute timing, peak memory, throughput and power.
Stress	Does the system remain correct under time, load and recovery pressure?	Isolated lab allocation	Nightly, weekly, pre-release	Soak duration, concurrency, memory pressure, injected faults, reset count, leak trend, temperature and terminal device state.
Qualification	Is the product robust across the deployed device population and operating envelope?	Characterization and production labs	Silicon, firmware, calibration or release change	Device sample plan, PVT corners, defect classes, drift/retention, guard bands, confidence intervals and signed release report.

GoogleTest, pytest, and test data

Choose the framework by the contract being protected

Recommended test subtreeNames describe scope, not just language

tests/
|-- cpp/
|   |-- unit/
|   |   |-- compiler/
|   |   |-- artifact/
|   |   |-- runtime/
|   |   |-- hal/
|   |   `-- firmware_model/
|   `-- integration/
|       |-- compiler_runtime/
|       `-- runtime_backend/
|-- python/
|   |-- unit/
|   |-- integration/
|   `-- package/
|-- smoke/
|   |-- simulator/
|   `-- hardware/
|-- sweeps/
|   |-- operators/
|   |-- numerical/
|   |-- faults/
|   `-- manifests/
|-- end_to_end/
|   |-- vision/
|   |-- transformer/
|   `-- application/
|-- performance/
|   |-- microbenchmarks/
|   |-- operators/
|   `-- models/
|-- stress/
|-- qualification/
|-- common/
|   |-- fixtures/
|   |-- device_pool/
|   |-- artifact_capture/
|   `-- comparisons/
`-- data/
    |-- golden/
    |-- models/
    |-- calibration/
    `-- target_descriptors/

Use	GoogleTest	pytest
Best fit	C++ IR utilities, encoders, allocators, queue/event state, HAL contracts, firmware host models	Python API, framework import, parameter sweeps, model execution, lab fixtures, report validation
Fixture scope	Process-local fake objects and deterministic temporary files	Function fixture by default; session fixture only for expensive immutable setup or reserved device lifecycle
Selection	Small executables or labels plus `--gtest_filter`	Markers such as `unit`, `hardware`, `smoke`, `nightly`, `performance`
Do not duplicate	Test a C++ invariant once in GoogleTest; add a Python test only for binding behavior or an end-user contract, not the same private branch.
Failure output	Expected/actual typed values, state transition, command bytes, seed and target descriptor	Node ID, parameters, backend, device identity, artifact hash, seed, trace path and minimized reproducer

TT-Metal precedent: its root test CMake enables CTest and GoogleTest, while focused C++ executables and extensive pytest directories cover different layers. The model CI tier document further separates end-to-end, module, sweep, device-performance, PR-gate, and stress coverage rather than treating all model tests as equivalent.

Performance discipline

A performance test is a controlled experiment, not a timed unit test

Pin the workloadModel/operator hash, shape, batch, sequence length, precision, layout, compiler options and fallback policy.

Pin the machineBoard SKU, device serial, firmware, clocks, power mode, temperature window, target descriptor, calibration and defect-map IDs.

Separate phasesReport compile, weight program, load, warmup and steady-state execution separately; do not hide setup in token or frame throughput.

Use distributionsRecord sample count, warmup count, p50, p95, variability and outliers. One best run is not a regression baseline.

Count causesCycles, bytes, queue wait, DMA, array active time, ADC occupancy, digital fallback, stalls, saturation and retries explain the headline metric.

Version thresholdsStore absolute limits and allowed regression percentages with owner, reason, baseline commit and expiry/review date.

Prove correctness firstA faster incorrect result fails. Associate accuracy output and numerical-policy identity with every accepted performance record.

Control instrumentationUse detailed traces to diagnose, then measure the official result with the declared low-overhead counter set.

ACiM-specific rule: report cold programming, warm resident-weight inference, recalibration, and degraded/remapped execution as different scenarios. A resident-weight MVM number cannot stand in for the latency or energy of deployment, model switching, digital fallback, or recovery.

Tracy and device profiling

Use Tracy directly on the host and preserve device time at the source

Tenstorrent is a useful reference because its tooling exposes both host work and device-side RISC-V work. The reusable architecture is a split design: normal Tracy instrumentation for code running on the host, plus bounded cycle-stamped records for bare-metal firmware and accelerator engines. A clock-correlation and export layer brings the evidence together without changing when the device work appears to have happened.

Recommended ACiM design: use upstream Tracy for host C++ and Python-visible runtime phases; use one fixed-record trace buffer per device producer; drain through the UMD/KMD path; fit each device clock to the host monotonic clock; then export an accurate device trace. Put device intervals into the Tracy UI only through a maintained bridge that accepts their original timestamps.

What the Tenstorrent implementation teaches

The current TT-Metalium Tracy guide covers host C++, Python, and device views. The more detailed device program profiler guide shows the device mechanism: kernels emit bounded records, and the host runtime drains them for analysis. These are related views, but not the same instrumentation path.

TT path	Observed interface	Reusable lesson for ACiM
Host C++	Normal Tracy `ZoneScoped` or named zones execute in the host process.	Instrument compiler, SDK, UMD, queue, DMA request, wait, fallback and postprocess directly.
Device kernel	`DeviceZoneScopedN` writes device-side scope records through TT's kernel profiler support.	Use a device-native emit primitive around dispatch, data movement, MVM, converter and epilogue phases.
Runtime enable	`TT_METAL_DEVICE_PROFILER=1` enables collection; device close drains automatically, or `ReadDeviceProfilerResults(device)` drains explicitly.	Make profile mode and drain points explicit runtime policy, not unconditional firmware behavior.
Evidence	The documented device output includes `profile_log_device.csv` and can be inspected in device timeline lanes.	Keep a machine-readable raw trace even when a richer profiler UI is available.
Limits	The guide warns about annotation cost, finite L1 records, inter-core skew, unsynchronized devices, and conflicts with DPRINT/Watcher.	Publish buffer, clock, overhead and tool-exclusion contracts for every ACiM profiling mode.

TT host patternRuns where the host work occurs

#include <tracy/Tracy.hpp>

void submit_program() {
  ZoneScopedN("submit_program");
  // Pack, enqueue, submit, and wait phases.
}

TT device-profiler patternReference pattern, not an ACiM header

#include <tools/profiler/kernel_profiler.hpp>

void kernel_main() {
  DeviceZoneScopedN("device_kernel");
  // Device RISC-V work.
}

Choose the path from the execution context

1. LocateWhere did it run?Host process, firmware controller, analog engine, converter, DSP, DMA, or fabric

2. Host?Use Tracy zonesInstrument the C++ call where it executes; preserve thread, nesting, frame, and lock context

3. Device?Read device cyclesFirmware records the local cycle immediately before and after the operation

4. TransportDrain bounded recordsPer-producer SRAM buffer, sequence, drop count, capture ID, ABI, and clock metadata

5. AlignCorrelate clocksMultiple ping samples fit device cycles to the host monotonic timeline

6. InspectJoin by command IDHost Tracy capture plus accurate device trace, counters, numerical result, and machine state

Two data paths, one correlated investigation

Host path: timestamps are created by Tracy in the process that performs the work.

ApplicationRequest and framePreprocess, compile/load policy, inference request, postprocess

SDK / UMDScoped host zonesSession, buffer, queue, submit, wait, callbacks, locks, DMA requests

ClientTracyClientOn-demand event collection in each instrumented process

TransportLocal socketLive event stream from the host process to the profiler

CaptureTracy profilerThreads, zones, frames, plots, messages, memory and OS context

ResultHost attributionCompile, queue, DMA request, wait, callback, fallback, and CPU overhead

Device path: timestamps are created on the controller or engine and survive the later drain.

ControllerRead cycle CSRCommand dispatch, validation, programming, calibration, sequencer work

EnginesEmit IDs and countersDAC/input drive, array MVM, settle, ADC, accumulation, DMA, retries

Device SRAMFixed trace recordsOne bounded single-producer buffer per source; no strings or allocation

KMD / UMDDrain and validateDMA or MMIO, ABI sizes, sequence, drop count, capture and command IDs

RuntimeFit clock and decodePer clock domain affine map plus generated event dictionary

ExporterPreserve historyCSV/JSON, Perfetto, or explicit-timestamp Tracy bridge

Do not create ordinary host zones while draining records and label them as device work. Those zones describe the drain time. Upstream Tracy's normal scoped-zone API is excellent for host execution, but it is not a generic public API for injecting an arbitrary accelerator interval that finished earlier. Keep the raw correlated device timeline authoritative unless the profiler bridge explicitly supports external timestamps.

Use each output for the question it can answer

Question	Recommended mechanism	What it preserves	Important limit
Why is the host request slow?	Upstream Tracy zones, frames, plots, locks, messages, allocation and OS scheduling	Original host timestamps, thread nesting and live context	It cannot infer what happened inside an opaque accelerator command.
Which device phase consumed cycles?	Firmware/device records with begin/end cycles and fixed event IDs	Original device time, source/core identity, sequence, counters and dropped records	Instrumentation consumes device SRAM and cycles; a finite buffer can overflow.
How do host and device overlap?	Clock-correlated device export joined to host command/capture IDs	Queue, DMA, command, MVM, ADC, DSP and completion ordering	Every independent or variable-frequency clock domain needs its own fit.
Must everything appear in Tracy?	Maintained profiler fork or bridge with explicit timestamp/context support	A unified interactive view when the integration is tested and versioned	This is profiler integration code, not ordinary application-zone instrumentation.
Is a regression real?	Uninstrumented benchmark plus low-overhead hardware counters	Official latency, throughput, energy, accuracy and machine configuration	A rich diagnostic capture may perturb the result and cannot be the only baseline.

End-to-end capture sequence and ownership

Step	Application / SDK	UMD / KMD	Firmware / ACiM device	Collector
1. Configure	Select profiling mode, capture ID, command ID and event mask.	Reserve DMA-visible trace storage and enforce ownership/isolation.	Reset selected per-producer buffers and dropped counters.	Open host Tracy capture and the raw device-trace output.
2. Synchronize	Record host send and receive monotonic timestamps around several clock pings.	Transport the ping without interpreting profiler semantics.	Return local device cycle and clock-domain ID with the host cookie.	Reject malformed samples; retain round-trip uncertainty.
3. Submit	Open host zones for pack, enqueue, submit and wait; attach the command ID.	Map buffers, ring doorbell, account bytes, and record interrupt path.	Record dispatch receive and descriptor validation cycles.	Keep host and device records under one capture manifest.
4. Execute	Remain asynchronous where possible; do not hide blocking in a broad request zone.	Expose queue and DMA progress counters.	Record programming/calibration, input drive, settle/MVM, ADC, accumulation, fallback and output DMA.	Detect sequence gaps, mismatched begin/end, and nearing capacity.
5. Complete	Record wait completion, callback, D2H visibility and postprocess.	Acknowledge interrupt/fence and make trace bytes visible safely.	Close zones, publish record count, drop count, clock rate and terminal status.	Associate terminal status, output hash and numerical counters.
6. Drain	Request a drain after completion or at an explicit live-trace watermark.	DMA records; KMD owns privileged transport, not event-name policy.	Hand off a completed buffer; switch to the other buffer for live collection.	Validate ABI before decoding any record.
7. Correlate	Provide the host monotonic time base and command intervals.	Provide transport counters and reset/DVFS epochs.	Provide device cycles, source ID, clock domain and frequency metadata.	Fit each domain, calculate residual/error bounds, and map cycles to host time.
8. Publish	Save the Tracy capture reference and run manifest.	Save driver counters and faults.	Preserve raw records and firmware/event-dictionary versions.	Write an accurate device trace, summary, and optional unified-view capture.

Add Tracy to host code without making it a hard dependency

The working example under examples/acim-minimal/ implements this pattern. Profiling macros compile to no-ops by default; the opt-in preset fetches Tracy 0.13.1 at an exact commit, enables on-demand collection, and limits the client to localhost.

CMake integrationTarget-scoped and opt-in

option(ACIM_ENABLE_TRACY "Instrument host targets" OFF)
option(ACIM_FETCH_TRACY "Fetch pinned Tracy" OFF)

if(ACIM_ENABLE_TRACY)
  if(ACIM_FETCH_TRACY)
    include(FetchContent)
    set(TRACY_ENABLE ON CACHE BOOL "" FORCE)
    set(TRACY_ON_DEMAND ON CACHE BOOL "" FORCE)
    set(TRACY_ONLY_LOCALHOST ON CACHE BOOL "" FORCE)
    set(TRACY_NO_BROADCAST ON CACHE BOOL "" FORCE)
    FetchContent_Declare(tracy
      URL https://github.com/wolfpld/tracy/archive/05cceee0df3b8d7c6fa87e9638af311dbabc63cb.zip)
    FetchContent_MakeAvailable(tracy)
  else()
    find_package(Tracy CONFIG REQUIRED)
  endif()

  target_compile_definitions(acim_profiling
    INTERFACE ACIM_TRACY_ENABLED=1)
  target_link_libraries(acim_profiling
    INTERFACE Tracy::TracyClient)
endif()

Host instrumentationZones match owned phases

#include "acim/profiling.hpp"

Result Runtime::submit(const Request& request) {
  ACIM_PROFILE_ZONE("Runtime::submit");

  {
    ACIM_PROFILE_ZONE("validate_and_pack");
    validate(request);
    pack_descriptors(request);
  }
  {
    ACIM_PROFILE_ZONE("queue_and_doorbell");
    queue_.submit(request.command_id());
  }
  {
    ACIM_PROFILE_ZONE("wait_for_completion");
    completion_.wait(request.command_id());
  }

  ACIM_PROFILE_PLOT("ACiM queue depth", queue_.depth());
  ACIM_PROFILE_FRAME("ACiM inference");
  return collect_result(request);
}

Zone the phase, not every loop iterationStart with request, compile, artifact load, H2D, submit, queue wait, D2H, fallback, and postprocess. Add inner zones only after evidence identifies a broad phase.

Carry IDs as messages or structured metadataCommand, graph, artifact, device, queue, capture, calibration, and defect-map IDs make host zones joinable to runtime logs and device records.

Use frames deliberatelyA frame can mean one image inference, one decode token, or one service iteration. Do not mix meanings in one capture.

Use plots for low-rate stateQueue depth, outstanding DMA, temperature, power, fallback count, and device utilization are useful; per-element plots create noise and overhead.

Build and run the included host integrationRun inside examples/acim-minimal/

cmake --preset tracy
cmake --build --preset tracy
ctest --preset tracy

# Start the Tracy profiler, connect locally, then run:
./build/tracy/acim_smoke-debug       # Linux
build\tracy\acim_smoke-debug.exe     # Windows

The fetched CMake target builds and links TracyClient; obtain or build the desktop Profiler from the same Tracy version separately. With on-demand collection and broadcast disabled, connect to 127.0.0.1 before starting the workload.

Add a firmware-safe device trace

Controller or engine instrumentationNo strings, allocation, socket, or host timestamp

#include "acim/device_events.h"
#include "acim/device_trace.h"

void run_tile(Command* command, AcimTraceBuffer* trace) {
  const uint16_t source = command->tile_id;
  acim_trace_set_command_id(trace, command->id);

  acim_trace_zone_begin(trace, read_device_cycles(),
                        ACIM_EVENT_INPUT_DRIVE_DAC, source);
  drive_inputs(command);
  acim_trace_zone_end(trace, read_device_cycles(),
                      ACIM_EVENT_INPUT_DRIVE_DAC, source);

  acim_trace_zone_begin(trace, read_device_cycles(),
                        ACIM_EVENT_ARRAY_SETTLE_MVM, source);
  start_array_mvm(command);
  wait_array_done();
  acim_trace_zone_end(trace, read_device_cycles(),
                      ACIM_EVENT_ARRAY_SETTLE_MVM, source);

  acim_trace_counter(trace, read_device_cycles(),
                     ACIM_EVENT_ADC_SATURATION_COUNT,
                     source, read_adc_saturation_count());
}

Runtime drain and correlationPseudocode for the real UMD path

samples = []
for cookie in range(sync_sample_count):
    host_send_ns = monotonic_ns()
    device_cycle = firmware.clock_ping(cookie)
    host_receive_ns = monotonic_ns()
    samples.append((host_send_ns,
                    device_cycle,
                    host_receive_ns))

correlation = fit_clock_correlation(samples)
batch = device.dma_read_trace(capture_id)

validate(batch.abi_version,
         batch.header_size,
         batch.record_size,
         batch.record_count,
         batch.dropped_records)

for record in batch.records:
    host_ns = correlation.map(record.cycle)
    exporter.write(host_ns, record.source_id,
                   event_name(record.event_id),
                   record.kind, record.value)

The trace ABI is a hardware/software contract

Field	Why it exists	Validation rule	Typical failure it exposes
`abi_version`, header/record size	Allows firmware and runtime to reject incompatible layouts before parsing.	Exact supported version and sizes; little-endian encoding declared by the platform ABI.	Old UMD with new firmware, wrong packing, stale generated reader.
`capture_id`, command ID	Joins a batch to the host capture and every record to its queued command, including interleaved commands.	The header capture ID identifies the epoch; every record carries a command ID that is unique within that capture.	Trace from a previous capture or another queued command attached to the current result.
`cycle`, clock rate/domain	Preserves when work happened in the source clock.	Monotonic inside a source and epoch; re-synchronize after reset, wrap, rate change, or power transition.	Negative duration, cross-domain ordering error, DVFS-induced drift.
`sequence`	Distinguishes record order when timestamps tie and detects missing records.	Monotonic modulo declared wrap behavior for each producer.	Partial DMA, overwrite, malformed live-buffer ownership.
`event_id`, kind	Represents begin, end, counter or sync without device-side strings.	ID dictionary is generated/versioned; every begin has a legal matching end for the same source.	Unknown firmware event, unbalanced zone, decoder mismatch.
`source_id`	Names controller, tile, array, converter, DSP, DMA engine or virtual lane.	Must be legal for the target descriptor and capture mode.	Wrong routing, unexpected engine, overlapping use of one resource.
`value`, flags	Carries signed counters, host sync cookie, error state or event attributes.	Meaning is fixed by event dictionary and unit metadata.	Saturation, retry, remap, timeout, converter contention, fallback.
`record_count`, dropped count	Makes a truncated profile visibly incomplete.	Count cannot exceed capacity; a nonzero drop count marks the timeline incomplete.	Trace mask too broad, buffer too small, drain too late.

Concurrency rule: make each device buffer single-producer so the hot emit path needs no shared atomic reservation. For live profiling, use two buffers with explicit device-owned, ready-for-host, host-owned, and free states. The device never writes a host-owned buffer; the host never decodes a device-owned buffer.

Correlate clocks instead of assuming they are synchronized

For sync sample i, record host send t0[i], device cycle c[i], and host receive t1[i]. Approximate the host time of the device read as m[i] = t0[i] + (t1[i] - t0[i]) / 2, then fit host_ns = reference_host_ns + ns_per_cycle * (cycle - reference_cycle). Save maximum fit residual and round-trip time; they are evidence about alignment quality, not decoration.

Use several samplesTwo points define a line but cannot reveal jitter. Collect samples before and after the workload, reject malformed/outlier exchanges by an explicit policy, and retain the raw samples.

Fit each domain independentlyController, array sequencer, converter, DSP, and a second chip may have different clocks. Never order unrelated raw cycle values directly.

Start a new epochReset, clock gating, DVFS, wrap, suspend/resume, firmware restart, or device replacement invalidates the old map unless the hardware contract proves continuity.

Expose uncertaintyRound-trip asymmetry and fit residual bound confidence in cross-domain ordering. Events closer than the uncertainty may be concurrent; do not invent a precise order.

Record the phases that distinguish ACiM from a digital NPU

Layer	Zones	Counters and state	Likely diagnosis
Host/compiler	Import, partition, quantize, map arrays, schedule, allocate, package, cache lookup	Graph ops, fallback ops, compile cache hit, artifact bytes, mapped arrays, padding/expansion	Compile overhead, unsupported op, poor utilization, excess digital fallback.
Runtime/transport	Open, program artifact, H2D, enqueue, doorbell, wait, interrupt, D2H, postprocess	Queue depth, DMA bytes, descriptors, interrupts, retries, IOMMU faults	Host serialization, PCIe/DMA limit, command starvation, completion delay.
Weight lifecycle	Erase/program, verify, calibration, defect scan, remap, refresh	Pulses, verify failures, bad rows/columns, spare use, calibration age, drift estimate	Cold-start/model-switch cost, endurance pressure, degraded capacity.
Analog datapath	Input encode/DAC, word-line drive, array settle/MVM, sense, ADC conversion	Active arrays, input slices, output slices, converter occupancy, saturation, clipping, retries	ADC/DAC bottleneck, settle-time increase, limited parallel converters, range mismatch.
Digital epilogue	Bit-slice reconstruction, partial-sum reduction, bias, activation, normalization, unsupported DSP op	DSP/vector cycles, SRAM bytes, NoC bytes, reduction steps, fallback percentage	Analog core is fast but digital accumulation or data movement sets end-to-end latency.
Reliability/power	Thermal throttle, recalibration, recovery, reset, power-state transition	Temperature, voltage, clock, power, ECC, fault, retry, remap, reset reason	Performance drift across time/PVT, intermittent stalls, instrumentation-induced power change.

Read the combined timeline as a causality graph

Host gapRequest to submit is longInspect preprocessing, compile/cache miss, artifact packing, allocation, locks and queue policy.Evidence: host Tracy zones and lock timeline

Dispatch gapDoorbell to device begin is longInspect queue backlog, PCIe/MMIO, descriptor DMA, firmware dispatch, power wake and command validation.Evidence: command ID across host and device

Input starveArray waits for inputInspect H2D, on-chip DMA, SRAM banking, input slicing, DAC/input-driver availability and scheduling.Evidence: bytes, DMA zone, array idle cycles

Converter boundMVM is short; ADC is longInspect ADC sharing, resolution/cycles, output slicing, saturation, range selection and converter scheduling.Evidence: ADC occupancy and conversion cycles

Digital tailAnalog work ends earlyInspect partial-sum reduction, dequantization, activation, normalization, routing and unsupported-operator DSP fallback.Evidence: DSP cycles, SRAM/NoC bytes, fallback

Accuracy taxRetries or calibration dominateInspect program/verify, drift, remap, redundant reads, high-resolution re-read and layer precision policy.Evidence: retries, saturation, remap, profile ID

Make profiling depth an explicit runtime mode

Mode	Enabled evidence	Use	Acceptance rule
`off`	No Tracy client activity; minimal architectural counters only if always-on by design	Official latency, throughput, energy and accuracy benchmark	Configuration is recorded and reproducible.
`host`	Host Tracy zones, frames, plots, messages, locks and allocation	Compiler, runtime, queue, callback and CPU diagnosis	A/B overhead versus `off` is measured.
`device-summary`	Command begin/end plus aggregate DMA, array, ADC, DSP, retry and saturation counters	Low-overhead continuous regression explanation	Counter read cost and wrap behavior are characterized.
`device-scoped`	Selected phase begin/end records per source plus counters	One operator/model bottleneck investigation	Zero drops for accepted capture; buffer and clock-fit quality reported.
`deep-debug`	Fine engine/NoC/converter records, waypoints, selected data snapshots	Short controlled reproduction only	Resource conflicts and probe effect are documented; never used as headline performance.

Budget device instrumentation before enabling it. Tenstorrent's device-profiler documentation describes finite per-core L1 storage, small inter-core clock skew, generally unsynchronized clocks across devices, annotation overhead, and resource conflicts with DPRINT and Watcher. An ACiM implementation should publish its own bytes/record, records/source, emit cycles, drain bandwidth, exclusions, and overflow behavior.

Measurement protocol and exit gates

1. FreezeWorkload and machineArtifact, input, precision, firmware, clocks, power, temperature, calibration

2. WarmSeparate setupCompile, program weights, calibration, allocator and cache warmup reported apart

3. CaptureDiagnostic profileSelected host zones, device events, counters, correctness and IDs

4. ValidateTrace qualityABI, sequence, balanced zones, zero unexpected drops, clock residual, no faults

5. CompareMeasure probe effectSame run with profile off, summary, and selected scoped mode

6. ReportOfficial evidenceUninstrumented distribution plus diagnostic explanation and archived manifests

Trace completenessEvery accepted command has submit, device begin/end, terminal status and output correlation; all device zones balance; record drop count is zero unless the report is explicitly marked incomplete.

Clock qualitySync sample count, maximum round trip, fit residual, frequency/epoch, and cross-domain ambiguity are reported against a project-defined budget. Compare 1e9 / ns_per_cycle with the header clock rate to catch a wrong domain or unreported rate change.

Numerical validityThe profiled output still passes the same accuracy policy; saturation, retries, remap, fallback and calibration identity accompany the performance trace.

Performance validityWarmup, sample count, p50/p95, temperature/power state and profile mode are fixed. Rich capture explains the result; the uninstrumented run supplies the claim.

Working code: the included example provides opt-in host zones, the pinned Tracy CMake target, a C-compatible 32-byte device record, event IDs, overflow accounting, clock-sync samples, affine correlation, and unit tests. It intentionally stops before pretending to be a real PCIe drain or a custom Tracy protocol extension; those require the actual device memory map, timer domains, firmware ownership protocol, and profiler API.

Continuous integration

Route cheap certainty early and expensive evidence deliberately

TT-Metal explicitly describes its pull-request gate as a fast minimum bar and relies on broader post-commit coverage. Use the same principle with declared time budgets and changed-path routing.

DeveloperLocal loopFormat, focused unit, functional simulator; target under 2 minutes

Pull requestFast host gateStatic, build, sanitizers, affected unit and golden tests; target under 8 minutes

Selected PRHardware smokeChanged-path route on reserved SKU; one known operation and cleanup

Post-commitIntegrationAll host tests, packaged SDK, simulator E2E, broader device matrix

NightlyDepthSweeps, models, performance, stress, sanitizer variants, multiple SKUs

ReleaseQualificationCompatibility, population/PVT, recovery, packages, reproducibility, reports

Changed path	Always run	Selected hardware	Deferred depth
`docs/ examples/`	Format, spelling/link, code-snippet compile or execution	Only when an example declares hardware	Documentation build and packaged tutorial run
`schemas/ generated/`	Regeneration diff, old/new compatibility, C++/Python/firmware round-trip	Artifact-load smoke if command or ABI semantics changed	Full consumer matrix and upgrade tests
`src/compiler/`	Compiler unit, verifier, golden IR, deterministic artifact, functional reference	One affected operator if lowering or command output changed	Operator sweeps, model tier, compile-time and artifact-size trends
`src/runtime/ src/hal/`	GoogleTest lifecycle, fake backend conformance, ASan/UBSan, multithread tests	Open/load/submit/wait/timeout/reset smoke	Concurrency, stress, recovery and multi-device tests
`src/driver/ src/firmware/`	Build, static analysis, ABI/layout checks, firmware host model	Boot, DMA/interrupt, bad descriptor, timeout and reset	Soak, fault injection, power transitions and upgrade/rollback
`kernels/`	Compile, static checks, numerical golden, simulator comparison	Affected kernel correctness and health monitor	Shape/precision sweep, device profiler and regression benchmark
`python/ bindings/`	Wheel build, import, pytest unit, API and exception behavior	Minimal public Python execution path	Supported Python/platform matrix and framework integration
`models/`	Manifest validation, reference/module tests, preprocessing checks	Critical model tier selected by manifest	Full accuracy, E2E performance, stress and fallback accounting

Workflow ownershipKeep routing visible instead of hiding every lane in one CI file

.github/workflows/pr-fast.yml                  # CPU-only minimum bar
.github/workflows/pr-hardware-smoke.yml        # changed-path device route
.github/workflows/post-commit.yml              # broad integration
.github/workflows/nightly-sweeps.yml           # parameter and model depth
.github/workflows/nightly-performance.yml      # controlled runner baselines
.github/workflows/release-qualification.yml    # compatibility and product evidence

configs/test-matrices/*.yaml                   # SKU, topology, marker, timeout
configs/performance-thresholds/*.yaml          # metric, limit, baseline, owner
scripts/ci/changed_path_router.py               # path -> required lane mapping
scripts/ci/collect_failure_bundle.py            # standard artifacts on failure

Executable example: .github/workflows/acim-minimal.yml applies this pattern to the included simulator. It defines GCC Debug, Clang Debug, Clang ASan+UBSan, staged-install smoke, and pinned GoogleTest and Tracy integration jobs, plus clang-format and clang-tidy checks, whenever the example or its workflow changes.

Contribution and code conduct

Make the engineering agreement executable and easy to find

CODE_OF_CONDUCT.md defines respectful project behavior. CONTRIBUTING.md defines how technical changes are proposed, built, tested, measured, reviewed, and merged. They solve different problems.

1. FrameProblem and evidenceReproducer, requirement, trace or measured bottleneck

2. DesignContract impactADR for ABI, architecture, schema or ownership changes

3. TestFailure and gateSmallest deterministic test plus required deeper lanes

4. ImplementFocused commitsEach commit builds and preserves relevant behavior

5. ProveCorrectness and metricsReports, traces, compatibility and performance evidence

6. ReviewOwners and follow-throughCODEOWNERS, docs, release note and monitored merge

Repository file	What it should answer	Practical content
`README.md`	What runs, for whom, and what is the shortest successful path?	Supported targets, architecture map, build/install, first simulator example, first hardware example, test commands, documentation links.
`CONTRIBUTING.md`	How does a change become trusted code?	Environment, style, commit rules, test selection, PR categories, design/ADR threshold, review, performance evidence, debugging bundle.
`CODE_OF_CONDUCT.md`	How do contributors work together?	Expected behavior, unacceptable behavior, reporting path, enforcement responsibility and scope.
`.github/CODEOWNERS`	Who understands and approves each contract?	Named owners for schema/ABI, compiler, runtime, driver, firmware, kernels, models, CI, performance and qualification.
`docs/adr/`	Why does a durable architecture decision exist?	Context, options, decision, consequences, compatibility, migration, tests, owners and review date.
PR template	What changed and what proves it?	Category, affected boundaries, test commands/results, hardware/SKU, accuracy, performance, docs, compatibility and rollback.

Code styleFormat automatically; compile with strong warnings; use static analysis; keep public headers minimal; prefer typed units, enums and spans over unstructured integers and pointers.

Error behaviorStable status category, subsystem, operation/graph ID, device ID, cause, recovery hint and machine-readable fields. Do not make users decode a generic timeout.

LoggingStructured events with severity and correlation IDs. Avoid hot-loop logging and sensitive/raw tensor dumps by default.

Commit qualityDescriptive subject, one coherent purpose, builds independently, relevant tests pass, generated output and docs remain synchronized.

Review depthArchitecture/schema changes need contract owners; kernel performance changes need correctness and profiler evidence; model claims need reproducible manifests.

Definition of doneImplementation, focused regression test, required lanes, observable failure path, documentation, compatibility decision, and measured acceptance evidence.

Debugging playbook

Find the first layer that diverges, then instrument only that layer

TT-Metal treats debug print, Watcher, Inspector, Tracy, device profiling, assertions, triage and NoC dumps as explicit tools. An ACiM repository needs equivalent host, compiler, runtime, device, numerical and performance views.

1. ReproduceFreeze the runCommit, target, artifact, calibration, seed, inputs, backend

2. ClassifyName the symptomBuild, compile, load, hang, wrong result, intermittent, performance

3. CompareFind first divergenceReference, ideal array, nonideal model, queued sim, RTL, silicon

4. InspectNarrow the layerIR, artifact, buffers, queue, firmware, array, digital epilogue

5. ProveMinimize and fixSmall vector, deterministic failure, explicit invariant

6. PreserveAdd regressionFocused test, failure bundle, docs and applicable deeper lane

Host C++Memory, lifetime, threadsDebug symbols, GDB/LLDB, ASan, UBSan, TSan, core dumps, queue/event state and ownership logs.Start: failing GoogleTest under sanitizer

Python and bindingsAPI, conversion, exceptionspytest -x -vv, one node ID, faulthandler, typed boundary logs, native debugger for binding crashes.Start: installed package, one parametrization

CompilerSemantics and loweringVerify after every pass, dump canonical IR, pass bisect, target legality report, mapping explanation, deterministic artifact diff.Start: first different IR operation

Artifact and runtimeCompatibility and executionArtifact inspector, schema/ABI versions, buffer ownership, command list, correlation IDs, queue timeline, timeout reason and cleanup trace.Start: load transcript and command ID

Firmware and deviceHang, fault, bad commandWatcher-style health monitor, waypoints, device print, bounded asserts, command validation, register snapshot, interrupt history, safe reset and triage bundle.Start: last completed waypoint

Analog numerical pathAccuracy and variabilityLayerwise reference, ideal-array, characterized nonideality and silicon outputs; converter saturation, slice sums, calibration, remap and defect-map identity.Start: first tensor exceeding tolerance

PerformanceLatency and utilizationSame artifact and machine state, warmup, host/device timeline, DMA bytes, queue waits, array active time, ADC/DSP occupancy, stalls and thermal history.Start: phase with changed cycles or bytes

Intermittent failuresRace, thermal, state leakRepeat with fixed seed, record preceding tests, serialize device use, vary reset depth, track temperature, allocator state, calibration age and firmware logs.Start: smallest repetition count that fails

Scale-outRoute and collective failureTopology epoch, logical/physical map, route table, per-link counters, collective step, timeout ownership and before/after recovery trace.Start: first node missing progress

Every failure should produce one portable evidence bundle

Proposed triage bundleMachine-readable files plus a short human summary

failure-20260711T143100Z/
|-- summary.md                    # symptom, first divergence, next action
|-- run-manifest.json             # commit, command, seed, environment
|-- model-manifest.json           # model, inputs, preprocessing, accuracy policy
|-- target-descriptor.json        # SKU, revision, topology and capabilities
|-- device-state.json             # serial, firmware, clocks, temperature, health
|-- calibration-manifest.json     # calibration and defect/remap identity
|-- executable.acx                # exact artifact when data policy permits
|-- artifact-report.json          # sections, versions, hashes, requirements
|-- compiler/
|   |-- pipeline.txt
|   |-- first-before.mlir
|   |-- first-after.mlir
|   `-- mapping-report.json
|-- runtime/
|   |-- commands.jsonl
|   |-- buffers.json
|   `-- queue-timeline.json
|-- device/
|   |-- firmware.log
|   |-- waypoints.json
|   |-- fault-snapshot.json
|   `-- reset-transcript.json
|-- numerical/
|   |-- comparison.json
|   `-- first-divergent-tensor.bin
`-- performance/
    |-- counters.json
    `-- trace.atrace

Do not enable every device tool at once. Instrumentation consumes memory, bandwidth, timing margin, or device resources and can change a race or performance result. TT-Metal documentation explicitly notes conflicts among some device debug/profiling modes; document equivalent exclusions in the new stack.

Developer loop

Give contributors short, memorable commands for every test depth

These are proposed command names for the blueprint, not commands currently supplied by TT-Metal.

Local build and focused verificationIllustrative interface to implement in scripts and presets

# Configure, build and install a host-only development tree
cmake --preset dev
cmake --build --preset dev --parallel
cmake --install build/dev --prefix build/install

# Fast checks before a pull request
pre-commit run --all-files
ctest --preset host-unit --output-on-failure
python -m pytest tests/python/unit -m "unit and not hardware"

# One complete simulator path
python -m pytest tests/smoke/simulator -m smoke
acim-compile models/manifests/matmul.yaml -o build/matmul.acx
acim-run build/matmul.acx --backend functional --trace build/matmul.atrace

# Reserved hardware path
python -m pytest tests/smoke/hardware -m "hardware and smoke" --device auto

# Broader correctness and performance lanes
python -m pytest tests/sweeps -m nightly --maxfail=20
python -m pytest tests/end_to_end -m "model_tier_1"
python -m pytest tests/performance -m performance --benchmark-manifest configs/performance.yaml

# Diagnostic host/device profile; compare against profile-off benchmark
cmake --preset tracy
cmake --build --preset tracy
ctest --preset tracy

# Reproduce a failure bundle
acim-triage collect --run-id RUN_ID --output build/triage/RUN_ID
acim-triage compare --left functional --right hardware --first-divergence

A real implementation should wrap lab reservation, timeout, artifact capture and reset in fixtures or runner services so individual tests cannot leave a shared device unusable.

Repository readiness workbook

Eleven checks before broad operator or model development

Complete these in order. Every checked item should point to code, a command, and saved evidence in the repository.

0%

0 / 11 complete

Primary references

Repository, build, tests, CI, profiling, contribution and debug sources

TT-Metal observations and build-tool behavior link to primary project documentation. The ACiM tree, commands, sample architecture and CI budgets on this page are proposed engineering guidance.

Tenstorrent TT-Metal repositoryCurrent top-level source, model, test, documentation, build, tooling and automation organization.

TT-Metal contribution guideBuild/test expectations, GoogleTest use, Python testing, debugging, formatting, commits, pull requests, review and architecture plans.

TT-Metal tests READMESweep framework, vector generation, result handling, dry runs, validation statuses and test-runner concepts.

Metalium test organizationSubsystem-oriented test placement, naming, fixtures and file organization.

TT-Metal model CI tiersTiered end-to-end, module, sweep, device-performance, PR-gate and stress coverage.

TT-Metal root test CMakeCTest, GoogleTest, common test libraries and component test inclusion.

TT-NN GoogleTest targetsFocused object libraries and smoke/basic/unit executable composition.

Metalium GoogleTest targetsSubsystem test executables for API, data movement, debug tools, device, dispatch, integration, NoC and performance.

TT-Metal pull-request gateFast minimum-bar philosophy, changed-file routing, build/sanitizer, smoke, documentation and analysis jobs.

TT-Metal smoke workflowConfiguration-driven hardware matrices, SKU routing, time budgets, sanitizer modes and focused executables.

TT-Metalium debugging toolsDebug checkpoints, device print, Watcher, Tracy, device profiler, asserts, Inspector, triage, NoC dump and TT-SMI.

TT-Metalium Tracy profilerHost C++, Python and device-side profiling and trace capture.

TT-Metalium device program profilerDevice scoped annotations, runtime drain, CSV output, finite L1 buffers, timing caveats, overhead and debug-tool conflicts.

TT performance toolsDevice timelines, operation summaries, cycle analysis, DRAM/NoC traffic, barriers and per-RISC execution evidence.

Tenstorrent Tracy forkThe maintained TT-Metalium profiler fork, showing that unified accelerator visualization is a deliberate integration component.

Upstream TracyProfiler source, C/C++ instrumentation APIs, client library, server applications, documentation and supported platform integrations.

Tracy 0.13.1 releaseThe exact upstream release pinned by the minimal example at commit 05cceee0df3b8d7c6fa87e9638af311dbabc63cb.

Perfetto Trace SDKPrimary documentation for writing timestamped trace events when an accurate portable device timeline is preferable to a custom Tracy bridge.

TT-Metalium getting startedCurrent high-level component, example, documentation and contributor entry points.

TT-Metalium architecture guideProgramming model, hardware concepts, host workflow and low-level execution context.

CMake tutorialOfficial target-oriented configure, build, library, testing, installation and packaging progression.

CMake presetsChecked-in project presets, local user presets, configure/build/test workflows and toolchain selection.

CMake toolchainsCompiler detection, capability checks, sysroots and GCC/Clang cross-compilation contracts.

CTest moduleNative test enablement, BUILD_TESTING, labels, dashboard integration and common runner behavior.

CMake FetchContentConfigure-time dependencies, package integration and the recommendation to pin remote content by commit hash.

Ninja manualGenerated dependency graph execution, incremental builds, command tracking and parallel scheduling.

GCC C++ optionsLanguage-standard, diagnostics and GNU compiler behavior used by the Linux baseline lane.

Clang user manualCompiler driver, target and sanitizer options used by the independent Clang lane.

Clang AddressSanitizerMemory-error instrumentation, symbolization, runtime behavior and limitations.

clang-tidyAST-based diagnostics and check configuration for bug-prone, performance and portability analysis.

Clang source-based coverageCoverage instrumentation, profile collection, llvm-profdata, llvm-cov reports and format limitations.

GoogleTest CMake quickstartCMake target setup, CTest discovery, assertions and dependency options.

pybind11 build systemsCMake helpers and build-system integration for a C++ Python extension.

scikit-build-corepyproject.toml-based CMake wheel and editable-install integration.

CMake FindDoxygenImported Doxygen/Graphviz tools and explicit API-documentation build targets.

Linux KbuildOfficial kernel build system used for an ACiM Linux kernel-mode driver.

Minimal ACiM build exampleWorking C++20 simulator, CMake/Ninja presets, CTest, optional GoogleTest and Tracy, device trace ABI, clock correlation, installation and cross-build notes.

ACiM SDK and compiler implementationTarget, IR, artifact, HAL, runtime, calibration, simulator and vertical-slice contracts used by this repository blueprint.

ACiM board bring-up guideKMD, UMD, PCIe, DMA, interrupts, firmware, reset, analog lifecycle and YOLO bring-up requirements.