Triton + CUDA Kernel Portfolio¶
A hands-on path from ordinary Python loops to Triton and CUDA kernels. Work through small numerical examples, implement the exercises, then inspect the optimized operators and measurements on an NVIDIA RTX 4080 SUPER.
Read it as a website: https://buicongnguyen.github.io/triton-cuda-lab/
New to kernels? Start with the beginner course. It has ten lessons, ten editable exercises, hints, separate reference answers, and a checker that explains which property failed. The first seven exercises need only Python's standard library.
Already comfortable? Optimization labs · GPU setup · measured case studies · validation.
Next levels: Intermediate and advanced workshops add strided softmax, an explicit backward pass, fused GEMM, online normalization, wide-row split softmax and streaming attention. Each has a starter, separate working answer and checks; GPU experiments preserve raw measurements.
The path, step by step¶
A session is 45–90 minutes. Do the steps in order; each one links to the guide that walks you through it.
| Step | What you do | Guide | Terminal | Sessions |
|---|---|---|---|---|
| 1 | Indices, strides, reductions, softmax, RMSNorm, GEMM and measurement in plain Python | Beginner lessons 0–7 | PowerShell or WSL; no GPU packages | 8–10 |
| 2 | Install the GPU environment once | SETUP.md | PowerShell (or WSL) | 1 |
| 3 | Write three Triton kernels, then read the CUDA version | Lessons 8–9 | GPU environment; PowerShell to build CUDA | 3–5 |
| 4 | Change configurations and explain measurements | Labs, case studies | GPU environment | 3–6 |
| 5 | Harder kernels: views, backward, fused GEMM, online softmax, attention | Workshops | GPU environment (A1 is CPU-only) | 2–5 each |
| 6 | Explain your own work | Journal, case studies | Any | 2 |
The 18-session schedule breaks steps 1–4 into sessions. Stuck on a word such as stride, L2, CUDA graph or constexpr? See the glossary.
Which terminal? (Windows)¶
| Task | Use | Why |
|---|---|---|
Lessons 0–7, workshop A1, --hint, learning.explain |
Any PowerShell with python (or WSL with python3) |
Standard library only |
Anything using Triton: lesson 8, GPU workshops, kernel-bench, tests/ |
The GPU environment: PowerShell after .\.venv-win\Scripts\Activate.ps1, or WSL after source .venv/bin/activate (setup) |
Needs PyTorch with CUDA plus Triton (triton-windows on Windows) |
Building and running cuda/kernels.cu |
PowerShell with scripts\build_cuda_windows.cmd, or CMake on Linux |
Uses the Windows CUDA Toolkit and Visual Studio |
Everything can be done from PowerShell. WSL is an alternative for the GPU work, and
it is what produced the recorded results. A checker line reading UNAVAILABLE
(exit code 3) means the current terminal has no PyTorch, Triton or NVIDIA GPU. It is
not a failure in your code: activate the GPU environment
(every new terminal).
First session — no GPU needed¶
From the repository root:
python -m learning.check --list
python -m learning.explain indexing
# Edit learning/exercises/offsets.py, then check your answer:
python -m learning.check offsets
Read offsets and masks alongside the exercise.
An unfinished starter reports TODO; that is expected. Use --hint when stuck
and --solution only to check the separate reference answer.
| Learn first | Then implement | Finally explain |
|---|---|---|
| Indices and masks | Ceiling division and block ownership | Why N=257 needs a tail mask |
| Storage and strides | Read cropped and transposed views | Why shape alone cannot locate a row |
| Reduction trees | Pairwise summation with neutral padding | Why FP32 accumulation and order matter |
| Stable softmax | Max → shift → exp → sum → divide | Why zero padding gives wrong probabilities |
| RMSNorm and GEMM | Scalar formulas and tile ranges | Precision semantics and input reuse |
| Measurement | Speedup, GB/s and Amdahl's law | Why graph and Python-dispatch results differ |
| Triton and CUDA | Three GPU exercise kernels | How block-level values map to thread cooperation |
Use the 18-session schedule, answer explanations and personal journal to turn each lesson into evidence you can discuss.
What is implemented¶
| Stage | Code | Optimization question |
|---|---|---|
| 1. Vector add | add, hello_triton.py |
How do masks, coalescing and block size affect a bandwidth-heavy kernel? |
| 2. Row sum | _row_sum |
How do reduction order, row width and accumulation dtype matter? |
| 3. Softmax | _softmax |
When does fusion beat eager and compiled PyTorch? |
| 4. Residual RMSNorm | _rmsnorm |
Can an intermediate tensor be eliminated without changing precision semantics? |
| Wide rows | _row_plan, looped and split kernels |
How do rows wider than one block (up to 2^20) stay fast, and when must a row be split across programs? |
| Training | _softmax_bwd, _rmsnorm_bwd, _matmul_strided, autograd registration, train_tiny.py |
What does a backward kernel need? How is a transposed GEMM read without a copy, and how does a custom op train inside torch.compile? |
| 5. GEMM | _matmul |
When do grouped tile order and autotuning beat a fixed configuration or cuBLAS? |
| torch.compile | library.py |
How do custom ops let a compiled model replay many kernels as one CUDA graph? |
| CUDA comparison | kernels.cu |
How do threads, warp shuffles, shared-memory barriers, online statistics and float4 loads implement the same ideas? |
Each operator has independent reference checks. Benchmarks keep raw samples, correctness errors, hardware/software metadata and source hashes. Results include losing cases; no universal speedup is claimed. See the measurement method.
GPU portfolio quick start¶
Native Windows PowerShell and Ubuntu WSL2 both work; SETUP.md compares them and has native CUDA build commands. In PowerShell:
.\scripts\setup_windows.ps1 # once: creates .venv-win (about 3 GB)
.\.venv-win\Scripts\Activate.ps1 # in each new terminal
python examples/hello_triton.py
python -m learning.check all --include-gpu --solution
$env:KERNEL_REQUIRE_GPU = "1"; python -m unittest discover -s tests -v
kernel-bench --op softmax --dtype float32 --compile --output results/local/softmax.json
In WSL, after the WSL setup, the same commands
apply, with source .venv/bin/activate and KERNEL_REQUIRE_GPU=1 python -m unittest ....
Every operator trains: given a gradient-tracking input, it runs through a custom op
whose backward pass is a Triton kernel (softmax, residual_rmsnorm, matmul), the
identity (add) or a broadcast (row_sum), in eager code and inside torch.compile.
examples/train_tiny.py trains a small model that uses all
five beside a plain-PyTorch twin. Double backward is not supported. All operators reject
mixed devices, unsupported layouts and out-of-range shapes. Row kernels support
FP32/FP16/BF16 and widths 1–1,048,576 (rows wider than 8192 loop over chunks, or split
across programs when there are fewer rows than SMs);
GEMM supports contiguous FP16/BF16 inputs with FP32 accumulation. For use inside
torch.compile, import kernel_portfolio.library registers the same kernels as
torch.ops.kernel_portfolio.* custom ops. Read the full
contracts before using them.
Document map¶
Guides to learn from, in reading order
- Beginner course: lessons, exercises, answer explanations, schedule, journal template and glossary.
- Workshops: six deeper tasks, their experiment worksheet and answer explanations.
Guides for running and measuring
- SETUP.md: environments, GPU and CUDA builds, operator contracts, troubleshooting.
- LABS.md: six experiments on the finished operators.
- BENCHMARKING.md: what the timing modes measure and what they exclude.
- PROFILING.md: compiler IR/PTX, Nsight and Compute Sanitizer.
- CASE_STUDIES.md and results/SUMMARY.md: measured results and their interpretation.
Reading
- REFERENCES.md and book previews: Manning's GPU Programming with Triton, Fregly's AI Systems Performance Engineering and official Triton/NVIDIA documentation.
Project records (optional: how the repo was planned, reviewed and validated)
- BEGINNER_UPGRADE.md and INTERMEDIATE_ADVANCED_PLAN.md: design plans.
- REVIEW.md: review findings and fixes, including why the results were re-measured.
- results/VALIDATION.md: exactly what was executed, with logs.
Evidence and limits¶
The results folder contains actual local GPU runs, not estimated numbers. GPU tests cover ragged dimensions, dtype differences, row strides, empty inputs, stable softmax, gradients of every operator against FP64 autograd, invalid calls, a non-default stream, kernel reuse across shapes and a small model trained beside a plain-PyTorch twin. CPU CI checks reference math and contracts on Python 3.10 and 3.12, and runs the Triton kernels in Triton's CPU interpreter; it does not certify GPU behavior or performance.
The teaching path's reference solutions passed 40 CPU checks and 36 GPU checks; the checker has its own regression tests. See beginner validation. Reference checks validate the material and do not mark your unfinished exercises as complete.
This is an AI-assisted learning project. Reproduce the experiments and add your own explanations before presenting any of it as your own work. Double backward, batched attention, non-NVIDIA accelerators, distributed collectives and production model integration are future work. Every operator trains through Triton backward kernels or their exact formulas; the workshops add an explicit softmax VJP exercise and bounded single-head streaming attention. Hardware-counter profiling requires permissions unavailable in the local validation environment; that limitation is recorded.
License¶
MIT; see LICENSE.