Triggers: adding or modifying Metal kernel / shader / dispatcher; LLM decode/prefill performance optimization; operator fusion (export-side declaration + backend single dispatch); per-op profiling to locate bottlenecks; running Metal LLM tests or cross-checking.
Boundaries: do not read or modify
schema/private/、source/internal/。
Usage
First locate the specific sub-doc by task type, this file only serves as an index and router, and carries no technical content.
Recording rules (by default, do not record)
The results of each experiment do not need to be written into the skill. The skill is operational knowledge for the next person, not a running log of experiments. The vast majority of A/B runs are done once finished; just write them in the commit message and PR description.
Only when one of the following is met is it worth writing into the skill, and only write "the reusable sentence", without copying data:
| Worth writing | Where to write | What to write |
|---|---|---|
| Onecriterion that still holds across shapes/chips(e.g. "the coop-tensor path gains by widening K and loses by widening N, because destination capacity only follows N") | the methodology section of the corresponding sub-doc | one sentence of principle + applicable boundary, without percentages |
Onepitfall that will make people repeat the same mistake(compile-time macros synchronized in four places, memory aliasing,run_test.out using precision 1 silently disables the fp16 kernel, zsh not word-splitting causes the switch to not take effect) |
kernel-dev-and-optimize.md §1.6 or "General principles at a glance" in this file |
symptom + root cause + avoidance pattern |
| Oneenv switch that is added/removed/renamed | env-registry.md |
only write name / default value / function / conditions for taking effect |
| Onenew routing or default value(which shape goes to which kernel) | kernel-dev-and-optimize.md routing quick reference |
condition → which kernel to use |
| Onedirection that has been disproven and is very likely to be retried by others | only when it can be distilled intoa reusable criterion, write it into the "applicable boundary" of the corresponding means | "this type of means does not hold under condition Z, because Y"—leave no experiment archive |
Do not write into any skill file:
- percentages, ms, GB/s, tok/s, pairing details, sweep tables, per-rep variance of a single A/B;
- narrative of the calibration process (what was measured first and what after, which time it was overturned, which time was a drift artifact);
- numbers that only hold on one machine and one model;
- implementation details of reverted experimental code.
kernel-dev-and-optimize.md has additional hard constraints: it recordswhy each means can gain—which type of resource it gains on (bandwidth / parallelism / instructions / synchronization), under what conditions it holds, and what pitfalls there are.Do not write any performance numbers(percentages, ms, GB/s, TFLOPS, tok/s, sweep tables, pairing details),do not write dates / commit hashes / specific machine models and model names(devices and shapes appear only as "categories", such as "tensor-coop devices", "models with larger GQA groups"),do not write counterexample archives and calibration narratives. See the recording rules at the beginning of that file for details.
runtime-scheduling.md Record only optimization methods and principles: preserve the source of overhead, applicable conditions, safety boundaries, code entry points, and verification methods;do not write any measured data or experimental narratives, including elapsed time, throughput, proportions, measured counts, A/B results, and number of tests passed. See the recording rules at the beginning of that file for details.
env-registry.md has additional hard constraints: it is merely a lookup table of "which switches currently exist, what their defaults are, what they do, and under what conditions they take effect".Any performance gain or loss brought about by any environment variable must never be written into that file—including percentages, pairing results, promotion/falsification conclusions, commit hashes, dates, and device and model names. See the constraint list at the beginning of that file for details.
Sub-doc structure
| File | When to read | Content |
|---|---|---|
op-bench-and-diagnosis.md |
slower than the competitor but don't know where the slowness is; or a certain kernel is known to be slow and you want to know "how much room is left / where to change / when to stop"—the first stop for any optimization task | Diagnostic workflow: why "op unit test (signal-to-noise ratio) + competitor baseline (absolute ruler)" must both be used together; how to build an op unit test (5 construction requirements, whether to choose GB/s or TFLOPS as the figure of merit); how to build a competitor image (6 comparability checks,lazy graph donation trap); selection among the three diagnostics (absolute ruler / fixed-streaming decomposition t=a+b·N / ablation ladder);calibration scale division-of-labor table(path switches use e2e, kernel-internal knobs use op unit tests); verification (changing reduction order cannot use byte-compare,env override as zero-rebuild baseline); stopping criteria; measurement hygiene + pairing script template; list of ready-made assets; summary table of 5 cases |
kernel-dev-and-optimize.md |
writing or modifying any Metal kernel beforealways read Part One first; when doing kernel-level performance optimization, read Part Two | Part One Development Specifications: core principles, dispatcher structure, shader organization and kernel naming conventions,variable/macro names must equal the real physical quantity, four-place synchronization of compile-time macros and pipeline cache keys, Execution skeleton and registration, 9 general traps (A–I), packed weight design, modification workflow, correctness verification, tensor API cooperative tensor layout, Apple GPU levers.Part Two Optimization Knowledge Base (write only mechanisms and principles, no numbers / dates / counterexample archives): optimization master outline and decode profile,kernel-internal ablation ladder (delete work category by category to locate internal bottlenecks + floor value vs competitor), GEMV (deferred dequant / dual SG / pre-scaling / split-K / GEMV_SG_OC_QUADS_2 / W2W3 / short-sequence gap), GEMM (fused Q4 / M64 / in-shader threshold), Attention (causal-tri / FA / tensor-API tc / single-pass SDPA / SDPA_QH_PER_TG / QSPLIT / GQA / routing quick reference), other kernels (RMSNorm small batch / LinearAttention scan evolution / gated norm), §2.5 kernel techniques methodology (principles/applicability/pitfalls/validation, for topic selection and design review) |
graph-fusion.md |
Need multiple operators to be fused into a single dispatch; change FusedLinear / GatedRMSNorm; troubleshoot "fusion not hit / wrong output after fusion" |
Full fusion pipeline: Python export-time group declaration → converter LN absorption → geometry fallback splitting → Metal setupFusion assembles leader/follower. Includes memory aliasing iron rules and STATIC re-home, chained gating dependencies,GatedRMSNorm independent pipeline, removed backend graph-matching history, troubleshooting checklist, §9 fusion methodology (isomorphic projection merging / packed grid / correctness threshold) |
runtime-scheduling.md |
Suspect decode has CPU blocking / GPU idle bubbles; change resize timing, commit cadence, H2D, Encode Replay | per-backend fence, content-cache, in-queue H2D upload, sampling de-frameworkization (host ArgMax/topK), completion-wait spinning, commit cadence, Encode Replay (safety model / attention and LinearAttention integration / KV dangling pointer pitfall), validation routine for scheduling changes, §9 scheduling methodology (sync point governance / automatic thresholds / switch convergence discipline) |
build-and-test.md |
After changing code, need to build / run tests / do cross-checking | cmake build commands, model export commands, performance test commands, CPU/Metal cross-checking |
env-registry.md |
Look up / add / remove Metal-related environment variable switches | env centrally registered,only four columns: name / default value / function / effective condition: attention prefill, attention decode, LayerNorm, decode GEMV, quantized GEMM, fusion, scheduling, profiling. Includes new switch conventions. ⛔ Forbidden to write any performance numbers or experimental narratives |
Quick task→sub-doc index
| What you want to do | Read first |
|---|---|
| Add a new Metal op / kernel (what it should be named, where to write it, how to register it) | kernel-dev-and-optimize.md §1.3 / §1.5 |
| Add a compile-time variant macro | kernel-dev-and-optimize.md §1.4(Four places must be kept in sync) |
| Add a new quant bit / change dispatcher path | kernel-dev-and-optimize.md §1.2 / §1.6 / §1.7 |
| Want to know Metal pitfalls (macro alias / weight byte order / getDequantScale coef) | kernel-dev-and-optimize.md §1.6 |
| Choosing Apple GPU optimization levers (sg_matrix / sg_reduce / tensor API) | kernel-dev-and-optimize.md §1.11 / §1.10 |
| Slower than the competitor, but don't know where the slowness is | op-bench-and-diagnosis.md——First stop for optimization tasks, includes the complete diagnostic procedure |
| Need to create a new op-level speed unit test / competitor mirror | op-bench-and-diagnosis.md §1 / §2 (5 construction requirements + 6 comparability checks) |
| Gap varies with scale (ctx/seq), don't know whether it's fixed overhead or streaming rate | op-bench-and-diagnosis.md §3.2 fixed/streaming decomposition t = a + b·N |
| Whether knobs should be calibrated on op unit tests or e2e | op-bench-and-diagnosis.md §4 scale division-of-labor table |
| Deciding whether to invest in a certain optimization direction (read this first) | kernel-dev-and-optimize.md §2.0——first self-test GPU busy vs wall; if already GPU-bound and occupancy-limited, sub-5us GPU savings won't materialize as wall |
| Know a certain kernel is slow, but don't know which segment inside it is slow | kernel-dev-and-optimize.md §2.0 "kernel internal ablation ladder"——remove work category by category and read the difference to get a per-item cost table + floor value |
| Optimization topic selection / design review / retrospective on optimization direction | Three methodology sections:kernel-dev-and-optimize.md §2.5、graph-fusion.md §9、runtime-scheduling.md §9 (principles/applicability/pitfalls/verification only, no numbers) |
| GEMV optimization (the main battlefield for decode) | kernel-dev-and-optimize.md §2.1 |
| GEMM / prefill optimization | kernel-dev-and-optimize.md §2.2 |
| Attention optimization / want to know which path is currently taken | kernel-dev-and-optimize.md §2.3 (§2.3.8 is the routing quick reference) |
Modify prefill_flash_attn_tc(coop-tensor prefill attention) |
Mechanism see kernel-dev-and-optimize.md §2.3.4; for FATC switch default values and activation conditions see env-registry.md |
| LinearAttention(Qwen3.5 gated delta rule) | kernel-dev-and-optimize.md §2.4.2 / §2.4.3 |
| Operator fusion: why it didn't hit / output wrong after fusion | graph-fusion.md §8 troubleshooting checklist |
| New model structure needs added fusion | graph-fusion.md §1 (export-side declaration) + §4 (backend assembly) |
| Output differs run to run after fusion | graph-fusion.md §4.3 memory aliasing |
| CPU blocking / sync overhead per token in decode | runtime-scheduling.md |
| Encode Replay related (whether a new op should be integrated / why it was banned) | runtime-scheduling.md §7 |
| cmake build options / model export commands | build-and-test.md |
| Look up the default value and semantics of an env switch | env-registry.md |
Topic selection decision path
瓶颈在哪?
├─ 还不知道 / 只知道"比对手慢" → op-bench-and-diagnosis.md(造 op 单测 + 对手镜像,再做固定/流式分解)
├─ 每 token 固定开销大(dispatch 多 / CPU 段长) → 融合方法论 graph-fusion.md §9 / 调度方法论 runtime-scheduling.md §9
├─ 权重带宽未吃满(decode GEMV) → kernel 方法论 §2.5.3/§2.5.4
├─ attention 随 KV 变慢 → kernel 方法论 §2.5.4/§2.5.5/§2.5.8
├─ GPU→CPU 同步密集 → runtime-scheduling.md §9.2/§9.1
├─ 知道是哪个 kernel,不知道它内部哪段慢 → 消融阶梯 kernel-dev-and-optimize.md §2.0
└─ 不确定 → 先 profile(§2.0 GPU busy vs wall)
General principles at a glance (details see kernel-dev-and-optimize.md)
- shader is an embedded C++ string(
R"metal(...)metal";in*Shader.hpp), not a standalone.metalfile; common headers rely onstring concatenationfor sharing. After editing, just make. - Variants use
preprocessorMacros, not function constants. Adding a macro requires synchronously changing four places: shader#ifdef, pipeline cache key, macro dictionary, andonResize's grid/threadgroup. - The dispatcher must be understood first: one op often has multiple kernels; before extending, first decide which ones to support + make the other paths explicitly fall back.
- Apple GPU ≠ Android: M3/M4/M5 cannot be inferred from each other, let alone Vulkan/OpenCL. For device tiers see
architecture.name/isSupportTensorApi()。 - Correctness oracle before performance:fp32(
precision: high) bit-identical is the strongest evidence; fp16 greedy comparison is the next strongest.token-level consistency does not equal bit-level consistency. A wrong kernel can also "get faster"; any gains measured before the comparison passes are untrustworthy. - Fusion must check memory aliasing: when folding a predecessor into a successor, the predecessor's input may already have been reused by the allocator as the successor's output.
- A/B must alternate in pairs: thermal drift can create spurious gains of several times; absolute numbers from profile build are artifacts.
7b. zsh does not word-split unquoted parameter expansions。
run() { env $3 ./bench; }when passing"A=1 B=2",envtreats the whole string asoneassignment (A="1 B=2"), the second variableis silently lost—A/B looks like "no difference", but in fact the probe was never turned on. This caused a misjudgment once in this session. For multiple variables use${=3}to force word splitting, or directly write theA=1 B=2 ./benchprefix.For any "change has no effect" conclusion, first confirm the switch was actually passed in. - Locate the bottleneck segment before acting: GPU busy vs wall, kernel timing vs e2e, and bandwidth realization rate are three different measures; optimization must target the real segment, otherwise the kernel gets faster but e2e doesn't move.
8b. To attribute within a kernel, delete work category by category, don't guess forward: Temporarily remove one class of work with a switch (output is intentionally wrong), removing all the way down to only MMA/FMA, and read the adjacent differences to get the per-component cost table andfloor value. Thencompare the floor value against the opponent, not total time—opponent ≈ your floor ⇒ the gap is entirely in overlap efficiency (you should do batched pre-launch / raise the compute:memory ratio); opponent < your floor ⇒ the gap is in algorithm and workload, and changing overlap is wasted effort. See
kernel-dev-and-optimize.md§2.0. The probe must guard against DCE (removing a small piece but saving half a kernel = downstream was eliminated wholesale, not a discovery). 8c. Calibration scale is split by switch type:path switch(whether to go through kernel X) is judged by e2e—to prevent the false gain of "faster at kernel level / flat at e2e";internal parameters of a kernel already confirmed to be on the critical path(nsg / tile / batch) are the opposite—first calibrate with an op test in which that kernel occupies the full measured time, then verify with e2e that there is no regression. Otherwise you will miscalibrate a knob that e2e cannot resolve: one default value of decode attention's nsg was wrong for a long time precisely because it was calibrated on e2e—attention occupies only a small slice of e2e in that scenario, and e2e simply cannot resolve this knob; switching to an op test immediately separates the winner from the loser. 8d. The opponent baseline must first do two things: fixed/streaming decomposition + confirm the lazy graph did not change semantics. (a) When the gap varies with scale, first fit two clean scale points witht = a + b·Nto separateper-iteration fixed overhead a andstreaming rate b—this once revealed that MNN decode attention's b already beat the opponent baseline and the entire gap was in a, so all inner-loop ILP directions could be skipped. (b) In a lazy graph framework,accumulating N steps into one graph and then evaluating changes in-place donation semantics: an in-place slice update degrades into a full copy, and based on this I misjudged the opponent baseline's KV append as O(ctx) (it is actually O(1)). To measure "per-step" cost you mustevaluate each step。 - Layout is a global invariant: a data layout change affects all readers and writers and must land atomically; intermediate states must not be runnable. 9b. Both multipliers of a simdgroup MMA must be half, and only the accumulator uses float. Promoting either multiplier to float drops you to the float MMA rate—fused attention's PV became noticeably slower because of this (see kernel methodology §2.3.9).
- Falsification is not documented; only reusable criteria are kept: the vast majority of negative/neutral experiments are done once they finish (writing them in the commit / PR is enough). Only when it can be distilled into one sentence ofa criterion that still holds across shapesdoes it enter the skill, and it is written as the "applicable boundary" of the corresponding technique—"this class of technique does not hold under condition Z, because Y". Percentages, paired details, reversals, and calibration narratives never enter the skill (see "recording rules" above).
- A name must equal the real physical quantity: when changing any backend operator / kernel code,first check whether the variables, members, macros, and env names along the way are exactly the quantity they actually hold(not just names you add yourself). Misleading names won't raise errors—compilation, differential testing, and unit tests all pass, but the tiering, thresholds, and gating built on top of them will all be wrong together. The most expensive case was writing the tiering independent variable as "head count", while the kernel's actual parallelism is the number of dispatched threadgroups (
batch*head_num/qh), and the default value reverses when you switch models. If a name doesn't match the index arithmetic, rename it (sync across three places: macro name / cache key / macro dictionary). Seekernel-dev-and-optimize.md§1.3 "Variable / field / macro names must equal the real physical quantity".
Related Skills
skills/general-debug/— entry point for correctness bugs / regression diagnosis (volumes organized by symptom); the two volumes commonly used for Metal aregeneral-debug/memory-aliasing.md(§1 memory aliasing / lifetime, including aliasing races introduced by fusion) andgeneral-debug/gpu-oob.md(§6 shader out-of-bounds / command buffer failures)skills/opencl-optimize/、skills/vulkan-optimize/、skills/cpu/— other backendsskills/support-new-llm/— complete workflow for adding a new LLM modelskills/test-ci/— unit tests / regression tests