Follow goal-first: the mainline is the algorithm conclusion; performance optimization serves to reduce waiting. Deduplication, batching, grouping, fused kernels, and pipelining are on-demand means, not a graduation checklist every project must complete; stop once research needs are met, and do not extend the task just because optimization headroom remains.
0. Boundaries and success criteria
- Success criteria: known obvious waste is addressed, correctness/reproducibility passes, and actual waiting supports the current research pace; no uniform threshold for utilization, power, or speedup multiple is set.
- By default only the execution path is changed; model structure, dtype, samples, and scoring protocol are kept, and floating-point differences are judged against a predetermined tolerance.
- Batch/padding/calibration/algorithm changes are compared separately; replacing fake quantization with real quantization or integer kernels is a change in numerical and deployment semantics and must not be passed off as a same-basis speedup.
- Record jointly within the same phase: effective throughput/host time, device busyness, power, device memory, and allocated/reserved peaks; for individual operators, use Events separately. Preserve metric definitions, windows, and device mappings together.
- Use remaining run count × time saveable per run to roughly estimate whether the investment in development, validation, maintenance, and attention is worthwhile; loading and per-configuration preparation/compilation also count. Give a short budget; deep engine/tool research and development gets a separate goal.
1. Break down time: do not guess
- For a new execution path, first do a quick scan for useless computation, memory copies, batching, serialization, synchronization/transfers, and consecutive small operations, then go deeper based on measurement; cover the inspection perspectives without running an experiment for each item. This also applies to batch pre-research; existing ready evidence can be reused.
- Use a small fixed input to separate loading, preparation, warmup, online loop, and scoring time; find actionable hotspots first.
- For individual operators, first reuse the project's validated device timer: specify stream, dependencies, warmup, call count, and statistic. Wrap a batch of calls in a pair of Events at the start and end, then divide by the count; 100 is a usable starting point, not a fixed contract for every task.
- Measure several rounds and keep the raw values. A normal Event window may include device execution and queue gaps and is not pure host cost; when necessary, add a dummy or graph-replay control and report each basis separately, without substituting asynchronous host dispatch time for device execution.
- Use a profiler only when targeted timing still does not localize the issue, focusing on host gaps, dense small kernels, and memcpy; do not collect heavy counters by default.
- After confirming the physical device, process, and other jobs, sample utilization/power/memory at low frequency alongside a representative scoring window; do not merge loading, compilation, and the online phase into one average, and do not insert synchronous monitoring in the hot loop.
- Classify conclusions as "avoidable waste / reasonably constrained under the current basis / insufficient evidence" with supporting rationale; localizing to an actionable hotspot is enough, and full busyness or power is not a completion condition.
- Large-matrix stress tests can only verify sustained load capability; low AICore does not mean the whole card is idle, and long inputs do not guarantee higher effective throughput. Only when this would change the judgment, follow
references/device-observation.mdto do a bounded-scale comparison.
2. Remove first: unneeded tensors and computation
| Item | Default handling |
|---|---|
| Gradient/backward graph for pure evaluation | model.eval() + torch.inference_mode(); training and calibration that need gradients keep the computation graph |
| KV cache for pure scoring | use_cache=False; generate/decode only then enabled |
| Full-vocabulary logits | First select the needed positions, then apply log_softmax;logits_to_keep |
| Hidden states/attention outputs | Do not request output_hidden_states/attentions |
| Excessive padding | Truncate to actual length; dynamic padding/bucketing |
| Offline preparation | Calibrate, quantize, tokenize, and compile once, then cache to disk |
| Unused outputs/intermediates | Do not retain; do not use output objects to hold cache long-term |
| Training memory | Trade memory for activation checkpointing; not needed for eval |
- For pure scoring, disable KV and verify logits match bitwise or within tolerance; the generation path cannot disable it.
- Completion criteria: peak allocation decreases, outputs align; "computing less" is not conflated into accuracy conclusions.
3. Deduplication: keep only one copy of weights and persistent memory
- During pure evaluation, release weight copies that are truly no longer needed; if necessary, under no-grad
weight.copy_(converted); training master weights, optimizer states, or original weights that need to be restored later must not be deleted arbitrarily. - Packed/transposed/grouped weights share storage with the original Parameter; do not
stackand then copy the whole thing. - For persistent packed weights that need to be modified in place later, create and bind slices under
torch.inference_mode(False); an outer inference context may turn it into an inference tensor, and afterward onlyno_gradcannot lift the write restriction. - Deduplicate by
(device, tensor.untyped_storage().data_ptr()), then add storage'snbytes(); a slice's tensordata_ptr()can differ, so this cannot be used to conclude there are multiple weight copies. - Distinguish necessary residency, redundant live tensors, allocated, and reserved; reserved minus allocated includes reusable cache and fragmentation and cannot be directly counted as "dead memory".
empty_cache()Use only as needed at stage boundaries. - Large temporary tensors, once used,
del; check whether closures, hooks, caches, or the profiler still reference them. - Completion criteria: one copy of necessary weights, reduced redundant residency; use the headroom to increase effective batch or lower peak, and do not judge good or bad directly by memory usage level.
4. Batch: find the throughput knee first
- batch 1 is not the capacity limit. Try a small number of candidates near the current batch and record tokens/s and peak; stop once the gain is sufficient, without chasing maximum capacity or a full curve.
- Sort/bucket variable-length inputs by length to reduce padding; use varlen operators when necessary.
- Check whether batching changes semantics: dynamic scaling/calibration, padding statistics, reduction order, RNG, sample grouping.
- To preserve the original protocol, compute per-sample statistics or fix calibration; otherwise record the batch change separately as a protocol change.
- Completion criteria: a small throughput comparison + selected batch + explanation of the impact on accuracy/calibration.
5. Serial → batched/grouped/fused
- Symptom: Python
forcalls small matrix multiplies one by one per expert/head/segment; each item incurs one launch, and the device is underutilized. - Approach: after sorting/grouping, use framework or vendor grouped GEMM; merge shared input projections (QKV, gate_up). Check the actual device dispatch/short trace; an interface called grouped or fused may still expand item by item internally.
- Fixed shape + mask can replace ragged dynamic indexing; when reduce/scatter/index fall back, zero-padding may be faster, but maximum routing capacity will amplify buffers for all groups. Measure actual time and peak memory; do not uniformly switch to dense zero-padding.
- Consider multiple streams only for independent branches; parallelism of small operators will be eaten by launch overhead.
- Completion criteria: the number of small kernels in a single module decreases, end-to-end speedup with the same input, and numerical consistency.
6. Reduce synchronization blocking the hot path
During asynchronous execution, the following operations may make the CPU wait for the device; whether they block depends on the actual backend and timeline:
| Type | Example | Handling |
|---|---|---|
| Value retrieval | .item()、.cpu()、.tolist()、.numpy() |
Move out of the loop, batch/once after completion |
| Boolean control flow | if tensor、bool(tensor) |
Change to tensor operations; only tensor > 0 generating the tensor itself does not require host value retrieval |
| Dynamic shape | nonzero()、unique(), single-parameter torch.where(mask) |
Fixed shape + three-parameter where / sort grouping |
| Explicit synchronization | synchronize()、empty_cache() |
Only at stage boundaries |
| Implicit | print(tensor)、assert tensor..., per-item logging |
Remove or defer |
| Transfer | In-loop .to(device)、.cpu() |
Prefetch, pin memory, non_blocking |
- Keep dependencies/synchronization required for correctness; once per layer
max().item()May be acceptable; judge by the timeline whether it causes major gaps. - Put accumulable metrics on the device side, reduce value retrieval frequency for necessary logs; do not retrieve loss/acc step by step and create waits.
- Completion criteria: major synchronization points improved, results unchanged, end-to-end faster; do not aim to eliminate all host gaps.
7. Operators, backends, compilation
- Use SDPA/flash attention, fused optimizer, vendor libraries, and
torch.compileas ready-made candidates; it is not required to try them all before writing a fused kernel. - Measure on the target device; CUDA documentation does not represent NPU, and running on CPU does not mean the device is fast, and vice versa.
- Check fallback: one op falling back to CPU may be slower than a zero-padded/dense implementation.
- dtype/layout: BF16/FP16, channels-last, contiguous; avoid transpose + contiguous at every step.
- CUDA graphs / graph mode only with static shapes and clear benefits.
- High-frequency consecutive PyTorch fragments (combinations of elementwise, broadcast, and reduction) can generate fused kernels; preserve efficient library boundaries, and compare the original chain, the fused kernel, and the complete fragment reconnected to the original call chain.
- Completion criteria: dominant operators are handled, end-to-end faster, numerics within tolerance.
Small-kernel timing: first distinguish the window
- When host submission cannot keep up, a plain Event will include idle waits between kernels; the device work of small kernels cannot be directly inferred from this.
- The dummy method enqueues sufficiently long work before start on the same stream, giving the host time to submit the target batch; actual measurement must cover the margin, and note the possible impact of dummy on cache/frequency; do not copy a fixed number of matmuls.
- A relatively large plain/dummy value suggests submission impact, but it is not a universal bottleneck threshold, nor does it prove device optimization is worthless; neither dummy nor graph equals the end-to-end benefit of a normal call.
- Professional measurement and available timers read from an independent
kernel-knowledgepipeline; for localization seekernel-worker/references/kernel-knowledge.md. When the library is missing, use the project's trusted timer and declare the measurement basis; do not install automatically; do not duplicate its implementation.
Agent generates and iterates kernels by model/shape
- Distinguish changing offline algorithms from reusable online primitives. A change in scale, rotation matrix, or weights does not mean the quantization/dequantization kernels must also be rewritten.
- Collect shape, dtype, stride, reduction axes, mask/grouping, and call frequency from actual runs of the target model; determine fusion segments around measured hot spots.
- Split kernels by computation form, then generate or tune for the model's shape families; allow different implementations and tiles, and do not insist that one interface or one set of optimal parameters covers all models.
- Reuse formulas, correct baselines, candidate source code, and optimization experience across models; after adaptation, validate on the new model. An existing implementation of the same form can be used directly as a candidate without rewriting from scratch.
- When the hot spot is independent and the reference/replay/interface is clear, follow
kernel-workerto fill in the task brief and explicitly invoke a host sub-Agent to develop in an independent workspace; the main session handles semantics, resource budget, integration, and full-model acceptance. - Correct baseline → a few relevant directions, trying a few candidates each → verify each time → same-card A/B/B/A → keep or revert. Change only one class of factor at a time; stop once the gain is clear, and do not keep climbing for small gains.
- Fix the reference and samples; compare values one by one for rounding boundaries, tail blocks, statistical grouping, and real layer inputs. For training, additionally verify backward/STE.
- Prefer reusing optional expertise in professional implementation, measurement, and platforms
kernel-knowledge; external engines such as Atrex are invoked only when the backend/evaluator and budget are ready. For discovery and missing-handling seekernel-worker/references/kernel-knowledge.md, and for research task trade-offs seereferences/kernel-decision.md; do not mix platform building into daily work.
8. Multi-GPU: distinguish placement from scheduling
- Explicitly set visible devices;
device_maptiered residency mainly shares storage and does not automatically enable tensor parallelism or pipelining. Investigate two GPUs taking turns being busy and two GPUs being jointly idle separately; do not conclude that communication is the main cause merely because multiple GPUs are used. - When the model must span GPUs, different complete batches can overlap across the preceding and following stages; prefer keeping existing bucketing/calibration and do not split into small batches first. Validate threads with bounded tests first, and consider a separate process only if there is a real dispatch bottleneck.
- Handle stage devices, Event dependencies, source tensor lifetimes, and thread contexts explicitly; first validate stage serialization against the original model, then compare serial versus overlapped with the same split. For details see
references/multi-device-pipeline.md。 - DataLoader: workers, pin_memory, non_blocking, prefetch; CPU preprocessing must not block the device.
- When model replicas are allowed, eval can be sharded by independent samples; for training use appropriate DDP/FSDP etc. Choose the parallel method based on capacity, dependencies, and goals; trying all of them is not required; follow the directions already constrained by this task.
- For minute-level probes, use Bash + Python directly and just save logs; only follow project conventions for managed hosting when there is a genuine multi-hour lifecycle issue, and do not create services for small experiments.
- Completion criteria: improvement in effective throughput and end-to-end time under the same measurement basis, with utilization/power explaining device behavior; not "the process is still alive" or higher hardware readings.
9. Verification and reporting
- Use the same input and the same warmup, and confirm gains with a small number of interleaved retests (e.g., A/B/B/A); do not add repeats once stable. For hour-level full runs, use one same-input comparison and state that there were no repeats.
- Check numerical values bitwise/by tolerance as agreed, and pair task responses item by item; the same total score does not prove per-question consistency. Mark batch/calibration changes separately; they cannot be counted as lossless.
- Record the version, device, batch, dtype, peak, and command; retain failed attempts as well.
- Do not extrapolate single-layer/single-module multipliers to the whole model; compare against the real model with the same inputs.
- Benefits from scattered-branch GMM, fused kernels, and pipelining cannot be multiplied together; after connecting them at a common entry point, run bounded acceptance with identical inputs and adopt only verified combinations.
- For whole-model acceptance, use the host clock to record the complete flow and wait on the relevant devices at boundaries; report loading, preparation, cold compilation, and steady state separately. Device microbenchmarks do not replace whole-model acceptance.
- One-off JIT cannot be mixed into steady-state comparisons, nor silently removed from cold-start end-to-end results: report cold start and steady state after explicit warmup separately. Only report the first sample separately when it is confirmed to bear compilation and an exclusion rule is declared in advance; it cannot be discarded just because it is slow, and actual call volume must still cover the compilation cost.
- Completion criterion: others can reproduce it from the record, and know which part is a protocol change.
10. Stop efficiency anxiety: freeze the usable path
- Correct, with trustworthy gains, and waiting is sufficient: fix the version and configuration and return to algorithm research; remaining optimization opportunities and low utilization are not unfinished items. It is also acceptable to close out earlier with the reference path, without first writing a kernel or building a pipeline.
- When the budget is exhausted or the cost of the next step exceeds the foreseeable savings in waiting: keep the current reliable implementation and stop this round; do not automatically switch hotspots, switch backends, or build another engine to continue spending time.
- Bounded integration of existing results counts as delivery; correctness issues in integration must be fixed, but do not casually start a new round of performance R&D.
- Reopen only when a new model/new path is abnormally slow, OOMs/regresses, waiting again blocks specific research, or there is a clear low-cost gain; "monitoring is not full yet" does not trigger a reopen by itself.
- Professional inference engines also do not guarantee that all workloads run at full capacity. Algorithm research also weighs ease of modification, numerical transparency, and maintenance cost; if performance work becomes the goal, agree on it separately and do not upgrade the current task by default.
Anti-patterns
- Changing without measuring; changing five places at once; treating doing the whole checklist or maxing out monitoring as delivery, crowding out algorithm research.
- Treating batch 1 as the capacity limit; treating reserved as a leak; judging code quality solely by utilization/power/memory level.
- Keeping multiple weight copies during conversion/quantization; adding abstractions for "later"; forcing a one-size-fits-all kernel to cover all models.
- Rejecting a shared online kernel because the offline algorithm is changing; using FLOP to guess latency; changing rounding semantics without a reference implementation; rewriting vendor matmul without comparison.
References
references/device-observation.md: lightweight hardware observation and judgment;references/checklist.md: on-demand code troubleshooting, timing, and memory templates.references/multi-device-pipeline.md: cross-full-batch overlap, context and dependencies, numerical/timeline/host acceptance.references/case-moe-eval.md: staged case studies on deduplication, batch, native GMM, pipelining, and input sizes; not a required route or universal multipliers.references/kernel-decision.md: select candidates by model shape, hand off professional capabilities, and accept locally/whole-model. Experience-evolving type: general performance methods flow back to the main body, specialized measurement/implementation goes back to the domain maintenance source, and summaries are synchronized; raw evidence stays with the project.