external-cannbot-model-model-infer-runtime-debug

NPU Inference Runtime Error Diagnosis and Repair

Core principle:Locate first, then fix. Does not cover precision issues; for precision tuning, see model-infer-precision-debug. —— Gradually narrow down the fault scope through bisection to avoid wasting time in the wrong direction.

Diagnostic Path Overview

问题发生
  │
  ├── 有明确错误信息 → 通用诊断流程
  │     ├── aicore timeout (507014) → 二分法定位
  │     │     ├── 定位到阶段 → 层 → 模块 → 算子
  │     │     └── 查表「常见算子约束」→ 修复策略 → 验证
  │     ├── OOM → 检查参数量 / batch_size / seq_len / 中间 buffer
  │     ├── Shape 不匹配 → 检查 TP/EP 切分维度 / Prefill vs Decode 分支
  │     └── HCCL timeout → 检查各 rank 代码路径一致性 / 通信组创建
  │
  └── 无明确错误 → npu-smi 状态检查
       ├── 部分 rank 缺失 / HBM 不均 → 多卡部署诊断
       │     ├── 权重加载 crash → TP 切分越界 / vocab pad
       │     ├── Config 字段缺失 → Config 兼容性
       │     └── 静默失败 → 逐 rank 检查日志
       ├── 有进程 + HBM 不变 + CPU 100% → 推理卡住诊断 - 模型构造阶段
       ├── 有进程 + HBM 不变 + CPU 0% → 推理卡住诊断
       │     ├── 检查残留进程
       │     ├── eager 也卡 → sync+print 定位（通用流程第二步）
       │     └── 仅 graph 卡 → 比较 eager/graph 代码路径差异
       └── 0 进程 → 查 rank 0 日志 → 通用诊断流程

npu-smi Status Check

When there is no clear error message, determine the problem type through device status:

npu-smi info

npu-smi Behavior	Meaning	Diagnostic Path
First N processes exist, later ones are missing, HBM uneven	High-order ranks crash during weight loading	→ Multi-card deployment diagnosis
8 processes, HBM even, AICore all 0%	All ranks are stuck waiting for communication	→ Inference hang diagnosis
Processes exist, HBM unchanged at ~3GB, CPU 100%+	CPU-intensive operations (model construction/weight initialization)	→ Inference hang diagnosis - model construction stage
Processes exist, HBM unchanged at ~3GB, CPU 0%	Deadlock/communication wait	→ Inference hang diagnosis
8 processes, HBM continuously growing	Weight loading in progress	Wait
0 processes	All crashed	Check rank 0 logs → general diagnostic process

General Diagnostic Process

Applicable to scenarios with clear error messages (aicore timeout, HCCL error, OOM, shape mismatch, etc.). Proceed linearly through classification → localization → table lookup → repair → verification.

Step 1: Error Classification

After obtaining the error message, first determine which category it belongs to. Different categories have completely different root causes and troubleshooting paths; misclassification wastes a lot of time.

A. aicore timeout (error code 507014)

Characteristics：AclrtSynchronizeDeviceWithTimeout、error code is 507014、aicore timeout、fftsplus aivector error、Kernel task happen error, retCode=0x25

Meaning: A certain NPU operator times out and does not return during execution on the device. This is the most common and also the most difficult error to troubleshoot, because the location where the error is reported (the synchronize call site) is usually not the location where the error occurred (a specific operator).

Common Root Causes：

Operator input parameters violate hardware constraints (e.g., MC2 requires ep_world_size >= 16, see "Common Operator Constraints")
Operator shape exceeds hardware limits (e.g., the M/N/K dimensions of a single matmul exceed limits)
Deadlock: some ranks take different communication paths, causing collective communication to never be fully synchronized
Out-of-bounds memory access: slot_mapping/block_table indices exceed the allocated KV cache range

B. HCCL Communication Errors

Characteristics：HCCL_CONNECT_TIMEOUT、HCCL error、AllReduce/AllToAll timeout、HCCL_EXEC_TIMEOUT

Meaning: Distributed communication operations time out or fail.

Common Root Causes：

Ranks enter communication operations in inconsistent order/number of times (code branches cause some ranks to skip a certain communication)
Incorrect parameters when creating the communication group (group_stride, group_num do not match world_size)
Network issues (incorrect HCCL_IF_IP configuration during cross-node communication)
Some ranks have already crashed while other ranks are still waiting for them to participate in communication

C. OOM (Out of Memory)

Symptoms：NPU out of memory、Tried to allocate X GiB、ENOMEM

Meaning: Insufficient NPU HBM.

Common root causes：

The sum of model parameters/KV cache/activation exceeds single-card memory
Intermediate tensors are not released in time (common in MoE EP all-to-all intermediate buffers)
batch_size or seq_len exceeds expectations

D. Shape mismatch

Symptoms：RuntimeError: shape mismatch、expected size X but got Y、mat1 and mat2 shapes cannot be multiplied

Meaning: The tensor dimensions are incorrect.

Common root causes：

After TP partitioning, the dimensions are not correctly divided by tp_size
After EP partitioning, the number of experts per card is calculated incorrectly
The Prefill/Decode branch is passed a tensor with the wrong shape

E. Operator constraint violation

Symptoms: It does not directly report a constraint error; it usually manifests as aicore timeout or SIGABRT. It needs to be located by bisection and then confirmed by consulting the operator documentation.

Meaning: NPU custom operators have implicit constraints on input arguments (data type, dimension range, hardware platform limitations), and behavior is undefined when these are violated.

Step 2: Bisection localization (sync+barrier checkpoint method)

Core method. When the error occurs in torch.npu.synchronize() , the error message only tells you "something on the device went wrong"; it does not tell you which specific operator. Narrow down the range by inserting checkpoints step by step.

2.1 Locate the phase (Prefill vs Decode)

In the runner's model_inference method, insert a synchronization check between the prefill and decode calls:

# 在 prefill 之后、decode 之前
torch.npu.synchronize()
logging.info("prefill passed")

Key pitfall: NPU operations are asynchronous.model.prefill() returning does not mean prefill is complete -- only torch.npu.synchronize() returning means all submitted operations are complete. So the error location (the synchronize call site) is not necessarily the location where the error occurred (some specific operator). Make sure the checkpoints cover all possible asynchronous operations.

2.2 Locate the layer

After confirming the error phase (e.g., Decode), insert a checkpoint between each decoder layer:

for i, layer in enumerate(self.layers):
    hidden_states = layer(hidden_states, ...)
    torch.npu.synchronize()
    logging.info(f"layer {i} passed")

If Layer 0 times out -- the problem is inside the first layer. If Layer 2 times out -- the problem is in Layer 2 or its submodules.

2.3 Locate the module

Enter the failing layer and insert checkpoints between the submodules:

# 在 DecoderLayer.forward 中
hidden_states = self.input_layernorm(hidden_states)
torch.npu.synchronize(); logging.info("  norm passed")

hidden_states = self.self_attn(hidden_states, ...)
torch.npu.synchronize(); logging.info("  attention passed")

hidden_states = self.mlp(hidden_states)  # MoE
torch.npu.synchronize(); logging.info("  MoE passed")

2.4 Locate the operator

Enter the failing module and insert a checkpoint between each NPU operator call. At this point you can usually locate the specific operator and its input arguments, and determine the root cause with reference to "Common operator constraints" below.

2.5 Notes

First use torch.npu.synchronize() for single-card localization, to confirm which operator is failing. Do not add dist.barrier()right from the start -- a barrier will cause other ranks to hang forever when one rank has already crashed
When you suspect multi-rank desynchronization, use dist.barrier() or torch.distributed.monitored_barrier() to confirm whether the ranks reach the same position.monitored_barrier After a timeout, it reports which rank did not arrive
Check the logs of all ranks, to confirm whether the checkpoints reached by each rank are consistent (inconsistency indicates a conditional branch difference -> communication deadlock)
Do not insert too many checkpoints at once -- each sync has overhead, and a large number of syncs may change the timing. Start coarse-grained, then fine-grained
Preserve logs: Save the checkpoint output located in each round for subsequent analysis

Step 3: Quick reference for common NPU operator constraints

The following constraints come from actual debugging experience in this repository. After locating a specific operator, you should check them against the signature of the currently called API and the official documentation, and do not apply rules across operators. The typical symptom of violating a constraint is aicore timeout.

MoE-related operators

Operator	Constraint	Violation symptom	Solution
`npu_moe_distribute_dispatch_v2` (MC2)	Atlas A2: `ep_world_size` must be {16,32,64,128,256} (fullmesh) or {16,32,64} (hierarchy)	aicore timeout, hang during the Decode phase	When EP<16, fall back to the double_routing path (npu_moe_init_routing_v2 + manual all_to_all)
`npu_moe_distribute_combine_v2` (MC2)	Same as above	Same as above	Same as above
`npu_moe_init_routing_v2`	There is no minimum EP limit, but `moe_chunk_max_len` being 0 may cause an empty tensor	SIGABRT or shape error	Set a reasonable moe_chunk_max_len (e.g., 1024)
`npu_moe_gating_top_k`	`k` cannot exceed the total number of experts	Silent incorrect output or crash	Check whether the reduced model's moe_topk has been adjusted

FA (Flash Attention) operators

Constraint	Description
`head_num` must match the actual Q heads	In MLA models, head_num = num_attention_heads, not kv_heads
Prefill `sparse_mode=3` must be used with `atten_mask`	mask shape must be `[1, 1, max_s, max_s]`, and cannot be omitted
`input_layout` varies by API and phase	`npu_fusion_attention` uses BSH/SBH/BSND/BNSD/TND;`npu_fused_infer_attention_score` uses composite layouts such as TND_NTD and BSND_NBSD, subject to the actual API signature
In Decode PA mode, `block_table`	shape `[batch, max_blocks]`, and the value cannot exceed the number of blocks allocated to the cache
NZ format cache	Requires `torch_npu.npu_format_cast` conversion, and hidden_dim must be an integer multiple of 16

actual_seq_lengths related parameters are subject to "the signature of the currently called operator and the official API documentation", and rules are not applied across operators:

torch_npu.npu_fused_infer_attention_score / torch_npu.npu_fusion_attention：actual_seq_lengths* is List[int](the documentation specifies int64 semantics)

torch_npu.npu_kv_quant_sparse_flash_attention：actual_seq_lengths_query/kv is Tensor(the documentation specifies int32)

Communication operators

Constraint	Description
`all_to_all_single` split sizes	The sums of input_splits and output_splits must equal the 0th dimension of input and output, respectively
All ranks in the communication group must arrive at the same time	Any difference in conditional branches may cause some ranks to skip communication -> deadlock
HCCL group creation order	All ranks must create the same communication groups in the same order

Step 4: Fix strategy patterns

After locating the root cause, choose an appropriate fix strategy.

Pattern 1: Operator Fallback

When a high-performance operator is unavailable under the current hardware/configuration, fall back to an alternative operator with equivalent functionality but fewer restrictions.

# 示例：MC2 MoE 算子 EP<16 时回退到 double_routing
def forward(self, hidden_states, is_prefill):
    topk_indices, topk_weights, _ = self.router(hidden_states)
    if is_prefill:
        return self.moe_infer_double_routing(hidden_states, topk_indices, topk_weights)
    else:
        if self.moe_ep_size < 16:
            # MC2 requires ep_world_size >= 16 on A2; fall back to double_routing
            return self.moe_infer_double_routing(hidden_states, topk_indices, topk_weights)
        return self.moe_infer_dispatch_combine(hidden_states, topk_indices, topk_weights)

Applicable scenarios: hardware constraints, EP/TP size does not meet operator requirements Note: fallback usually incurs a performance loss and must be recorded in the optimization report

Mode 2: Parameter Fix

Incorrect operator input parameter computation (e.g., FA head_num used KV heads instead of Q heads, dimensions not divided by tp_size after TP partitioning). After locating the specific operator, check the differences between its input parameters and the operator documentation/reference model one by one.

Applicable scenarios: shape mismatch, dtype error, dimension computation error

Mode 3: Path Unification

Some ranks take different code paths, causing communication deadlock.

# 错误：不同 rank 可能在不同 step 进入 prefill/decode
if is_prefill:
    self.all_reduce(...)  # rank 0 执行
# rank 1 已进入 decode，不执行 all_reduce → 死锁

# 修复：确保所有 rank 同步状态后再分支
dist.barrier()
is_prefill_tensor = torch.tensor([int(is_prefill)], device="npu")
dist.broadcast(is_prefill_tensor, src=0)
is_prefill = bool(is_prefill_tensor.item())

Applicable scenarios: multi-rank deadlock, HCCL timeout

Mode 4: Configuration Downgrade

When the configuration combination is incompatible, adjust YAML parameters (e.g., lower moe_chunk_max_len, reduce batch_size）。

Applicable scenarios: OOM, abnormal performance

Step 5: Verification Checklist

After fixing, verify according to the following checklist:

Single-step verification: add sync+barrier before and after the error location to confirm the timeout no longer occurs
Full-process verification: remove all debugging checkpoints and run the complete warmup + inference
All-rank verification: check the logs of all ranks and confirm all succeeded (grep "model run success" log_*.log）
Output consistency: the final outputs of all ranks should be consistent (for DP mode, ranks within the same DP group should have consistent outputs)
Performance recording: record the Prefill/Decode latency after the fix and compare with before the fix
Rollback documentation: if operator rollback was used, record the performance impact and future conditions for recovery

Scenario-Specific Diagnosis

The following scenarios have independent diagnostic paths and do not follow the general process. Determine via the entry npu-smi table and jump directly.

Multi-card Deployment Diagnosis

Multi-card deployment introduces a class of problems that do not exist on a single card: weight loading partitioning, vocab alignment, partial rank crashes, config compatibility. Common characteristics:Some ranks succeed while some ranks fail, and the error occurs during the weight loading stage rather than the forward stage.

Weight Loading TP Partition Out-of-Bounds

Typical error：

RuntimeError: start (105984) + length (35328) exceeds dimension size (131125)

or:

ValueError: 131125 is not divisible by 8

Root cause: ColumnParallelLinear / VocabParallelEmbedding split the weight into tp_size parts along the output dimension. If the actual dimension of the checkpoint is not equal to the dimension of the model parameters, or cannot be evenly divided by tp_size, the narrow() operation of the higher ranks will go out of bounds.

Common scenarios：

Scenario	Description	Fix
Multimodal model text inference	`embed_tokens` uses full vocab (282624) but `lm_head` uses text+special (131125), and the two dimensions differ	lm_head uses `text_vocab_plus_multimodal_special_token_size`
vocab not evenly divisible by tp_size	131125 % 8 ≠ 0	Pad up to the nearest multiple of tp_size:`padded = ((raw + tp - 1) // tp) * tp`
padded parameters vs original checkpoint	Model parameters are 131128 (padded) but the checkpoint weight is only 131125	Pad the checkpoint weight first when loading, then pass it to weight_loader

Fix template (vocab pad + weight pad)：

# 1. 创建 lm_head 时 pad vocab size
lm_head_raw = getattr(config, "text_vocab_plus_multimodal_special_token_size", config.vocab_size)
self.lm_head_vocab_size = ((lm_head_raw + tp_size - 1) // tp_size) * tp_size

# 2. 加载权重时 pad checkpoint tensor
if "lm_head" in name and loaded_weight.shape[0] < self.lm_head_vocab_size:
    padded = torch.zeros(self.lm_head_vocab_size, *loaded_weight.shape[1:],
                         dtype=loaded_weight.dtype, device=loaded_weight.device)
    padded[:loaded_weight.shape[0]] = loaded_weight
    loaded_weight = padded

Troubleshooting tips: If only high ranks (rank 3-7) crash while low ranks (0-2) are normal, it is almost certainly a split out-of-bounds — because low ranks' start_idx is still within the valid range.

Config Compatibility Issues

When loading from the original HuggingFace model, config.json may lack fields the code expects, or the field format may differ.

Common Issues：

Issue	Symptom	Fix
Missing `rope_parameters`	`TypeError: 'NoneType' object is not subscriptable` at RotaryEmbedding init	Config `__init__` when `rope_parameters=None` construct default:`{"rope_type": "default", "rope_theta": self.rope_theta}`
`vocab_size` contains multimodal tokens	Embedding too large / lm_head dimension mismatch	Distinguish `vocab_size`(used for embedding) and `text_vocab_plus_multimodal_special_token_size`(used for lm_head)
`num_hidden_layers` ambiguity	Under the Dual-sublayer architecture, HF may store 28 (number of attention layers) but there are only 14 physical layers	Override with a property:`@property num_hidden_layers -> return self.num_layers`
`auto_map` points to the original model code	On load, try to import the original modeling file	Pass a custom config class to `from_pretrained`

Best Practices: At the end of the config class's __init__ , add defensive defaults for missing fields. For parameters that differ across models (such as rope_theta), read from config.json rather than hardcoding default values, and raise an error if missing.

Some Ranks Fail Silently

Sometimes a rank crashes but the error is swallowed, and the surviving ranks hang on the next dist.barrier() or collective communication, which looks like "inference is stuck".

Diagnostic Flow：

# 1. npu-smi 确认进程数
npu-smi info | grep "Process id"

# 2. 逐 rank 检查日志末尾
for i in $(seq 0 7); do
    echo "=== Rank $i ===" 
    tail -5 logs/log_${i}.log
done

# 3. 检查是否所有 rank 都到达了同一个里程碑
grep "model run success" logs/log_*.log
# 或
grep "Loading weights took" logs/log_*.log

Multi-Card Weight Loading Checklist

Before starting multi-card inference, confirm item by item:

embed_tokens dimension: In the checkpoint, embed_tokens.weight.shape[0] matches config.vocab_size
lm_head dimension: In the checkpoint, lm_head.weight.shape[0] may be smaller than embed_tokens (common in multimodal models)
vocab divisible by tp_size: The output_size of both lm_head and embed_tokens is divisible by their respective tp_size (pad if not)
ignore rule coverage：_keys_to_ignore_on_load_unexpected includes all unnecessary weight prefixes (ngram, visual, audio, mtp, etc.)
config fields complete：rope_parameters、num_layers、zero_expert_num and other fields required by the code are present in config.json or have defaults
Model path absolute path: In the executor framework, relative paths may resolve differently under different ranks' cwd

Inference Stuck Diagnosis

Inference being stuck is different from a crash — the process is still running but produces no results. You need to distinguish between truly stuck and falsely stuck. First determine the state via the entry npu-smi table, then troubleshoot along the following paths.

Reproduce in eager mode

卡住 → 换 eager YAML 跑同样模型
  ├── eager 也卡 → bug 在模型代码 → 用 sync+print 定位
  ├── eager 正常 → bug 在图模式适配
  │     ├── 看报错：ERR03007 / graph break → 找不兼容的算子
  │     └── 无报错但慢/卡 → 比较 eager 和 graph 模式代码路径差异
  └── 两者都正常但上次卡了 → 检查是否有残留进程占 NPU

Stuck during model construction phase (CPU-intensive)

Symptom: Logs stop after HCCL init, HBM does not grow, but CPU is 100%+

Common Root Causes：

Cause	Mechanism	Fix
`nn.Embedding` large vocab initialization	For large vocab, `nn.Embedding` uses `normal_()` to fill random values; multiple embeddings accumulated can take several minutes	Use `no_init_weights()` Context wrapping, skip parameter initialization that will be overwritten by checkpoint
`VocabParallelEmbedding` Transient memory	Inherit `nn.Embedding`，`super().__init__(full_vocab, dim)` First create the full vocab tensor, then replace it with per-partition tensors; transient CPU memory far exceeds final usage	Use `nn.Module` + `torch.empty(per_partition_size)` Create per-partition weights directly to avoid transient full-vocab allocation
`post_init()` Iterate over all parameters	HuggingFace `PreTrainedModel.post_init()` Call `_init_weights` to initialize each parameter; for models with many parameters this is very slow	For large parameter modules loaded from checkpoint, use `no_init_weights()` to skip

Diagnostic Tips：

# 区分真卡和 CPU 密集慢
ps aux | grep "infer.py" | awk '{print $2, $3"%CPU", int($6/1024)"MB"}'
# CPU 100%+ = 还在算（慢但没死）
# CPU 0%    = 真卡住（等通信/IO）

Fix Template (Large embedding module)：

from transformers.modeling_utils import no_init_weights

# 用 no_init_weights 包裹，避免对即将被 checkpoint 覆盖的参数做随机初始化
with no_init_weights():
    self.large_embeddings = LargeEmbeddingModule(config, ...)

Residual processes cause HCCL hang

Symptom: Newly started inference hangs at HCCL init, no error and no timeout

Root cause: The previous inference process did not exit correctly and is still occupying the NPU device or HCCL port

Fix：

# 1. 杀死残留进程
ps aux | grep "infer.py" | grep -v grep | awk '{print $2}' | xargs kill -9

# 2. 等待 NPU 释放
sleep 3

# 3. 确认 NPU 完全空闲
npu-smi info | grep "No running"
# 期望输出 8 行 "No running processes found"

# 4. 重新启动推理

Prevention: Before each inference start, check for and clean up residual processes.

Appendix

Runtime Monitoring and Efficiency Improvement

The following command examples are based on the current repository's default log paths and startup script conventions; adjustments are needed after changes to executor or log organization.

Core Principles

Judge completion status by processes, not the log stream. Multi-card inference launches N processes in the background via infer.sh and then wait. Rank 0's stdout is piped to the terminal via tee , and may be buffered and truncated; other ranks only write to log files.

Completion Detection Methods

Method	Command	Description
Check NPU processes	`npu-smi info \| grep "running process"`	No processes = finished
Check non-rank0 logs	`grep "model run success" logs/*/log_1.log`	Rank 1-7 logs do not go through tee and are written completely.Do not check only rank 0
Check all ranks	`for i in $(seq 0 7); do grep -l "model run success" logs/*/log_${i}.log; done \| wc -l`	Expected output = world_size

Log Keywords to Determine Stage

grep "Loading safetensors" logs/*/log_1.log     # 正在加载权重
grep "Loading weights took" logs/*/log_1.log    # 加载完成
grep "inference time cost of prefill" logs/*/log_1.log  # 推理已开始
grep "model run success" logs/*/log_1.log       # 推理已结束

Get global status with one command

echo "=== NPU ===" && npu-smi info | grep -E "Process|HBM" | head -5 && \
echo "=== Log ===" && tail -3 logs/*/log_1.log && \
echo "=== Done? ===" && grep -c "model run success" logs/*/log_*.log 2>/dev/null

Common Misjudgment Scenarios

Symptom	Cause	Correct approach
No new output in the terminal, assume it is running	Rank 0's tee pipe has closed or the buffer has not been flushed	Check `npu-smi` or check rank 1-7 logs
Rank 0 log has no "model run success"	tee truncation	Check rank 1's log
`sleep 120` not completed yet after	Large model / slow first compilation	First `npu-smi` to determine liveness, then check HBM changes to determine the stage
grep log cannot find the keyword	The process crashed and did not write success	Check `npu-smi` to confirm the process is gone → check `tail -20 log_*.log` for errors

Agent Working Mode

Start inference（bash infer.sh, in the foreground or run_in_background）
Do not blindly sleep — use run_in_background to wait for notification, or view the output directly in the foreground
No progress for 10 minutes → proactively investigate: If inference startup exceeds 10 minutes with still no model run success output, do not keep waiting; troubleshoot according to the "Stuck Inference Diagnosis" process (npu-smi → determine status → triage)
After a timeout, the first step is to check npu-smi; do not check the logs first
When the check completes, check rank 1 (not rank 0)— rank 1 logs are complete and not truncated
Check multiple signals at once：npu-smi + tail log_1.log + grep "model run success" log_*.log run in parallel

About this skill

Diagnostic Path Overview

npu-smi Status Check

Step 1: Error Classification

A. aicore timeout (error code 507014)

B. HCCL Communication Errors

C. OOM (Out of Memory)

D. Shape mismatch

E. Operator constraint violation

Step 2: Bisection localization (sync+barrier checkpoint method)

2.1 Locate the phase (Prefill vs Decode)

2.2 Locate the layer

2.3 Locate the module

2.4 Locate the operator

2.5 Notes

Step 3: Quick reference for common NPU operator constraints

MoE-related operators

FA (Flash Attention) operators

Communication operators

Step 4: Fix strategy patterns

Pattern 1: Operator Fallback

Mode 2: Parameter Fix

Mode 3: Path Unification

Mode 4: Configuration Downgrade

Step 5: Verification Checklist

Multi-card Deployment Diagnosis

Weight Loading TP Partition Out-of-Bounds

Config Compatibility Issues

Some Ranks Fail Silently

Multi-Card Weight Loading Checklist

Inference Stuck Diagnosis

Reproduce in eager mode

Stuck during model construction phase (CPU-intensive)

Residual processes cause HCCL hang

Runtime Monitoring and Efficiency Improvement

Core Principles

Completion Detection Methods

Log Keywords to Determine Stage

Get global status with one command

Common Misjudgment Scenarios

Agent Working Mode