SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsohos-issue-graphics-sysfreeze-analysis

ohos-issue-graphics-sysfreeze-analysis

General & other

View repository

Updated 10-16 00:04

About this skill

Task and Boundaries

Problem Solved

Analyze Sysfreeze logs of OpenHarmony graphics-related modules to diagnose the root causes of process freeze and thread blocking issues.

Input

  • Sysfreeze/Appfreeze log files
  • Related source code (code/ directory)
  • so files with symbol tables (lib.unstripped/ directory)
  • Optional: hilog and hilog_kmsg logs

Output

  • Fault type confirmation
  • Blocking chain path
  • Root cause analysis
  • Optimization suggestions

Inapplicable Cases

  • Pure performance optimization issues (no blocking)
  • Application-layer logic errors (no system-level blocking)
  • Freeze issues with call stacks unrelated to graphics modules
  • Log information is severely missing and cannot be analyzed

Prerequisites

The user environment provides llvm tools (llvm-readelf/llvm-addr2line/llvm-objdump, etc.).

Trigger Signals
  • The user requests analysis of sysfreeze and appfreeze logs
  • SERVICE_BLOCK or THREAD_BLOCK faults occur
  • Need to trace thread blocking chains and deadlocks
  • Need to parse task queues and resource states
  • Keywords: process stuck, thread blocking, deadlock, freeze, block
Pre-checks

Confirm log integrity

  • Check whether there are two stack capture records (SERVICE_WARNING/SERVICE_BLOCK or THREAD_BLOCK_3S/THREAD_BLOCK_6S)
  • Confirm that key fields such as TIMESTAMP and TID exist

Confirm symbol table availability

  • Check whether the lib.unstripped/ directory contains so files with symbol tables
  • Parsing command:llvm-addr2line -Cfpie path/to/so offset
  • Note:ld-musl-xxx.so i.e. libc.so
Execution Strategy

1. Basic Information Analysis

Confirm fault type

  • Check the Reason field to confirm whether it is SERVICE_BLOCK or THREAD_BLOCK
  • Understand the detection principles and thresholds of the corresponding problem

Record key information

  • TIMESTAMP: The moment the fault occurs, used to lock the time range in hilog logs
  • TID: The ID of the thread where the problem occurs; problem analysis starts from this thread

Check overall device resource status

  • ReclaimAvailBuffer: Remaining available memory of the whole device, to determine whether it is a low-memory issue
    • Note the memory information capture time; if it is after the freeze time, it has no reference value
  • ThermalLevel info: When the thermal level is ≥ 5, the system may be affected by thermal throttling and performance may degrade
  • If there is an overall device resource problem and business analysis shows no anomaly, the problem may be prioritized as related to it

2. Task Queue Analysis

Analyze the current task

Check the Current Running task of the two detections (SERVICE_WARNING/SERVICE_BLOCK, or THREAD_BLOCK_3S/THREAD_BLOCK_6S):

  • If the tasks in the two detections are the same: the same task is blocked
  • If the tasks in the two detections are different: analyze the task that takes longer based on the task queue information

Verify time reasonableness

Assume the last watchdog task is submitted at time T:

  • First stack capture: SERVICE_WARNING expected time T+5s, THREAD_BLOCK_3S expected time T+3s
  • Second stack capture: SERVICE_BLOCK expected time T+10s, THREAD_BLOCK_6S expected time T+6s
  • If there is a large difference between the actual time and the expected time: DFX capture has a time deviation, and there may be overall device issues such as low memory/high load

Analyze the task queue

Check task queues of each priority (VIP, Immediate, High, Low, Idle), focusing on:

  • Task submission time
  • Task execution time (starts at trigger time, completes at completeTime)
  • Task priority
  • Task submitter
  • Task name

3. Blocking chain tracing

Before analyzing, ask yourself three questions：

  1. What is the blocking type? — Is the top-of-stack function waiting for a lock, waiting for a semaphore, or an IPC call?
  2. Where is the source of the blocking? — Who holds the resource, and who is the IPC peer?
  3. Is the blocking chain closed? — Can it be traced back to the final thread that is not blocked?

Analyze thread blocking relationships based on the call stacks captured twice,The stack must be unwound during analysis。

Important: Analysis starts from the Tid:<TID> thread in the basic information.

Call stack comparison

Compare the two stack captures

  • The two stack captures are consistent: stuck in the same service
  • The two stack captures are inconsistent: it may be two instantaneous stacks of different tasks, or the same task that is not stuck but takes a long time to execute.

Analyze thread state(if available)

  • View the thread state field:state、utime、stime、priority、nice、clk
  • Key judgment:
    • If the top of the thread stack is not waiting for a resource/entering the kernel but is not in a running state (state=S): it may be caused by an overall device issue (low memory/IO problem, etc.)
    • Combine utime/stime and the stack capture time interval to determine the thread's execution duration in user mode and kernel mode
    • Compare the running time of the two stack captures. If there is no change, it means the thread has not been scheduled.

Analyze stack blocking patterns

Identify the blocking function at the top of the stack, refer to Common blocking patterns

Blocking chain analysis

Identify the blocking type

  • Determine the blocking cause based on the top-of-stack function (waiting for a lock, semaphore, IPC call, etc.)

Find the blocking source

  • Top of stack waiting for a lock/semaphore/condition variable: find the thread currently holding the resource or the triggering thread
  • Top of stack is an IPC call: find the IPC peer thread
  • The blocking thread is usually in the same so or the same class of service as the blocked top of stack

Recursively analyze blocking

  • If the blocking thread is also in a blocking condition, continue analyzing the thread that blocks it
  • Until a thread that is not in a blocking state is found, which is the final problem thread

If it is IPC blocking handling

  • View BinderCatcher information to obtain the synchronous call whose source is 当前进程号:当前线程号 synchronous call
  • View the blocking time
  • View 对端进程号:对端线程号 information, find the corresponding process stack capture in PeerBinder Stacktrace corresponding process stack capture
  • Start analyzing the blocking situation from the thread corresponding to the peer thread ID

4. Pipeline log analysis (if logs are available)

Obtain pipeline logs

  • Check whether hilog and hilog_kmsg files exist

Analyze pipeline logs

  • Analyze logs near TIMESTAMP logs near the time the problem occurred
  • Identify possibly related exception logs, refer to common flow logs
  • Identify the business scenario and operation flow

5. Comprehensive analysis and report

Determine the root cause

  • Synthesize stack information, task queue, thread state, and resource state
  • Determine the related business information of the problem, or the related whole-device problem

Draw the blocking path

  • Describe the complete call chain from the blocked thread to the final problem thread
  • Mark the key blocking points and the resources being waited for

Analyze business information

  • Parse the business code corresponding to the stack
  • Combine with key logs to explain the stuck business scenario and operation flow

Generate an analysis report

  • Fault analysis summary
  • Blocking path analysis
  • Root cause of the problem (if the result is incomplete, give the next troubleshooting direction)
  • Suggested optimization direction

Important: After completing the analysis, generate a report and save it to the document.

Prohibited practices

Do not skip the stack unwinding step

  • Incorrect practice: directly guessing the business logic based on the so name and offset
  • Reason: The code line number corresponding to the function name and offset is the key to locating the problem; not unwinding the stack will lead to the wrong analysis direction

Do not ignore the whole-device resource state

  • Incorrect practice: only focus on the business stack and ignore resource information such as memory and temperature
  • Reason: Whole-device problems such as low memory and overheating may be the real root cause

Do not analyze a single thread in isolation

  • Incorrect practice: only analyze the stack of the problem thread without tracing the blocking chain
  • Reason: Blocking is usually caused by lock contention or an IPC call chain, and the final problem thread needs to be found

Do not force a conclusion when information is insufficient

  • Incorrect practice: still give a definite root cause when symbol tables or logs are missing
  • Reason: Incorrect analysis conclusions will mislead the subsequent troubleshooting direction

Do not ignore time information

  • Incorrect practice: ignore the time difference between two stack captures and the task submission time
  • Reason: Time information can determine whether it is instantaneous blocking and whether DFX capture is normal
Exceptions and fallback

Handling when information is insufficient

Missing information Handling method
Symbol table does not exist Point out which so symbol tables are needed, and the code line number cannot be precisely located
Only one stack capture Explain that comparison information is missing, and it cannot be determined whether the same task is blocked
hilog missing Rely only on stack and task queue analysis, and note the missing logs in the report
BinderCatcher missing IPC blocking analysis is limited; try to infer the peer from the stack

Handling when stack unwinding fails

  • Check whether the so file path is correct
  • Confirm whether the offset is valid (hexadecimal format)
  • Use llvm-readelf to verify whether the so contains a symbol table
  • If it is confirmed that there is no symbol table, clearly note it in the report

Handling when the blocking chain cannot be closed

  • Check whether there are IPC cross-process calls (need to inspect the peer process)
  • Consider kernel-mode blocking (such as IO waiting)
  • In the report, state at which node the blocking chain is interrupted, and give the next troubleshooting direction

Priority determination for whole-device issues

When both business issues and whole-device issues exist:

  • If whole-device resources are severely abnormal (such as memory nearing exhaustion), report the whole-device issue first
  • If whole-device resources are slightly abnormal but the business has obvious blocking logic, analyze the business issue first
  • If it cannot be determined, list both possibilities in the report at the same time
Reference materials

When to read：

  • Common blocking patterns: read when identifying the blocking function at the top of the stack (such as pthread_mutex_timedlock_inner, pthread_cond_timedwait)
  • Common pipeline logs: read when analyzing abnormal hilog logs (such as GetSaWrap, vsync signal)

When not to read：

  • Only perform basic information analysis (fault type, whole-device resources); no reference files need to be loaded
  • The blocking chain is already clear and there is no need to identify the blocking pattern; do not load blocking-patterns.md
  • There is no hilog file or log analysis is already complete; do not load hilog-refernce.md