Logs are what the system says. Most of the time it is saying nonsense, but when problems occur, the answers are hidden in that nonsense.
Analysis Process
拿到日志
│
├─ 1. 识别日志格式和结构
├─ 2. 定位异常和错误
├─ 3. 构建时间线
├─ 4. 分析根因
└─ 5. 给出结论和建议
Step 1: Identify the Log Format
| Format | Characteristics | Handling Method |
|---|---|---|
| JSON structured | One JSON object per line | Parse JSON, filter by fields |
| Standard log format | [时间] [级别] [模块] 消息 |
Extract fields with regular expressions |
| Plain text | No fixed format | Match by keywords and timestamp patterns |
| Stack trace | Error + at file:line | Identify error type and call stack |
If the log is too long (>500 lines), first ask the user:
- Roughly what time period did the problem occur?
- Are there any related error keywords?
- The affected user/request ID?
Step 2: Locate Anomalies
2.1 Error Level Scan
Scan by priority from high to low:
FATAL / CRITICAL → ERROR → WARN → 异常模式（超时、重试、拒绝）
2.2 Key Pattern Recognition
| Pattern | Keywords | Possible Problem |
|---|---|---|
| Exception/Error | Error, Exception, FATAL, panic |
Code error |
| Timeout | timeout, ETIMEDOUT, deadline exceeded |
Network/dependency slow |
| Connection failed | ECONNREFUSED, connection reset, ECONNRESET |
Service unreachable |
| Resource exhaustion | OOM, too many open files, pool exhausted |
Resource leak |
| Permission issue | 403, permission denied, unauthorized |
Permission configuration |
| Data issue | null, undefined, NaN, invalid |
Data quality |
| Retry/degradation | retry, fallback, circuit breaker |
Dependency unstable |
2.3 Count Error Distribution
## 错误统计
| 错误类型 | 次数 | 首次出现 | 最后出现 |
|---------|------|---------|---------|
| ConnectionTimeout | 45 | 14:23:01 | 14:28:33 |
| NullPointerError | 3 | 14:25:12 | 14:25:14 |
| AuthTokenExpired | 12 | 14:20:00 | 14:30:00 |
Step 3: Build a Timeline
Arrange key events in chronological order to reconstruct how the problem occurred:
## 事件时间线
14:20:00 [正常] 服务启动，连接数据库成功
14:23:01 [异常] 首次出现 ConnectionTimeout，目标：redis:6379
14:23:15 [异常] ConnectionTimeout 频率上升，10s 内 15 次
14:24:00 [告警] Redis 连接池耗尽，开始排队
14:25:12 [错误] NullPointerError — getUserSession() 返回 null
↑ 因为 Redis 不可用，session 查询失败
14:25:30 [降级] 熔断器触发，Redis 请求直接返回 fallback
14:28:33 [恢复] Redis 连接恢复，ConnectionTimeout 消失
14:30:00 [正常] 错误率归零
Step 4: Analyze Root Cause
Analysis Methods
- Reason backward from the result: what the user saw → direct cause → root cause
- Time correlation: around the time the problem started, was there any deployment/configuration change/traffic spike?
- 5 Whys: keep asking "why" until the root cause is found
Output Format
## 根因分析
### 现象
用户登录页报 500 错误
### 直接原因
getUserSession() 返回 null，后续代码未做空值处理导致 NullPointerError
### 根本原因
Redis 实例在 14:23 出现网络抖动（可能是运维变更），
导致连接超时 → 连接池耗尽 → session 查询失败 → 空值未处理 → 500
### 因果链
Redis 网络抖动 → 连接超时 → 连接池耗尽 → session 查询失败
→ 返回 null → 未做空值检查 → NullPointerError → 500 响应
Step 5: Conclusions and Recommendations
## 结论
1. **根因**：Redis 网络抖动导致连接超时，session 查询失败
2. **影响面**：14:23-14:28 期间约 200 个请求受影响
3. **恢复方式**：Redis 连接自动恢复后问题消失
## 建议
### 短期（立即修复）
- getUserSession() 增加空值检查，返回 null 时走降级逻辑而非抛异常
### 中期（防止复发）
- Redis 连接池增加健康检查和自动重连机制
- 添加 Redis 连接状态监控告警
### 长期（提升韧性）
- session 查询增加本地缓存兜底
- 考虑 Redis Sentinel 或 Cluster 提高可用性
Analysis Guidelines
- Global first, then details: first scan the overall log picture (error distribution, time range), then dig into specific errors
- Focus on time correlation: what changed before and after the problem? Deployment? Configuration? Traffic?
- Distinguish causation from coincidence: two events being close in time does not mean there is a causal relationship
- Do not guess: if the information is not in the log, say "needs further confirmation" and do not make up a cause
- Give actionable recommendations: not just "add a log", but "in
src/services/auth.ts:45Add a null check after the return value of getUserSession()"