emergency-response

Incident Response Methodology

An intrusion investigation operations manual for both Linux and Windows platforms. Proceed in phases, and only move to the next step after each step produces a clear conclusion.

Phase 1: Initial Response Decision

主机疑似被入侵？
├─ 1. 确认事件范围
│   ├─ 单台主机 → 直接排查
│   └─ 多台/不明 → 先网络隔离，再逐台排查
├─ 2. 判断操作系统
│   ├─ Linux → Phase 2
│   └─ Windows → Phase 3
├─ 3. 确认当前权限
│   ├─ root/Administrator → 可完整排查
│   └─ 普通用户 → 先提权或协调管理员权限
└─ 4. 保全证据（排查前）
    ├─ 内存快照（如有条件）
    ├─ 磁盘镜像（关键服务器）
    └─ 记录当前时间戳，后续所有发现标注时间

Key Principles：

Collect first, analyze second, and remove last — do not directly delete suspicious files during the investigation process
Record findings at every step to form a timeline
Prioritize investigation of high-threat items: accounts → processes/network → persistence → files → logs

Phase 2: Linux Incident Investigation

2.1 Account Audit

Investigation Checklist：

# 1. 查找特权用户（uid=0 应该只有 root）
awk -F: '$3==0{print $1}' /etc/passwd

# 2. 查找可远程登录的账户（有密码 hash 的账户）
awk '/\$1|\$6/{print $1}' /etc/shadow

# 3. 检查 sudo 权限异常（非管理员不应有 ALL 权限）
grep -v "^#\|^$" /etc/sudoers | grep "ALL=(ALL)"
cat /etc/sudoers.d/*

# 4. 检查 SSH 授权密钥（攻击者常植入公钥）
find / -name "authorized_keys" -exec ls -la {} \; -exec cat {} \;

# 5. 查看最近登录记录
lastlog           # 所有用户最后登录时间
last              # 历史登录/注销记录
lastb             # 失败登录记录（暴力破解痕迹）

Judgment Criteria：

Finding	Risk Level	Handling
Non-root user with uid=0	High	Disable immediately:`usermod -L <user>`
Unknown user has sudo ALL privileges	High	Remove the sudoers entry
Unknown SSH public key	High	Record the public key fingerprint, then delete it
Login records during abnormal time periods	Medium	Correlate IP for traceability

2.2 Processes and Network

# 1. 检查异常网络连接（重点关注 ESTABLISHED 和 LISTEN）
netstat -antlp | grep -E "ESTABLISHED|LISTEN"
# 或
ss -antlp

# 2. 根据可疑连接 PID 定位进程
ls -la /proc/<PID>/exe        # 进程对应的可执行文件
cat /proc/<PID>/cmdline       # 完整命令行
ls -la /proc/<PID>/fd         # 打开的文件描述符

# 3. 检查异常进程
ps aux --sort=-%cpu | head -20    # CPU 占用排序（挖矿检测）
ps aux --sort=-%mem | head -20    # 内存占用排序

# 4. 检查隐藏进程（进程名以 . 开头或伪装为内核线程）
ps -ef | grep -E "^\S+\s+\d+.*\[.*\]" | grep -v "\[kworker"

Judgment Criteria：

ESTABLISHED connections to non-business IPs → highly suspicious (C2 communication)
Process executable files under /tmp, /dev/shm, /var/tmp → highly suspicious
Process executable file has been deleted (/proc/PID/exe -> (deleted)) → malicious process residing in memory

2.3 Startup Items and Scheduled Tasks

# 1. 检查 crontab（所有用户）
for user in $(cut -f1 -d: /etc/passwd); do
  crontab -l -u $user 2>/dev/null && echo "=== $user ==="
done
cat /etc/crontab
ls -la /etc/cron.d/ /etc/cron.daily/ /etc/cron.hourly/ /etc/cron.monthly/ /etc/cron.weekly/

# 2. 检查 systemd 服务（关注非官方和最近修改的）
systemctl list-unit-files --type=service | grep enabled
find /etc/systemd/system/ /usr/lib/systemd/system/ -name "*.service" -mtime -7

# 3. 检查 rc.local 和 init.d
cat /etc/rc.local
ls -la /etc/init.d/
ls -la /etc/rc.d/rc*.d/ 2>/dev/null

# 4. 检查 bashrc/profile 植入
grep -r "curl\|wget\|python\|bash -i\|/dev/tcp" /etc/profile* /etc/bashrc /root/.bashrc /home/*/.bashrc 2>/dev/null

Key Focus Areas：

Recently added cron tasks (correlate creation time with the intrusion time window)
Service files in non-standard paths (/tmp, user home directories)
Suspicious scripts in rc.local

2.4 File System Investigation

# 1. 查找最近修改的文件（以 3 天为例，根据事件时间调整）
find / -mtime -3 -type f -not -path "/proc/*" -not -path "/sys/*" | head -100

# 2. 查找 SUID/SGID 文件（提权后门常见手段）
find / -perm -4000 -type f 2>/dev/null
find / -perm -2000 -type f 2>/dev/null

# 3. 检查 /tmp 及隐藏目录
ls -la /tmp/ /var/tmp/ /dev/shm/
find / -name ".. " -o -name "..." -o -name ".hidden" 2>/dev/null

# 4. WebShell 检测（Web 服务器目录）
find /var/www/ -name "*.php" -mtime -7
grep -rl "eval\|system\|exec\|passthru\|shell_exec\|base64_decode" /var/www/ 2>/dev/null

# 5. 检查文件时间戳篡改（mtime 与 ctime 不一致）
stat <可疑文件>   # 对比 Modify 和 Change 时间

2.5 Log Analysis

Log File	Purpose	How to View
/var/log/auth.log or /var/log/secure	SSH login, sudo, su	`grep "Accepted\|Failed" /var/log/auth.log`
/var/log/wtmp	Login/logout records	`last -f /var/log/wtmp`
/var/log/btmp	Failed logins	`lastb -f /var/log/btmp`
/var/log/cron	Scheduled task execution records	`cat /var/log/cron`
/var/log/messages or /var/log/syslog	System events	`grep -i "error\|warning\|fail" /var/log/messages`
~/.bash_history	Command history	`cat /root/.bash_history`

# 暴力破解检测：统计失败登录 IP
grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

# 成功登录排查：关注非常规 IP 和时段
grep "Accepted" /var/log/auth.log | awk '{print $1,$2,$3,$9,$11}'

# 提权行为检测
grep "sudo:" /var/log/auth.log | grep -v "session opened\|session closed"

2.6 Rootkit Detection

# 方法 1: chkrootkit
apt install chkrootkit -y   # 或 yum install chkrootkit
chkrootkit

# 方法 2: rkhunter
apt install rkhunter -y     # 或 yum install rkhunter
rkhunter --update
rkhunter --check --sk

# 方法 3: 手动检查（工具不可信时）
# 比较系统命令 hash（与已知干净环境对比）
md5sum /usr/bin/ps /usr/bin/netstat /usr/bin/ls /usr/bin/find /usr/bin/top
# 检查 LD_PRELOAD 劫持
echo $LD_PRELOAD
cat /etc/ld.so.preload
# 检查内核模块
lsmod | grep -v "^Module"

Phase 3: Windows Incident Investigation

3.1 Account Audit

:: 1. 查看当前登录会话
query user

:: 2. 列出所有用户和管理员组成员
net user
net localgroup administrators

:: 3. 检查隐藏/克隆账户
:: 隐藏账户名末尾带 $
wmic useraccount list full
:: 对比注册表 SAM 与 net user 结果
:: HKLM\SAM\SAM\Domains\Account\Users（需 SYSTEM 权限）

:: 4. 踢出可疑会话
logoff <SessionID>

Auxiliary Tools：

D Shield: detects hidden accounts and cloned accounts
View SID history:wmic useraccount get name,sid — administrators whose SID does not end in 500/501 need verification

3.2 Processes and Network

:: 1. 查看所有网络连接及对应进程
netstat -ano | findstr "ESTABLISHED LISTENING"

:: 2. 根据 PID 查进程信息
tasklist /FI "PID eq <PID>" /V
wmic process where processid=<PID> get name,executablepath,commandline

:: 3. 检查可疑进程
tasklist /V
wmic process get name,processid,executablepath,commandline /format:list

:: 4. 查看网络代理配置（是否被篡改）
REG QUERY "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings"

:: 5. 查看路由表
route print

Judgment Criteria：

Executable file paths under %TEMP%, %APPDATA%, or user Downloads → suspicious
Process names imitating system processes (such as svchost.exe but not under C:\Windows\System32) → highly suspicious
ESTABLISHED connections to abnormal external IPs → correlate with threat intelligence

3.3 Startup Items and Services

:: 1. 检查注册表启动项
REG QUERY "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run"
REG QUERY "HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run"
REG QUERY "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce"
REG QUERY "HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Run"

:: 2. 检查计划任务
schtasks /query /fo LIST /v

:: 3. 检查启动文件夹
dir "%APPDATA%\Microsoft\Windows\Start Menu\Programs\Startup"
dir "%ProgramData%\Microsoft\Windows\Start Menu\Programs\Startup"

:: 4. 检查服务（关注非标准服务和最近创建的服务）
wmic service list brief
wmic service get name,displayname,pathname,startmode | findstr /i "auto"

:: 5. 检查启动信息
wmic startup get command,caption

Key Focus Areas：

Entries in Run/RunOnce pointing to temporary directories or non-standard paths
Scheduled tasks that execute PowerShell, cmd /c, mshta, etc.
Service executable paths containing spaces without quotes (Unquoted Service Path)

3.4 File System Investigation

:: 1. 查看最近访问的文件
dir %USERPROFILE%\Recent /a /o:-d

:: 2. 检查临时目录
dir %TEMP% /a /o:-d
dir %SYSTEMROOT%\Temp /a /o:-d

:: 3. 检查浏览器下载目录
dir "%USERPROFILE%\Downloads" /a /o:-d

:: 4. 查看回收站
dir C:\$Recycle.Bin /s /a

:: 5. 检查 ADS（备用数据流，可隐藏恶意代码）
dir /r <可疑目录>

Check Prefetch Files (Prefetch)：

Path:C:\Windows\Prefetch\
Can reconstruct program execution history even if the executable file has been deleted

3.5 Log Analysis

Open Event Viewer:eventvwr.msc

Log Locations：

Log Type	Default Path	Content of Interest
Security Log	`%SystemRoot%\System32\Winevt\Logs\Security.evtx`	Logon events, account management, privilege use
System Log	`%SystemRoot%\System32\Winevt\Logs\System.evtx`	Service installation, driver loading, system errors
Application Log	`%SystemRoot%\System32\Winevt\Logs\Application.evtx`	Application crashes, program errors

Key Event IDs：

Event ID	Meaning	Investigation Value
4624	Successful Logon	Determine attack method based on logon type
4625	Failed Logon	Traces of brute force attacks
4634	Logoff	Session duration analysis
4672	Privileged Logon	Administrator logon events
4720	User Created	Backdoor account detection
7045	Service Installed	Traces of persistence/lateral movement

Logon Type Reference：

Type	Description	Common Scenarios
2	Interactive Logon	Local console logon
3	Network Logon	Shared folders, PsExec
4	Batch	Scheduled tasks
5	Service	Service startup
7	Unlock	Screensaver unlock
10	Remote Interactive	RDP Remote Desktop

# PowerShell 查询安全日志（筛选登录失败）
Get-WinEvent -FilterHashtable @{LogName='Security';ID=4625} |
  Select-Object TimeCreated,@{N='IP';E={$_.Properties[19].Value}} |
  Group-Object IP | Sort-Object Count -Descending

# 查询新建用户事件
Get-WinEvent -FilterHashtable @{LogName='Security';ID=4720} |
  Format-List TimeCreated,Message

Phase 4: Tracing and Hardening

4.1 Timeline Reconstruction

收集所有发现 → 按时间排序 → 构建攻击链：
1. 初始入侵时间点（最早的异常记录）
2. 攻击路径（如何从入口扩展）
3. 持久化手段（留了哪些后门）
4. 数据泄露（是否有大量数据外传）

4.2 IOC Extraction

Extract the following indicators from the investigation for threat intelligence correlation:

IOC Type	Source	Purpose
Malicious IP/Domain	Network connections, logs	Blocking + intelligence queries
File Hash	Suspicious file MD5/SHA256	VirusTotal/ThreatBook queries
Attacker Accounts	New/anomalous accounts	Internal network lateral investigation
Malicious File Paths	File system investigation	Scan the entire network for similar files

4.3 Hardening Recommendations

Immediate Measures：

Change passwords for all affected accounts
Remove persistence mechanisms implanted by the attacker (scheduled tasks, services, startup items, SSH keys)
Block malicious IPs (firewall/security groups)
Patch the vulnerabilities exploited during the intrusion

Long-Term Hardening：

Enable centralized log collection (to prevent local logs from being cleared)
Deploy HIDS/EDR monitoring
Implement the principle of least privilege
Regularly audit accounts and services

Precautions

Do not restart the server during the investigation—malicious processes and network connection information in memory will be lost
Do not directly delete suspicious files—back them up for forensics before removing them
If you suspect that system commands (ps/netstat/ls) have been replaced (Rootkit), use a statically compiled busybox or copy tools from a trusted source for investigation
Logs may be cleared or tampered with by attackers—cross-verify from multiple sources (system logs + network device logs + application logs)

About this skill

Phase 1: Initial Response Decision

Phase 2: Linux Incident Investigation

2.1 Account Audit

2.2 Processes and Network

2.3 Startup Items and Scheduled Tasks

2.4 File System Investigation

2.5 Log Analysis

2.6 Rootkit Detection

Phase 3: Windows Incident Investigation

3.1 Account Audit

3.2 Processes and Network

3.3 Startup Items and Services

3.4 File System Investigation

3.5 Log Analysis

Phase 4: Tracing and Hardening

4.1 Timeline Reconstruction

4.2 IOC Extraction

4.3 Hardening Recommendations

Precautions