Overview
Disaster recovery plans, RTO/RPO, failover skills.
Core Concepts
RTO and RPO
RPO (Recovery Point Objective)
- 可接受的数据丢失量
- 决定备份频率
RTO (Recovery Time Objective)
- 可接受的恢复时间
- 决定恢复策略
Example:
- RPO = 1 hour → hourly backup
- RTO = 4 hours → requires hot standby or fast recovery
### 恢复策略
Cold Standby (Cold)
- Lowest cost
- Longest RTO
- Suitable for non-critical systems
Warm Standby (Warm)
- Moderate cost
- Moderate RTO
- Periodic data synchronization
Hot Standby (Hot)
- Highest cost
- Shortest RTO
- Real-time synchronization
## 数据库恢复
### MySQL 恢复
```bash
# 从备份恢复
mysql -u root -p < full_backup.sql
# 应用 binlog
mysqlbinlog mysql-bin.000001 | mysql -u root -p
# 时间点恢复
mysqlbinlog --stop-datetime="2024-01-15 10:00:00" mysql-bin.* | mysql -u root -p
# 主从切换
# 在从库执行
STOP SLAVE;
RESET SLAVE ALL;
# 应用程序切换连接
PostgreSQL Recovery
# 从备份恢复
pg_restore -d database backup.dump
# PITR 恢复
# recovery.conf
restore_command = 'cp /archive/%f %p'
recovery_target_time = '2024-01-15 10:00:00'
# 主从切换
pg_ctl promote -D /var/lib/postgresql/data
Redis Recovery
# 从 RDB 恢复
cp backup.rdb /var/lib/redis/dump.rdb
systemctl restart redis
# 从 AOF 恢复
cp backup.aof /var/lib/redis/appendonly.aof
redis-check-aof --fix appendonly.aof
systemctl restart redis
System Recovery
File System Recovery
# 从 tar 备份恢复
tar -xzvf /backup/system.tar.gz -C /
# 从 rsync 备份恢复
rsync -avz /backup/system/ /
# 恢复权限
restorecon -Rv /
Boot Repair
# 进入救援模式
# 挂载根分区
mount /dev/sda1 /mnt
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt
# 修复 GRUB
grub-install /dev/sda
update-grub
Failover
Keepalived Failover
# 检查状态
systemctl status keepalived
ip addr show | grep -w inet
# 手动切换
# 降低主节点优先级
# /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
priority 50 # 降低
}
systemctl reload keepalived
DNS Failover
# 修改 DNS 记录
# 降低 TTL（提前）
# 切换 A 记录指向备用 IP
# 验证
dig +short example.com
nslookup example.com
Common Scenarios
Scenario 1: Full Recovery Process
#!/bin/bash
# 1. 评估损失
echo "检查系统状态..."
# 2. 通知相关人员
# send_alert "开始灾难恢复"
# 3. 恢复基础设施
echo "恢复网络配置..."
# 4. 恢复数据
echo "恢复数据库..."
mysql -u root -p < /backup/latest.sql
# 5. 恢复应用
echo "启动应用服务..."
systemctl start application
# 6. 验证
echo "验证服务状态..."
curl -s http://localhost/health
# 7. 通知恢复完成
# send_alert "灾难恢复完成"
Scenario 2: DR Drill
#!/bin/bash
# DR 演练脚本
LOG="/var/log/dr-drill.log"
echo "$(date): 开始 DR 演练" >> $LOG
# 1. 切换到备用站点
echo "切换 DNS..." >> $LOG
# 2. 验证服务
echo "验证服务可用性..." >> $LOG
curl -s http://dr-site/health >> $LOG
# 3. 测试数据一致性
echo "验证数据一致性..." >> $LOG
# 4. 记录 RTO
echo "实际 RTO: $(计算时间)" >> $LOG
# 5. 切回主站点
echo "切回主站点..." >> $LOG
Scenario 3: Automatic Failover
# Keepalived 配置
vrrp_script chk_app {
script "/usr/local/bin/check_app.sh"
interval 2
weight -20
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
track_script {
chk_app
}
virtual_ipaddress {
192.168.1.100
}
}
DR Checklist
| Item | Check Content |
|---|---|
| Backup | Backup integrity, recoverability |
| Documentation | Recovery steps, contacts |
| Network | DNS, IP, firewall |
| Data | Data consistency, synchronization status |
| Application | Configuration, dependencies, certificates |
Troubleshooting
# 检查备份状态
ls -la /backup/
md5sum /backup/latest.tar.gz
# 检查复制状态
# MySQL
SHOW SLAVE STATUS\G
# PostgreSQL
SELECT * FROM pg_stat_replication;
# 检查网络连通性
ping dr-site
traceroute dr-site