标签归档：obet patch_scn

不当数据库恢复操作导致一个月数据丢失

发表于 2026 年 9 月 4 日 由 惜分飞

今天分享一个本不该发生的故障,结果由于现场人员没有合适的判断导致客户丢失近一个月数据,事故经过大概是这样的
1. 客户数据库由于断电,启动之后报ORA-600 kcratr_scan_lastbwr错误

2026-09-03T11:22:58.998379+08:00
Beginning crash recovery of 2 threads
 parallel recovery started with 31 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 37738 block 7724), scn 0
 Thread 2: Recovery starting at checkpoint rba (logseq 43821 block 37937), scn 0
2026-09-03T11:22:59.158232+08:00
Started redo scan
Hex dump of (file 20, block 2720) in trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_34049.trc

Reading datafile '+DATA/ORCL/DATAFILE/undotbs01.dbf' for corrupt data at rdba: 0x05000aa0 (file 20, block 2720)
Reread Not Encrypted (file 20, block 2720) found same data 
physically corrupt:No, logically corrupt:No, and is not expected version by upper layer. 
Write verification failed for File 20 Block 2720 (rdba 0x5000aa0)
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_34049.trc  (incident=430645):
ORA-00600: internal error code, arguments: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_430645/orcl1_ora_34049_i430645.trc
2026-09-03T11:23:00.484486+08:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2026-09-03T11:23:00.510512+08:00
Slave encountered ORA-10388 exception during crash recovery

这个是一个比较常见的错误一般是由于redo写丢失导致(数据库open报ORA-600 kcratr_scan_lastbwr故障处理)

2. 现场操作的技术人员看了下这个错误,觉得数据库有备份，直接决定使用备份进行恢复
rman_bak

结果发现读取的是20260719的备份,而且原始的数据文件在asm里面，已经被覆盖。这个时候如果归档或者增量备份完整,还能够最大限度抢救数据,检查备份发现只有8月7日的增量备份,没有任何归档的备份(使用crontab 直接清理3天之前的归档)
1811
基于这样的情况,应用到最后一个增量备份(也就是8月7日),后面的数据由于归档已经被清理(而且没有备份)
arch
出现这个问题的原因是由于备份脚本本身备份归档,导致备份无法按照策略清理历史备份,导致磁盘空间满,从而后续备份无法继续(直接清理归档了)
ORA-27027
恢复只能到此为止,查询最终的文件头恢复时间点
header_rman
3. 基于这种情况只能尝试强制打开数据库,结果报ORA-600 kcbzibmlt_kcrsds_1错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzibmlt_kcrsds_1], [], [], []
Process ID: 30365
Session ID: 2180 Serial number: 43414

4. 这种错误类似ORA-600 kcbzib_kcrsds_1的处理思路,使用obet的patch_scn功能直接进行处理

OBET> patch_scn 89103 0x60017e98 3182707622

=== Oracle SCN Patch ===
PID:     89103
Address: 0x60017e98
New Value: 0xbdb443a6 (3182707622)
Original SCN: 0x0

Are you sure you want to modify Oracle SCN? (yes/no): y

SCN modified: 0x0 -> 0xbdb443a6
Oracle SCN successfully modified.

然后打开数据库

SQL> alter database open;

Database altered.

导出数据,导入新库完成本次恢复任务,可惜没有办法帮客户找回来丢失的那一个月数据.

案例总结:
1. rman备份脚本尽量不要有直接删除归档的操作,尽可能是先备份然后再删除(确认备份成功之后删除)
2. 备份还原操作之前,需要看备份文件和日志,大概评估下备份的可用性
3. 尽可能不要在损坏的现场环境上直接做rman还原操作(万一备份无法正常还原,比如中途有备份文件异常呢？)
4. 这个故障,如果不做还原操作,直接异常恢复,基本上也就丢失了当前redo中的数据,损失是最小的

发表在 Oracle备份恢复 | 标签为 , , | 评论关闭