惜分飞

学校客户由于机房断电,导致数据库异常.现场工程师进行了现场备份,然后尝试强制拉库,结果没有成功,大概报错和操作过程如下:
断电之后启动数据库,数据库报ORA-600 kcratr_nab_less_than_odr故障解决错误,这个是一种非常常见的操作,一般是由于写丢失导致,以前有过很多类似恢复经历:
ORA-600 kcratr_nab_less_than_odr故障解决
差点被误操作的ORA-600 kcratr_nab_less_than_odr故障
ORA-600 kcratr_nab_less_than_odr和ORA-600 2662故障处理
ORA-600 kcratr_nab_less_than_odr和ORA-600 4193故障处理

Tue Aug 18 23:59:19 2026
ALTER DATABASE OPEN
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 19 processes
Started redo scan
Completed redo scan
 read 21634 KB redo, 800 data blocks need recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc  (incident=703037):
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_703037/orcl2_ora_275950_i703037.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Abort recovery for domain 0
Aborting crash recovery due to error 600
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_275950.trc:
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6916], [3], [4], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN...

现场恢复人员上来之后,直接尝试做强制resetlogs操作,数据库报ORA-600 ORA-600 krsi_al_hdr_update.15错误,主要是由于redo写丢失导致无法resetlogs成功,具体参考：Alter Database Open Resetlogs returns error ORA-00600: [krsi_al_hdr_update.15], (Doc ID 2026541.1) Oracle断电故障处理

Wed Aug 19 00:28:47 2026
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38935 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
ARCH: All Archive destinations made inactive due to error 742
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+ARCHDG/2_19951_1224781365.arc' (error 742)(orcl2)
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_287355.trc  (incident=727038):
ORA-00600: internal error code, arguments: [krsi_al_hdr_update.15], [4294967295], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_727038/orcl2_ora_287355_i727038.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Master archival failure: 600
Archive all online redo logfiles failed:600
ORA-600 signalled during: alter database open resetlogs...

通过 ALTER DATABASE RECOVER database using backup controlfile until cancel之后,继续尝试强制打开库，报ORA-600 2662错误.

Wed Aug 19 00:40:25 2026
Checker run found 32 new persistent data failures
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 29721342127
Archived Log entry 38936 added for thread 1 sequence 6915 ID 0xceea62af dest 1:
Archived Log entry 38937 added for thread 1 sequence 6916 ID 0xceea62af dest 1:
Archived Log entry 38938 added for thread 2 sequence 19951 ID 0xceea62af dest 1:
Archived Log entry 38939 added for thread 2 sequence 19950 ID 0xceea62af dest 1:
Clearing online redo logfile 1 +DATADG/orcl/onlinelog/group_1.319.1224781365
Clearing online log 1 of thread 1 sequence number 6915
Wed Aug 19 00:40:34 2026
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATADG/orcl/onlinelog/group_2.320.1224781367
Clearing online log 2 of thread 1 sequence number 6916
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 +DATADG/orcl/onlinelog/group_3.323.1224781463
Clearing online log 3 of thread 2 sequence number 19951
Clearing online redo logfile 3 complete
Clearing online redo logfile 4 +DATADG/orcl/onlinelog/group_4.324.1224781465
Clearing online log 4 of thread 2 sequence number 19950
Clearing online redo logfile 4 complete
Resetting resetlogs activation ID 3471467183 (0xceea62af)
Online log +DATADG/orcl/onlinelog/group_1.319.1224781365: Thread 1 Group 1 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_1.4574.1224781367: Thread 1 Group 1 was previously cleared
Online log +DATADG/orcl/onlinelog/group_2.320.1224781367: Thread 1 Group 2 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_2.10729.1224781369: Thread 1 Group 2 was previously cleared
Online log +DATADG/orcl/onlinelog/group_3.323.1224781463: Thread 2 Group 3 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_3.12866.1224781463: Thread 2 Group 3 was previously cleared
Online log +DATADG/orcl/onlinelog/group_4.324.1224781465: Thread 2 Group 4 was previously cleared
Online log +ARCHDG/orcl/onlinelog/group_4.7527.1224781465: Thread 2 Group 4 was previously cleared
Wed Aug 19 00:40:43 2026
Setting recovery target incarnation to 3
Wed Aug 19 00:40:43 2026
Assigning activation ID 3488382552 (0xcfec7e58)
Thread 2 opened at log sequence 1
  Current log# 3 seq# 1 mem# 0: +DATADG/orcl/onlinelog/group_3.323.1224781463
  Current log# 3 seq# 1 mem# 1: +ARCHDG/orcl/onlinelog/group_3.12866.1224781463
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 19 00:40:43 2026
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc  (incident=739007):
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl2/incident/incdir_739007/orcl2_ora_291405_i739007.trc
Wed Aug 19 00:40:45 2026
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_291405.trc:
ORA-00600: internal error code, arguments: [2662], [6], [3951555809], [6], [3951556555], [12583040], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 291405): terminating the instance due to error 600
Instance terminated by USER, pid = 291405
ORA-1092 signalled during: alter database open resetlogs...

到这一步,现场停止了继续尝试,我接手故障处理.先dbv检测坏块,发现system有两个坏块

[oracle@db3 ~]$ dbv userid=sys/oracle file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 10:39:26 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Page 94587 is marked corrupt
Corrupt block relative dba: 0x0041717b (file 1, block 94587)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0

Page 95021 is marked corrupt
Corrupt block relative dba: 0x0041732d (file 1, block 95021)
Bad header found during dbv: 
Data in bad block:
 type: 11 format: 2 rdba: 0x00400001
 last change scn: 0x0000.00000000 seq: 0x1 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x00000b01
 check value in block header: 0xd49
 computed block checksum: 0x0



DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72079
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21854
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 2
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 0 (0.0)

OBET> dbv file 1

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
Endian:     little-endian (x86)
Target File: #1(only)
===============================================

Verifying file #1: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (157441 blocks) - Started: 2026-08-22 11:14:48
File #32: rfile=1 (0x00000001)  header_block_num=157440 (0x00026700)  filesize_status:OK
  Progress: 100000 / 157441 blocks checked...
  File #1completed: 0 all zero, 0 soft corrupted, 2 tailchk error, 0 checksum error, 0 rdba error

DBV completed at: 2026-08-22 11:14:59

===============================================
DBV Summary:
Total blocks checked: 157439
Total all zero blocks found: 0
Total all rdba error blocks found: 0
Total all tailchk error blocks found: 2
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 0
Total bad blocks found: 2
Execution time: 11.00 seconds
Throughput: 111.82 MB/s
===============================================

Detailed report saved to: dbv_file_32_20260822111448.log
Files processed: 1

OBET> list corrupt
file 1(/datapool/cold_backup_20260818/SYSTEM.314.1224781265) total bad blocks: 2
block#   bad block type
94587   tailchk
95021   tailchk

使用obet修复坏块
Oracle Block Editor Tool使用手册

OBET> set file 32
filename set to: /datapool/cold_backup_20260818/SYSTEM.314.1224781265 (file#1)

OBET> set block 94587
block set to: 94587

OBET> set mode edit
mode set to: edit

OBET> repair block
Warning: Missing value for 'block', using global setting: 94587

Repairing block 94587 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 94587:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0x0106C0E4 -> needs repair (0xE4C00601)
3. Checksum check: 0x5EC5 -> needs repair (0x7DE6)

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 94587
Operations needed: fix tailchk, fix checksum
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x0106C0E4 OK
3. Checksum: 0x7DE6 OK

Block 94587 repair completed successfully.

OBET> set block 95021
block set to: 95021

OBET> repair block 
Warning: Missing value for 'block', using global setting: 95021

Repairing block 95021 in file /datapool/cold_backup_20260818/SYSTEM.314.1224781265...

Repair analysis for block 95021:
1. seq_kcbh check: 0x01 -> OK
2. Tailchk check: 0xF8FA0601 -> needs repair (0xE06C0601)
3. Checksum check: 0x81BA -> OK

Confirm repair operations:
File: /datapool/cold_backup_20260818/SYSTEM.314.1224781265
Block: 95021
Operations needed: fix tailchk
Confirm? (Y/YES to proceed): y

Verification after repair:
1. seq_kcbh: 0x01 OK
2. Tailchk: 0x01066CE0 OK
3. Checksum: 0x81BA OK

Block 95021 repair completed successfully.

dbv检查确认坏块修复成功

[oracle@db3 ~]$ dbv file=/datapool/cold_backup_20260818/SYSTEM.314.1224781265

DBVERIFY: Release 11.2.0.4.0 - Production on Sat Aug 22 11:17:30 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /datapool/cold_backup_20260818/SYSTEM.314.1224781265


DBVERIFY - Verification complete

Total Pages Examined         : 157440
Total Pages Processed (Data) : 72080
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 21855
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 49484
Total Pages Processed (Seg)  : 1
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 14021
Total Pages Marked Corrupt   : 0
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 3951556890 (6.3951556890)

后面的恢复比较简单,使用客户恢复之前的备份,直接重建ctl,然后打开库成功,并且做rman校验没有异常,直接把恢复之后的库备份还原到asm里面,完成本次恢复任务