分类目录归档：Oracle备份恢复

硬件故障数据库异常恢复

发表于 2019 年 2 月 10 日 由 惜分飞

硬件故障数据库crash
有客户由于硬件故障导致数据库异常ORA-00345 ORA-00312 ORA-27070 OSD-04016

Tue Feb 05 16:58:26 2019
Thread 1 advanced to log sequence 17139 (LGWR switch)
  Current log# 12 seq# 17139 mem# 0: S:\ORADATA\ORCL\REDO12A.LOG
  Current log# 12 seq# 17139 mem# 1: S:\ORADATA\ORCL\REDO12B.LOG
Tue Feb 05 19:47:24 2019
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_lgwr_2420.trc:
ORA-00345: redo log write error block 152097 count 8
ORA-00312: online log 12 thread 1: 'S:\ORADATA\ORCL\REDO12A.LOG'
ORA-27070: async read/write failed
OSD-04016: 异步 I/O 请求排队时出错。
O/S-Error: (OS 1) 函数不正确。
ORA-00345: redo log write error block 152097 count 8
ORA-00312: online log 12 thread 1: 'S:\ORADATA\ORCL\REDO12B.LOG'
ORA-27070: async read/write failed
OSD-04016: 异步 I/O 请求排队时出错。
O/S-Error: (OS 1) 函数不正确。
ORA-00345: redo log write error block 152105 count 1
ORA-00312: online log 12 thread 1: 'S:\ORADATA\ORCL\REDO12A.LOG'
ORA-27070: async read/write failed
OSD-04016: 异步 I/O 请求排队时出错。
O/S-Error: (OS 1) 函数不正确。

直接启动数据库报错
修复好硬件之后,直接启动数据库报ORA-00600 kcratr_scan_lastbwr错误 

Fri Feb 08 20:58:15 2019
alter database mount exclusive
Successful mount of redo thread 1, with mount id 1527506791
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database mount exclusive
alter database open
Beginning crash recovery of 1 threads
Started redo scan
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_3672.trc  (incident=41353):
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_41353\orcl_ora_3672_i41353.trc
Aborting crash recovery due to error 600
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_3672.trc:
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_3672.trc:
ORA-00600: ??????, ??: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open...
Fri Feb 08 20:58:24 2019
Trace dumping is performing id=[cdmp_20190208205824]
Fri Feb 08 20:59:04 2019
alter database open 
Beginning crash recovery of 1 threads
Started redo scan
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_1696.trc  (incident=41354):
ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_41354\orcl_ora_1696_i41354.trc
Aborting crash recovery due to error 600
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_1696.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_ora_1696.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: alter database open ...

recover database报错
执行recover database报错ORA-00600 6101,ORA-00600 kdourp_inorder2,ORA-00600 ktbsdp1,ORA-00600 3020

Fri Feb 08 21:09:20 2019
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 4 slaves
Fri Feb 08 21:09:21 2019
Recovery of Online Redo Log: Thread 1 Group 12 Seq 17139 Reading mem 0
  Mem# 0: S:\ORADATA\ORCL\REDO12A.LOG
  Mem# 1: S:\ORADATA\ORCL\REDO12B.LOG
Fri Feb 08 21:09:21 2019
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr02_3780.trc  (incident=49379):
ORA-00600: internal error code, arguments: [6101], [17], [21], [0], [], [], [], [], [], [], [], []
Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49379\orcl_pr02_3780_i49379.trc
Fri Feb 08 21:09:21 2019
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr01_2040.trc  (incident=49371):
ORA-00600: internal error code, arguments: [kdourp_inorder2], [34], [0], [0], [44], [], [], [], [], [], [], []
Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49371\orcl_pr01_2040_i49371.trc
Fri Feb 08 21:09:21 2019
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr03_1068.trc  (incident=49387):
ORA-00600: internal error code, arguments: [ktbsdp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49387\orcl_pr03_1068_i49387.trc
Fri Feb 08 21:09:24 2019
Trace dumping is performing id=[cdmp_20190208210924]
Slave exiting with ORA-10562 exception
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr03_1068.trc:
ORA-10562: Error occurred while applying redo to data block (file# 4, block# 1716972)
ORA-10564: tablespace USERS
ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 204127
ORA-00600: internal error code, arguments: [ktbsdp1], [], [], [], [], [], [], [], [], [], [], []
Slave exiting with ORA-10562 exception
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr02_3780.trc:
ORA-10562: Error occurred while applying redo to data block (file# 4, block# 1738552)
ORA-10564: tablespace USERS
ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 211606
ORA-00600: internal error code, arguments: [6101], [17], [21], [0], [], [], [], [], [], [], [], []
Slave exiting with ORA-10562 exception
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr01_2040.trc:
ORA-10562: Error occurred while applying redo to data block (file# 4, block# 1725898)
ORA-10564: tablespace USERS
ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 73907
ORA-00600: internal error code, arguments: [kdourp_inorder2], [34], [0], [0], [44], [], [], [], [], [], [], []
Recovery Slave PR03 previously exited with exception 10562
Fri Feb 08 21:09:28 2019
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr04_2608.trc  (incident=49395):
ORA-00600: internal error code, arguments: [3020], [4], [1739291], [18516507], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 4, block# 1739291, file offset is 1363369984 bytes)
ORA-10564: tablespace USERS
ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 211552
Incident details in: c:\oracle\diag\rdbms\orcl\orcl\incident\incdir_49395\orcl_pr04_2608_i49395.trc
Slave exiting with ORA-600 exception
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr04_2608.trc:
ORA-00600: internal error code, arguments: [3020], [4], [1739291], [18516507], [], [], [], [], [], [], [], []
ORA-10567: Redo is inconsistent with data block (file# 4, block# 1739291, file offset is 1363369984 bytes)
ORA-10564: tablespace USERS
ORA-01110: data file 4: 'S:\ORADATA\ORCL\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 211552
Media Recovery failed with error 448
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr00_1548.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
Slave exiting with ORA-283 exception
Errors in file c:\oracle\diag\rdbms\orcl\orcl\trace\orcl_pr00_1548.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
ORA-10562 signalled during: ALTER DATABASE RECOVER  database  ...

出现上述问题主要是由于硬件突然故障,数据写丢失导致相关问题.

处理思路

RMAN> recover datafile 1;

启动 recover 于 09-2月 -19
使用通道 ORA_DISK_1

正在开始介质的恢复
介质恢复完成, 用时: 00:00:01

完成 recover 于 09-2月 -19

RMAN> recover datafile 2;

启动 recover 于 09-2月 -19
使用通道 ORA_DISK_1

正在开始介质的恢复
介质恢复完成, 用时: 00:00:01

完成 recover 于 09-2月 -19

RMAN> recover datafile 3;

启动 recover 于 09-2月 -19
使用通道 ORA_DISK_1

正在开始介质的恢复
介质恢复完成, 用时: 00:00:02

完成 recover 于 09-2月 -19

RMAN> recover datafile 4;

启动 recover 于 09-2月 -19
使用通道 ORA_DISK_1

正在开始介质的恢复
无法恢复介质
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: recover 命令 (在 02/09/2019 21:48:19 上) 失败
ORA-00283: recovery session canceled due to errors
RMAN-11003: 在分析/执行 SQL 语句期间失败: alter database recover if needed
 datafile 4
ORA-00283: 恢复会话因错误而取消
ORA-10562: Error occurred while applying redo to data block (file# 4, block# 172
5913)
ORA-10564: tablespace USERS
ORA-01110: 数据文件 4: 'S:\ORADATA\ORCL\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 73907
ORA-00600: 内部错误代码, 参数: [kdourp_inorder2], [34], [43], [44], [44], [], []
, [], [], [], [], []

SQL> recover datafile 4;
ORA-00283: 恢复会话因错误而取消
ORA-10562: Error occurred while applying redo to data block (file# 4, block#
1725913)
ORA-10564: tablespace USERS
ORA-01110: 数据文件 4: 'S:\ORADATA\ORCL\USERS01.DBF'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 73907
ORA-00600: 内部错误代码, 参数: [kdourp_inorder2], [34], [43], [44], [44], [],
[], [], [], [], [], []

--通过bbed修改异常文件,屏蔽文件恢复,直接open库
SQL> alter database open;

数据库已更改。

数据库open之后,逻辑方式导出数据,重建新库,导入数据.

发表在 非常规恢复 | 标签为 , , , , , , , , | 评论关闭

ORA-600 16513故障恢复

发表于 2019 年 1 月 30 日 由 惜分飞

接到一个朋友恢复请求，由于硬件故障，通过底层技术恢复数据文件之后，数据库启动报错，但是dbv检查没有发现坏块
ORA-600 16513报错

[oracle@ora11g ~]$ ss

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jan 25 15:06:15 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> startup
ORACLE instance started.

Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [],
[], [], [], [], [], []
Process ID: 20487
Session ID: 191 Serial number: 3

数据库alert日志报错

Fri Jan 25 14:15:18 2019
ALTER DATABASE OPEN
Thread 1 opened at log sequence 11
  Current log# 2 seq# 11 mem# 0: /u01/app/oracle/oradata/orcl11g/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc  (incident=4953):
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/incident/incdir_4953/orcl11g_ora_12712_i4953.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 12712): terminating the instance due to error 704
Instance terminated by USER, pid = 12712
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (12712) as a result of ORA-1092

trace文件报错

---orcl11g_ora_12712.trc文件报错
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []

ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []

*** 2019-01-25 14:15:20.875
USER (ospid: 12712): terminating the instance due to error 704

---orcl11g_ora_12712_i4953.trc文件报错
Dump continued from file: /u01/app/oracle/diag/rdbms/orcl11g/orcl11g/trace/orcl11g_ora_12712.trc
ORA-00600: internal error code, arguments: [16513], [1403], [20], [], [], [], [], [], [], [], [], []

========= Dump for incident 4953 (ORA 600 [16513]) ========

*** 2019-01-25 14:15:19.247
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=1h50ks4ncswfn) -----
ALTER DATABASE OPEN

----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+41        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksedst1()+103        call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksedst()+39          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbkedDefDump()+2746  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksedmp()+41          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
ksfdmp()+69          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFD7A0B37B0 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgexProcessError()  call     dbgexPhaseII()       7F06FBCD2730 ? 7F06FBCD5520 ?
+2680                                              7FFD7A0BCE08 ? 7FFD7A0B3888 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgeExecuteForError  call     dbgexProcessError()  7F06FBCD2730 ? 7F06FBCD5520 ?
()+88                                              000000001 ? 000000000 ?
                                                   7FFD7A0B8330 ? 000000002 ?
dbgePostErrorKGE()+  call     dbgeExecuteForError  7F06FBCD2730 ? 7F06FBCD5520 ?
2136                          ()                   000000001 ? 000000001 ?
                                                   000000000 ? 000000002 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   00C114E60 ? 7F06FB7B0040 ?
71                                                 000000258 ? 000000001 ?
                                                   000000000 ? 000000002 ?
kgeade()+351         call     dbkePostKGE_kgsf()   00C114E60 ? 7F06FB7B0040 ?
                                                   000000258 ? 000000001 ?
                                                   000000000 ? 000000002 ?
kgeriv_int()+125     call     kgeade()             00C114E60 ? 00C115020 ?
                                                   7F06FB7B0040 ? 000000258 ?
                                                   000000000 ? 000000002 ?
kgeriv()+17          call     kgeriv_int()         00C114E60 ? 00C115020 ?
                                                   7F06FB7B0040 ? 000000258 ?
                                                   000000000 ? 000000002 ?
kgesiv()+115         call     kgeriv()             00C114E60 ? 00C115020 ?
                                                   7F06FB7B0040 ? 000000258 ?
                                                   000000000 ? 000000002 ?
ksesic2()+199        call     kgesiv()             00C114E60 ? 7F06FB7B0040 ?
                                                   000004081 ? 000000002 ?
                                                   7FFD7A0BDAC0 ? 000000002 ?
kqdpts()+522         call     ksesic2()            00C114E60 ? 000000000 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kqrlfc()+685         call     kqdpts()             1158C5BC0 ? 000000000 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kqlbplc()+180        call     kqrlfc()             1158C5BC0 ? 7FFD7A0BDEC0 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kqlblfc()+280        call     kqlbplc()            000000000 ? 7FFD7A0BDEC0 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
adbdrv()+57408       call     kqlblfc()            000000000 ? 7FFD7A0C4BD4 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
opiexe()+18724       call     adbdrv()             000000000 ? 7FFD7A0C4BD4 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
opiosq0()+4303       call     opiexe()             000000004 ? 000000000 ?
                                                   00000057B ? 000000000 ?
                                                   00000003B ? 000008000 ?
kpooprx()+274        call     opiosq0()            000000003 ? 00000000E ?
                                                   7FFD7A0C65B0 ? 0000000A4 ?
                                                   00000003B ? 000008000 ?
kpoal8()+842         call     kpooprx()            7FFD7A0C9D94 ? 7FFD7A0C7DB8 ?
                                                   000000013 ? 000000001 ?
                                                   000000000 ? 000008000 ?
opiodr()+917         call     kpoal8()             00000005E ? 7FFD7A0C7DB8 ?
                                                   000000013 ? 000000001 ?
                                                   000000000 ? 000008000 ?
ttcpip()+2183        call     opiodr()             00000005E ? 00000001C ?
                                                   7FFD7A0C9D90 ? 000000001 ?
                                                   000000000 ? 000008000 ?
opitsk()+1710        call     ttcpip()             00C132AB0 ? 0099D6610 ?
                                                   7FFD7A0C9D90 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opiino()+969         call     opitsk()             00C132AB8 ? 000000000 ?
                                                   7FFD7A0C9D90 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opiodr()+917         call     opiino()             00000003C ? 000000004 ?
                                                   7FFD7A0CB588 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opidrv()+570         call     opiodr()             00000003C ? 000000004 ?
                                                   7FFD7A0CB588 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
sou2o()+103          call     opidrv()             00000003C ? 000000004 ?
                                                   7FFD7A0CB588 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
opimai_real()+133    call     sou2o()              7FFD7A0CB560 ? 00000003C ?
                                                   000000004 ? 7FFD7A0CB588 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
ssthrdmain()+265     call     opimai_real()        000000002 ? 7FFD7A0CB750 ?
                                                   000000004 ? 7FFD7A0CB588 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
main()+201           call     ssthrdmain()         000000002 ? 7FFD7A0CB750 ?
                                                   000000001 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
__libc_start_main()  call     main()               000000002 ? 7FFD7A0CB8F8 ?
+253                                               000000001 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
_start()+41          call     __libc_start_main()  000A2A5B4 ? 000000002 ?
                                                   7FFD7A0CB8E8 ? 000000000 ?
                                                   7FFD7A0C97E8 ? 7FFD7A0C9D8C ?
--------------------- Binary Stack Dump ---------------------

根据错误提示含ORA-00704错误,可以初步确定很可能是在数据库核心基表有问题导致该问题.trace数据库启动过程,发现在obj$中异常,通过跟踪定位到启动报错所在位置,然后通过bbed(参考:bbed 文章汇总)进行修改,再次尝试启动数据库
ORA-704 ORA-1555

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 10 with name
"_SYSSMU10_1197734989$" too small
Process ID: 17898
Session ID: 191 Serial number: 5

该错误比较熟悉,通过bbed继续修改相关信息(参考:在数据库open过程中常遇到ORA-01555汇总)启动数据库成功

SQL> startup mount;
ORACLE instance started.

Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
SQL> alter database open;

Database altered.

尝试ddl操作报错误
ORA-600 kkdlcob-objn-exists

SQL>  create table t1 as select * from dual;
 create table t1 as select * from dual
                                  *
ERROR at line 1:
ORA-00600: internal error code, arguments: [kkdlcob-objn-exists], [87520], [],
[], [], [], [], [], [], [], [], []

该错误是由于字典obj$的dataobj#太小导致,修复该问题之后,有出现以下问题
ORA-8102

SQL> create table t1 as select * from dual;
create table t1 as select * from dual
                                 *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 39, file 1, block 54781 (2)

这个错误比较明显由于obj$上的index异常导致,通过文章:bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决修复之后一切恢复正常

SQL> CREATE TABLE T1 AS SELECT * FROM DUAL;

Table created.

SQL> DROP TABLE T1 PURGE;

Table dropped.

至此数据库恢复基本完成，建议客户逻辑方式导出数据，导入到新库

发表在 非常规恢复 | 标签为 , , | 评论关闭

tab$恢复错误汇总

发表于 2019 年 1 月 11 日 由 惜分飞

在以前多以前发现的tab$被恶意脚本删除的问题(ORA-600 16703故障解析—tab$表被清空,警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),虽然多次强调注意Oracle安装介质安全,但是很不幸,还是大量客户中招.我们这一年多对于tab$的故障进行了大量case处理,拯救了大量客户的核心数据,也积累了一些常见的可能遭遇的错误.主要恢复思路是使用bbed处理异常block,让数据库open起来.
ORA-00704 ORA-39700
有核心基表处理异常导致

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Process ID: 1603
Session ID: 1 Serial number: 5

Sun Jan 06 21:30:14 2019
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 1603): terminating the instance due to error 704
Instance terminated by USER, pid = 1603
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (1603) as a result of ORA-1092
Sun Jan 06 21:30:14 2019
ORA-1092 : opitsk aborting process

ora-704 ora-604 ora-01555
由于scn异常导致

SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name
"_SYSSMU2_2996391332$" too small
Process ID: 26520
Session ID: 1 Serial number: 5

Sun Jan 06 19:49:12 2019
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: bqbdby3c400p7, SCN: 0x0022.1117ef75):
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 26520): terminating the instance due to error 704
Instance terminated by USER, pid = 26520
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (26520) as a result of ORA-1092

ORA-600 13304
有核心基表处理异常导致

SQL> startup mount;
ORACLE instance started.

Total System Global Area  521936896 bytes
Fixed Size                  2254824 bytes
Variable Size             352323608 bytes
Database Buffers          163577856 bytes
Redo Buffers                3780608 bytes
Database mounted.
SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 1724
Session ID: 1 Serial number: 5

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Jan 06 21:31:04 2019
SMON: enabling cache recovery
[1724] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2239884804 end:2239884864 diff:60 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc  (incident=61755):
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_61755/orcl_ora_1724_i61755.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1724): terminating the instance due to error 600
Instance terminated by USER, pid = 1724
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (1724) as a result of ORA-1092
Sun Jan 06 21:31:06 2019
ORA-1092 : opitsk aborting process

ORA-00704 ORA-600 kdBlkCheckError
恢复的block有逻辑坏块

SQL> startup         
ORACLE instance started.

Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497],
[6121], [], [], [], [], [], [], [], []
Process ID: 76932
Session ID: 191 Serial number: 3

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc  (incident=6153):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_6153/xifenfei_ora_76932_i6153.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 76932): terminating the instance due to error 704
Instance terminated by USER, pid = 76932
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (76932) as a result of ORA-1092
Sat Feb 22 11:04:19 2014
ORA-1092 : opitsk aborting process
发表在 非常规恢复 | 标签为 , , , , , , , , , , | 评论关闭