tab$恢复错误汇总

在以前多以前发现的tab$被恶意脚本删除的问题(ORA-600 16703故障解析—tab$表被清空,警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),虽然多次强调注意Oracle安装介质安全,但是很不幸,还是大量客户中招.我们这一年多对于tab$的故障进行了大量case处理,拯救了大量客户的核心数据,也积累了一些常见的可能遭遇的错误.主要恢复思路是使用bbed处理异常block,让数据库open起来.
ORA-00704 ORA-39700
有核心基表处理异常导致

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Process ID: 1603
Session ID: 1 Serial number: 5

Sun Jan 06 21:30:14 2019
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 1603): terminating the instance due to error 704
Instance terminated by USER, pid = 1603
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (1603) as a result of ORA-1092
Sun Jan 06 21:30:14 2019
ORA-1092 : opitsk aborting process

ora-704 ora-604 ora-01555
由于scn异常导致

SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name
"_SYSSMU2_2996391332$" too small
Process ID: 26520
Session ID: 1 Serial number: 5

Sun Jan 06 19:49:12 2019
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: bqbdby3c400p7, SCN: 0x0022.1117ef75):
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 26520): terminating the instance due to error 704
Instance terminated by USER, pid = 26520
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (26520) as a result of ORA-1092

ORA-600 13304
有核心基表处理异常导致

SQL> startup mount;
ORACLE instance started.

Total System Global Area  521936896 bytes
Fixed Size                  2254824 bytes
Variable Size             352323608 bytes
Database Buffers          163577856 bytes
Redo Buffers                3780608 bytes
Database mounted.
SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 1724
Session ID: 1 Serial number: 5

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Jan 06 21:31:04 2019
SMON: enabling cache recovery
[1724] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2239884804 end:2239884864 diff:60 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc  (incident=61755):
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_61755/orcl_ora_1724_i61755.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1724): terminating the instance due to error 600
Instance terminated by USER, pid = 1724
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (1724) as a result of ORA-1092
Sun Jan 06 21:31:06 2019
ORA-1092 : opitsk aborting process

ORA-00704 ORA-600 kdBlkCheckError
恢复的block有逻辑坏块

SQL> startup         
ORACLE instance started.

Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497],
[6121], [], [], [], [], [], [], [], []
Process ID: 76932
Session ID: 191 Serial number: 3

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc  (incident=6153):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_6153/xifenfei_ora_76932_i6153.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 76932): terminating the instance due to error 704
Instance terminated by USER, pid = 76932
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (76932) as a result of ORA-1092
Sat Feb 22 11:04:19 2014
ORA-1092 : opitsk aborting process
发表在 非常规恢复 | 标签为 , , , , , , , , , , | 评论关闭

10g数据库遭遇ORA-600 16703

ORA-600 16703
有客户反馈10g数据库启动报ORA-00704 ORA-00600,具体如下

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi
PL/SQL Release 10.2.0.3.0 - Production
CORE	10.2.0.3.0	Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio
NLSRTL Version 10.2.0.3.0 - Production
Fri Jan  4 10:11:05 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 10:11:05 2019
SMON: enabling cache recovery
Fri Jan  4 10:11:05 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 1826952
ORA-1092 signalled during: alter database open...

看到这个错误第一想到以前的ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [],但是有几个地方不同:1)根据以往恢复经验这个错误都出现在11.2.0.4版本数据库中;2)以往的ORA-600 16703错误最后值是20,而这次是28

分析数据库故障过程
节点一因为其他错误重启,重启之后报大量错误,然后abort掉.

--节点1
Fri Jan  4 06:11:00 2019
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=2883786
Fri Jan  4 06:11:07 2019
SMON: Parallel transaction recovery tried
Fri Jan  4 06:11:07 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2769106.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020E4E7258], [257], [0], [], [], [], []
Fri Jan  4 06:11:09 2019
ORA-600 signalled during: ALTER DATABASE OPEN...
Fri Jan  4 06:11:09 2019
Trace dumping is performing id=[cdmp_20190104061109]
Fri Jan  4 06:11:30 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2240550.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:12:35 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793508.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [65], [], [], []
Fri Jan  4 06:12:40 2019
Trace dumping is performing id=[cdmp_20190104061240]
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1933540.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793546.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:11 2019
Trace dumping is performing id=[cdmp_20190104061611]
Fri Jan  4 06:16:15 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:16:16 2019
Non-fatal internal error happenned while SMON was doing logging scn->time mapping.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Fri Jan  4 06:17:28 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2035774.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
…………
Fri Jan  4 06:56:47 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:56:49 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_3072030.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_pmon_3092524.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
PMON: terminating instance due to error 474
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_lms0_3178590.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
System state dump is made for local instance
System State dumped to trace file /oracle/admin/xifenfei/bdump/xifenfei1_diag_434192.trc
Fri Jan  4 06:56:51 2019
Shutting down instance (abort)

节点一重启触发故障之后,节点二也开始报错,然后数据库直接挂掉

--节点2
Fri Jan  4 06:21:19 2019
Trace dumping is performing id=[cdmp_20190104062118]
Fri Jan  4 06:21:22 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j000_1253440.trc:
ORA-00600: internal error code, arguments: [kghGetHpSz1], [0x7000001DF886220], [], [], [], [], [], []
Fri Jan  4 06:21:24 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_1835262.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:21:56 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_1777888.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Fri Jan  4 06:24:38 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:24:44 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-12012: error on auto execute of job 351
ORA-12008: error in materialized view refresh path
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:28:48 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j001_1417456.trc:
ORA-00600: internal error code, arguments: [kkzufst], [18446744073709551615], [], [], [], [], [], []
Fri Jan  4 06:28:49 2019
ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x1104B2580], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 07:30:25 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_2121800.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kggfaAllocFunc1], [], [], [], [], [], [], []
…………
Fri Jan  4 07:31:23 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
PMON: terminating instance due to error 472
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_569432.trc:
ORA-00472: PMON  process terminated with error

两个节点再次启动报ORA-600 16703错误

Fri Jan  4 07:31:50 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 07:31:50 2019
SMON: enabling cache recovery
Fri Jan  4 07:31:50 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_3780764.trc:
ORA-00704: bootstrap process failure
Instance terminated by USER, pid = 3813516
ORA-1092 signalled during: ALTER DATABASE OPEN...

10046跟踪启动过程
ora-600-16703-10046


分析故障原因
通过分析,确认该数据库被注入了恶意脚本,当发生重启之后导致数据库核心基表被破坏,从而使得数据库无法正常启动

procedure     DBMS_DBMONITOR wrapped
a000000
369
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
7
15a 171
VKGnETw5GkgNlUvXadIYpQ2thwAwgzKur9zWZ47SR+pHN0bgoPvQ8ezdufYxFkzCmAQA8xT2
IQkFph+2IhlEBrPrXe14giDow/HzZI43FwLiMlynCYjnzQh3aXRSIVOalcwGAfUvgCip6Eng
OWA8Vq49YJ38WCPZjq2P5Dc428Wa1ZOPMb+E7GPs8ZOWM7RWsQdMzx/pqFncbX/tLwp0NY5E
Uu4E54MZ34yVtDQybwljVqp06KHqWN/ZwZJpvT+2gO4hRNX2UyE7laWCXzM2IR05BTGf2yoZ
+E7eIn6kciinFmhcUiBuszxE0pykt+moWZuuDuj9ebUXmj+0Mx7A+eQc6tp7wLHRCHYb6p0D
VtDc4f6

通过对损坏的字典进行恢复,实现数据0丢失
open-database


发表在 非常规恢复 | 标签为 , , , , , , , | 评论关闭

2019恢复第一单—ORA-704-ORA-702恢复

19年元旦刚过就有客户反馈数据库无法正常启动报ORA-00704和ORA-00702错误
ora-00702


Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3756.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3756.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Error 704 happened during db open, shutting down database
USER (ospid: 3756): terminating the instance due to error 704
Instance terminated by USER, pid = 3756

通过分析确认,确认由于数据库遭受到而已脚本注入,破坏数据库基表数据,导致数据库重启之后就无法正常启动,
1


这类故障可以通过对oracle基表进行恢复,实现数据库完美open,数据0丢失.如果无法自行解决,需要恢复支持请联系我们
Phone:13429648788    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com
2019年再次提醒:1)数据库做好备份容灾,2)请使用官方途径下载数据库介质和数据库访问工具,3)定期检查数据库是否被注入恶意脚本

发表在 非常规恢复 | 标签为 , , | 评论关闭

正常open库报ORA-600 2662,ORA-600 4097错误

又一起数据库正常open之后继续报ORA-600 2662

Sat Dec 29 04:30:40 2018
QMNC started with pid=36, OS id=12985 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open
Sat Dec 29 04:30:42 2018
Starting background process CJQ0
Sat Dec 29 04:30:42 2018
CJQ0 started with pid=41, OS id=12999 
Sat Dec 29 04:30:43 2018
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m000_12989.trc  (incident=365101):
ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], []
Sat Dec 29 04:30:43 2018
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m002_12993.trc  (incident=365117):
ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], []
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365101/orcl_m000_12989_i365101.trc
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365117/orcl_m002_12993_i365117.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m000_12989.trc  (incident=365102):
ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], []
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365102/orcl_m000_12989_i365102.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m002_12993.trc  (incident=365118):
ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], []
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365118/orcl_m002_12993_i365118.trc

在open状态处理掉ORA-00600 2662错误之后,又出现了ORA-600 4097 以及ORA-00353 ORA-00312错误,这个也是一种不常见的现象,一般都是数据库open之前出现类似错误,但是这里在open之后出现(以前类似处理参考:ORA-600 4097故障处理ORA-00322 ORA-00312 ,ORA-00354 ORA-00353 ORA-00312异常处理

Sat Dec 29 05:55:22 2018
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j000_19136.trc  (incident=432567):
ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], []
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432567/orcl_j000_19136_i432567.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j000_19136.trc  (incident=432568):
ORA-00353: log corruption near block 15682 change 193647101 time 12/29/2018 04:40:31
ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log'
ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log'
ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], []
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432568/orcl_j000_19136_i432568.trc
Errors in file /home/app/diag/rdbms/orcl/orcl/incident/incdir_432567/orcl_j000_19136_i432567.trc:
ORA-00399: corrupt change description in redo log
ORA-00353: log corruption near block 15682 change 193647101 time 12/29/2018 04:40:31
ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log'
ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log'
ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], []
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j000_19136.trc  (incident=432569):
ORA-00353: log corruption near block 15682 change 193647101 time 12/29/2018 04:40:31
ORA-00334: archived log: '/home/app/oradata/orcl/redo03.log'
ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log'
ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], []
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432569/orcl_j000_19136_i432569.trc

处理完成上面报错之后,继续出现比较少见的ORA-600 ktecgsc:kcbz_objdchk错误,具体参考:ORA-00600 [ktecgsc:kcbz_objdchk] on 11.2 (Doc ID 1562473.1)

Sat Dec 29 06:00:17 2018
DBMS_STATS: GATHER_STATS_JOB encountered errors.  Check the trace file.
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j001_19436.trc:
ORA-20011: Approximate NDV failed: ORA-08103: object no longer exists
Sat Dec 29 06:00:21 2018
Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j002_19479.trc  (incident=432631):
ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], []
Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432631/orcl_j002_19479_i432631.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
发表在 Oracle备份恢复 | 标签为 , , , , | 评论关闭

Oracle Exadata坏盘导致磁盘组无法mount恢复

接到朋友求救有客户oracle exadata一体机 的 asm磁盘组无法mount,希望我们提供恢复支持服务
经过分析和了解,大致问题是:磁盘空间已经超容量使用(部分数据不能完成ASM镜像),最近又损坏一块盘,导致asm 磁盘组无法mount。我们分析后,通过重构exadata celldisk数据,将asm 磁盘组 mount成功后,实现五套数据库全部open成功(由于底层磁盘部分数据损坏,导致部分数据访问报错,需要在oracle层面进行处理)。

本次问题的具体分析和处理如下:
存放数据库文件的磁盘组不能mount

Wed Dec 12 21:29:04 2018
SQL> alter diskgroup DATA_XFF mount force 
NOTE: cache registered group DATA_XFF number=1 incarn=0x5fe882cb
NOTE: cache began mount (first) of group DATA_XFF number=1 incarn=0x5fe882cb
NOTE: Assigning number (1,36) to disk (o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03)
NOTE: Assigning number (1,34) to disk (o/192.168.10.5/DATA_XFF_CD_10_XFFCEL03)
NOTE: Assigning number (1,37) to disk (o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03)
NOTE: Assigning number (1,38) to disk (o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03)
NOTE: Assigning number (1,39) to disk (o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03)
NOTE: Assigning number (1,40) to disk (o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03)
NOTE: Assigning number (1,41) to disk (o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03)
NOTE: Assigning number (1,42) to disk (o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03)
NOTE: Assigning number (1,43) to disk (o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03)
NOTE: Assigning number (1,44) to disk (o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03)
NOTE: Assigning number (1,45) to disk (o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03)
NOTE: Assigning number (1,46) to disk (o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03)
NOTE: Assigning number (1,22) to disk (o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02)
NOTE: Assigning number (1,18) to disk (o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02)
NOTE: Assigning number (1,19) to disk (o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02)
NOTE: Assigning number (1,15) to disk (o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02)
NOTE: Assigning number (1,20) to disk (o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02)
NOTE: Assigning number (1,17) to disk (o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02)
NOTE: Assigning number (1,16) to disk (o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02)
NOTE: Assigning number (1,23) to disk (o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02)
NOTE: Assigning number (1,12) to disk (o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02)
NOTE: Assigning number (1,21) to disk (o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02)
NOTE: Assigning number (1,13) to disk (o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02)
NOTE: Assigning number (1,14) to disk (o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02)
NOTE: Assigning number (1,1) to disk (o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01)
NOTE: Assigning number (1,2) to disk (o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01)
NOTE: Assigning number (1,3) to disk (o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01)
NOTE: Assigning number (1,4) to disk (o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01)
NOTE: Assigning number (1,5) to disk (o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01)
NOTE: Assigning number (1,6) to disk (o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01)
NOTE: Assigning number (1,7) to disk (o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01)
NOTE: Assigning number (1,8) to disk (o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01)
NOTE: Assigning number (1,9) to disk (o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01)
NOTE: Assigning number (1,10) to disk (o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01)
NOTE: Assigning number (1,11) to disk (o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01)
Wed Dec 12 21:29:10 2018
NOTE: GMON heartbeating for grp 1
GMON querying group 1 at 101 for pid 27, osid 62541
NOTE: Assigning number (1,0) to disk ()
GMON querying group 1 at 102 for pid 27, osid 62541
NOTE: process _user62541_+asm2 (62541) initiating offline of disk 0.3915937355 () with mask 0x7e[0x7f] in group 1
NOTE: initiating PST update: grp = 1, dsk = 0/0xe968764b, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 103 for pid 27, osid 62541
ERROR: Disk 0 cannot be offlined, since all the disks [0, 25] with mirrored data would be offline.
ERROR: too many offline disks in PST (grp 1)
WARNING: Offline of disk 0 () in group 1 and mode 0x7f failed on ASM inst 2
NOTE: cache dismounting (not clean) group 1/0x5FE882CB (DATA_XFF) 
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0x5FE882CB (DATA_XFF) 
NOTE: cache ending mount (fail) of group DATA_XFF number=1 incarn=0x5fe882cb
NOTE: cache deleting context for group DATA_XFF 1/0x5fe882cb
GMON dismounting group 1 at 104 for pid 27, osid 62541
ERROR: diskgroup DATA_XFF was not mounted
ORA-15032: not all alterations performed
ORA-15040: diskgroup is incomplete
ORA-15066: offlining disk "0" in group "DATA_XFF" may result in a data loss
ORA-15042: ASM disk "0" is missing from group number "1" 
ERROR: alter diskgroup DATA_XFF mount force

检查底层损坏情况

CellCLI>   list physicaldisk
         20:0            KN3VZL          normal
         20:1            KNAWLL          normal
         20:2            KN4E4L          warning - predictive failure, poor performance
         20:3            KNAN5L          normal
         20:4            KMJKYL          normal
         20:5            KN5DGL          normal
         20:6            KMDLWL          normal
         20:7            KMDKPL          normal
         20:8            KMDA7L          normal
         20:9            KN1YJL          normal
         20:10           KMH1YL          normal
         20:11           KMVHAL          normal

CellCLI>   list griddisk
         DATA_XFF_CD_00_XFFCEL01       active
         DATA_XFF_CD_01_XFFCEL01       active
         DATA_XFF_CD_02_XFFCEL01       proactive failure
         DATA_XFF_CD_03_XFFCEL01       active
         DATA_XFF_CD_04_XFFCEL01       active
         DATA_XFF_CD_05_XFFCEL01       active
         DATA_XFF_CD_06_XFFCEL01       active
         DATA_XFF_CD_07_XFFCEL01       active
         DATA_XFF_CD_08_XFFCEL01       active
         DATA_XFF_CD_09_XFFCEL01       active
         DATA_XFF_CD_10_XFFCEL01       active
         DATA_XFF_CD_11_XFFCEL01       active

在db节点无法发现异常磁盘的asm disk

[grid@ycdwdb01 grid]$ kfod disk=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group   
============================================================
   1:     433152 Mb o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 <unknown> <unknown>
   2:     433152 Mb o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 <unknown> <unknown>
   3:     433152 Mb o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 <unknown> <unknown>
   4:     433152 Mb o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 <unknown> <unknown>
   5:     433152 Mb o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 <unknown> <unknown>
   6:     433152 Mb o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 <unknown> <unknown>
   7:     433152 Mb o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 <unknown> <unknown>
   8:     433152 Mb o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 <unknown> <unknown>
   9:     433152 Mb o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 <unknown> <unknown>
  10:     433152 Mb o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 <unknown> <unknown>
  11:     433152 Mb o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 <unknown> <unknown>

根据客户的反馈该磁盘组几乎全部被使用,asmcmd lsdg看到Usable_file_MB已经出现负值.证明该磁盘组本身的normal没有完全存储两份数据,在这样的情况下,继续坏盘会导致部分数据只有一份,因此也就出现了这里的磁盘组无法正常mount成功.

通过底层修复celldisk之后

CellCLI>  list griddisk
         DATA_XFF_CD_00_XFFCEL01       active
         DATA_XFF_CD_01_XFFCEL01       active
         DATA_XFF_CD_02_XFFCEL01       active
         DATA_XFF_CD_03_XFFCEL01       active
         DATA_XFF_CD_04_XFFCEL01       active
         DATA_XFF_CD_05_XFFCEL01       active
         DATA_XFF_CD_06_XFFCEL01       active
         DATA_XFF_CD_07_XFFCEL01       active
         DATA_XFF_CD_08_XFFCEL01       active
         DATA_XFF_CD_09_XFFCEL01       active
         DATA_XFF_CD_10_XFFCEL01       active
         DATA_XFF_CD_11_XFFCEL01       active

[grid@ycdwdb01 grid]$ kfod disk=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group   
============================================================
   1:     433152 Mb o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 <unknown> <unknown>
   2:     433152 Mb o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 <unknown> <unknown>
   3:     433152 Mb o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01 <unknown> <unknown>
   4:     433152 Mb o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 <unknown> <unknown>
   5:     433152 Mb o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 <unknown> <unknown>
   6:     433152 Mb o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 <unknown> <unknown>
   7:     433152 Mb o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 <unknown> <unknown>
   8:     433152 Mb o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 <unknown> <unknown>
   9:     433152 Mb o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 <unknown> <unknown>
  10:     433152 Mb o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 <unknown> <unknown>
  11:     433152 Mb o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 <unknown> <unknown>
  12:     433152 Mb o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 <unknown> <unknown>

data磁盘组直接mount成功

Fri Dec 14 14:04:59 2018
SQL> alter diskgroup DATA_XFF mount 
NOTE: cache registered group DATA_XFF number=1 incarn=0x78a886e7
NOTE: cache began mount (not first) of group DATA_XFF number=1 incarn=0x78a886e7
NOTE: Assigning number (1,36) to disk (o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03)
NOTE: Assigning number (1,34) to disk (o/192.168.10.5/DATA_XFF_CD_10_XFFCEL03)
NOTE: Assigning number (1,37) to disk (o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03)
NOTE: Assigning number (1,38) to disk (o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03)
NOTE: Assigning number (1,39) to disk (o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03)
NOTE: Assigning number (1,40) to disk (o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03)
NOTE: Assigning number (1,41) to disk (o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03)
NOTE: Assigning number (1,42) to disk (o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03)
NOTE: Assigning number (1,43) to disk (o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03)
NOTE: Assigning number (1,44) to disk (o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03)
NOTE: Assigning number (1,45) to disk (o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03)
NOTE: Assigning number (1,46) to disk (o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03)
NOTE: Assigning number (1,22) to disk (o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02)
NOTE: Assigning number (1,18) to disk (o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02)
NOTE: Assigning number (1,19) to disk (o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02)
NOTE: Assigning number (1,15) to disk (o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02)
NOTE: Assigning number (1,20) to disk (o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02)
NOTE: Assigning number (1,17) to disk (o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02)
NOTE: Assigning number (1,16) to disk (o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02)
NOTE: Assigning number (1,23) to disk (o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02)
NOTE: Assigning number (1,12) to disk (o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02)
NOTE: Assigning number (1,21) to disk (o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02)
NOTE: Assigning number (1,13) to disk (o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02)
NOTE: Assigning number (1,14) to disk (o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02)
NOTE: Assigning number (1,1) to disk (o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01)
NOTE: Assigning number (1,2) to disk (o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01)
NOTE: Assigning number (1,3) to disk (o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01)
NOTE: Assigning number (1,4) to disk (o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01)
NOTE: Assigning number (1,5) to disk (o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01)
NOTE: Assigning number (1,6) to disk (o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01)
NOTE: Assigning number (1,7) to disk (o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01)
NOTE: Assigning number (1,8) to disk (o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01)
NOTE: Assigning number (1,9) to disk (o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01)
NOTE: Assigning number (1,10) to disk (o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01)
NOTE: Assigning number (1,11) to disk (o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01)
NOTE: Assigning number (1,0) to disk (o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01)
Fri Dec 14 14:04:59 2018
GMON querying group 1 at 78 for pid 28, osid 76016
NOTE: Assigning number (1,24) to disk ()
NOTE: Assigning number (1,25) to disk ()
NOTE: Assigning number (1,26) to disk ()
NOTE: Assigning number (1,27) to disk ()
NOTE: Assigning number (1,28) to disk ()
NOTE: Assigning number (1,29) to disk ()
NOTE: Assigning number (1,30) to disk ()
NOTE: Assigning number (1,31) to disk ()
NOTE: Assigning number (1,32) to disk ()
NOTE: Assigning number (1,33) to disk ()
NOTE: Assigning number (1,35) to disk ()
GMON querying group 1 at 79 for pid 28, osid 76016
NOTE: cache opening disk 0 of grp 1: DATA_XFF_CD_02_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01
NOTE: cache opening disk 1 of grp 1: DATA_XFF_CD_05_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01
NOTE: cache opening disk 2 of grp 1: DATA_XFF_CD_03_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01
NOTE: F1X0 found on disk 2 au 5 fcn 0.15948262
NOTE: cache opening disk 3 of grp 1: DATA_XFF_CD_06_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01
NOTE: cache opening disk 4 of grp 1: DATA_XFF_CD_09_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01
NOTE: cache opening disk 5 of grp 1: DATA_XFF_CD_04_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01
NOTE: cache opening disk 6 of grp 1: DATA_XFF_CD_07_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01
NOTE: cache opening disk 7 of grp 1: DATA_XFF_CD_11_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01
NOTE: cache opening disk 8 of grp 1: DATA_XFF_CD_01_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01
NOTE: cache opening disk 9 of grp 1: DATA_XFF_CD_00_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01
NOTE: cache opening disk 10 of grp 1: DATA_XFF_CD_10_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01
NOTE: cache opening disk 11 of grp 1: DATA_XFF_CD_08_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01
NOTE: cache opening disk 12 of grp 1: DATA_XFF_CD_00_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02
NOTE: cache opening disk 13 of grp 1: DATA_XFF_CD_01_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02
NOTE: cache opening disk 14 of grp 1: DATA_XFF_CD_02_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02
NOTE: cache opening disk 15 of grp 1: DATA_XFF_CD_03_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02
NOTE: cache opening disk 16 of grp 1: DATA_XFF_CD_04_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02
NOTE: cache opening disk 17 of grp 1: DATA_XFF_CD_05_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02
NOTE: cache opening disk 18 of grp 1: DATA_XFF_CD_06_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02
NOTE: cache opening disk 19 of grp 1: DATA_XFF_CD_07_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02
NOTE: cache opening disk 20 of grp 1: DATA_XFF_CD_08_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02
NOTE: cache opening disk 21 of grp 1: DATA_XFF_CD_09_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02
NOTE: F1X0 found on disk 21 au 2 fcn 0.15948262
NOTE: cache opening disk 22 of grp 1: DATA_XFF_CD_10_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02
NOTE: cache opening disk 23 of grp 1: DATA_XFF_CD_11_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02
NOTE: cache opening disk 36 of grp 1: DATA_XFF_CD_11_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03
NOTE: cache opening disk 37 of grp 1: DATA_XFF_CD_04_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03
NOTE: cache opening disk 38 of grp 1: DATA_XFF_CD_00_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03
NOTE: cache opening disk 39 of grp 1: DATA_XFF_CD_03_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03
NOTE: cache opening disk 40 of grp 1: DATA_XFF_CD_05_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03
NOTE: cache opening disk 41 of grp 1: DATA_XFF_CD_08_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03
NOTE: cache opening disk 42 of grp 1: DATA_XFF_CD_01_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03
NOTE: cache opening disk 43 of grp 1: DATA_XFF_CD_09_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03
NOTE: cache opening disk 44 of grp 1: DATA_XFF_CD_06_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03
NOTE: F1X0 found on disk 44 au 2 fcn 0.15948262
NOTE: cache opening disk 45 of grp 1: DATA_XFF_CD_07_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03
NOTE: cache opening disk 46 of grp 1: DATA_XFF_CD_02_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03
NOTE: cache mounting (not first) normal redundancy group 1/0x78A886E7 (DATA_XFF)
Fri Dec 14 14:04:59 2018
kjbdomatt send to inst 2
Fri Dec 14 14:04:59 2018
NOTE: attached to recovery domain 1
NOTE: redo buffer size is 512 blocks (2101760 bytes)
Fri Dec 14 14:04:59 2018
NOTE: LGWR attempting to mount thread 2 for diskgroup 1 (DATA_XFF)
NOTE: LGWR found thread 2 closed at ABA 98.4672
NOTE: LGWR mounted thread 2 for diskgroup 1 (DATA_XFF)
NOTE: LGWR opening thread 2 at fcn 0.18931129 ABA 99.4673
NOTE: cache mounting group 1/0x78A886E7 (DATA_XFF) succeeded
NOTE: cache ending mount (success) of group DATA_XFF number=1 incarn=0x78a886e7
GMON querying group 1 at 80 for pid 19, osid 9805
Fri Dec 14 14:04:59 2018
NOTE: Instance updated compatible.asm to 11.2.0.3.0 for grp 1
SUCCESS: diskgroup DATA_XFF was mounted
SUCCESS: alter diskgroup DATA_XFF mount

恢复后的asm磁盘状态

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  NORMAL  Y         512   4096  4194304  15160320  4776184          5197824         -210820             12             N  DATA_XFF/
MOUNTED  NORMAL  N         512   4096  4194304    864896   863400           298240          282580              0             Y  DBFS_DG/
MOUNTED  NORMAL  N         512   4096  4194304   3787840  2157232          1298688          429272              0             N  RECO_XFF/

后续数据库open成功,有部分坏块通过技术手段进行二次处理,至此数据库恢复完成,成功抢救了客户Oracle Exadata中的绝大部分数据.如果有类似xd故障恢复,无法自行解决,需要恢复支持请联系我们
Phone:13429648788    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

发表在 非常规恢复 | 标签为 , , , , , , , , | 评论关闭