tab$恢复错误汇总

在以前多以前发现的tab$被恶意脚本删除的问题(ORA-600 16703故障解析—tab$表被清空,警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),虽然多次强调注意Oracle安装介质安全,但是很不幸,还是大量客户中招.我们这一年多对于tab$的故障进行了大量case处理,拯救了大量客户的核心数据,也积累了一些常见的可能遭遇的错误.主要恢复思路是使用bbed处理异常block,让数据库open起来.
ORA-00704 ORA-39700
有核心基表处理异常导致

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Process ID: 1603
Session ID: 1 Serial number: 5

Sun Jan 06 21:30:14 2019
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 1603): terminating the instance due to error 704
Instance terminated by USER, pid = 1603
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (1603) as a result of ORA-1092
Sun Jan 06 21:30:14 2019
ORA-1092 : opitsk aborting process

ora-704 ora-604 ora-01555
由于scn异常导致

SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name
"_SYSSMU2_2996391332$" too small
Process ID: 26520
Session ID: 1 Serial number: 5

Sun Jan 06 19:49:12 2019
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: bqbdby3c400p7, SCN: 0x0022.1117ef75):
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 26520): terminating the instance due to error 704
Instance terminated by USER, pid = 26520
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (26520) as a result of ORA-1092

ORA-600 13304
有核心基表处理异常导致

SQL> startup mount;
ORACLE instance started.

Total System Global Area  521936896 bytes
Fixed Size                  2254824 bytes
Variable Size             352323608 bytes
Database Buffers          163577856 bytes
Redo Buffers                3780608 bytes
Database mounted.
SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 1724
Session ID: 1 Serial number: 5

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Jan 06 21:31:04 2019
SMON: enabling cache recovery
[1724] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2239884804 end:2239884864 diff:60 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc  (incident=61755):
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_61755/orcl_ora_1724_i61755.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1724): terminating the instance due to error 600
Instance terminated by USER, pid = 1724
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (1724) as a result of ORA-1092
Sun Jan 06 21:31:06 2019
ORA-1092 : opitsk aborting process

ORA-00704 ORA-600 kdBlkCheckError
恢复的block有逻辑坏块

SQL> startup         
ORACLE instance started.

Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497],
[6121], [], [], [], [], [], [], [], []
Process ID: 76932
Session ID: 191 Serial number: 3

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc  (incident=6153):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_6153/xifenfei_ora_76932_i6153.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 76932): terminating the instance due to error 704
Instance terminated by USER, pid = 76932
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (76932) as a result of ORA-1092
Sat Feb 22 11:04:19 2014
ORA-1092 : opitsk aborting process
发表在 非常规恢复 | 标签为 , , , , , , , , , , | 评论关闭

10g数据库遭遇ORA-600 16703

ORA-600 16703
有客户反馈10g数据库启动报ORA-00704 ORA-00600,具体如下

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi
PL/SQL Release 10.2.0.3.0 - Production
CORE	10.2.0.3.0	Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio
NLSRTL Version 10.2.0.3.0 - Production
Fri Jan  4 10:11:05 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 10:11:05 2019
SMON: enabling cache recovery
Fri Jan  4 10:11:05 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 1826952
ORA-1092 signalled during: alter database open...

看到这个错误第一想到以前的ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [],但是有几个地方不同:1)根据以往恢复经验这个错误都出现在11.2.0.4版本数据库中;2)以往的ORA-600 16703错误最后值是20,而这次是28

分析数据库故障过程
节点一因为其他错误重启,重启之后报大量错误,然后abort掉.

--节点1
Fri Jan  4 06:11:00 2019
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=2883786
Fri Jan  4 06:11:07 2019
SMON: Parallel transaction recovery tried
Fri Jan  4 06:11:07 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2769106.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020E4E7258], [257], [0], [], [], [], []
Fri Jan  4 06:11:09 2019
ORA-600 signalled during: ALTER DATABASE OPEN...
Fri Jan  4 06:11:09 2019
Trace dumping is performing id=[cdmp_20190104061109]
Fri Jan  4 06:11:30 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2240550.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:12:35 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793508.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [65], [], [], []
Fri Jan  4 06:12:40 2019
Trace dumping is performing id=[cdmp_20190104061240]
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1933540.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793546.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:11 2019
Trace dumping is performing id=[cdmp_20190104061611]
Fri Jan  4 06:16:15 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:16:16 2019
Non-fatal internal error happenned while SMON was doing logging scn->time mapping.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Fri Jan  4 06:17:28 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2035774.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
…………
Fri Jan  4 06:56:47 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:56:49 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_3072030.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_pmon_3092524.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
PMON: terminating instance due to error 474
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_lms0_3178590.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
System state dump is made for local instance
System State dumped to trace file /oracle/admin/xifenfei/bdump/xifenfei1_diag_434192.trc
Fri Jan  4 06:56:51 2019
Shutting down instance (abort)

节点一重启触发故障之后,节点二也开始报错,然后数据库直接挂掉

--节点2
Fri Jan  4 06:21:19 2019
Trace dumping is performing id=[cdmp_20190104062118]
Fri Jan  4 06:21:22 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j000_1253440.trc:
ORA-00600: internal error code, arguments: [kghGetHpSz1], [0x7000001DF886220], [], [], [], [], [], []
Fri Jan  4 06:21:24 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_1835262.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:21:56 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_1777888.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Fri Jan  4 06:24:38 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:24:44 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-12012: error on auto execute of job 351
ORA-12008: error in materialized view refresh path
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:28:48 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j001_1417456.trc:
ORA-00600: internal error code, arguments: [kkzufst], [18446744073709551615], [], [], [], [], [], []
Fri Jan  4 06:28:49 2019
ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x1104B2580], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 07:30:25 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_2121800.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kggfaAllocFunc1], [], [], [], [], [], [], []
…………
Fri Jan  4 07:31:23 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
PMON: terminating instance due to error 472
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_569432.trc:
ORA-00472: PMON  process terminated with error

两个节点再次启动报ORA-600 16703错误

Fri Jan  4 07:31:50 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 07:31:50 2019
SMON: enabling cache recovery
Fri Jan  4 07:31:50 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_3780764.trc:
ORA-00704: bootstrap process failure
Instance terminated by USER, pid = 3813516
ORA-1092 signalled during: ALTER DATABASE OPEN...

10046跟踪启动过程
ora-600-16703-10046


分析故障原因
通过分析,确认该数据库被注入了恶意脚本,当发生重启之后导致数据库核心基表被破坏,从而使得数据库无法正常启动

procedure     DBMS_DBMONITOR wrapped
a000000
369
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
7
15a 171
VKGnETw5GkgNlUvXadIYpQ2thwAwgzKur9zWZ47SR+pHN0bgoPvQ8ezdufYxFkzCmAQA8xT2
IQkFph+2IhlEBrPrXe14giDow/HzZI43FwLiMlynCYjnzQh3aXRSIVOalcwGAfUvgCip6Eng
OWA8Vq49YJ38WCPZjq2P5Dc428Wa1ZOPMb+E7GPs8ZOWM7RWsQdMzx/pqFncbX/tLwp0NY5E
Uu4E54MZ34yVtDQybwljVqp06KHqWN/ZwZJpvT+2gO4hRNX2UyE7laWCXzM2IR05BTGf2yoZ
+E7eIn6kciinFmhcUiBuszxE0pykt+moWZuuDuj9ebUXmj+0Mx7A+eQc6tp7wLHRCHYb6p0D
VtDc4f6

通过对损坏的字典进行恢复,实现数据0丢失
open-database


发表在 非常规恢复 | 标签为 , , , , , , , | 评论关闭

2019恢复第一单—ORA-704-ORA-702恢复

19年元旦刚过就有客户反馈数据库无法正常启动报ORA-00704和ORA-00702错误,使用自研的ORA-702_Recovery Tool小工具可以实现一键恢复,下载地址:ORA-702_Recovery下载
ora-00702


Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3756.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3756.trc:
ORA-00704: 引导程序进程失败
ORA-00702: 引导程序版本 '' 与版本 '8.0.0.0.0' 不一致
Error 704 happened during db open, shutting down database
USER (ospid: 3756): terminating the instance due to error 704
Instance terminated by USER, pid = 3756

通过分析确认,确认由于数据库遭受到而已脚本注入,破坏数据库基表数据,导致数据库重启之后就无法正常启动,
1


这类故障可以通过对oracle基表进行恢复,实现数据库完美open,数据0丢失.如果无法自行解决,需要恢复支持请联系我们
Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com
2019年再次提醒:1)数据库做好备份容灾,2)请使用官方途径下载数据库介质和数据库访问工具,3)定期检查数据库是否被注入恶意脚本

发表在 非常规恢复 | 标签为 , , , | 评论关闭