最近连续遇到两次数据库启动报ORA-600 16703错误,而导致数据库无法正常启动的问题
ORA-600 16703报错
Completed: ALTER DATABASE RECOVER database alter database open Beginning crash recovery of 1 threads parallel recovery started with 32 processes Started redo scan Completed redo scan read 0 KB redo, 0 data blocks need recovery Started redo application at Thread 1: logseq 993752, block 2, scn 14872376881763 Recovery of Online Redo Log: Thread 1 Group 6 Seq 993752 Reading mem 0 Mem# 0: /oracle/oradata/xifenfei/redo06.log Completed redo application of 0.00MB Completed crash recovery at Thread 1: logseq 993752, block 3, scn 14872376901765 0 data blocks read, 0 data blocks written, 0 redo k-bytes read Tue Jul 04 23:13:29 2017 Thread 1 advanced to log sequence 993753 (thread open) Thread 1 opened at log sequence 993753 Current log# 7 seq# 993753 mem# 0: /oracle/oradata/xifenfei/redo07.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Tue Jul 04 23:13:29 2017 SMON: enabling cache recovery Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc (incident=163595): ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_163595/xifenfei_ora_15886_i163595.trc Tue Jul 04 23:13:30 2017 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Error 704 happened during db open, shutting down database USER (ospid: 15886): terminating the instance due to error 704 Instance terminated by USER, pid = 15886 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (15886) as a result of ORA-1092
这里报错比较明显ORA-600 16703,而且是在启动时bootstrap$中的对象出现该问题.
10046分析启动过程
===================== select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1 END OF STMT PARSE #140048443935120:c=0,e=390,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905161433 ===================== select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=00 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91b87bd0 bln=22 avl=02 flg=05 value=20 EXEC #140048443932288:c=2000,e=2686,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1218588913,tim=1499185905249810 WAIT #140048443932288: nam='db file sequential read' ela= 6205 file#=1 block#=337 blocks=1 obj#=36 tim=1499185905256132 WAIT #140048443932288: nam='db file sequential read' ela= 3739 file#=1 block#=338 blocks=1 obj#=36 tim=1499185905266704 WAIT #140048443932288: nam='db file sequential read' ela= 4966 file#=1 block#=241 blocks=1 obj#=18 tim=1499185905271761 FETCH #140048443932288:c=0,e=21976,p=3,cr=3,cu=0,mis=0,r=1,dep=1,og=4,plh=1218588913,tim=1499185905271820 STAT #140048443932288 id=1 cnt=1 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=3 pr=3 pw=0 time=21993 us)' STAT #140048443932288 id=2 cnt=1 pid=1 pos=1 obj=36 op='INDEX RANGE SCAN I_OBJ1 (cr=2 pr=2 pw=0 time=16923 us)' CLOSE #140048443932288:c=0,e=63,dep=1,type=0,tim=1499185905271941 BINDS #140048443935120: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91c07de8 bln=22 avl=02 flg=05 value=20 EXEC #140048443935120:c=1000,e=795,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=2970138452,tim=1499185905272802 WAIT #140048443935120: nam='db file sequential read' ela= 3197 file#=1 block#=169 blocks=1 obj#=3 tim=1499185905276069 WAIT #140048443935120: nam='db file sequential read' ela= 3459 file#=1 block#=170 blocks=1 obj#=3 tim=1499185905404084 WAIT #140048443935120: nam='db file sequential read' ela= 6358 file#=1 block#=145 blocks=1 obj#=4 tim=1499185905410548 FETCH #140048443935120:c=999,e=137805,p=3,cr=3,cu=0,mis=0,r=0,dep=1,og=4,plh=2970138452,tim=1499185905410635 STAT #140048443935120 id=1 cnt=0 pid=0 pos=1 obj=4 op='TABLE ACCESS CLUSTER TAB$ (cr=3 pr=3 pw=0 time=137810 us)' STAT #140048443935120 id=2 cnt=1 pid=1 pos=1 obj=3 op='INDEX UNIQUE SCAN I_OBJ# (cr=2 pr=2 pw=0 time=131330 us)' *** 2017-07-05 00:31:46.094 Incident 176395 created, dump file: /oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_176395/xifenfei_ora_51261_i176395.trc ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
报错信息明显,由于select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1无法正常执行.从而出现了ORA-600 16703的错误,更加直接一点的解释就是obj#=20的对象在tab$中找不到记录，从而出现此类报错.和官方解释ORA-600 TAB$和obj$不匹配一致.
分析system文件
通过dul等工具分析system文件发现tab$表记录为空
Data UnLoader: 11.2.0.1.5 - Internal Only - on Wed Jul 05 01:28:53 2017 with 64-bit io functions and the decompression option Copyright (c) 1994 2017 Bernard van Duijnen All rights reserved. Strictly Oracle Internal Use Only Found db_id = 1334610369 Found db_name = xifenfei DUL> unload table TAB$( OBJ# number, DATAOBJ# number, 2 TS# number, FILE# number, BLOCK# number, 3 BOBJ# number, TAB# number, COLS number, CLUCOLS number, 4 PCTFREE$ ignore, PCTUSED$ ignore, INITRANS ignore, MAXTRANS ignore, 5 FLAGS ignore, AUDIT$ ignore, ROWCNT ignore, BLKCNT ignore, 6 EMPCNT ignore, AVGSPC ignore, CHNCNT ignore, AVGRLN ignore, 7 AVGSPC_FLB ignore, FLBCNT ignore, 8 ANALYZETIME ignore, SAMPLESIZE ignore, 9 DEGREE ignore, INSTANCES ignore, 10 INTCOLS ignore, KERNELCOLS number, PROPERTY number) 11 cluster C_OBJ#(OBJ#) 12 storage ( tablespace 0 segobjno 2 tabno 1 file 1 block 144); . unloading table TAB$ 0 rows unloaded DUL> unload table OBJ$( OBJ# number, DATAOBJ# number, OWNER# number, 2 NAME clean varchar2(30), NAMESPACE ignore, SUBNAME clean varchar2(30), 3 TYPE# number, CTIME ignore, MTIME ignore, STIME ignore, 4 STATUS ignore, REMOTEOWNER ignore, LINKNAME ignore, 5 FLAGS ignore, OID$ hexraw) 6 storage ( tablespace 0 segobjno 18 file 1 block 240); . unloading table OBJ$ 89804 rows unloaded DUL>
发现在obj$中有创建表ORACHKBEC66CBE055000000000001(ORACHK+16进制24位)的一个表名字
该表用途通过分析数据库日志发现
create table ORACHKBEC66CBE055000000000001 tablespace system as select * from sys.tab$;
也就是说,这个orachk的表是用来备份tab$的,然后进一步发现有delete from tab$.至此基本上分析清楚,tab$表备份到ORACHK表中,然后delete tab$表数据.实现数据库破坏以及难以恢复的效果.有点类似plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案的破坏案例
我们可以通过使用bbed对tab$表数据数据进行恢复实现数据库正常启动,实现数据0丢失,最大限度抢救您的数据和减少业务恢复时间
Phone:17813235971 Q Q:107644445
E-Mail:dba@xifenfei.com
源头已经分析清楚,是有人篡改了oracle安装介质,导致数据库运行300天之后出现该问题,具体见:警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
加我QQ(107644445)
