接到客户请求,存储异常断电之后,一个40多T的经分数据库无法正常启动,通过各方一系列操作之后,数据库依旧无法open,报错信息为:ORA-00600: internal error code, arguments: [16513], [1403], [28]
Sun Feb 14 00:20:09 BEIST 2021 SMON: enabling cache recovery Sun Feb 14 00:20:09 BEIST 2021 ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0f27.13cd4fc3): Sun Feb 14 00:20:09 BEIST 2021 select ctime, mtime, stime from obj$ where obj# = :1 Sun Feb 14 00:20:09 BEIST 2021 Errors in file /oracle10g/db/admin/xifenfei/udump/xifenfei1_ora_177254.trc: ORA-00600: internal error code, arguments: [16513], [1403], [28], [], [], [], [], [] Sun Feb 14 00:20:10 BEIST 2021 Errors in file /oracle10g/db/admin/xifenfei/udump/xifenfei1_ora_177254.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16513], [1403], [28], [], [], [], [], [] Error 704 happened during db open, shutting down database USER: terminating instance due to error 704 Instance terminated by USER, pid = 177254 ORA-1092 signalled during: ALTER DATABASE OPEN...
通过对启动过程进行跟踪
=====================
PARSING IN CURSOR #5 len=52 dep=1 uid=0 oct=3 lid=0 tim=194171381576991 hv=429618617 ad='afcee60'
select ctime, mtime, stime from obj$ where obj# = :1
END OF STMT
PARSE #5:c=0,e=257,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,tim=194171381576990
BINDS #5:
kkscoacd
Bind#0
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=1104b8ed0 bln=22 avl=02 flg=05
value=28
EXEC #5:c=0,e=422,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,tim=194171381577472
WAIT #5: nam='db file sequential read' ela= 274 file#=1 block#=110 blocks=1 obj#=36 tim=194171381577809
WAIT #5: nam='db file sequential read' ela= 257 file#=1 block#=78943 blocks=1 obj#=36 tim=194171381578123
WAIT #5: nam='db file sequential read' ela= 253 file#=1 block#=111 blocks=1 obj#=36 tim=194171381578416
WAIT #5: nam='db file sequential read' ela= 226 file#=1 block#=62 blocks=1 obj#=18 tim=194171381578692
=====================
PARSING IN CURSOR #6 len=142 dep=2 uid=0 oct=3 lid=0 tim=194171381579134 hv=361892850 ad='df87eb0'
select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,
DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1
END OF STMT
PARSE #6:c=0,e=368,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=3,tim=194171381579133
BINDS #6:
kkscoacd
Bind#0
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=1105af9a8 bln=22 avl=02 flg=05
value=92
EXEC #6:c=0,e=531,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=3,tim=194171381579736
WAIT #6: nam='db file sequential read' ela= 232 file#=1 block#=102 blocks=1 obj#=34 tim=194171381580011
WAIT #6: nam='db file sequential read' ela= 251 file#=1 block#=54 blocks=1 obj#=15 tim=194171381580307
FETCH #6:c=0,e=615,p=2,cr=2,cu=0,mis=0,r=1,dep=2,og=3,tim=194171381580369
STAT #6 id=1 cnt=1 pid=0 pos=1 obj=15 op='TABLE ACCESS BY INDEX ROWID UNDO$ (cr=2 pr=2 pw=0 time=584 us)'
STAT #6 id=2 cnt=1 pid=1 pos=1 obj=34 op='INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=1 pw=0 time=282 us)'
WAIT #5: nam='db file sequential read' ela= 260 file#=4290 block#=365090 blocks=1 obj#=0 tim=194171381580721
FETCH #5:c=10000,e=3327,p=7,cr=7,cu=0,mis=0,r=0,dep=1,og=4,tim=194171381580817
*** 2021-02-14 02:32:40.430
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [16513], [1403], [28], [], [], [], [], []
Current SQL statement for this session:
alter database open
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst+001c bl ksedst1 70000090FC00AE4 ? 000000000 ?
ksedmp+0290 bl ksedst 104C1CA58 ?
ksfdmp+02d8 bl 03F4BD0C
kgeriv+0108 bl _ptrgl
kgesiv+0080 bl kgeriv 7000008F6DB5890 ? 00000000C ?
7000008F6DB5890 ? 0FFFFFDC3 ?
0FFFFFFBF ?
ksesic2+0060 bl kgesiv 5FFFEC580 ? 500000000 ?
FFFFFFFFFFEC590 ? 0F5FAF808 ?
7000008F5FAF7E8 ?
kqdpts+0158 bl ksesic2 408100004081 ? 000000000 ?
00000057B ? 000000000 ?
00000001C ?
A006150571B05EF0 ?
1100CF0A8 ? 000000001 ?
kqrlfc+0274 bl kqdpts 000000000 ?
kqlbplc+00b4 bl 03F4E6A8
kqlblfc+0230 bl kqlbplc 00000009D ?
adbdrv+1a9c bl 03F4B66C
opiexe+2db4 bl adbdrv
opiosq0+1ac8 bl opiexe 1103B418C ? 000000000 ?
FFFFFFFFFFF9008 ?
kpooprx+016c bl opiosq0 300F1E1D4 ? 000000000 ?
000000000 ? A4FFFFFFFF9798 ?
000000000 ?
kpoal8+03cc bl kpooprx FFFFFFFFFFFB814 ?
FFFFFFFFFFFB5C0 ?
1300000013 ? 100000001 ?
000000000 ? A40000000000A4 ?
000000000 ? 1103B4378 ?
opiodr+0b2c bl _ptrgl
ttcpip+1020 bl _ptrgl
opitsk+117c bl 01FBD04C
opiino+09d0 bl opitsk 1EFFFFD7E0 ? 000000000 ?
opiodr+0b2c bl _ptrgl
opidrv+04a4 bl opiodr 3C102B1398 ? 404C6FF30 ?
FFFFFFFFFFFF7A0 ? 0102B1390 ?
sou2o+0090 bl opidrv 3C02705B3C ? 440663000 ?
FFFFFFFFFFFF7A0 ?
opimai_real+01bc bl 01FB9BF4
main+0098 bl opimai_real 000000000 ? 000000000 ?
__start+0098 bl main 000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------
报错比较明显,数据库在查询obj$取obj#=28的数据的时候无法正常获取到该数据,从而报错ORA-600 16513错误.通过对相关block进行分析,发现有事务异常
BBED> p ktbbh.ktbbhitl[1]
struct ktbbhitl[1], 24 bytes @44
struct ktbitxid, 8 bytes @44
ub2 kxidusn @44 0x5c00
ub2 kxidslt @46 0x0a00
ub4 kxidsqn @48 0x48b4fc01
struct ktbituba, 8 bytes @52
ub4 kubadba @52 0x22928531
ub2 kubaseq @56 0xe383
ub1 kubarec @58 0x01
ub2 ktbitflg @60 0x0120 (NONE)
union _ktbitun, 2 bytes @62
b2 _ktbitfsc @62 0
ub2 _ktbitwrp @62 0x0000
ub4 ktbitbas @64 0x95c2ff13
通过一些技巧处理规避掉该事务,然后启动库报熟悉的ORA-01555错误
相对比较简单参考(在数据库open过程中常遇到ORA-01555汇总),数据库顺利open成功,完成春节后第一个大库的恢复
加我微信(17813235971)
加我QQ(107644445)
