-
加我微信(17813235971)
加我QQ(107644445)
标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-00742 ORA-01110 ORA-01555 ORA-01578 ORA-01595 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (103)
- 数据库 (1,769)
- DB2 (22)
- MySQL (77)
- Oracle (1,610)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (24)
- ORA-xxxxx (166)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (15)
- ORACLE 21C (3)
- Oracle 23ai (8)
- Oracle ASM (69)
- Oracle Bug (8)
- Oracle RAC (54)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (29)
- Oracle备份恢复 (592)
- Oracle安装升级 (97)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (86)
- PostgreSQL (30)
- pdu工具 (6)
- PostgreSQL恢复 (9)
- SQL Server (32)
- SQL Server恢复 (13)
- TimesTen (7)
- 达梦数据库 (3)
- 达梦恢复 (1)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (39)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (22)
-
最近发表
- 2025年的Oracle 8.0.5数据库恢复
- ORA-600 kokiasg1故障分析(obj$中核心字典序列全部被恶意删除)
- ORA-00756 ORA-10567故障数据0丢失恢复
- 数据库文件变成32k故障恢复
- tcp连接过多导致监听TNS-12532 TNS-12560 TNS-00502错误
- 文件系统格式化MySQL数据库恢复
- .sstop勒索加密数据库恢复
- 解决一次硬件恢复之后数据文件0kb的故障恢复case
- Error in invoking target ‘libasmclntsh19.ohso libasmperl19.ohso client_sharedlib’问题处理
- ORA-01171: datafile N going offline due to error advancing checkpoint
- linux环境oracle数据库被文件系统勒索加密为.babyk扩展名溯源
- ORA-600 ksvworkmsgalloc: bad reaper
- ORA-600 krccfl_chunk故障处理
- Oracle Recovery Tools恢复案例总结—202505
- ORA-600 kddummy_blkchk 数据库循环重启
- 记录一次asm disk加入到vg通过恢复直接open库的案例
- CHECKDB 发现了 N 个分配错误和 M 个一致性错误
- 达梦数据库dm.ctl文件异常恢复
- Oracle Recovery Tools修复ORA-00742、ORA-600 ktbair2: illegal inheritance故障
- 可能是 tempdb 空间用尽或某个系统表不一致故障处理
分类目录归档：ORA-xxxxx
收集统计信息出现ORA-00600[ksxprqfre3]
alert日志出现ORA-00600[ksxprqfre3]错误
Sun Mar 18 22:11:44 2012 Errors in file /oracle9/app/admin/offon/bdump/offon2_p001_8220790.trc: ORA-00600: internal error code, arguments: [ksxprqfre3], [0x11033D5A8], [], [], [], [], [], [] Sun Mar 18 22:11:44 2012 Errors in file /oracle9/app/admin/offon/bdump/offon2_p001_8220790.trc: ORA-00600: internal error code, arguments: [ksxprqfre3], [0x11033E178], [], [], [], [], [], []
通过这里可以发现，是由于并发进程导致异常
trace文件内容
Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /oracle9/app/product/9.2.0
System name: AIX
Node name: zwq_offon2
Release: 3
Version: 5
Machine: 00CA56A44C00
Instance name: offon2
Redo thread mounted by this instance: 2
Oracle process number: 328
Unix process pid: 8220790, image: oracle@zwq_offon2 (P001)
*** 2012-03-18 22:11:43.994
*** SESSION ID:(381.10338) 2012-03-18 22:11:43.989
krqh = 11033d5a8
rqh trace:
Slot: 6 Op: 114 ksxpirqh: irqh requested
Slot: 7 Op: 130 ksxpirqh: rqh SUCC
Slot: 8 Op: 123 ksxpwait: receive mv to doneq
Slot: 9 Op: 121 ksxprcv: rqh found on doneq
Slot: 10 Op: 102 ksxprqfre: freed to queue
Slot: 11 Op: 101 ksxprqalo: allocated
Slot: 12 Op: 105 ksxpdteq: enqueue to delta q
Slot: 13 Op: 117 ksxpvsnd: send via ksxpisnd
Slot: 14 Op: 106 ksxpisnd: send requested
Slot: 0 Op: 107 ksxpisnd: send submit OK
Slot: 1 Op: 153 pre send completion cbk
Slot: 2 Op: 102 ksxprqfre: freed to queue
Slot: 3 Op: 114 ksxpirqh: irqh requested
Slot: 4 Op: 130 ksxpirqh: rqh SUCC
Slot: 5 Op: 153 pre send completion cbk
Dump of memory from 0x000000011033D5A8 to 0x000000011033D660
11033D5A0 00000001 1033D5A8 [.....3..]
11033D5B0 00000001 1033D5A8 00000001 1033D5B8 [.....3.......3..]
11033D5C0 00000001 1033D5B8 00000006 00000001 [.....3..........]
11033D5D0 00000000 00000000 00000000 00000000 [................]
11033D5E0 00000572 00000000 07000000 EE473B28 [...r.........G;(]
11033D5F0 00010000 00000000 00000001 1029CEA8 [.............)..]
11033D600 00000001 104382A0 07000000 F279B5A8 [.....C.......y..]
11033D610 00000000 00000572 00000000 00000000 [.......r........]
11033D620 00000000 00000000 00000000 13DCD225 [...............%]
11033D630 13DA7A25 00010001 26750000 00000000 [..z%....&u......]
11033D640 00000001 10339C70 6B996672 82997282 [.....3.pk.fr..r.]
11033D650 7B796665 69756A06 00000001 1033E228 [{yfeiuj......3.(]
*** 2012-03-18 22:11:43.996
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [ksxprqfre3], [0x11033D5A8], [], [], [], [], [], []
Current SQL statement for this session:
SELECT /*+ PIV_SSF */ SYS_OP_MSR(SYS_OP_COUNTCHG(SUBSTRB(ROWIDTOCHAR(A1.C0),1,15),1),COUNT(DISTINCT A1.C3),COUNT(*)) C0 FROM (SELECT /*+ INDEX_RRS(A2 "PK_TINEXWORK_DATA
") */ A2.ROWID C0,A2."WORKSHTSN" C1,A2."EXWORKSN" C2,SYS_OP_LBID(8574,'L',A2.ROWID) C3 FROM "OFFON"."TINEXWORK_DATA" PX_GRANULE(0, BLOCK_RANGE, DYNAMIC) A2) A1
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp+0148 bl ksedst 102973B94 ?
ksfdmp+0018 bl 01FD34D8
kgerinv+00e8 bl _ptrgl
kgeasnmierr+004c bl kgerinv 127C7F9D7226029B ?
9770A0AFAF4BDC0 ?
12E0BE826D694B2F ?
000000000 ? 000000010 ?
ksxprqfre+0054 bl kgeasnmierr 110006728 ? 110338C88 ?
1030069D4 ? 100000001 ?
000000002 ? 11033D5A8 ?
080000000 ?
7F7F7F7F7F7F7F7F ?
ksxpwait+06bc bl ksxprqfre 7000000D6D793D8 ?
ksliwat+02e0 bl ksxpwait 1014F4508 ? 1101FB1C8 ?
kslwaitns+001c bl ksliwat FFFFFFFFFFFC4B0 ? 000000004 ?
FFFFFFFFFFFC590 ? 11030BC60 ?
FFFFFFFFFFFC530 ?
7000000DDF1FBE0 ? 110006868 ?
kskthbwt+01b4 bl kslwaitns 000000002 ? 7000000EAEEB7A0 ?
000000000 ? 000000000 ?
000000868 ? 7000000DAFEFB30 ?
7000000F4979B10 ?
kslwait+005c bl kskthbwt FFFFFFFFFFFC6E0 ?
2422424800000000 ?
1015EDD34 ? 110305194 ?
FFFF0000000003 ? 000000000 ?
000000001 ? FFFFFFFFFFFF920 ?
ksxprcv+0184 bl kslwait 102970928 ? 000000006 ?
000000000 ? 000000000 ?
110305148 ? 110006728 ?
kxfpqidqr+0600 bl ksxprcv 11029CEA8 ? 000000008 ?
FFFFFFFFFFFD020 ?
FFFFFFFFFFFCFC0 ?
162DFEF4180 ? 000000000 ?
000000000 ? 000000000 ?
kxfpqdqr+02c0 bl kxfpqidqr 11029CF80 ? 11029CF20 ?
FFFFFFFFFFFD7C0 ?
7000000EAE2DA30 ?
kxfxgs+0050 bl kxfpqdqr 101518AF0 ? 000002000 ?
FFFFFFFFFFFD9F0 ? 000000000 ?
kxfxmai+0100 bl kxfxgs 102970928 ? 000000060 ?
7000000EEF059F0 ?
70000000006D670 ?
kxfprdp+05b0 bl _ptrgl
opirip+0390 bl kxfprdp
opidrv+0300 bl opirip 000000018 ? 0101FB1C8 ?
000000000 ?
sou2o+0028 bl opidrv 320C000000 ? 0A00E8B50 ?
000000000 ?
main+01a4 bl 01FD2EF0
__start+0098 bl main 000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------
SO: 7000000cf267198, type: 4, owner: 7000000cd2a53e0, flag: INIT/-/-/0x00
(session) trans: 0, creator: 7000000cd2a53e0, flag: (48000041) USR/- -/-/-/-/-/-
DID: 0002-0148-003F8809, short-term DID: 0000-0000-00000000
txn branch: 0
oct: 0, prv: 0, sql: 0, psql: 0, user: 25/MONITOR
O/S info: user: maint1, term: , ospid: 8220790, machine: zwq_acc2
program: oracle@zwq_offon2 (P001)
application name: SQL*Plus, hash value=3669949024
waiting for 'PX Deq: reap credit' blocking sess=0x0 seq=182 wait_time=0
=0, =0, =0
temporary object counter: 0
这里可以得出信息如下：
1)系统环境AIX 5.3 9.2.0.8.0 RAC
2)报错的语句是收集TINEXWORK_DATA表相关统计信息
3)是由于SQL*Plus收集统计信息使用并发导致该错误
查询MOS发现
Bug 5887047: ORA-00600 [KSXPRQFRE3] TRACE WHEN GATHERING OPTIMIZER STATISTICS描述相符：
1）数据库版本9.2.0.8
2）收集统计信息操作导致
3）STACK TRACE一致
处理建议
针对官方没有给出明确的解决方案，因为该问题出现概率比较低，如果经常出现需要考虑升级数据库版本
10.2.0.5出现ORA-00600[kcblasm_1]
今天朋友执行一条如下sql
SELECT YF,
KHBH,
MAX(KHXM),
MAX(YYBDM),
MAX(YYBMC),
MIN(ZJZH),
SUM(CASE
WHEN RQ = 20120321 THEN
NVL(ZJYE, 0)
ELSE
0
END),
SUM(CASE
WHEN RQ = 20120301 THEN
NVL(ZJYE, 0)
ELSE
0
END),
SUM(CASE
WHEN RQ = 20120321 THEN
NVL(ZQSZ, 0)
ELSE
0
END),
SUM(CASE
WHEN RQ = 20120301 THEN
NVL(ZQSZ, 0)
ELSE
0
END),
SUM(CASE
WHEN RQ = 20120321 THEN
NVL(ZC, 0)
ELSE
0
END),
SUM(CASE
WHEN RQ = 20120301 THEN
NVL(ZC, 0)
ELSE
0
END),
SUM(CASE
WHEN RQ = 20120321 THEN
NVL(DWJZ, 1)
ELSE
0
END),
SUM(CASE
WHEN RQ = 20120301 THEN
NVL(DWJZ, 1)
ELSE
0
END),
MAX(NVL(ZC, 0)) ZGZC,
SUM(NVL(ZRZC, 0)) ZRZC,
SUM(NVL(ZCZC, 0)) ZCZC,
SUM(NVL(FDYK, 0)) FDYK,
SUM(NVL(ZRGF, 0)) ZRGF,
SUM(NVL(ZCGF, 0)) ZCGF,
AVG(NVL(ZC, 0)) PJZC,
SUM(NVL(CJJE, 0)) CJJE,
SUM(NVL(YJ, 0)) YJ,
SUM(NVL(ZJFSS, 0)) ZJFSS,
SUM(NVL(ZYJ, 0)) ZYJ,
SUM(NVL(CJCS, 0)) CJCS
FROM CRM_FX2_KHZJHZ P
WHERE YF = 201203
GROUP BY YF, KHBH;
出现ORA-00600: internal error code, arguments: [kcblasm_1], [103], [], [], [], [], []错误
数据库版本
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi PL/SQL Release 10.2.0.5.0 - Production CORE 10.2.0.5.0 Production TNS for Linux: Version 10.2.0.5.0 - Production NLSRTL Version 10.2.0.5.0 - Production
通过MOS发现该错误在10.2.0.5上有相关描述
This problem is hit only in 10.2.0.5, up to PSU 10.2.0.5.4 in which problem is fixed. Problem is affecting 10.2.0.5 to 10.2.0.5.3. Cause This is is a known and common problem hit in 10.2.0.5, investigated and corrected in unpublished Bug 7612454. The problem was introduced in 10.2.0.5, by the fix of Bug:7523755. This has been corrected in PSU 10.2.0.5.4 and 11.2.
解决方法
To quickest way to the solution of this problem is to apply PSU 10.2.0.5.4, Patch 12419392 or later. All alternative solutions for this problem are listed below: • Upgrade the database to 11.2. OR • Apply 10.2.0.5.4 Patch Set Update (Patch 12419392) or later PSUs where bug is fixed. The available PSUs are mentioned in "10.2.0.5 Patch Set Updates - List of Fixes in each PSU" (Document 1337394.1) OR • Apply interim Patch 7612454 on top of 10.2.0.5 (10.2.0.5.0-10.2.0.5.3): o For UNIX / Linux platforms apply Patch 7612454 available for download on MOS. o For Windows platforms apply Patch 3 or higher. Please check Document 342443.1 for latest patches available for Windows on top of 10.2.0.5.
这个错误在10.2.0.5上很容易发生，官方也没有给出发生这个错误的原因，以及如何改写相关sql，给出的方案是升级到10.2.0.5.4或者更高。这里可以看出升级需要慎重，可能导致原库有小部分sql不能正常执行。
ORA-7445[__milli_memcpy]分析
帮朋友分析日志的过程中发现在两个节点上多次出现类此错误
节点1日志分析
Mon Mar 5 08:19:20 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc: ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Mon Mar 5 08:19:21 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc: ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Mon Mar 5 08:19:22 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc: ORA-07445: exception encountered: core dump [kghalf()+464] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] [] ORA-07445: exception encountered: core dump [kghalf()+961] [SIGSEGV] [Invalid permissions for mapped object] [0x000000000] [] [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
对应trace文件
/u01/app/oracle/admin/hddms/udump/hddms1_ora_18556.trc
Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production
ORACLE_HOME = /u01/app/oracle/product/9.2.0
System name: HP-UX
Node name: HDDB1
Release: B.11.23
Version: U
Machine: ia64
Instance name: hddms1
Redo thread mounted by this instance: 1
Oracle process number: 42
Unix process pid: 18556, image: oracle@HDDB1 (TNS V1-V3)
*** 2012-03-05 08:19:20.675
*** SESSION ID:(100.40701) 2012-03-05 08:19:20.602
Exception signal: 11 (SIGSEGV), code: 1 (Address not mapped to object), addr: 0x9fffffffbf580000, PC: [0x400000000182a9f0, __milli_memcpy()+2448]
r1: 6000000000524900 r20: 8 br5: 0
r2: 0 r21: 9fffffffbf4c21ec br6: 40000000018ad000
r3: 37 r22: 8 br7: e000000120001400
r4: 0 r23: c00000005e3bac98 ip: 400000000182a9f0
r5: 60000000005311d8 r24: 9fffffffbf4c2268 iipa: 0
r6: 40000004 r25: 9fffffffbf4c2268 cfm: 2a54295428
r7: 9fffffffffff1ba8 r26: 9fffffffbf580000 um: 1a
r8: 9fffffffbf4c2268 r27: 9fffffffbf57ff00 rsc: 1f
r9: 8000 r28: 9fffffffbf580008 bsp: 9fffffffbf802150
r10: 9fffffffbf4ba260 r29: 9fffffffbf57ff08 bspstore: 9fffffffbf802150
r11: 0 r30: 0 rnat: 0
r12: 9ffffffffffe5240 r31: 0 ccv: 20000000
r13: 9fffffffbf5a1420 NaTs: 0 unat: 0
r14: 9fffffffbf4c23e8 PRs: ffffffffffff4837 fpsr: 9804c8274433f
r15: 9fffffffbf580028 br0: 400000000202a7e0 pfs: c000000000001b3e
r16: 3fffffffffffdff br1: 0 lc: 3ffffffffffce89
r17: 9fffffffbf4c2258 br2: 0 ec: 5
r18: 0 br3: 0 isr: 9fffffffbf802150
r19: 80 br4: 0 ifa: 0
Reason code: 0008
*** 2012-03-05 08:19:20.735
ksedmp: internal or fatal error
ORA-07445: exception encountered: core dump [__milli_memcpy()+2448] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
Current SQL statement for this session:
update pmis.PM_BZ_DC_DWRFHSJ set ZGFDFH=:v_result where PARINST_ID=:v_pid and T_ID>:v_mdtbegin and T_ID<=:v_mdtend
----- PL/SQL Call Stack -----
object line object
handle number name
c00000008975ebc0 49 procedure PMIS.PM_DC_SJJGGC_EXECINF
c0000000971ef530 2 anonymous block
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedmp()+528 call ksedst() 000000001 ?
C050000000000996 ?
4000000002A53E60 ?
ssexhd()+1184 call ksedmp() 000000003 ?
C0500000000010A9 ?
40000000025E1420 ?
FFFFFFFFFFFF9075 ?
9FFFFFFFFFFDF2E0 ?
6000000000514428 ?
6000000000514420 ?
60000000005178E0 ?
<kernel> call ssexhd() 400000000086EFB0 ?
000000015 ? 000000007 ?
C050000000000085 ?
__milli_memcpy()+24 call <kernel>
48
$cold_updgrh()+1088 call __milli_memcpy() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
9FFFFFFFBF564348 ?
000000000 ? 000000000 ?
000000000 ?
upduaw()+224 call $cold_updgrh() C0000000A2665550 ?
9FFFFFFFBF558FA0 ?
C000000000000C1E ?
4000000001916600 ?
000000000 ? 00000E137 ?
kdusru()+720 call upduaw() C0000000A2665550 ?
C000000000005BBF ?
4000000001912AC0 ?
00000E135 ?
9FFFFFFFFFFE52B0 ?
C0000000A2665660 ?
9FFFFFFFFFFE63F0 ?
9FFFFFFFBF5590D8 ?
kauupd()+544 call kdusru() 9FFFFFFFBF4C5D24 ?
000000000 ?
9FFFFFFFBF558FA0 ?
000000000 ?
C00000008B5D1108 ?
C00000008B5D1138 ?
000000018 ?
9FFFFFFFBF5591E0 ?
updrow()+3024 call kauupd() 9FFFFFFFFFFE65D0 ?
000000000 ?
6000000000530010 ?
000000000 ?
C00000009202CF38 ?
00000010B ? 00001AA7A ?
C000000086F2B778 ?
qerupFetch()+608 call updrow() 9FFFFFFFBF4C89F0 ?
C0000000A2665550 ?
C000000000000C1D ?
40000000018F5BF0 ?
000000000 ?
updaul()+1008 call qerupFetch() C0000000A0DB24D0 ?
000000000 ?
C0000000A2665550 ?
000007FFF ?
updThreePhaseExe()+ call updaul() C000000091A57FA0 ?
432 000000006 ? 000000000 ?
C0000000000019BB ?
40000000019DF9B0 ?
000008E23 ?
60000000001D30F0 ?
0000000BD ?
updexe()+624 call updThreePhaseExe() C000000091A57FA0 ?
000000000 ?
9FFFFFFFBF558FA0 ?
9FFFFFFFFFFF1AF0 ?
C000000000000FA6 ?
40000000019DFCA0 ?
000000000 ?
opiexe()+6944 call updexe() C000000091A57FA0 ?
9FFFFFFFBF4C8A00 ?
C000000000002858 ?
4000000001857620 ?
000008E25 ?
9FFFFFFFFFFF1AD0 ?
9FFFFFFFBF558FA0 ?
9FFFFFFFBF4C88D4 ?
opipls()+3488 call opiexe() 000000004 ?
9FFFFFFFFFFF2F50 ?
9FFFFFFFFFFF1D30 ?
C000000000002C60 ?
40000000019852D0 ?
000000000 ? 000007123 ?
9FFFFFFFFFFF1B40 ?
opiodr()+3088 call opipls() 9FFFFFFFFFFF2F50 ?
9FFFFFFFFFFF2F58 ?
000000004 ?
C00000000000214A ?
40000000018747B0 ?
00000E123 ?
9FFFFFFFFFFF2E70 ?
9FFFFFFFFFFF2DE0 ?
rpidrus()+304 call opiodr() 9FFFFFFFFFFF3810 ?
9FFFFFFFFFFF37E0 ?
9FFFFFFFFFFF36F8 ?
9FFFFFFFFFFF36F4 ?
skgmstack()+288 call rpidrus() 9FFFFFFFFFFF4DA0 ?
C000000000000716 ?
4000000001862FE0 ?
9FFFFFFFFFFF47F0 ?
600000000004F3C0 ?
9FFFFFFFFFFF4DB8 ?
6000000000524900 ?
9FFFFFFFFFFF4DA0 ?
rpidru()+256 call skgmstack() 9FFFFFFFFFFF4DC0 ?
600000000004EDF0 ?
00000F618 ?
400000000129E400 ?
9FFFFFFFFFFF4DA0 ?
rpiswu2()+784 call rpidru() 00000057B ?
C000000000001329 ?
4000000001870CE0 ?
rpidrv()+2096 call rpiswu2() C00000007D7B3708 ?
9FFFFFFFFFFF5510 ?
6000000000515CE8 ?
60000000000502E8 ?
9FFFFFFFFFFF5520 ?
000000205 ?
6000000000530010 ?
6000000000531090 ?
psddr0()+256 call rpidrv() 000000000 ? 000000066 ?
9FFFFFFFFFFF6630 ?
00000003A ?
psdnal()+672 call psddr0() 000000000 ? 000000066 ?
9FFFFFFFFFFF6630 ?
000000030 ?
pevm_EXIM()+400 call psdnal() 9FFFFFFFFFFF84A0 ?
9FFFFFFFFFFF8440 ?
6000000000531A04 ?
600000000064BAA0 ?
C000000090880CC8 ?
600000000063EA80 ?
pfrrun()+37120 call pevm_EXIM() 9FFFFFFFBF567AD8 ?
C000000090880CC8 ?
C000000000001532 ?
400000000173DAD0 ?
000006269 ?
4000000002190B40 ?
9FFFFFFFBF4BA068 ?
9FFFFFFFBF5C08A0 ?
peicnt()+544 call pfrrun() 9FFFFFFFBF567CDC ?
C000000000000D1D ?
4000000001676FE0 ?
kkxexe()+832 call peicnt() 9FFFFFFFFFFF84A0 ?
9FFFFFFFBF567AD8 ?
C000000000000818 ?
40000000021A3A80 ?
00000FEAB ?
9FFFFFFFFFFF7EB0 ?
9FFFFFFFBF567CDC ?
000000000 ?
opiexe()+12592 call kkxexe() 9FFFFFFFFFFF8430 ?
C000000000002858 ?
4000000001858C30 ?
0000080AB ?
9FFFFFFFFFFF8420 ?
9FFFFFFFBF4F3A6C ?
9FFFFFFFBF4F3A68 ?
9FFFFFFFFFFF84A0 ?
opiall0()+3456 call opiexe() 000000004 ?
9FFFFFFFFFFF9A80 ?
9FFFFFFFFFFF8830 ?
C0000000000024D1 ?
40000000017A3350 ?
000000000 ? 00000C82B ?
9FFFFFFFFFFF8640 ?
kpoal8()+2272 call opiall0() 000000002 ?
9FFFFFFFFFFF9A10 ?
9FFFFFFFFFFFA2B4 ?
000000000 ?
9FFFFFFFFFFF9954 ?
9FFFFFFFFFFFA2BC ?
000000000 ?
FFFFFFFFFFFFDFFF ?
opiodr()+3088 call kpoal8() 9FFFFFFFFFFFA0B0 ?
000000000 ? 000000000 ?
C00000000000214A ?
40000000018747B0 ?
00000F82F ?
9FFFFFFFFFFFA130 ?
9FFFFFFFFFFFA1D0 ?
ttcpip()+1888 call opiodr() 9FFFFFFFFFFFA460 ?
9FFFFFFFFFFFA430 ?
9FFFFFFFFFFFA348 ?
9FFFFFFFFFFFA344 ?
C0000000000018B9 ?
400000000172CB70 ?
000000000 ? 00000C0AB ?
opitsk()+1920 call ttcpip() 6000000000052C40 ?
000000002 ?
9FFFFFFFFFFFB950 ?
6000000000273AA0 ?
9FFFFFFFFFFFBAB0 ?
9FFFFFFFFFFFB8C4 ?
9FFFFFFFBF75A660 ?
4000000000D2C7AA ?
opiino()+2656 call opitsk() 000000000 ? 000000000 ?
C000000000000D1F ?
400000000236CE90 ?
000000000 ?
opiodr()+3088 call opiino() 60000000005DEC18 ?
6000000000531170 ?
9FFFFFFFFFFFF4C0 ?
C00000000000214A ?
40000000018747B0 ?
00000E825 ?
9FFFFFFFFFFFD7C0 ?
9FFFFFFFFFFFF4D0 ?
opidrv()+1088 call opiodr() 9FFFFFFFFFFFDF10 ?
9FFFFFFFFFFFDEE0 ?
9FFFFFFFFFFFDDF8 ?
9FFFFFFFFFFFDDF4 ?
sou2o()+48 call opidrv() 9FFFFFFFFFFFEF60 ?
000000004 ?
9FFFFFFFFFFFF4C0 ?
main()+352 call sou2o() 9FFFFFFFFFFFF4E0 ?
00000003C ? 000000004 ?
9FFFFFFFFFFFF4C0 ?
main_opd_entry()+80 call main() 000000000 ?
9FFFFFFFFFFFF9B0 ?
C000000000000004 ?
C00000000002FA60 ?
--------------------- Binary Stack Dump ---------------------
观察节点2，发现类此错误不同之处有
Wed Mar 7 08:19:11 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Wed Mar 7 08:19:12 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Wed Mar 7 08:19:13 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] Wed Mar 7 08:19:14 2012 Errors in file /u01/app/oracle/admin/hddms/udump/hddms2_ora_16729.trc: ORA-07445: exception encountered: core dump [kghalf()+993] [SIGSEGV] [Address not mapped to object] [0xC0000000A8B01DA0] [] [] ORA-07445: exception encountered: core dump [kgidmp()+1568] [SIGSEGV] [Address not mapped to object] [0xB38F00000000D9] [] [] ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x9FFFFFFFBF55A550], [], [], [], [], [], [] ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] []
对应trace文件不同之处
*** 2012-03-07 08:19:11.582 ksedmp: internal or fatal error ORA-07445: exception encountered: core dump [__milli_memcpy()+2496] [SIGSEGV] [Address not mapped to object] [0x9FFFFFFFBF580000] [] [] No current SQL statement being executed. ----- PL/SQL Call Stack ----- object line object handle number name c000000090c4a818 1356 procedure PMIS.PM_DC_SJJGGC_QDX c0000000a25512c0 43 procedure PMIS.PM_DC_SJJGGC_EXECINF c00000009a8306e8 2 anonymous block
通过这两个节点的错误记录，都包含PMIS.PM_DC_SJJGGC_EXECINF和anonymous，所以初步怀疑是某个程序在这个时间定运行某个任务(包含PMIS.PM_DC_SJJGGC_EXECINF过程)导致。
继续观察trace日志发现两个节点都是类此情况相同的程序，相同的主机
SO: c00000007d7b3708, type: 4, owner: c00000007d6d0190, flag: INIT/-/-/0x00
(session) trans: c000000082974470, creator: c00000007d6d0190, flag: (100041) USR/- BSY/-/-/-/-/-
DID: 0001-002A-0000693D, short-term DID: 0000-0000-00000000
txn branch: 0000000000000000
oct: 6, prv: 0, sql: c0000000a229ab38, psql: c000000084e6b510, user: 517/PMIS
O/S info: user: Administrator, term: DMIS_SWITCH2, ospid: 5600:3020, machine: WORKGROUP\DMIS_SWITCH2
program: PMTDCtrl.exe
application name: PMTDCtrl.exe, hash value=0
last wait for 'enqueue' blocking sess=0x0 seq=32849 wait_time=127
name|mode=54540004, id1=1, id2=10
temporary object counter: 0
现在更加怀疑是PMTDCtrl.exe运行某过程导致数据库出现这些错误
错误总结
查看MOS发现该问题是一个bug导致(Bug 6166690:ORA-7445 [MILLI_MEMCPY] WHEN REF CURSOR AS OUT PARAMETER)
错误原因:It is caused by a Dynamic SQL used to obtain a REF CURSOR from a procedure (that opens the cursor).
处理建议:通过个bug，我们可以分析PMIS.PM_DC_SJJGGC_EXECINF过程，看看是否符合bug描述,如果符合建议改写过程或者打上补丁(Patch 2709343)
补丁:p2709343_92080_HPUX-IA64