标签归档：kgegpa

ORA-600 kkkicreatecgmap:!efn3

发表于 2025 年 10 月 23 日 由 惜分飞

raid故障恢复之后,数据库recover成功,但是open报ORA-03113: end-of-file on communication channel错误

SQL> recover database;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 20394
Session ID: 191 Serial number: 3

对应的alert日志错误为ORA-600 [kkkicreatecgmap:!efn3]错误

ALTER DATABASE RECOVER  database
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 4 slaves
Mon Oct 20 18:51:06 2025
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Media Recovery Complete (orcl)
Completed: ALTER DATABASE RECOVER  database
Mon Oct 20 18:51:16 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 41 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 32119, block 34978
Recovery of Online Redo Log: Thread 1 Group 1 Seq 32119 Reading mem 0
  Mem# 0: /u01/oradata/redo01.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 32119, block 35061, scn 17375938230308
 0 data blocks read, 0 data blocks written, 41 redo k-bytes read
Mon Oct 20 18:51:16 2025
Thread 1 advanced to log sequence 32120 (thread open)
Thread 1 opened at log sequence 32120
  Current log# 2 seq# 32120 mem# 0: /u01/oradata/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:51:16 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile /u01/oradata/temp01.dbf
Database Characterset is ZHS16GBK
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x2297750, kgegpa()+40][flags: 0x0, count: 1]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0x3999DC33][PC:0x229597B, kgebse()+279][flags: 0x2, count: 2]
No Resource Manager plan active
Mon Oct 20 18:51:16 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc  (incident=3649):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_3649/orcl_dbrm_20286_i3649.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20286.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20286): terminating the instance due to error 56710
Instance terminated by DBRM, pid = 20286

对应的trace文件内容


----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
skdstdst()+36        call     kgdsdst()            000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   7FFD08773E78 ? 000000000 ?
ksedst1()+98         call     skdstdst()           000000000 ? 000000000 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedst()+34          call     ksedst1()            000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkedDefDump()+2736  call     ksedst()             000000000 ? 000000001 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksedmp()+36          call     dbkedDefDump()       000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
ksfdmp()+64          call     ksedmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexPhaseII()+1764  call     ksfdmp()             000000003 ? 000000002 ?
                                                   7FFD0876F978 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgexProcessError()  call     dbgexPhaseII()       7F16CCF3E6F0 ? 7F16CA6F2598 ?
+2279                                              7FFD0877BC68 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbgeExecuteForError  call     dbgexProcessError()  7F16CCF3E6F0 ? 7F16CA6F2598 ?
()+83                                              000000001 ? 000000000 ?
                                                   7FFD00000000 ? 000000000 ?
dbgePostErrorKGE()+  call     dbgeExecuteForError  7F16CCF3E6F0 ? 7F16CA6F2598 ?
1615                          ()                   000000001 ? 000000001 ?
                                                   000000000 ? 000000000 ?
dbkePostKGE_kgsf()+  call     dbgePostErrorKGE()   000000000 ? 7F16CA560040 ?
63                                                 000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgeadse()+383        call     dbkePostKGE_kgsf()   00A99D360 ? 7F16CA560040 ?
                                                   000000258 ? 7F16CA6F2598 ?
                                                   000000000 ? 000000000 ?
kgerinv_internal()+  call     kgeadse()            00A99D360 ? 7F16CA560040 ?
45                                                 000000258 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kgerinv()+33         call     kgerinv_internal()   00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000258 ?
                                                   000000000 ? 000000000 ?
kgeasnmierr()+143    call     kgerinv()            00A99D360 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 000000000 ?
kkkicreatecgmap()+3  call     kgeasnmierr()        00A99D360 ? 7F16CA560040 ?
125                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskirefreshcgmap()+  call     kkkicreatecgmap()    121DC3050 ? 7F16CA560040 ?
104                                                877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskreload()+1150     call     kskirefreshcgmap()   121DC3050 ? 7F16CA560040 ?
                                                   877420000000000 ? 000000000 ?
                                                   000000000 ? 00000057B ?
kskdbrmpa()+378      call     kskreload()          7FFD0877D428 ? 000000001 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbabs()+465         call     kskdbrmpa()          7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
ksbrdp()+923         call     ksbabs()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opirip()+618         call     ksbrdp()             7FFD0877D418 ? 000000058 ?
                                                   000000000 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opidrv()+598         call     opirip()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
sou2o()+98           call     opidrv()             000000032 ? 000000004 ?
                                                   7FFD0877E598 ? 000000001 ?
                                                   000000000 ? 00000020A ?
opimai_real()+261    call     sou2o()              7FFD0877E570 ? 000000032 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
ssthrdmain()+209     call     opimai_real()        000000000 ? 7FFD0877E760 ?
                                                   000000004 ? 7FFD0877E598 ?
                                                   000000000 ? 00000020A ?
main()+196           call     ssthrdmain()         000000003 ? 7FFD0877E760 ?
                                                   000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
__libc_start_main()  call     main()               000000003 ? 7FFD0877E900 ?
+253                                               000000001 ? 000000000 ?
                                                   000000000 ? 00000020A ?
_start()+36          call     __libc_start_main()  0009D3D94 ? 000000001 ?
                                                   7FFD0877E8F8 ? 000000000 ?
                                                   000000000 ? 00000020A ?

虽然ORA-600 kkkicreatecgmap:!efn3没有见过但是数据库open过程中kgegpa、kgebse的故障还是遇到不少,大部分可能和undo有一定关系,处理undo问题之后,继续尝试open库,依旧报ORA-03113: end-of-file on communication channel

SQL> alter database Open;
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 20586
Session ID: 191 Serial number: 3

但是后台的alert日志已经改变ORA-600 4193,ORA-600 kkkicreatecgmap:!efn3错误

Mon Oct 20 18:54:39 2025
Thread 1 advanced to log sequence 32121 (thread open)
Thread 1 opened at log sequence 32121
  Current log# 3 seq# 32121 mem# 0: /u01/oradata/redo03.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Oct 20 18:54:39 2025
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4905):
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4905/orcl_smon_20492_i4905.trc
No Resource Manager plan active
Mon Oct 20 18:54:39 2025
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc  (incident=4849):
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4849/orcl_dbrm_20478_i4849.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_dbrm_20478.trc:
ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403], [0], [], [], [], [], [], [], [], [], []
DBRM (ospid: 20478): terminating the instance due to error 56710
Some DDE async actions failed or were cancelled
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc  (incident=4906):
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_4906/orcl_smon_20492_i4906.trc
Errors in file /u01/app/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_smon_20492.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 8 change 17375938190767 time 10/03/2025 00:20:34
ORA-00312: online log 1 thread 1: '/u01/oradata/redo01.log'
ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
Instance terminated by DBRM, pid = 20478

虽然ORA-600 kkkicreatecgmap:!efn3还在,但是看到了比较熟悉的ORA-600 4193错误,处理undo异常回滚段,数据库open成功,重建undo,尝试导出数据,完成数据恢复任务.

补充说明,对于ORA-00600: internal error code, arguments: [kkkicreatecgmap:!efn3], [1403]网络上没有任何资料,查询了mos发现一个有一点类似的报错信息
Bug 28167557 – bigscn_dbim_tm_def – trc – kkkdchkcriticalobj – ORA-700 [kkkdchkcriticalob (Doc ID 28167557.8)

Description
Information about new symptoms:
 
- Signaling function: kkkdchkcriticalobj (kkkd.c)
- Symptom: ORA-700 [kkkdchkcriticalobj:fail]
- Owner: SUELEE
- Special Run:  BIGSCN_DBIM_TM_DEF (owner: WAI-SZE.TAM, sr_id: 2709)
- Release: 19.1
- Job id: 22488605
- Base label:  RDBMS_MAIN_LINUX.X64_180607
 
ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 
 REDISCOVERY INFORMATION:
 Symptoms are any of the following:
 ORA-600 [KKKICHKRMAPPRI:0=NR PEND] 
 ORA-600 kkkicreatecgmap:!group
 ORA-700 kkkdchkcriticalobj
  
 .
 WORKAROUND:
 None

由于这个是硬件故障恢复出来的数据文件（涉及磁盘坏道，磁盘顺序，磁盘在raid中均衡,raid的cache等因素可能会一起Oracle各种非常规问题）,分析原因意义不大,重点是快速解决问题,不做过多分析

发表在 Oracle备份恢复 | 标签为 , , , , , , | 评论关闭

ORA-600 4194引起SMON encountered 100 out of maximum 100 non-fatal internal errors故障

发表于 2025 年 9 月 23 日 由 惜分飞

客户数据库11.2.0.3版本,由于机房停电导致数据库启动之后一段时间自动crash

Sat Sep 20 20:31:14 2025
QMNC started with pid=39, OS id=10637 
Completed: ALTER DATABASE OPEN
Starting background process CJQ0
Sat Sep 20 20:31:14 2025
CJQ0 started with pid=44, OS id=10654 
Setting Resource Manager plan SCHEDULER[0x318E]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Starting background process VKRM
Sat Sep 20 20:31:17 2025
VKRM started with pid=40, OS id=10680 
Sat Sep 20 20:38:01 2025
Starting background process SMCO
Sat Sep 20 20:38:01 2025
SMCO started with pid=38, OS id=10955 
Sat Sep 20 20:56:54 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_11564.trc (incident=148368):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Block recovery from logseq 1082886, block 29263 to scn 74449804596
Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo14.log
Block recovery stopped at EOT rba 1082886.29264.16
Block recovery completed at rba 1082886.29264.16, scn 17.1435360559
Block recovery from logseq 1082886, block 29263 to scn 74449804590
Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo14.log
Block recovery completed at rba 1082886.29264.16, scn 17.1435360559
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_11564.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Sat Sep 20 20:56:58 2025
Sweep [inc][148368]: completed
Sweep [inc2][148368]: completed
Sat Sep 20 21:00:20 2025
Exception[type:SIGSEGV,Address not mapped to object][ADDR:0xBC44AC1][PC:0x932F8EA,kgegpa()+40][flags:0x0,count:1]
Exception[type:SIGSEGV,Address not mapped to object][ADDR:0xBC44AC1][PC:0x932DEF3,kgebse()+771][flags:0x2,count:2]
Exception[type:SIGSEGV,Address not mapped to object][ADDR:0xBC44AC1][PC:0x932DEF3,kgebse()+771][flags:0x2,count:2]
Sat Sep 20 21:00:21 2025
Block recovery from logseq 1082886, block 29263 to scn 74449804596
Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo14.log
……………………
Sat Sep 20 21:05:00 2025
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc(incident=148296):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Block recovery from logseq 1082886, block 32045 to scn 74449805729
Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo14.log
Block recovery completed at rba 1082886.32056.16, scn 17.1435361698
Block recovery from logseq 1082886, block 32045 to scn 74449806046
Recovery of Online Redo Log: Thread 1 Group 14 Seq 1082886 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo14.log
Block recovery completed at rba 1082886.32321.16, scn 17.1435362015
Non-fatal internal error happenned while SMON was doing flushing of monitored table stats.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
*******
Block recovery completed at rba 1082898.52054.16, scn 17.1444838013
Non-fatal internal error happenned while SMON was doing flushing of monitored table stats.
SMON encountered 99 out of maximum 100 non-fatal internal errors.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc(incident=164458):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Block recovery from logseq 1082898, block 52038 to scn 74459282045
Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo08.log
Block recovery completed at rba 1082898.52054.16, scn 17.1444838014
Block recovery from logseq 1082898, block 52038 to scn 74459282088
Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo08.log
Block recovery completed at rba 1082898.52104.16, scn 17.1444838057
Non-fatal internal error happenned while SMON was doing flushing of monitored table stats.
SMON encountered 100 out of maximum 100 non-fatal internal errors.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc  (incident=164459):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Block recovery from logseq 1082898, block 52038 to scn 74459282045
Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo08.log
Block recovery completed at rba 1082898.52054.16, scn 17.1444838014
Block recovery from logseq 1082898, block 52038 to scn 74459282101
Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo08.log
Block recovery completed at rba 1082898.52130.16, scn 17.1444838070
Non-fatal internal error happenned while SMON was doing flushing of monitored table stats.
SMON exceeded the maximum limit of 100 internal error(s).
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_10516.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
SMON (ospid: 10516): terminating the instance due to error 474
Mon Sep 22 04:05:28 2025
System state dump requested by(instance=1,osid=10516 (SMON)),summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_10463.trc
Instance terminated by SMON, pid = 10516

错误原因比较明显是由于“Non-fatal internal error happenned while SMON was doing flushing of monitored table stats.”smon进程对MON_MODS$/MON_MODS_ALL$操作异常触发ORA-600 4194错误使得该操作失败,默认情况smon尝试100次(_smon_internal_errlimit该参数控制)依旧没有成功,会强制终止smon进程,从而导致实例crash.然后尝试重启数据库无法启动成功

Mon Sep 22 09:00:03 2025
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 1360 KB redo, 405 data blocks need recovery
Started redo application at
 Thread 1: logseq 1082898, block 49410
Recovery of Online Redo Log: Thread 1 Group 8 Seq 1082898 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo08.log
Completed redo application of 0.60MB
Completed crash recovery at
 Thread 1: logseq 1082898, block 52130, scn 74459302102
 405 data blocks read, 405 data blocks written, 1360 redo k-bytes read
Thread 1 advanced to log sequence 1082899 (thread open)
Thread 1 opened at log sequence 1082899
  Current log# 9 seq# 1082899 mem# 0: /oracledb/oradata/orcl/redo09.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_78465.trc  (incident=164779):
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Block recovery from logseq 1082899, block 3 to scn 74459302111
Recovery of Online Redo Log: Thread 1 Group 9 Seq 1082899 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo09.log
Block recovery stopped at EOT rba 1082899.5.16
Block recovery completed at rba 1082899.5.16, scn 17.1444858077
Block recovery from logseq 1082899, block 3 to scn 74459302108
Recovery of Online Redo Log: Thread 1 Group 9 Seq 1082899 Reading mem 0
  Mem# 0: /oracledb/oradata/orcl/redo09.log
Block recovery completed at rba 1082899.5.16, scn 17.1444858077
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_78465.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_78465.trc:
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 78465): terminating the instance due to error 600
Instance terminated by USER, pid = 78465
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (78465) as a result of ORA-1092
Mon Sep 22 09:00:08 2025
ORA-1092 : opitsk aborting process

客户再次尝试几次之后,最后直接无法正常mount库

Mon Sep 22 19:14:14 2025
ALTER DATABASE   MOUNT
USER (ospid: 11679): terminating the instance
System state dump requested by (instance=1, osid=11679), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_11615.trc
Dumping diagnostic data in directory=[cdmp_20250922191419],requested by(instance=1,osid=11679)
   ,summary=[abnormal instance termination].
Instance terminated by USER, pid = 11679

这个故障处理起来相对比较简单:
1)根据当前损坏的的ctl以及操作系统的控制数据文件,redo以及alert日志中数据库字符集信息,构造重建控制文件语句,对该库进行rectl
2)由于ORA-600 4194错误,明显指向undo异常,通过对异常undo的回滚段处理,打开数据库

发表在 Oracle备份恢复 | 标签为 , , , | 评论关闭

存储宕机导致Oracle异常故障处理

发表于 2024 年 7 月 28 日 由 惜分飞

存储突然掉线，导致数据库crash,报大量ORA-00206 ORA-00202 ORA-15081以及Linux-x86_64 Error: 5: Input/output error之类的错误

Sun Jul 21 20:00:11 2024
Thread 1 advanced to log sequence 1594398 (LGWR switch)
  Current log# 5 seq# 1594398 mem# 0: +DATA/xff/onlinelog/group_5.412.906718739
Sun Jul 21 20:53:17 2024
WARNING: Write Failed. group:2 disk:0 AU:506916 offset:49152 size:16384
Sun Jul 21 20:53:17 2024
WARNING: Read Failed. group:2 disk:2 AU:506931 offset:49152 size:16384
WARNING: failed to read mirror side 1 of virtual extent 4 logical extent 0 of file 415 in group [2.34109396] 
from disk ORACLE_DATA_0002  allocation unit 506931 reason error; if possible, will try another mirror side
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_ckpt_42142.trc:
ORA-15080: 与磁盘的同步 I/O 操作失败
ORA-27061: 异步 I/O 等待失败
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0 
of file 415 in group 2 on disk 0 allocation unit 506916 
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_ckpt_42142.trc:
ORA-00206: 写入控制文件时出错 (块 3, # 块 1)
ORA-00202: 控制文件: ''+DATA/xff/controlfile/current.415.906718737''
ORA-15081: 无法将 I/O 操作提交到磁盘
ORA-15081: 无法将 I/O 操作提交到磁盘
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_ckpt_42142.trc:
ORA-00221: 写入控制文件时出错
ORA-00206: 写入控制文件时出错 (块 3, # 块 1)
ORA-00202: 控制文件: ''+DATA/xff/controlfile/current.415.906718737''
ORA-15081: 无法将 I/O 操作提交到磁盘
ORA-15081: 无法将 I/O 操作提交到磁盘
CKPT (ospid: 42142): terminating the instance due to error 221
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_lmon_42087.trc:
ORA-00202: 控制文件: ''+DATA/xff/controlfile/current.415.906718737''
ORA-15081: 无法将 I/O 操作提交到磁盘
ORA-27072: 文件 I/O 错误
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1038194784
Additional information: -1
Sun Jul 21 20:53:19 2024
ORA-1092 : opitsk aborting process
Sun Jul 21 20:53:24 2024
ORA-1092 : opitsk aborting process
Sun Jul 21 20:53:24 2024
License high water mark = 59
Sun Jul 21 20:53:28 2024
Instance terminated by CKPT, pid = 42142
USER (ospid: 64660): terminating the instance
Instance terminated by USER, pid = 64660

存储恢复之后启动数据库报ORA-600 2131错误

Mon Jul 22 09:10:04 2024
ALTER DATABASE   MOUNT
This instance was first to mount
Mon Jul 22 09:10:04 2024
Sweep [inc][490008]: completed
Sweep [inc2][490008]: completed
NOTE: Loaded library: System 
SUCCESS: diskgroup ORACLE_DATA was mounted
NOTE: dependency between database rac and diskgroup resource ora.ORACLE_DATA.dg is established
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_ora_14301.trc  (incident=492409):
ORA-00600: ??????, ??: [2131], [33], [32], [], [], [], [], [], [], [], [], []
Incident details in: /users/oracle/app/db/diag/rdbms/xff/xff1/incident/incdir_492409/xff1_ora_14301_i492409.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: ALTER DATABASE   MOUNT...

客户尝试重建ctl进行恢复,结果由于分析不正确,导致在重建ctl的时候,遗漏了3个数据文件,并且在屏蔽一致性的情况下,强制resetlogs操作,结果数据库没有被正常打开,而是报ORA-600 2662错误

alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 9965567206652
Clearing online redo logfile 1 +DATA/xff/onlinelog/group_1.414.906718739
Clearing online log 1 of thread 1 sequence number 0
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATA/xff/onlinelog/group_2.413.906718739
Clearing online log 2 of thread 1 sequence number 0
Clearing online redo logfile 2 complete
Clearing online redo logfile 5 +DATA/xff/onlinelog/group_5.412.906718739
Clearing online log 5 of thread 1 sequence number 0
Clearing online redo logfile 5 complete
Expanded controlfile section 2 from 1 to 63 records
The number of logical blocks in section 2 remains the same
Expanded controlfile section 1 from 4 to 66 records
Requested to grow by 62 records; added 32 blocks of records
Expanded controlfile section 30 from 1 to 63 records
The number of logical blocks in section 30 remains the same
Expanded controlfile section 29 from 1 to 63 records
The number of logical blocks in section 29 remains the same
Control file has been expanded to support 63 threads
Mon Jul 22 23:04:07 2024
Redo thread 2 enabled by open resetlogs or standby activation
Online log +DATA/xff/onlinelog/group_1.414.906718739: Thread 1 Group 1 was previously cleared
Online log +DATA/xff/onlinelog/group_2.413.906718739: Thread 1 Group 2 was previously cleared
Online log +DATA/xff/onlinelog/group_3.501.1175036643: Thread 2 Group 3 was previously cleared
Online log +DATA/xff/onlinelog/group_4.502.1175036645: Thread 2 Group 4 was previously cleared
Online log +DATA/xff/onlinelog/group_5.412.906718739: Thread 1 Group 5 was previously cleared
Mon Jul 22 23:04:08 2024
Setting recovery target incarnation to 2
Initializing SCN for created control file
Database SCN compatibility initialized to 3
Warning - High Database SCN: Current SCN value is 9965567206655, threshold SCN value is 0
If you have not previously reported this warning on this database, 
please notify Oracle Support so that additional diagnosis can be performed.
Mon Jul 22 23:04:09 2024
Assigning activation ID 2763017873 (0xa4b04e91)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: +DATA/xff/onlinelog/group_1.414.906718739
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Mon Jul 22 23:04:10 2024
SMON: enabling cache recovery
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_ora_64210.trc  (incident=624374):
ORA-00600: 内部错误代码, 参数: [2662], [2320], [1243079939], [2320], [1243211805], [12583040], [], [], [], [], [], []
Incident details in: /users/oracle/app/db/diag/rdbms/xff/xff1/incident/incdir_624374/xff1_ora_64210_i624374.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_ora_64210.trc:
ORA-00600: 内部错误代码, 参数: [2662], [2320], [1243079939], [2320], [1243211805], [12583040], [], [], [], [], [], []
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_ora_64210.trc:
ORA-00600: 内部错误代码, 参数: [2662], [2320], [1243079939], [2320], [1243211805], [12583040], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 64210): terminating the instance due to error 600
Instance terminated by USER, pid = 64210
ORA-1092 signalled during: alter database open resetlogs...

操作到这里,后续问题就比较麻烦了,因为在asm磁盘组中数据文件重建ctl的时候遗漏3个并且还被resetlogs操作过,导致这三个文件的resetlogs scn和其他数据文件不一致,对于这个问题,解决办法通过Oracle Recovery Tools工具或者bbed修改相关resetlogs scn,然后重建ctl

SQL> @rectl.sql

Control file created.

SQL> RECOVER DATABASE;
Media recovery complete

然后解决之前数据库启动报ORA-600 2662问题,通过修改数据库scn进行解决,可以使用Patch_SCN工具进行快速解决,然后open数据库成功

SQL> ALTER DATABASE OPEN;
 
Database altered.

但是查看alert日志数据库报大量ORA-600 4194、ORA-01595和Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xC21D511] [PC:0x97F4EFA, kgegpa()+40]之类错误

Wed Jul 24 15:24:21 2024
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
…………
Database Characterset is ZHS16GBK
No Resource Manager plan active
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_smon_40279.trc  (incident=777938):
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed Jul 24 15:24:40 2024
QMNC started with pid=79, OS id=54632 
Block recovery from logseq 2, block 74 to scn 9965587206835
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/redo02
LOGSTDBY: Validating controlfile with logical metadata
Wed Jul 24 15:24:40 2024
Block recovery stopped at EOT rba 2.82.16
Block recovery completed at rba 2.82.16, scn 2320.1263080114
Block recovery from logseq 2, block 74 to scn 9965587206833
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/redo02
Block recovery completed at rba 2.82.16, scn 2320.1263080114
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_smon_40279.trc:
ORA-01595: 释放区 (4) 回退段 (20) 时出错
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
LOGSTDBY: Validation complete
Wed Jul 24 15:24:41 2024
Sweep [inc][777938]: completed
Sweep [inc2][777938]: completed
Wed Jul 24 15:24:41 2024
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_q001_54657.trc  (incident=778362):
ORA-00600: 内部错误代码, 参数: [4194], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Starting background process SMCO
Wed Jul 24 15:24:42 2024
SMCO started with pid=83, OS id=54691 
Block recovery from logseq 2, block 74 to scn 9965587206835
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/redo02
Block recovery completed at rba 2.82.16, scn 2320.1263080118
Block recovery from logseq 2, block 74 to scn 9965587206838
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATA/xff/onlinelog/redo02
Block recovery completed at rba 2.83.16, scn 2320.1263080119
Error 600 in kwqmnpartition(), aborting txn 
Errors in file /users/oracle/app/db/diag/rdbms/xff/xff1/trace/xff1_q001_54657.trc  (incident=778363):
ORA-25319: 队列表重新分区已中止
Completed: alter database open
Block recovery from logseq 2, block 74 to scn 9965587206835
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATA/rac/onlinelog/redo02
Block recovery completed at rba 2.82.16, scn 2320.1263080118
Block recovery from logseq 2, block 74 to scn 9965587207538
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: +DATA/rac/onlinelog/redo02
Block recovery completed at rba 2.1097.16, scn 2320.1263080819
Errors in file /users/oracle/app/db/diag/rdbms/rac/rac1/trace/rac1_cjq0_55657.trc  (incident=778427):
ORA-00600: 内部错误代码, 参数: [600], [ORA-00600: 内部错误代码, 参数: 
[4194], [], [], [], [], [], [], [], [], [], [], []], [], [], [], [], [], [], [], [], [], []
Incident details in: /users/oracle/app/db/diag/rdbms/xff/xff1/incident/incdir_778427/xff1_cjq0_55657_i778427.trc
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0xC21D511][PC:0x97F4EFA, kgegpa()+40][flags: 0x0, count: 1]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0xC21D511][PC:0x97F396E, kgebse()+776][flags: 0x2, count: 2]
Exception [type:SIGSEGV, Address not mapped to object][ADDR:0xC21D511][PC:0x97F396E, kgebse()+776][flags: 0x2, count: 2]

从报错分析是由于undo异常导致,处理异常undo回滚段之后,数据库open正常,安排逻辑迁移数据,完成本次恢复

发表在 Oracle备份恢复 | 标签为 , , , , , | 评论关闭