ORA-00600[3689]引起MRP进程异常

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:ORA-00600[3689]引起MRP进程异常

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

今天检查数据库发现一数据库因为添加数据文件导致ORA-00600[3689]错误,进入引起MRP进程异常,从而使得数据库可以正常接收归档日志,但是不能在备库应用归档日志
数据库版本

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bi
PL/SQL Release 10.2.0.1.0 - Production
CORE    10.2.0.1.0      Production
TNS for Solaris: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 – Production

检查日志

Fri Oct 19 12:00:22 2012
RFS[1]: No standby redo logfiles created
RFS[1]: Archived Log: '/baceldba/archive/1_30374_730819916.dbf'
Fri Oct 19 12:00:28 2012
Media Recovery Log /baceldba/archive/1_30374_730819916.dbf
WARNING: File being created with same name as in Primary
Existing file may be overwritten
Fri Oct 19 12:00:50 2012
Recovery created file /data/oracle/oradata/vodapp/NETTVCLPS02.dbf
Successfully added datafile 32 to media recovery
Datafile #32: '/data/oracle/oradata/vodapp/NETTVCLPS02.dbf '
WARNING: File being created with same name as in Primary
Existing file may be overwritten
Fri Oct 19 12:01:15 2012
Recovery created file /data/oracle/oradata/vodapp/UCICMS01_DATA04.dbf
Successfully added datafile 33 to media recovery
Datafile #33: '/data/oracle/oradata/vodapp/UCICMS01_DATA04.dbf'
Fri Oct 19 12:01:16 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Errors with log /baceldba/archive/1_30374_730819916.dbf
MRP0: Background Media Recovery terminated with error 600  <--mrp进程异常
Fri Oct 19 12:01:16 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail
Fri Oct 19 12:01:18 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
Fri Oct 19 12:01:18 2012
Errors in file /data/oracle/admin/vodapp/bdump/vodapp_mrp0_21304.trc:
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []

通过观察日志发现,数据库的mrp0进程因为ora-600的错误导致异常终止,从而使得dg备库无法正常使用归档日志

分析trace文件得出

*** 2012-10-19 12:01:16.327
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [3689], [33], [], [], [], [], [], []
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedmp()+744         CALL     ksedst()             000000840 ?
                                                   FFFFFFFF7FFFA08C ?
                                                   000000000 ?
                                                   FFFFFFFF7FFF6B80 ?
                                                   FFFFFFFF7FFF58E8 ?
                                                   FFFFFFFF7FFF62E8 ?
kgeriv()+220         PTR_CALL 0000000000000000     000106000 ? 106323304 ?
                                                   106323000 ? 000106323 ?
                                                   000106000 ? 106323304 ?
kgesiv()+112         CALL     kgeriv()             10631DCD0 ? 000000000 ?
                                                   000000E69 ? 000000001 ?
                                                   FFFFFFFF7FFFA5E8 ?
                                                   000001430 ?
ksesic1()+96         CALL     kgesiv()             10631DCD0 ? 10646AD00 ?
                                                   000000E69 ? 000000001 ?
                                                   FFFFFFFF7FFFA5E8 ?
                                                   000001420 ?
kcvadc1_10gr2()+265  CALL     ksesic1()            000000E69 ? 00010631D ?
2                                                  1063232F8 ? 000106000 ?
                                                   106323000 ? 000106323 ?
kcoapl()+5664        PTR_CALL 0000000000000000     FFFFFFFF7A5570F8 ?
                                                   FFFFFFFF7FFFB320 ?
                                                   FFFFFFFF76667E18 ?
                                                   000000000 ? 000000021 ?
                                                   000000020 ?
kcramr()+5352        CALL     kcoapl()             FFFFFFFF76667E00 ?
                                                   000000003 ? 1055136B0 ?
                                                   105513340 ? 00000001B ?
                                                   000000017 ?
krddmr()+2688        CALL     kcramr()             000000000 ?
                                                   FFFFFFFF7A55FF78 ?
                                                   000002000 ? 000106000 ?
                                                   0000001F6 ? 464897868 ?
krsmdp()+2332        CALL     krddmr()             105502000 ?
                                                   FFFFFFFF7A5570F8 ?
                                                   000000001 ? 10646C760 ?
                                                   464897868 ? 000000000 ?
ksbrdp()+896         PTR_CALL 0000000000000000     000380000 ? 000380000 ?
                                                   105517000 ? 000106328 ?
                                                   000106000 ? 00000FFFF ?
opirip()+824         CALL     ksbrdp()             000106320 ? 380007734 ?
                                                   000380000 ? 00038000E ?
                                                   000106000 ? 1005DCF80 ?
opidrv()+1200        CALL     opirip()             10632A000 ? 000106000 ?
                                                   000106332 ? 380007000 ?
                                                   00010632A ? 1063D73A0 ?
sou2o()+80           CALL     opidrv()             10632D460 ? 000000001 ?
                                                   000000032 ? 000000000 ?
                                                   000000032 ? 000106000 ?
opimai_real()+268    CALL     sou2o()              FFFFFFFF7FFFF968 ?
                                                   000000032 ? 000000004 ?
                                                   FFFFFFFF7FFFF990 ?
                                                   105C1F000 ? 000105C1F ?
main()+152           CALL     opimai_real()        000000003 ?
                                                   FFFFFFFF7FFFFA68 ?
                                                   000000000 ? 000000000 ?
                                                   0023F9764 ? 000014400 ?
_start()+380         CALL     main()               000000001 ?
                                                   FFFFFFFF7FFFFB78 ?
                                                   000000000 ?
                                                   FFFFFFFF7FFFFA70 ?
                                                   FFFFFFFF7FFFFB80 ?
                                                   FFFFFFFF7CF00200 ?
 
--------------------- Binary Stack Dump ---------------------

查询mos发现
Bug 5230162 : ORA-600[3689][42] ON PHYSICAL STANDBY PREVENT MEDIA RECOVERY描述相符

处理建议
1.10.2.0.1数据库不稳定,bug较多,建议升级数据库到10.2.0.4或者10.2.0.5
2.当前standby_file_management为true,建议修改为false,每次手工两边增加数据文件
3.主库添加数据文件后,观察备库的mrp进程是否正常,如果异常
1)尝试启动mrp进程,如果因为新增加的数据文件导致异常,那先使用rman或者其他方法拷贝主库新建立数据文件到备库,然后启动mrp进程
2)如果因为某种原因导致归档日志出现gap,可以从备份中找出归档日志,如果备份中没有,那可以使用基于scn的备份方式来修复该故障。
3)如果scn相差较大,建议直接重建备库

此条目发表在 Data Guard 分类目录。将固定链接加入收藏夹。

ORA-00600[3689]引起MRP进程异常》有 1 条评论

  1. 惜分飞 说:

    Bug 5230162 : ORA-600[3689][42] ON PHYSICAL STANDBY PREVENT MEDIA RECOVERY

    Hdr: 5230162 10.2.0.1 RDBMS 10.2.0.1 DICTIONARY PRODID-5 PORTID-226 ORA-600
    
    Abstract: ORA-600[3689][42] ON PHYSICAL STANDBY PREVENT MEDIA RECOVERY
     
    *** 05/16/06 07:17 pm ***
    TAR:
    ----
     
    PROBLEM:
    --------
    Physical standby media recovery mrp0 failed with
    ORA-600[3689][42]
    This cause physical standby stop recovery.
     
    Ct performed add 3 datafiles to tablespace bilanpuissance around May 15 
    21:41:15. Also did resize of the datafiles. 13min later at May 15 21:58:14, 
    they drop those 3 datafiles and create another tablespace 
    BILANPUISSANCE_2006.
    After that they did data load to above tablespaces. 
    But at May 16 08:15:57, both tablespaces are dropped.
     
    On standby site, we can see datafiles are added, straight after that 
    ora-600[3689][42] is complained:
     
    Mon May 15 22:02:09 2006
    Successfully added datafile 42 to media recovery
    Datafile #42: '+DATA/pp_stby/datafile/bilanpuissance.461.590536923'
    Successfully added datafile 46 to media recovery
    Datafile #46: '+DATA/pp_stby/datafile/bilanpuissance.462.590536929'
    on May 15 22:05:06 2006
    Successfully added datafile 47 to media recovery
    Datafile #47: '+DATA/pp_stby/datafile/bilanpuissance.463.590537069'
    Mon May 15 22:05:39 2006
    Errors in file /opt/oracle/admin/coparppr/bdump/pp_stby_mrp0_3600.trc:
    ORA-600: internal error code, arguments: [3689], [42], [], [], [], [], [], 
    []
     
     
    Primary Database are 2 nodes RAC, standby is single node. All are 10.2
     
    It is currently happening on ct site.
     
    they
     
    DIAGNOSTIC ANALYSIS:
    --------------------
    ora-600[3689][42] is complaining about file #42 is already part of media 
    recovery, but we are trying to add a file with #42 again.
     
    Try to offline drop file #42, #46 and #46 on standby. Not help.
     
    Trying to recreate a standby controlfile from primary. But all datafile names 
    are different (OMF), too troublesome to alter all file name (on ASM), so this 
    approach is not tried.
     
    WORKAROUND:
    -----------
    None
     
    RELATED BUGS:
    -------------
    None. Only bug 2802219 has similar error but fixed in 10.1
     
    REPRODUCIBILITY:
    ----------------
    currently on ct site
     
    TEST CASE:
    ----------
     
    STACK TRACE:
    ------------
    ksedst ksedmp ksfdmp kgeriv kgesiv ksesic1 kcvadc1_10gr2 kcvafc_10gr2 kcoapl 
    kcramr krddmr krsmdp ksbrdp opirip opidrv sou2o
     
    SUPPORTING INFORMATION:
    -----------------------
    alert log - both primary and standby
    pp_stby_mrp0_30732.trc - mrp0 trace. more mrp0 trace available in RDA
    RDA.RDA_coprac4_20060516.zip - RDA of standby
     
    24 HOUR CONTACT INFORMATION FOR P1 BUGS:
    ----------------------------------------
    support contact for next 2 hour - Joei Xu +61 3 86164121 AIM: jwxuau
     
    DIAL-IN INFORMATION:
    --------------------
     
    IMPACT DATE:
    ------------
    now