分享一次ORA-01113 ORA-01110故障处理过程

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:分享一次ORA-01113 ORA-01110故障处理过程

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

近期编写了小工具,对于此类问题,可以实现一键恢复,参考:一键恢复ORA-01113 ORA-01110—Oracle Recovery Tools

数据库报ORA-01113 ORA-01110无法正常open

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'

alert日志

Wed Jul 16 17:17:56 2014
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 1380870980
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Wed Jul 16 17:19:01 2014
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_m000_3540.trc  (incident=367575):
ORA-00700: soft internal error, arguments: [dbgrmblcp_corrupt_page], [D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\metadata\HM_FINDING.ams], 
[1245], [], [], [], [], [], [], [], [], []
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_367575\orcl_m000_3540_i367575.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_m000_3540.trc  (incident=367576):
ORA-00600: internal error code, arguments: [dbgrmblgp_get_page_1], [1245], [0], [80], [], [], [], [], [], [], [], []
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_367576\orcl_m000_3540_i367576.trc
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_m000_3540.trc:
ORA-51106: check failed to complete due to an error.  See error below
ORA-00600: internal error code, arguments: [dbgrmblgp_get_page_1], [1245], [0], [80], [], [], [], [], [], [], [], []
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'

这里出现ORA-00700[dbgrmblcp_corrupt_page]与ORA-00600[dbgrmblgp_get_page_1]是Bug 10321285 : ORA-600 [DBGRMBLCP_CORRUPT_PAGE]

尝试recover database 遭遇ORA-00283 ORA-16433

SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database must be opened in read/write mode.

alert日志信息

Wed Jul 16 17:36:33 2014
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Wed Jul 16 17:36:33 2014
Media Recovery failed with error 16433
Slave exiting with ORA-283 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr00_3868.trc:
ORA-00283: recovery session canceled due to errors
ORA-16433: The database must be opened in read/write mode.
Recovery Slave PR00 previously exited with exception 283
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...
Wed Jul 16 17:36:45 2014
alter database open
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3236.trc:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-1113 signalled during: alter database open...

重建控制文件后恢复,遭遇ORA-600 2662错误

SQL> shutdown abort
ORACLE instance shut down.
SQL> startup nomount
ORACLE instance started.

Total System Global Area      10255212544 bytes
Fixed Size                        2264088 bytes
Variable Size                  4529849320 bytes
Database Buffers               5704253440 bytes
Redo Buffers                     18845696 bytes
SQL> @d:\ctl.sql

Control file created.

SQL> recover database using backup controlfile;
ORA-00279: change 13953689551797 generated at 07/16/2014 17:53:41 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\ORCL\ARCHIVELOG\2014_07_16\O1_MF_1_1_%U_

.ARC
ORA-00280: change 13953689551797 for thread 1 is in sequence #1


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
d:\app\administrator\oradata\orcl\redo01.log
Log applied.
Media recovery complete.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [2662], [3248], [3635774399],
[3249], [3580470066], [4194545], [], [], [], [], [], []
Process ID: 3932
Session ID: 200 Serial number: 1

因为该数据库是11.2.0.3未打上scn patch,直接推进scn(可以使用event,隐含参数,oradebug,bbed等),数据库就正常open

此条目发表在 非常规恢复 分类目录,贴了 , , , , 标签。将固定链接加入收藏夹。

分享一次ORA-01113 ORA-01110故障处理过程》有 1 条评论

  1. 惜分飞 说:

    Bug 10321285 : ORA-600 [DBGRMBLCP_CORRUPT_PAGE]

    Hdr: 10321285 11.1.0.7.0 RDBMS 11.1.0.7.0 DIAG ADR PRODID-5 PORTID-212 ORA-600
    Abstract: ORA-600 [DBGRMBLCP_CORRUPT_PAGE]
     
    *** 11/21/10 07:41 am *** (CHG: RDBMS Ver.-> NULL -> 11.1.0.7.0)
    *** 11/21/10 07:41 am ***
    ----
     
    PROBLEM:
    --------
    1. Clear description of the problem encountered
     
    On Sep. 5th, the following internal errors were written to the alert log:
     
    ORA-600 [dbgrmblcp_corrupt_page], 
    [/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
    []
    ORA-600 [dbgrmblgp_get_page_1], [16], [675499854], [42], []
     
    can also see:
    ORA-51108: unable to access diagnostic repository - retry command
    ORA-48223: Interrupt Requested - Fetch Aborted - Return Code [51014]
    ORA-6512: at "SYS.DBMS_IR", line 522
     
    This was a weekend - should have been little to no activity
    These errors were raised several times on one day, did not reappear since 
    then.
     
    2. Pertinent configuration information (MTS/OPS/distributed/etc)
     
    3. Indication of the frequency and predictability of the problem
    So far, this happened only on Sep. 5th at Customer-side.
    Not reproducible
     
    4. Sequence of events leading to the problem
    undetermined
     
    5. Technical impact on the customer. Include persistent after effects.
    Customer is concerned about their ADR repository.
    Is this the visible sign of something being corrupt in their ADR Repository?
     
    DIAGNOSTIC ANALYSIS:
    --------------------
    Alert log shows:
     
    Sun Sep 05 11:36:41 2010
    Errors in file 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j001_3485708.trc  
    (incident=326889):
    ORA-600: internal error code, arguments: [dbgrmblcp_corrupt_page], 
    [/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
    [], [], [], [], [], [], [], [], []
    Incident details in: 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326889/pextapps_j
    001_3485708_i326889.trc
    Errors in file 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j001_3485708.trc  
    (incident=326890):
    ORA-600: internal error code, arguments: [dbgrmblgp_get_page_1], [16], 
    [675499854], [42], [], [], [], [], [], [], [], []
    Incident details in: 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326890/pextapps_j
    001_3485708_i326890.trc
    Sun Sep 05 11:36:45 2010
    Trace dumping is performing id=[cdmp_20100905113645]
    Sun Sep 05 11:36:45 2010
    Sweep Incident[326889]: completed
    Errors in file 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j001_3485708.trc:
    ORA-12012: error on auto execute of job 11375
    ORA-51108: unable to access diagnostic repository - retry command
    ORA-48223: Interrupt Requested - Fetch Aborted - Return Code [51014]
    ORA-6512: at "SYS.DBMS_IR", line 522
    Sun Sep 05 11:37:41 2010
    Errors in file 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
    (incident=326831):
    ORA-600: internal error code, arguments: [dbgrmblcp_corrupt_page], 
    [/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
    [], [], [], [], [], [], [], [], []
    Incident details in: 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326831/pextapps_j
    000_1769628_i326831.trc
    Sun Sep 05 11:37:45 2010
    Sweep Incident[326890]: completed
    Sun Sep 05 11:38:00 2010
    Errors in file 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
    (incident=326832):
    ORA-600: internal error code, arguments: [dbgrmblgp_get_page_1], [16], 
    [675499854], [42], [], [], [], [], [], [], [], []
    Incident details in: 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326832/pextapps_j
    000_1769628_i326832.trc
    Sun Sep 05 11:38:00 2010
    Trace dumping is performing id=[cdmp_20100905113800]
    Trace dumping is performing id=[cdmp_20100905113803]
    Sun Sep 05 11:38:43 2010
    Errors in file 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
    (incident=326833):
    ORA-600: internal error code, arguments: [dbgrmblcp_corrupt_page], 
    [/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
    [], [], [], [], [], [], [], [], []
    Incident details in: 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326833/pextapps_j
    000_1769628_i326833.trc
    Sun Sep 05 11:38:45 2010
    Sweep Incident[326832]: completed
    Sweep Incident[326831]: completed
    Errors in file 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
    (incident=326834):
    ORA-600: internal error code, arguments: [dbgrmblgp_get_page_1], [16], 
    [675499854], [42], [], [], [], [], [], [], [], []
    Incident details in: 
    /pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326834/pextapps_j
    000_1769628_i326834.trc
    Sun Sep 05 11:38:47 2010
    Trace dumping is performing id=[cdmp_20100905113847]
    Trace dumping is performing id=[cdmp_20100905113850]
    Sun Sep 05 11:39:45 2010
    Sweep Incident[326834]: completed
    Sweep Incident[326833]: completed
     
     
    Asked CT to run the following commands:
     adrci> set home rdbms
     adrci> show catalog
     adrci> describe hm_finding
     adrci> query (RUN_ID, NAME) HM_FINDING
    which all worked fine
     
    Obtained the content of their /diag/rdbms/.../.../metadata/' folder
     
    WORKAROUND:
    -----------
     
    RELATED BUGS:
    -------------
    Bug 10303857 - QCN: ORA-700[DBGRMBLCP_CORRUPT_PAGE] / 
    ORA-600[DBGRMBLGP_GET_PAGE_1] ON NT 
    Bug 6595098 - ORA-48132 AND ORA-600 [DBGRMBLCP_CORRUPT_PAGE] ETC DURING OPEN 
    DATABASE
     
    REPRODUCIBILITY:
    ----------------
    For the time being, these errors did not reappear.
     
    TEST CASE:
    ----------
     
    STACK TRACE:
    ------------
    skdstdst ksedst1 ksedst dbkedDefDump ksedmp ksfdmp dbgexPhaseII
    dbgexProcessError dbgeExecuteForError dbgePostErrorKGE dbkePostKGE_kgsf
    kgeadse kgerinv_internal kgesoftnmierr
    dbgrmblcp_corrupt_page dbgrmblgp_get_page dbgrmblrp_read_page 
    dbgrmqmgp_get_page dbgrmqmif_internal_fetch dbgrmqmf_fetch_real 
    dbgrmqmf_fetch
    dbgrip_fetch_record dbgrip_relation_iterator dbghmo_close_findin 
    dbkh_close_finding  kcidr_file_header_check dbkird_df_check 
    dbkh_run_check_internal dbkh_run_revalidate check dbkifRevalidateFinding 
    dbkifReevaluateOpenFailures dbkiicd_reeval pevm_icd_call_commo pfrinstr_ICAL 
    pfrrun_no_tool pfrrun plsql_run peidxr_run peidxexe kkxdexe kkxmpexe kgmexwi 
    kgmexec evapls evaopn2 kkxmexcs opiexe kpoal8 opiodr kpoodr upirtrc kpurcsc 
    kpuexec OCIStmtExecute jslvec_execcb1 jslvswu jslve_execute0 jslve_execute 
    rpiswu2 kkjex1e kkjsexe kkjrdp opirip opidrv sou2o opimai_real main start