标签归档：ORA-600 2662

分享一次ORA-01113 ORA-01110故障处理过程

发表于 2014 年 7 月 17 日 由 惜分飞

近期编写了小工具,对于此类问题,可以实现一键恢复,参考:一键恢复ORA-01113 ORA-01110—Oracle Recovery Tools

数据库报ORA-01113 ORA-01110无法正常open

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'

alert日志

Wed Jul 16 17:17:56 2014
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 1380870980
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Wed Jul 16 17:19:01 2014
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_m000_3540.trc  (incident=367575):
ORA-00700: soft internal error, arguments: [dbgrmblcp_corrupt_page], [D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\metadata\HM_FINDING.ams], 
[1245], [], [], [], [], [], [], [], [], []
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_367575\orcl_m000_3540_i367575.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_m000_3540.trc  (incident=367576):
ORA-00600: internal error code, arguments: [dbgrmblgp_get_page_1], [1245], [0], [80], [], [], [], [], [], [], [], []
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_367576\orcl_m000_3540_i367576.trc
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_m000_3540.trc:
ORA-51106: check failed to complete due to an error.  See error below
ORA-00600: internal error code, arguments: [dbgrmblgp_get_page_1], [1245], [0], [80], [], [], [], [], [], [], [], []
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'

这里出现ORA-00700[dbgrmblcp_corrupt_page]与ORA-00600[dbgrmblgp_get_page_1]是Bug 10321285 : ORA-600 [DBGRMBLCP_CORRUPT_PAGE]

尝试recover database 遭遇ORA-00283 ORA-16433

SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database must be opened in read/write mode.

alert日志信息

Wed Jul 16 17:36:33 2014
ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Wed Jul 16 17:36:33 2014
Media Recovery failed with error 16433
Slave exiting with ORA-283 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr00_3868.trc:
ORA-00283: recovery session canceled due to errors
ORA-16433: The database must be opened in read/write mode.
Recovery Slave PR00 previously exited with exception 283
ORA-283 signalled during: ALTER DATABASE RECOVER  database  ...
Wed Jul 16 17:36:45 2014
alter database open
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_3236.trc:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-1113 signalled during: alter database open...

重建控制文件后恢复,遭遇ORA-600 2662错误

SQL> shutdown abort
ORACLE instance shut down.
SQL> startup nomount
ORACLE instance started.

Total System Global Area      10255212544 bytes
Fixed Size                        2264088 bytes
Variable Size                  4529849320 bytes
Database Buffers               5704253440 bytes
Redo Buffers                     18845696 bytes
SQL> @d:\ctl.sql

Control file created.

SQL> recover database using backup controlfile;
ORA-00279: change 13953689551797 generated at 07/16/2014 17:53:41 needed for
thread 1
ORA-00289: suggestion :
D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\ORCL\ARCHIVELOG\2014_07_16\O1_MF_1_1_%U_

.ARC
ORA-00280: change 13953689551797 for thread 1 is in sequence #1


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
d:\app\administrator\oradata\orcl\redo01.log
Log applied.
Media recovery complete.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [2662], [3248], [3635774399],
[3249], [3580470066], [4194545], [], [], [], [], [], []
Process ID: 3932
Session ID: 200 Serial number: 1

因为该数据库是11.2.0.3未打上scn patch,直接推进scn(可以使用event,隐含参数,oradebug,bbed等),数据库就正常open

发表在 非常规恢复 | 标签为 , , , , | 一条评论

记录一次ORA-600 3004 恢复过程和处理思路

发表于 2014 年 7 月 15 日 由 惜分飞

10.1.0.2数据库在启动的时候报ORA-00600[3004]错误

SQL> startup
ORACLE 例程已经启动。
Total System Global Area 1042254360 bytes
Fixed Size 743960 bytes
Variable Size 503316480 bytes
Database Buffers 536870912 bytes
Redo Buffers 1323008 bytes
数据库装载完毕。
ORA-00600: 内部错误代码，参数: [3004], [1], [0], [0], [], [], [], []

alert日志信息

Tue Jul 15 10:57:11 2014
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE   MOUNT
Tue Jul 15 10:57:12 2014
ALTER DATABASE OPEN
Tue Jul 15 10:57:12 2014
Beginning crash recovery of 1 threads
 attempting to start a parallel recovery with 3 processes
 parallel recovery started with 3 processes
Tue Jul 15 10:57:12 2014
Started first pass scan
Tue Jul 15 10:57:12 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\udump\orcl_ora_1084.trc:
ORA-00600: 内部错误代码, 参数: [3004], [1], [0], [0], [], [], [], []

Tue Jul 15 10:57:13 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\bdump\orcl_p000_3780.trc:
ORA-10388: parallel query server interrupt (failure)

Tue Jul 15 10:57:13 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\bdump\orcl_p002_3420.trc:
ORA-10388: parallel query server interrupt (failure)

Tue Jul 15 10:57:14 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\bdump\orcl_p001_3784.trc:
ORA-10388: parallel query server interrupt (failure)

Tue Jul 15 10:57:14 2014
Aborting crash recovery due to error 600
Tue Jul 15 10:57:14 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\udump\orcl_ora_1084.trc:
ORA-00600: 内部错误代码, 参数: [3004], [1], [0], [0], [], [], [], []

ORA-600 signalled during: ALTER DATABASE OPEN...

根据老杨的blog,出现该问题，很可能是crontrolfile异常导致,所以这里可以考虑通过重建控制文件或者把当前控制文件当作备份控制文件来使用

Tue Jul 15 13:42:31 2014
ALTER DATABASE RECOVER  database using backup controlfile   
Tue Jul 15 13:42:31 2014
Media Recovery Start
WARNING! Recovering data file 1 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 2 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 3 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 4 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 5 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 6 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 7 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 8 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 9 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 10 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 11 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
 attempting to start a parallel recovery with 3 processes
 parallel recovery started with 3 processes
Starting datafile 1 with incarnation depth 0 in thread 1 sequence 794
Datafile 1: 'D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORCL\SYSTEM01.DBF'
Starting datafile 2 with incarnation depth 0 in thread 1 sequence 794
Datafile 2: 'D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORCL\UNDOTBS01.DBF'
Starting datafile 3 with incarnation depth 0 in thread 1 sequence 794
Datafile 3: 'D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORCL\SYSAUX01.DBF'
Starting datafile 4 with incarnation depth 0 in thread 1 sequence 794
Datafile 4: 'D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORCL\USERS01.DBF'
Starting datafile 5 with incarnation depth 0 in thread 1 sequence 794
Datafile 5: 'E:\ORADB\USER1_1.ORA'
Starting datafile 6 with incarnation depth 0 in thread 1 sequence 794
Datafile 6: 'E:\ORADB\USER1_2.ORA'
Starting datafile 7 with incarnation depth 0 in thread 1 sequence 794
Datafile 7: 'E:\ORADB\USER1_3.ORA'
Starting datafile 8 with incarnation depth 0 in thread 1 sequence 794
Datafile 8: 'E:\ORADB\USER1_4.ORA'
Starting datafile 9 with incarnation depth 0 in thread 1 sequence 794
Datafile 9: 'E:\ORADB\INDEX1_1.ORA'
Starting datafile 10 with incarnation depth 0 in thread 1 sequence 794
Datafile 10: 'E:\ORADB\INDEX1_2.ORA'
Starting datafile 11 with incarnation depth 0 in thread 1 sequence 794
Datafile 11: 'E:\ORADB\TEMP1_1.ORA'
Media Recovery Log 
ORA-279 signalled during: ALTER DATABASE RECOVER  database using backup cont...
Tue Jul 15 13:43:03 2014
ALTER DATABASE RECOVER    LOGFILE 'D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORCL\REDO01.LOG'  
Tue Jul 15 13:43:03 2014
Media Recovery Log D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORCL\REDO01.LOG
Completed: ALTER DATABASE RECOVER    LOGFILE 'D:\ORACLE\PRODU
Tue Jul 15 13:43:17 2014
alter database open resetlogs
RESETLOGS after complete recovery through change 3142208
Resetting resetlogs activation ID 1378452168 (0x522982c8)
Setting recovery target incarnation to 3
Tue Jul 15 13:43:18 2014
Setting recovery target incarnation to 3
Tue Jul 15 13:43:18 2014
Flashback Database Disabled 
Tue Jul 15 13:43:19 2014
Assigning activation ID 1380750902 (0x524c9636)
Maximum redo generation record size = 120832 bytes
Maximum redo generation change vector size = 116476 bytes
Private_strands 7 at log switch
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\ORACLE\PRODUCT\10.1.0\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
Tue Jul 15 13:43:19 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Jul 15 13:43:19 2014
SMON: enabling cache recovery
Tue Jul 15 13:43:20 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\udump\orcl_ora_1484.trc:
ORA-00600: 内部错误代码, 参数: [2662], [0], [3142214], [0], [3142438], [8388617], [], []

Tue Jul 15 13:43:21 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\udump\orcl_ora_1484.trc:
ORA-00600: 内部错误代码, 参数: [2662], [0], [3142214], [0], [3142438], [8388617], [], []
Tue Jul 15 13:43:21 2014
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600

这里出现ORA-600[2662]因为scn相差较小,直接重启几次,然后数据库出现如下启动正常但是出现ORA-01595和ORA-00600[4194]错误

Tue Jul 15 13:48:26 2014
Starting background process MMON
Starting background process MMNL
MMON started with pid=17, OS id=2896
MMNL started with pid=18, OS id=3828
Tue Jul 15 13:48:26 2014
Block recovery completed at rba 2.90.16, scn 0.3162303
Tue Jul 15 13:48:26 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\bdump\orcl_smon_3724.trc:
ORA-01595: error freeing extent (3) of rollback segment (1))
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [57], [6], [], [], [], [], []

Tue Jul 15 13:48:26 2014
Completed: alter database open
Tue Jul 15 13:48:27 2014
Errors in file d:\oracle\product\10.1.0\admin\orcl\bdump\orcl_j000_2956.trc:
ORA-00600: internal error code, arguments: [4193], [1375], [1379], [], [], [], [], []

通过设置undo_management=manual,重建undo表空间解决,至此本数据库恢复完成,建议对其进行逻辑方式重建

发表在 Oracle备份恢复 | 标签为 , , , | 评论关闭

ORACLE 12C redo异常恢复测试—部分pdb未正常open

发表于 2014 年 3 月 21 日 由 惜分飞

为了验证当前redo丢失的情况下ORACLE 12C CDB数据库恢复的情况,做了一个小实验，三个会话，分别操作为在root pdb中执行一个delete 不提交；另外一个会话在user pdb中delete记录不提交；最后一个会话中直接abort数据库，然后进行数据库恢复，验证数据库是否可以都正常open(所有pdb)
会话1(root pdb中操作)

CDB_CDB$ROOT@SYS> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB                            READ WRITE NO

CDB_CDB$ROOT@SYS> show con_name

CON_NAME
------------------------------
CDB$ROOT

CDB_CDB$ROOT@SYS> create table t_xifenfei as select * from dba_objects;

表已创建。

CDB_CDB$ROOT@SYS> delete from t_xifenfei;

已删除 91032 行。

CDB_CDB$ROOT@SYS>

会话2(user pdb中操作)

CDB_CDB$ROOT@SYS> show con_name

CON_NAME
------------------------------
PDB

CDB_CDB$ROOT@SYS> drop table t_xifenfei purge;

表已删除。

CDB_CDB$ROOT@SYS> create table t_xifenfei as select * from dba_objects;

表已创建。

CDB_CDB$ROOT@SYS> delete from t_xifenfei;

已删除 91144 行。

会话3(直接abort数据库)

CDB_CDB$ROOT@SYS> shutdown abort;
ORACLE 例程已经关闭。

删除数据库联机日志

E:\app\XIFENFEI\oradata\cdb>dir redo*.log
 驱动器 E 中的卷是 本地磁盘
 卷的序列号是 000C-3B41

 E:\app\XIFENFEI\oradata\cdb 的目录

2013-08-07  01:41        52,429,312 REDO01.LOG
2013-08-07  01:40        52,429,312 REDO02.LOG
2013-08-07  01:40        52,429,312 REDO03.LOG
2014-03-20  22:47        52,432,896 REDO04.LOG
2014-03-20  22:47        52,432,896 REDO05.LOG
2014-03-20  22:47        52,432,896 REDO06.LOG
               6 个文件    314,586,624 字节
               0 个目录 21,359,374,336 可用字节

E:\app\XIFENFEI\oradata\cdb>del redo*.log

E:\app\XIFENFEI\oradata\cdb>dir redo*.log
 驱动器 E 中的卷是 本地磁盘
 卷的序列号是 000C-3B41

 E:\app\XIFENFEI\oradata\cdb 的目录

找不到文件

启动数据库报ORA-00313错误

idle> startup
ORACLE 例程已经启动。

Total System Global Area  521936896 bytes
Fixed Size                  2404552 bytes
Variable Size             205524792 bytes
Database Buffers          306184192 bytes
Redo Buffers                7823360 bytes
数据库装载完毕。
ORA-00313: 无法打开日志组 6 (用于线程 1) 的成员
ORA-00312: 联机日志 6 线程 1: 'E:\APP\XIFENFEI\ORADATA\CDB\REDO06.LOG'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。

使用_allow_resetlogs_corruption参数尝试open数据库

idle> shutdown abort
ORACLE 例程已经关闭。
idle> startup pfile='d:/pfile.txt' mount
ORACLE 例程已经启动。

Total System Global Area  521936896 bytes
Fixed Size                  2404552 bytes
Variable Size             205524792 bytes
Database Buffers          306184192 bytes
Redo Buffers                7823360 bytes
数据库装载完毕。
idle> show parameter resetlogs;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
_allow_resetlogs_corruption          boolean     TRUE
idle> recover database until cancel;
ORA-00279: 更改 12696935641735 (在 03/20/2014 22:38:52 生成) 对于线程 1
是必需的
ORA-00289: 建议:
E:\APP\XIFENFEI\FAST_RECOVERY_AREA\CDB\ARCHIVELOG\2014_03_20\O1_MF_1_872_9LOZSK9
X_.ARC
ORA-00280: 更改 12696935641735 (用于线程 1) 在序列 #872 中


指定日志: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01194: 文件 1 需要更多的恢复来保持一致性
ORA-01110: 数据文件 1: 'E:\APP\XIFENFEI\ORADATA\CDB\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


idle> alter database open resetlogs;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [2956], [1012314778],
[2956], [1012314903], [268435600], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [2956], [1012314777],
[2956], [1012314903], [268435600], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [2956], [1012314768],
[2956], [1012314903], [268435600], [], [], [], [], [], []
进程 ID: 9268
会话 ID: 92 序列号: 3

ora-600[2662]很熟悉，文件头的scn小于文件中某个block的csn，通过bbed等工具修改文件scn，尝试启动数据库

CDB_CDB$ROOT@SYS> SHUTDOWN ABORT
ORACLE 例程已经关闭。
CDB_CDB$ROOT@SYS> STARTUP
ORACLE 例程已经启动。

Total System Global Area  521936896 bytes
Fixed Size                  2404552 bytes
Variable Size             205524792 bytes
Database Buffers          306184192 bytes
Redo Buffers                7823360 bytes
数据库装载完毕。
数据库已经打开。

尝试open user pdb

CDB_CDB$ROOT@SYS> alter session set container=pdb;

会话已更改。

CDB_CDB$ROOT@SYS> alter database open;
alter database open
*
第 1 行出现错误:
ORA-00600: 内部错误代码, 参数: [kcvfdb_pdb_set_clean_scn: cleanckpt], [3],
[2956], [1012312995], [2956], [1012334778], [2], [], [], [], [], []

查询mos得到结论:
Bug 16784143 ORA-600 [kcvfdb_pdb_set_clean_scn: cleanckpt] with PDBs
在12.2/12.1.0.2/12.1.0.1.1中修复，后续尝试打patch,来进一步恢复.
1

测试结果证明
1.root pdb的open过程和以前版本数据库无大差异,但是scn推进比较费劲,以前的event和隐含参数方法无效
2.user pdb再是看是因为bug无法open,后续继续验证
3.12.1的base 版本可能确实问题很多，生产使用慎重

发表在 ORACLE 12C, Oracle备份恢复 | 标签为 , , , , | 一条评论