标签归档：ORA-01207

服务器断电引起的一例ORA-01207故障处理

发表于 2025 年 9 月 6 日 由 惜分飞

有一个客户反馈服务器异常断电,导致数据库无法正常启动(启动报ORA-01207: file is more recent than control file – old control file 错误)

SQL> startup mount
ORACLE 例程已经启动。

Total System Global Area 5.4784E+10 bytes
Fixed Size                  2188768 bytes
Variable Size            4.2010E+10 bytes
Database Buffers         1.2751E+10 bytes
Redo Buffers               20566016 bytes
数据库装载完毕。
SQL> recover database
ORA-00283: recovery session canceled due to errors
ORA-01122: database file 6 failed verification check
ORA-01110: data file 6: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\HSHIS1'
ORA-01207: file is more recent than control file - old control file

尝试拷贝alert日志到本地进行分析,发现无法正常拷贝
alert-error

怀疑文件系统或者底层磁盘有损坏,进一步检查系统日志,确认发现大量ntfs层面异常提示（而且多个盘都有）
ntfs-err
对于这样的情况,先对数据文件进行了备份,运气还不错,文件全部备份成功,并通过Oracle Database Recovery Check 脚本进行检查当前故障库状态,确认很多文件的checkpoint点相差很远
11
对于这样的一个非归档数据库,肯定无法正常打开,只能通过屏蔽一致性尝试强制打开库,结果在打开过程中出现ORA-600 4194错误

SQL> alter database open ;
alter database open 
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
ORA-00600: internal error code, arguments: [4194], [], [], [], [], [], [], [],
[], [], [], []
进程 ID: 7820
会话 ID: 177 序列号: 147

解决异常undo,数据库正常open成功,但是在数据库导出数据过程中报ORA-600 2662和ORA-600 2663错误(这种情况一般比较少见,大部分ORA-600 2662/2663都是在open过程中,这种库已经open成功,后续查询报该错误的比较少见,但是原理上有可能[启动的时候没有读到这些block,后面放到这些表的block scn大于文件头scn,所以报该错误])

Fri Sep 05 01:04:12 2025
QMNC started with pid=54, OS id=7708 
Completed: alter database open
Fri Sep 05 01:04:13 2025
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_m001_8624.trc  (incident=10058):
ORA-00600: internal error code, arguments: [2662], [2], [1384717872], [2], [1385084764], [41943042],
Incident details in: d:\app\administrator\diag\rdbms\his\his\incident\incdir_10058\his_m001_8624_i10058.trc
Fri Sep 05 01:04:13 2025
Starting background process CJQ0
Fri Sep 05 01:04:13 2025
CJQ0 started with pid=62, OS id=6852 
Fri Sep 05 01:04:13 2025
db_recovery_file_dest_size of 3912 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_ora_9220.trc  (incident=10084):
ORA-00600: 内部错误代码, 参数: [2662], [2], [1384717989], [2], [1385446402], [37802453], [], [], [],
Fri Sep 05 01:04:18 2025
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_j017_11248.trc  (incident=10059):
ORA-00600: internal error code, arguments: [2662], [2], [1384717990], [2], [1385446402], [37802453],
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_j017_11248.trc:
ORA-00600: internal error code, arguments: [2662], [2], [1384717990], [2], [1385446402], [37802453],
ORA-06512: at "HOSPITAL.RPT_CASE", line 5684
ORA-06512: at line 3
Errors in file d:\app\administrator\diag\rdbms\his\his\trace\his_ora_9220.trc  (incident=10085):
ORA-00600: 内部错误代码, 参数: [2662], [2], [1384717990], [2], [1385446402], [37802453], [], [], [],

这个问题相对比较简单,通过Patch SCN小工具修改正确的Oracle SCN即可
Patch_SCN09

发表在 Oracle备份恢复 | 标签为 , , | 评论关闭

ORA-600 krhpfh_03-1210故障处理

发表于 2024 年 8 月 26 日 由 惜分飞

rac数据库多个节点均处于open状态,数据查询正常,但是应用入库有些时候会失败报类似ORA-01187: cannot read from file because it failed verification tests错误：
ora-01187

故障最初原因是由于有坏盘,换盘之后,有两个节点数据实例crash

Mon Aug 19 21:16:47 2024
Read of datafile '+DATA/xifenfei99.dbf' (fno 1399) header failed with ORA-01207
Rereading datafile 1399 header failed with ORA-01207
Errors in file /u01/app/oracle/diag/rdbms/xff/xff5/trace/xff5_ckpt_75779.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01122: database file 1399 failed verification check
ORA-01110: data file 1399: '+DATA/xifenfei99.dbf'
ORA-01207: file is more recent than control file - old control file
Errors in file /u01/app/oracle/diag/rdbms/xff/xff5/trace/xff5_ckpt_75779.trc:
ORA-01242: data file suffered media failure: database in NOARCHIVELOG mode
ORA-01122: database file 1399 failed verification check
ORA-01110: data file 1399: '+DATA/xifenfei99.dbf'
ORA-01207: file is more recent than control file - old control file
CKPT (ospid: 75779): terminating the instance due to error 1242
Mon Aug 19 21:16:47 2024
System state dump requested by (instance=5, osid=75779 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff5/trace/xff5_diag_75725.trc
Mon Aug 19 21:16:52 2024
ORA-1092 : opitsk aborting process
Mon Aug 19 21:16:53 2024
ORA-1092 : opitsk aborting process
Mon Aug 19 21:16:53 2024
License high water mark = 131
Termination issued to instance processes. Waiting for the processes to exit
Mon Aug 19 21:17:02 2024
Instance termination failed to kill one or more processes
Instance terminated by CKPT, pid = 75779
Mon Aug 19 21:17:03 2024
USER (ospid: 33495): terminating the instance
Termination issued to instance processes. Waiting for the processes to exit
Mon Aug 19 21:17:13 2024
Instance termination failed to kill one or more processes
Instance terminated by USER, pid = 33495

但是数据库人工启动成功,查询所有数据文件均处于online状态
20240820-182825

可是有部分入库进程非常慢大量等待在enq:HW – contention
20240826-120804
所有数据库节点alert日志偶尔报ORA-01186: file 1399 failed verification tests等错

Tue Aug 20 21:30:02 2024
Read of datafile '+DATA/xifenfei99.dbf' (fno 1399) header failed with ORA-01207
Rereading datafile 1399 header failed with ORA-01207
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_dbw0_43828.trc:
ORA-01186: file 1399 failed verification tests
ORA-01122: database file 1399 failed verification check
ORA-01110: data file 1399: '+DATA/xifenfei99.dbf'
ORA-01207: file is more recent than control file - old control file
File 1399 not verified due to error ORA-01122
Read of datafile '+DATA/xifenfei99.dbf' (fno 1399) header failed with ORA-01207
Rereading datafile 1399 header failed with ORA-01207
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_dbw0_43828.trc:
ORA-01186: file 1399 failed verification tests
ORA-01122: database file 1399 failed verification check
ORA-01110: data file 1399: '+DATA/xifenfei99.dbf'
ORA-01207: file is more recent than control file - old control file
File 1399 not verified due to error ORA-01122

基于这种情况,初步判断：
1. 是由于该集群本身多节点(6个节点),只要有节点是open状态,其他节点关闭再启动依旧可以正常启动,但是无法写入数据到报ORA-01207错误的数据文件中(可以读取数据).
2. 如果所有节点关闭关闭,然后数据库无法正常启动会报ORA-01207: file is more recent than control file错误

这样的情况,根据以往经验,ORA-01207: file is more recent than control file通过重建ctl即可恢复,先关闭所有节点,然后尝试启动一个节点

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01122: database file 1399 failed verification check
ORA-01110: data file 1399: '+DATA/xifenfei99.dbf'
ORA-01207: file is more recent than control file - old control file

alter database open
Wed Aug 21 14:14:22 2024
SUCCESS: diskgroup REDO was mounted
Wed Aug 21 14:14:22 2024
NOTE: dependency between database xff and diskgroup resource ora.REDO.dg is established
Wed Aug 21 14:14:27 2024
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_47884.trc:
ORA-01122: database file 1399 failed verification check
ORA-01110: data file 1399: '+DATA/xifenfei99.dbf'
ORA-01207: file is more recent than control file - old control file
ORA-1122 signalled during: alter database open...

和预期的一样,重试重建ctl,然后数据库报ORA-00600 [krhpfh_03-1210]错误

SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup nomount pfile='/tmp/xff/pfile';
ORACLE instance started.

Total System Global Area 1.3255E+11 bytes
Fixed Size		    2244832 bytes
Variable Size		 9.7442E+10 bytes
Database Buffers	 3.4897E+10 bytes
Redo Buffers		  208654336 bytes
SQL> @rectl

Control file created.

SQL> 
SQL> 
SQL> 
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done


SQL> recover database using backup controlfile;
ORA-00283: recovery session canceled due to errors
ORA-00600: internal error code, arguments: [krhpfh_03-1210], [fno =], [1399],
[fhcpc =], [274968], [fhccc =], [274983], [], [], [], [], []
ORA-01110: data file 1399: '+DATA/xifenfei99.dbf'

这里的提示是有fhcpc和fhccc值不对导致,通过bbed查看相关值

BBED> set file 1399
	FILE#          	1399

BBED> p kcvfhccc
ub4 kcvfhccc                                @148      0x00043227 ===>274983(10进制)

BBED> p kcvfhcpc
ub4 kcvfhcpc                                @140      0x00043218 ===>274968(10进制)

报错比较明显通过bbed修改这两个值

BBED> m /x 2a390400 offset 148
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
 File: /tmp/xff/1399.dbf.header (1399)
 Block: 1                Offsets:  148 to  659           Dba:0x5dc00001
------------------------------------------------------------------------
 2a390400 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 0c000000 0f004441 
 5441315f 5442535f 45515f30 31000000 00000000 00000000 00000000 78010000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 cfebdd33 01000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 419333df 81001c0a 6ab13046 06000000 
 c1520400 02000000 10000000 7e000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 0d000d00 0d000100 00000000 00000000 

 <32 bytes per line>

BBED> m /x 2b390400 offset 140
 File: /tmp/xff/1399.dbf.header (1399)
 Block: 1                Offsets:  140 to  651           Dba:0x5dc00001
------------------------------------------------------------------------
 2b390400 e6ef524d 2a390400 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 0c000000 0f004441 5441315f 5442535f 45515f30 31000000 00000000 00000000 
 00000000 78010000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 cfebdd33 01000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 419333df 81001c0a 
 6ab13046 06000000 c1520400 02000000 10000000 7e000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 0d000d00 0d000100 

 <32 bytes per line>

修改好这些值之后，recover database和open数据库成功,检查字典正常,业务读写也正常,完成本次恢复任务

SQL> @hcheck
HCheck Version 07MAY18 on 21-AUG-2024 15:13:02
----------------------------------------------
Catalog Version 11.2.0.3.0 (1102000300)
db_name: XFF

				   Catalog	 Fixed
Procedure Name			   Version    Vs Release    Timestamp
Result
------------------------------ ... ---------- -- ---------- --------------
------
.- LobNotInObj		       ... 1102000300 <=  *All Rel* 08/21 15:13:02 PASS
.- MissingOIDOnObjCol	       ... 1102000300 <=  *All Rel* 08/21 15:13:02 PASS
.- SourceNotInObj	       ... 1102000300 <=  *All Rel* 08/21 15:13:02 PASS
.- OversizedFiles	       ... 1102000300 <=  *All Rel* 08/21 15:13:02 PASS
.- PoorDefaultStorage	       ... 1102000300 <=  *All Rel* 08/21 15:13:02 PASS
.- PoorStorage		       ... 1102000300 <=  *All Rel* 08/21 15:13:02 PASS
.- TabPartCountMismatch        ... 1102000300 <=  *All Rel* 08/21 15:13:02 PASS
.- OrphanedTabComPart	       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- MissingSum$		       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- MissingDir$		       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- DuplicateDataobj	       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- ObjSynMissing	       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- ObjSeqMissing	       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- OrphanedUndo 	       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- OrphanedIndex	       ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- OrphanedIndexPartition      ... 1102000300 <=  *All Rel* 08/21 15:13:03 PASS
.- OrphanedIndexSubPartition   ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- OrphanedTable	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- OrphanedTablePartition      ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- OrphanedTableSubPartition   ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- MissingPartCol	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- OrphanedSeg$ 	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- OrphanedIndPartObj#	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- DuplicateBlockUse	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- FetUet		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- Uet0Check		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- SeglessUET		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- BadInd$		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- BadTab$		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- BadIcolDepCnt	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- ObjIndDobj		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- TrgAfterUpgrade	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- ObjType0		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- BadOwner		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- StmtAuditOnCommit	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- BadPublicObjects	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- BadSegFreelist	       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- BadDepends		       ... 1102000300 <=  *All Rel* 08/21 15:13:04 PASS
.- CheckDual		       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- ObjectNames		       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- BadCboHiLo		       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- ChkIotTs		       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- NoSegmentIndex	       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- BadNextObject	       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- DroppedROTS		       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- FilBlkZero		       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- DbmsSchemaCopy	       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- OrphanedObjError	       ... 1102000300 >  1102000000 08/21 15:13:05 PASS
.- ObjNotLob		       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- MaxControlfSeq	       ... 1102000300 <=  *All Rel* 08/21 15:13:05 PASS
.- SegNotInDeferredStg	       ... 1102000300 >  1102000000 08/21 15:13:06 PASS
.- SystemNotRfile1	       ... 1102000300 >   902000000 08/21 15:13:06 PASS
.- DictOwnNonDefaultSYSTEM     ... 1102000300 <=  *All Rel* 08/21 15:13:07 PASS
.- OrphanTrigger	       ... 1102000300 <=  *All Rel* 08/21 15:13:07 PASS
.- ObjNotTrigger	       ... 1102000300 <=  *All Rel* 08/21 15:13:07 PASS
---------------------------------------
21-AUG-2024 15:13:07  Elapsed: 5 secs
---------------------------------------
Found 0 potential problem(s) and 0 warning(s)

PL/SQL procedure successfully completed.

Statement processed.

Complete output is in trace file:
/u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_70961_HCHECK.trc
发表在 Oracle备份恢复 | 标签为 , , | 评论关闭

分布式存储故障导致数据库无法启动故障处理

发表于 2024 年 5 月 11 日 由 惜分飞

国内xx医院使用了国外医疗行业龙头的pacs系统,由于是一个历史库,存放在分布式存储中,由于存储同时多个节点故障,导致数据库多个文件异常,数据库无法启动,三方维护人员尝试通通过rman归档进行应用日志,结果发现日志有损坏报ORA-00354 ORA-00353,无法记录恢复,希望我们给予支持
ORA-00353

Mon Apr 29 13:28:40 2024
Media Recovery failed with error 354
Mon Apr 29 13:28:40 2024
Errors in file F:\XXXXXX_DB\ORACLE\ADMIN\diag\rdbms\xxx\msxxx1\trace\xxx_pr00_4568.trc:
ORA-00283: recovery session canceled due to errors
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 487424 change 8737273868 time 04/01/2024 01:38:25
ORA-00334: archived log: 'F:\XXXXXX_DB\ORADATA\XXX\LOGARC0000052184_0922116268.0001'
ORA-283 signalled during: alter database recover logfile 'F:\XXXXXX_DB\ORADATA\XXX\LOGARC0000052184_0922116268.0001'...

接手故障之后,通过尝试恢复发现除该错误之外,还有ORA-600 4552之类错误
ORA-600-4552

跳过这些异常文件恢复,最终确认异常文件有如下部分
20240511223413
这些文件由于日志无法正常应用(有日志损坏无法应用，有日志和数据文件block不匹配导致无法应用),这样的情况直接通过自研的Oracle Recovery Tools小工具直接修改文件头信息
orarecovery
然后尝试OPEN数据库结果报ORA-1207
ORA-1207
对于这个故障可以通过rectl或者using backup ctl方式处理,然后open数据库成功
20240510224845
由于该系统是历史库,不会有新业务写入,通过对异常表和索引进行处理之后,客户测试业务可以正常访问,完成本次恢复

发表在 Oracle备份恢复 | 标签为 , , , , | 评论关闭