惜分飞 » ORA-600 kcbzib_kcrsds_1

由于数据块scn大于数据库scn导致ORA-600 kcbzib_kcrsds_1错误

惜分飞 — Mon, 22 Dec 2025 16:20:42 +0000

标题：由于数据块scn大于数据库scn导致ORA-600 kcbzib_kcrsds_1错误

以前遇到ORA-600 kcbzib_kcrsds_1一般都是数据库强制打开的时候,这次在一个open的库中遇到该问题,做一个记录
在expdp过程中有一个表报ORA-01555错误

. . 导出了 "SRM_DEV"."AAAAAA"             3.256 MB    4636 行
ORA-31693: 表数据对象 "SRM_DEV"."XXXXXXX" 无法加载/卸载并且被跳过, 错误如下:
ORA-02354: 导出/导入数据时出错
ORA-01555: 快照过旧: 回退段号  (名称为 "") 过小

这种错误,根据以往经验,大部分情况是由于lob字段异常引起
查看表结构

SQL> desc "SRM_DEV"."XXXXXXX"
 名称                                      是否为空? 类型
 ----------------------------------------- -------- ----------------------------

 DELIVERY_ID                               NOT NULL NUMBER(19)
 ARRIVAL_CUSTOMER_ID                                NUMBER(19)
 ARRIVAL_LOCATION_TYPE                              VARCHAR2(255 CHAR)
 ARRIVAL_SUPPLIER_ID                                NUMBER(19)
 ARRIVAL_TIME                                       TIMESTAMP(6)
 ARRIVAL_USER                                       NUMBER(19)
 CREATE_BY                                          VARCHAR2(32 CHAR)
 CREATE_TIME                                        TIMESTAMP(6)
 DELIVERY_DATE                                      VARCHAR2(32 CHAR)
 DELIVERY_NO                                        VARCHAR2(32 CHAR)
 DELIVERY_TYPE                                      VARCHAR2(32 CHAR)
 INVOICE_TYPE                                       VARCHAR2(2)
 IS_CONFIRM                                         VARCHAR2(2)
 IS_ONEKEY_IO                                       VARCHAR2(2)
 IS_SUPPORTING                                      VARCHAR2(32 CHAR)
 MODI_BY                                            VARCHAR2(32 CHAR)
 MODI_TIME                                          TIMESTAMP(6)
 OP_ATTRIBUTE                                       VARCHAR2(1 CHAR)
 PRINT_TIMES                                        NUMBER(10)
 RECEIVE_COMPANY                                    VARCHAR2(64 CHAR)
 RECEIVE_LOCATION                                   VARCHAR2(64 CHAR)
 RECEIVE_SUPPLIER_ID                                NUMBER(19)
 REMARK                                             VARCHAR2(256 CHAR)
 STATUS                                             VARCHAR2(5 CHAR)
 SUPPLIER_CODE                                      VARCHAR2(64 CHAR)
 SUPPLIER_ID                                        NUMBER(19)
 OVERDUE_STATUS                                     VARCHAR2(255 CHAR)

确认没有lob字段而引起了该问题,进一步分析
对表进行全扫描看看报什么错误

SQL> select /*+full(t)*/ count(1) from "SRM_DEV"."XXXXXXX" t;
select /*+full(t)*/ count(1) from "SRM_DEV"."XXXXXXX" t
                    *
第 1 行出现错误:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [],
[], [], [], [], []

根据以往经验,怀疑是由于这个表中的某个block的scn大于数据库scn导致
对全表扫描过程进行跟踪

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期日 12月 21 22:18:46 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> oradebug setmypid
已处理的语句
SQL> alter session set db_file_multiblocK_read_count=1;

会话已更改。

SQL> oradebug EVENT 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12
已处理的语句
SQL> oradebug TRACEFILE_NAME
F:\APP\diag\rdbms\srmdev\srmdev\trace\srmdev_ora_14768.trc
SQL>
SQL>
SQL> select /*+full(t)*/ count(1) from "SRM_DEV"."XXXXXXX" t;
select /*+full(t)*/ count(1) from "SRM_DEV"."XXXXXXX" t
                    *
第 1 行出现错误:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [],
[], [], [], [], []

分析trace文件确认对应的block为20548

WAIT #758663789368: nam='db file sequential read' ela= 40 file#=7 block#=20545 blocks=1 obj#=73756 tim=47881962193
WAIT #758663789368: nam='db file sequential read' ela= 41 file#=7 block#=20546 blocks=1 obj#=73756 tim=47881962259
WAIT #758663789368: nam='db file sequential read' ela= 40 file#=7 block#=20547 blocks=1 obj#=73756 tim=47881962325
WAIT #758663789368: nam='db file sequential read' ela= 41 file#=7 block#=20548 blocks=1 obj#=73756 tim=47881962389
Encountered exception while getting args for function:0x00007FF6F0BC39A0
2025-12-21T22:19:17.768815+08:00
Incident 1562659 created, dump file: F:\APP\diag\rdbms\srmdev\srmdev\incdir_1562659\srmdev_ora_14768_i1562659.trc
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []

对异常block进行dump分析

PARSE #758663744160:c=0,e=97,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=0,tim=48020763236
Start dump data blocks tsn: 4 file#:7 minblk 20548 maxblk 20548
Block dump from cache:
Dump of buffer cache at level 3 for pdb=0 tsn=4 rdba=29380676
WAIT #758663744160: nam='db file sequential read' ela= 88 file#=7 block#=20548 blocks=1 obj#=73756 tim=48020763396
Block dump from disk:
buffer tsn: 4 rdba: 0x01c05044 (7/20548)
scn: 0x6a37b2cb8 seq: 0x02 flg: 0x06 tail: 0x2cb80602
frmt: 0x02 chkval: 0xe203 type: 0x06=trans data
Hex dump of block: st=0, typ_found=1
Dump of memory from 0x000000B0A274D000 to 0x000000B0A274F000
B0A274D000 0000A206 01C05044 A37B2CB8 06020006  [....DP...,{.....]
B0A274D010 0000E203 001B0001 0001201C A37B2CB4  [......... ...,{.]
B0A274D020 00068000 00321F02 01C05001 00160010  [......2..P......]
B0A274D030 0409F0A2 010001A8 000C0776 00002001  [........v.... ..]

通过这里可以确认当前block的scn为0x6a37b2cb8==>28512562360
查询当前数据库scn

SQL> select CURRENT_SCN  from v$database;

  CURRENT_SCN  
--------------
  28512556338

通过分析确认当前数据库的scn小于该block的scn,所以报ORA-600 kcbzib_kcrsds_1错误.解决该问题的方法有两种
1. 调整数据库是scn,让数据库的scn 大于该block scn即可
2. 通过把该block的scn修改小一些（目的就是数据库scn大于block scn）
通过处理之后实现表正常查询

SQL> alter system checkpoint;

系统已更改。

SQL> select checkpoint_change# from v$database;

  CHECKPOINT_CHANGE#
--------------------
         28512563830

SQL>  select /*+full(t)*/ count(1) from "SRM_DEV"."XXXXXXX" t;

            COUNT(1)
--------------------
               19560

对于这种情况除了ora-600 kcbzib_kcrsds_1的报错之外,还可能会遇到ora-600 kcbzibmlt_kcrsds_1的错误

win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

惜分飞 — Thu, 14 Aug 2025 09:59:14 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：win环境断电强制拉库报ORA-600 kcbzib_kcrsds_1故障处理

客户环境异常断电,导致数据库无法正常启动,执行recover提示缺少归档（ORA-00279 ORA-00289 ORA-00280错误)

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on 星期四 8月 14 11:17:08 2025
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


连接到:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


SQL> select open_mode from v$database;

OPEN_MODE
----------------------------------------
MOUNTED

SQL> select group#，sequence#,status from v$log;

    GROUP#  SEQUENCE# STATUS
---------- ---------- --------------------------------
         1      15082 ACTIVE
         3      15084 CURRENT
         2      15083 ACTIVE

SQL> recover database;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {=suggested | filename | AUTO | CANCEL}

ORA-00308: 无法打开归档日志
'D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001'
ORA-27041: 无法打开文件
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。

z这个库需要15080归档,但是由于非归档环境,redo最小为15082,所以这种情况,只能先恢复可以正常recover的数据文件，然后强制拉库

SQL> recover datafile 1;
完成介质恢复。
SQL> recover datafile 3,4,7;
完成介质恢复。
SQL> recover datafile 62;
完成介质恢复。

SQL> recover database until cancel;
ORA-00279: 更改 885427209 (在 08/11/2025 22:14:56 生成) 对于线程 1 是必需的
ORA-00289: 建议:
D:\APP\ADMINISTRATOR\PRODUCT\19.0.0\DBHOME_1\RDBMS\ARC0000015080_1051367880.0001
ORA-00280: 更改 885427209 (用于线程 1) 在序列 #15080 中


指定日志: {=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: 警告: RECOVER 成功但 OPEN RESETLOGS 将出现如下错误
ORA-01152: 文件 1 没有从过旧的备份中还原
ORA-01110: 数据文件 1: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\SYSTEM01.DBF'


ORA-01112: 未启动介质恢复


SQL> alter database open resetlogs ;
alter database open resetlogs
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
进程 ID: 6272
会话 ID: 3873 序列号: 46847

alert日志报错

2025-08-14T11:24:08.246798+08:00
alter database open resetlogs
2025-08-14T11:24:08.425971+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 28: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_P.DBF'
2025-08-14T11:24:08.643178+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 29: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\PRESCRIPTION_DETAIL_P.DBF'
2025-08-14T11:24:08.958486+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 885427209 time 
Resetting resetlogs activation ID 3572089731 (0xd4e9c383)
2025-08-14T11:24:08.988512+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 30: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ORDER_LIST_P.DBF'
2025-08-14T11:24:12.029432+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 31: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\CHARGE_STATIS_P.DBF'
2025-08-14T11:24:13.687023+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_mz00_7464.trc:
ORA-01110: 数据文件 32: 'D:\APP\ADMINISTRATOR\ORADATA\HIS\ECASETAB01.DBF'
2025-08-14T11:24:13.725060+08:00
Setting recovery target incarnation to 2
Checker run found 31 new persistent data failures
2025-08-14T11:24:14.701994+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Endian type of dictionary set to little
2025-08-14T11:24:14.793084+08:00
Assigning activation ID 3729861866 (0xde512cea)
2025-08-14T11:24:14.848137+08:00
TT00 (PID:8068): Gap Manager starting
2025-08-14T11:24:14.870158+08:00
Redo log for group 1, sequence 1 is not located on DAX storage
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\HIS\REDO01.LOG
Successful open of redo thread 1
2025-08-14T11:24:14.912199+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2025-08-14T11:24:15.849098+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717057):
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Please look for redo dump in pinned buffers history in incident trace file, 
if not dumped for what so ever reason, use the following command to dump it at the earliest.
ALTER SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
2025-08-14T11:24:17.402582+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 2392187 end: 2394046 diff: 1859 ms (1.9 seconds)
2025-08-14T11:24:17.528703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:17.529703+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\his\his\trace\his_ora_6272.trc(incident=717058):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2025-08-14T11:24:18.995110+08:00
opiodr aborting process unknown ospid (6272) as a result of ORA-603
2025-08-14T11:24:18.997112+08:00
ORA-603 : opitsk aborting process
License high water mark = 42
USER (ospid: (prelim)): terminating the instance due to ORA error 
2025-08-14T11:24:23.749679+08:00
Instance terminated by USER(prelim), pid = 6272

遭遇到比较明显的ORA-600 kcbzib_kcrsds_1错误,通过自研的Patch_SCN小工具,以前有过类似恢复案例：
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1

然后直接open数据库

SQL> recover database;
完成介质恢复。
SQL> oradebug setmypid
已处理的语句
SQL> oradebug DUMPvar SGA kcsgscn_
kscn8 kcsgscn_ [7FF6623FF310, 7FF6623FF318) = 40B272E2 00000000
SQL>
SQL> alter database open;

数据库已更改。

然后安排导出数据,完成本次恢复任务.

CDM备份缺少归档打开数据库报ORA-600 kcbzib_kcrsds_1故障处理

惜分飞 — Sun, 09 Feb 2025 08:19:05 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：CDM备份缺少归档打开数据库报ORA-600 kcbzib_kcrsds_1故障处理

有客户联系我们,说一个19c的库,由于产生归档较多在备份过程中把部分归档删除而导致没有备份成功,从而使得他们那边一个误操作需要恢复使用备份做不完全恢复,备份厂商进行恢复并尝试强制打开库报ORA-600 kcbzib_kcrsds_1错误

由于客户那边使用的是CDM方式备份,可以较快的准备出来一个新环境,观察客户在应用日志过程中,文件头的scn一直不变,怀疑文件头由于begin backup冻结,对其进行dump,发现确实做了hot backup操作,而且没有end backup

由于缺少归档,不完全恢复没有成功,begin backup 也无法正常结束,对于这种情况,先尝试调整到文件头最大的scn值,尝试打开库

SQL> alter database open resetlogs ;
alter database open resetlogs 
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
Process ID: 3949615
Session ID: 5111 Serial number: 30040


--重启到mount状态
SQL> set numw 16
col CHECKPOINT_TIME for a40
set lines 150
set pages 1000
SELECT status,
to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
count(*) ROW_NUM
FROM v$datafile_header
GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
ORDER BY status, checkpoint_change#, checkpoint_time;SQL> SQL> SQL> SQL>   2    3    4    5    6  

STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
ONLINE  2025-02-08 22:43:01                      YES     15626238353558               56

打开库失败,只能找出来数据库最大的block中最大scn,然后调整文件头scn的值,实现数据库open

SQL> oradebug setmypid
Statement processed.
SQL> oradebug DUMPvar SGA kcsgscn_
SQL> kscn8 kcsgscn_ [060017E98, 060017EA0) = 00000000 00000000
SQL> oradebug DUMPvar SGA kcsgscn_
kscn8 kcsgscn_ [060017E98, 060017EA0) = 8CD9C896 00000E4D
SQL> 
SQL> 
SQL> 
SQL> alter database open ;
alter database open 
*
ERROR at line 1:
ORA-01113: file 1 needs media recovery
ORA-01110: data file 1: '/cdmbak/db/xifenfei/ob_data_D-XIFENFEI-SYSTEM_FNO-1_t13930nd'

SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL>

使用exp导出客户需要的表,完成本次恢复任务
对于ORA-600 kcbzib_kcrsds_1恢复的情况,以前有过大量恢复案例,修改数据库scn即可
kcbzib_kcrsds_1报错汇总
 12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
 存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
 Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
此类故障处理太多,不一一列举,解决这个错误之后,数据库open成功,然后安排逻辑迁移即可

2025年首个故障恢复—ORA-600 kcbzib_kcrsds_1

惜分飞 — Sat, 04 Jan 2025 14:25:21 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：2025年首个故障恢复—ORA-600 kcbzib_kcrsds_1

一个12.2.0.1的库由于某种原因引起的双机切换,导致数据库无法正常mount

2025-01-04T15:45:44.424193+08:00
alter database mount
2025-01-04T15:45:48.491054+08:00
Network throttle feature is disabled as mount time

2025-01-04T15:45:48.601366+08:00
LGWR (ospid: 34014): terminating the instance
2025-01-04T15:45:48.602480+08:00
System state dump requested by (instance=1, osid=34014 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xifenfei/trace/xifenfei_diag_33978_20250104154548.trc
2025-01-04T15:45:48.790674+08:00
Dumping diagnostic data in directory=[cdmp_20250104154548], requested by (instance=1, osid=34014 (LGWR))
2025-01-04T15:45:49.915068+08:00
Instance terminated by LGWR, pid = 34014

这个错误相对比较明显,是由于ctl异常导致,通过重建ctl,然后mount库,利用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本进行检测发现所有数据文件头的checkpoint 信息被冻结在 2024-11-29 19:00:29 （scn 2112302221)

分析alert日志数据库在此后20天中正常提供服务,业务运行都正常,客户反馈在这个冻结checkpoint信息的时间点,使用备份一体机发起过备份,之后就没有再备份了.
当时急着恢复数据库,没有对文件头进行dump不然应该可以发现类似begin backup的信息,类似这样（测试环境重现）：

DATA FILE #1:
  name #7: /u01/app/oracle/oradata/xifenfei/system01.dbf
creation size=0 block size=8192 status=0xe head=7 tail=7 dup=1
 tablespace 0, index=1 krfil=1 prev_file=0
 unrecoverable scn: 0x0000.00000000 01/01/1988 00:00:00
 Checkpoint cnt:625 scn: 0x0105.0106deef 01/04/2025 22:02:50
 Stop scn: 0xffff.ffffffff 12/14/2024 08:15:07
 Creation Checkpointed at scn:  0x0000.00000007 08/24/2013 11:37:33
 thread:0 rba:(0x0.0.0)
 enabled  threads:  00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
 Offline scn: 0x0000.000e2005 prev_range: 0
 Online Checkpointed at scn:  0x0000.000e2006 03/20/2024 20:53:56
 thread:1 rba:(0x1.2.0)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
 Hot Backup end marker scn: 0x0000.00000000
 aux_file is NOT DEFINED
 Plugged readony: NO
 Plugin scnscn: 0x0000.00000000
 Plugin resetlogs scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign creation scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Foreign checkpoint scn/timescn: 0x0000.00000000 01/01/1988 00:00:00
 Online move state: 0
 V10 STYLE FILE HEADER:
        Compatibility Vsn = 186647552=0xb200400
        Db ID=1780931490=0x6a26dba2, Db Name='XIFENFEI'
        Activation ID=0=0x0
        Control Seq=32953021=0x1f6d2bd, File size=98560=0x18100
        File Number=1, Blksiz=8192, File Type=3 DATA
Tablespace #0 - SYSTEM  rel_fn:1
Creation   at   scn: 0x0000.00000007 08/24/2013 11:37:33
Backup taken at scn: 0x0105.0106deef 01/04/2025 22:02:50 thread:1    <====注意
 reset logs count:0x45636764 scn: 0x0000.000e2006
 prev reset logs count:0x3121c97a scn: 0x0000.00000001
 recovered at 12/14/2024 08:36:35
 status:0x2001 root dba:0x00400208 chkpt cnt: 625 ctl cnt:624
begin-hot-backup file size: 98560                        <====注意
Checkpointed at scn:  0x0105.0106deef 01/04/2025 22:02:50
 thread:1 rba:(0x205.fdd9.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
Backup Checkpointed at scn:  0x0105.0106df14 01/04/2025 22:03:20   <====注意
 thread:1 rba:(0x209.2.10)
 enabled  threads:  01000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
  00000000 00000000 00000000 00000000 00000000 00000000
External cache id: 0x0 0x0 0x0 0x0
Absolute fuzzy scn: 0x0000.00000000
Recovery fuzzy scn: 0x0000.00000000 01/01/1988 00:00:00
Terminal Recovery Stamp  01/01/1988 00:00:00
Platform Information:    Creation Platform ID: 13
Current Platform ID: 13 Last Platform ID: 13

基于上述情况,尝试强制打开库,报ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1]错误

对于这个情况,以前有过大量恢复案例,修改数据库scn即可
kcbzib_kcrsds_1报错汇总
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
此类故障处理太多,不一一列举,解决这个错误之后,数据库open成功,然后安排逻辑迁移即可

Oracle 19c异常恢复—ORA-01209/ORA-65088

惜分飞 — Tue, 17 Sep 2024 13:19:23 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 19c异常恢复—ORA-01209/ORA-65088

由于raid卡bug故障,导致文件系统异常,从而使得数据库无法正常启动,客户找到我之前已经让多人分析,均未恢复成功,查看alert日志,发现他们恢复的时候尝试resetlogs库,然后报ORA-600 kcbzib_kcrsds_1错误

2024-09-15T17:07:32.553215+08:00
alter database open resetlogs
2024-09-15T17:07:32.569110+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 274757454692 time 
.... (PID:8074): Clearing online redo logfile 1 /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log
.... (PID:8074): Clearing online redo logfile 2 /opt/oracle/oradata/XFF/onlinelog/o1_mf_2_j3k201h3_.log
.... (PID:8074): Clearing online redo logfile 3 /opt/oracle/oradata/XFF/onlinelog/o1_mf_3_j3k201hk_.log
Clearing online log 1 of thread 1 sequence number 0
Clearing online log 2 of thread 1 sequence number 0
Clearing online log 3 of thread 1 sequence number 0
2024-09-15T17:07:34.939550+08:00
.... (PID:8074): Clearing online redo logfile 1 complete
.... (PID:8074): Clearing online redo logfile 2 complete
.... (PID:8074): Clearing online redo logfile 3 complete
Online log /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log: Thread 1 Group 1 was previously cleared
Online log /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_1_j3k201l4_.log: Thread 1 Group 1 was previously cleared
Online log /opt/oracle/oradata/XFF/onlinelog/o1_mf_2_j3k201h3_.log: Thread 1 Group 2 was previously cleared
Online log /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_2_j3k201kw_.log: Thread 1 Group 2 was previously cleared
Online log /opt/oracle/oradata/XFF/onlinelog/o1_mf_3_j3k201hk_.log: Thread 1 Group 3 was previously cleared
Online log /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_3_j3k201mt_.log: Thread 1 Group 3 was previously cleared
2024-09-15T17:07:34.966674+08:00
Setting recovery target incarnation to 2
2024-09-15T17:07:34.992357+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Buffer Cache Full DB Caching mode changing from FULL CACHING DISABLED to FULL CACHING ENABLED 
2024-09-15T17:07:34.994329+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2024-09-15T17:07:34.994390+08:00
Crash Recovery excluding pdb 3 which was cleanly closed.
2024-09-15T17:07:34.994433+08:00
Crash Recovery excluding pdb 4 which was cleanly closed.
2024-09-15T17:07:34.994474+08:00
Crash Recovery excluding pdb 5 which was cleanly closed.
Initializing SCN for created control file
Database SCN compatibility initialized to 3
Endian type of dictionary set to little
2024-09-15T17:07:35.001752+08:00
Assigning activation ID 2966012017 (0xb0c9c071)
Redo log for group 1, sequence 1 is not located on DAX storage
2024-09-15T17:07:35.015921+08:00
TT00 (PID:8113): Gap Manager starting
2024-09-15T17:07:35.034047+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log
  Current log# 1 seq# 1 mem# 1: /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_1_j3k201l4_.log
Successful open of redo thread 1
2024-09-15T17:07:35.034573+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2024-09-15T17:07:35.063726+08:00
TT03 (PID:8119): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2024-09-15T17:07:35.129748+08:00
Undo initialization recovery: Parallel FPTR failed: start:2528681 end:2528684 diff:3 ms (0.0 seconds)
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc  (incident=146455) (PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/xff/XFF/incident/incdir_146455/XFF_ora_8074_i146455.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Please look for redo dump in pinned buffers history in incident trace file, if not dumped for what so ever reason,
use the following command to dump it at the earliest. ALTER SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 2528681 end: 2529341 diff: 660 ms (0.7 seconds)
2024-09-15T17:07:35.786923+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-09-15T17:07:35.786967+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc  (incident=146456) (PDBNAME=CDB$ROOT):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/xff/XFF/incident/incdir_146456/XFF_ora_8074_i146456.trc
2024-09-15T17:07:36.291884+08:00
opiodr aborting process unknown ospid (8074) as a result of ORA-603
2024-09-15T17:07:36.299928+08:00
ORA-603 : opitsk aborting process
License high water mark = 4
USER(prelim) (ospid: 8074): terminating the instance due to ORA error 600

然后他们又重建了ctl,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查,发现几个问题:
1. PDB$SEED不在该库记录中(由于该pdb中无业务数据,可以忽略)

2. 部分文件resetlogs 信息不正确(应该是对部分文件offline或者重建ctl的时候没有带上他们)

接手该库进行恢复,尝试resetlogs该库

[oracle@localhost check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 17 11:29:28 2024
Version 19.9.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-65088: database open should be retried
Process ID: 101712
Session ID: 105 Serial number: 4711

对应的alert日志报错

Endian type of dictionary set to little
2024-09-17T11:29:46.691904+08:00
Assigning activation ID 2966261119 (0xb0cd8d7f)
Redo log for group 1, sequence 1 is not located on DAX storage
2024-09-17T11:29:46.714594+08:00
TT00 (PID:101731): Gap Manager starting
2024-09-17T11:29:46.735407+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log
Successful open of redo thread 1
2024-09-17T11:29:46.736182+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2024-09-17T11:29:46.774207+08:00
TT03 (PID:101737): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2024-09-17T11:29:46.793381+08:00
Undo initialization recovery: Parallel FPTR complete: start:99831350 end:99831351 diff:1 ms (0.0 seconds)
Undo initialization recovery: err:0 start: 99831349 end: 99831351 diff: 2 ms (0.0 seconds)
Undo initialization online undo segments: err:0 start: 99831351 end: 99831353 diff: 2 ms (0.0 seconds)
Undo initialization finished serial:0 start:99831349 end:99831356 diff:7 ms (0.0 seconds)
Dictionary check beginning
2024-09-17T11:29:46.817810+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65106: Pluggable database #2 (PDB$SEED) is in an invalid state.
Pluggable Database PDB$SEED (#2) found in data dictionary,
but not in the control file. Adding it to control file.
Pluggable Database PDB1 (#3) found in data dictionary,
but not in the control file. Adding it to control file.
Pluggable Database PDB2 (#4) found in data dictionary,
but not in the control file. Adding it to control file.
Pluggable Database PDB3 (#5) found in data dictionary,
but not in the control file. Adding it to control file.
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
2024-09-17T11:29:46.878684+08:00
Read of datafile '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_system_j3kc9hl0_.dbf'(fno 9)header failed with ORA-01209
Rereading datafile 9 header failed with ORA-01209
2024-09-17T11:29:46.921314+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_dbw0_100632.trc:
ORA-01186: file 9 failed verification tests
ORA-01122: database file 9 failed verification check
ORA-01110: data file 9: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_system_j3kc9hl0_.dbf'
ORA-01209: data file is from before the last RESETLOGS
File 9 not verified due to error ORA-01122
…………
Read of datafile '/opt/oracle/oradata/XFF/datafile/users07.dbf' (fno 39) header failed with ORA-01209
Rereading datafile 39 header failed with ORA-01209
2024-09-17T11:29:46.983955+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_dbw0_100632.trc:
ORA-01186: file 39 failed verification tests
ORA-01122: database file 39 failed verification check
ORA-01110: data file 39: '/opt/oracle/oradata/XFF/datafile/users07.dbf'
ORA-01209: data file is from before the last RESETLOGS
File 39 not verified due to error ORA-01122
2024-09-17T11:29:46.987947+08:00
Dictionary check complete
Verifying minimum file header compatibility for tablespace encryption for pdb 1..
Verifying file header compatibility for tablespace encryption completed for pdb 1
*********************************************************************
WARNING: The following temporary tablespaces in container(CDB$ROOT)
         contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE  ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
2024-09-17T11:29:47.059806+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 9: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_system_j3kc9hl0_.dbf'
ORA-01209: data file is from before the last RESETLOGS
…………
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
   then disabled.
**********************************************************
Starting background process IMCO
2024-09-17T11:29:47.340660+08:00
2024-09-17T11:29:47.382153+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 13: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_users_j3kckos2_.dbf'
ORA-01209: data file is from before the last RESETLOGS
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
2024-09-17T11:29:47.464233+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 14: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_users_j3kckqfx_.dbf'
ORA-01209: data file is from before the last RESETLOGS
AQ Processes can not start in restrict mode
Could not open PDB$SEED error=65106
2024-09-17T11:29:47.522825+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65106: Pluggable database #2 (PDB$SEED) is in an invalid state.
ORA-65106: Pluggable database #2 (PDB$SEED) is in an invalid state.
2024-09-17T11:29:47.525249+08:00
db_recovery_file_dest_size of 65536 MB is 0.05% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
2024-09-17T11:29:47.529134+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65088: database open should be retried
2024-09-17T11:29:47.529202+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65088: database open should be retried
2024-09-17T11:29:47.529253+08:00
Error 65088 happened during db open, shutting down database
2024-09-17T11:29:47.545440+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 15: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_users_j3kckstd_.dbf'
ORA-01209: data file is from before the last RESETLOGS
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc(incident=775863)(PDBNAME=CDB$ROOT):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-65088: database open should be retried
2024-09-17T11:29:48.046698+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 21: '/opt/oracle/oradata/XFF/PDB2/datafile/o1_mf_users_j45x90oq_.dbf'
ORA-01209: data file is from before the last RESETLOGS
2024-09-17T11:29:48.073328+08:00
opiodr aborting process unknown ospid (101712) as a result of ORA-603
2024-09-17T11:29:48.081576+08:00
ORA-603 : opitsk aborting process
License high water mark = 122
USER(prelim) (ospid: 101712): terminating the instance due to ORA error 65088
2024-09-17T11:29:49.104770+08:00
Instance terminated by USER(prelim), pid = 101712

主要错误有两个
ORA-01209: data file is from before the last RESETLOGS 和
ORA-65088: database open should be retried
通过分析这两个错误

[oracle@ora19c:/home/oracle]$ oerr ora 65088
65088, 00000, "database open should be retried"
// *Cause:   An inconsistency between the control file and the data dictionary
//           was found and fixed during the database open. The database open
//           needs to be executed again.
// *Action:  Retry the database open.
//
[oracle@ora19c:/home/oracle]$ oerr ora 01209
01209, 00000, "data file is from before the last RESETLOGS"   
// *Cause:  The reset log data in the file header does not match the   
//         control file. If the database is closed or the file is offline,  
//         the backup is old because it was taken before the last ALTER   
//         DATABASE OPEN RESETLOGS command. If opening a database that is   
//         open already by another instance, or if another instance just   
//         brought this file online, the file accessed by this instance is 
//         probably a different version. Otherwise, a backup of the file 
//         probably was restored while the file was in use.   
// *Action: Make the correct file available to the database. Then, either open
//         the database, or execute ALTER SYSTEM CHECK DATAFILES.

ORA-65088参见官方:ORA-65088 while opening DB with resetlogs for multi-tenant DB in 12.2 (Doc ID 2449591.1),应该不是一个技术问题(由于重建ctl+resetlogs导致)
ORA-01209: data file is from before the last RESETLOGS 这个错误,可以简单理解resetlogs的信息比数据文件的checkpoint信息新,对于这种情况,以及结合上述的部分文件resetlogs信息不一致问题,索性直接使用m_scn小工具对其进行批量

再次使用Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查,确认resetlogs 问题修复

然后顺利打开数据库,并导出数据,完成本次恢复任务

19c库启动报ORA-600 kcbzib_kcrsds_1

惜分飞 — Sun, 25 Aug 2024 07:58:10 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：19c库启动报ORA-600 kcbzib_kcrsds_1

一套19c的库由于某种情况,发现异常,当时的技术使用隐含参数强制拉库,导致数据库启动报ORA-00704 ORA-600 kcbzib_kcrsds_1错误

2024-08-24T06:11:25.494304+08:00
ALTER DATABASE OPEN
2024-08-24T06:11:25.494370+08:00
TMI: adbdrv open database BEGIN 2024-08-24 06:11:25.494324
Smart fusion block transfer is disabled:
  instance mounted in exclusive mode.
2024-08-24T06:11:25.515306+08:00
Beginning crash recovery of 1 threads
 parallel recovery started with 7 processes
 Thread 1: Recovery starting at checkpoint rba (logseq 2 block 3), scn 286550073
2024-08-24T06:11:25.567011+08:00
Started redo scan
2024-08-24T06:11:25.587170+08:00
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
2024-08-24T06:11:25.595192+08:00
Started redo application at
 Thread 1: logseq 2, block 3, offset 0, scn 0x0000000011146839
2024-08-24T06:11:25.595552+08:00
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: /dbf/RLZY/redo02.log
2024-08-24T06:11:25.595712+08:00
Completed redo application of 0.00MB
2024-08-24T06:11:25.596058+08:00
Completed crash recovery at
 Thread 1: RBA 2.3.0, nab 3, scn 0x000000001114683a
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
Endian type of dictionary set to little
2024-08-24T06:11:25.648152+08:00
LGWR (PID:1614826): STARTING ARCH PROCESSES
2024-08-24T06:11:25.661738+08:00
TT00 (PID:1614908): Gap Manager starting
Starting background process ARC0
2024-08-24T06:11:25.677246+08:00
ARC0 started with pid=54, OS id=1614910 
2024-08-24T06:11:25.687525+08:00
LGWR (PID:1614826): ARC0: Archival started
LGWR (PID:1614826): STARTING ARCH PROCESSES COMPLETE
2024-08-24T06:11:25.687733+08:00
ARC0 (PID:1614910): Becoming a 'no FAL' ARCH
ARC0 (PID:1614910): Becoming the 'no SRL' ARCH
2024-08-24T06:11:25.696437+08:00
TMON (PID:1614886): STARTING ARCH PROCESSES
Starting background process ARC1
2024-08-24T06:11:25.711645+08:00
Thread 1 advanced to log sequence 3 (thread open)
Redo log for group 3, sequence 3 is not located on DAX storage
2024-08-24T06:11:25.715270+08:00
ARC1 started with pid=56, OS id=1614914 
Starting background process ARC2
Thread 1 opened at log sequence 3
  Current log# 3 seq# 3 mem# 0: /dbf/RLZY/redo03.log
Successful open of redo thread 1
2024-08-24T06:11:25.728586+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
2024-08-24T06:11:25.734124+08:00
ARC2 started with pid=57, OS id=1614916 
Starting background process ARC3
2024-08-24T06:11:25.752891+08:00
ARC3 started with pid=58, OS id=1614918 
2024-08-24T06:11:25.752979+08:00
TMON (PID:1614886): ARC1: Archival started
TMON (PID:1614886): ARC2: Archival started
TMON (PID:1614886): ARC3: Archival started
TMON (PID:1614886): STARTING ARCH PROCESSES COMPLETE
2024-08-24T06:11:25.802551+08:00
ARC0 (PID:1614910): Archived Log entry 2828 added for T-1.S-2 ID 0x74f18f91 LAD:1
2024-08-24T06:11:25.806845+08:00
TT03 (PID:1614922): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_1614892.trc  (incident=124865):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/diag/rdbms/xff/xff/incident/incdir_124865/xff_ora_1614892_i124865.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-08-24T06:11:25.871925+08:00
2024-08-24T06:11:26.772652+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2024-08-24T06:11:26.872265+08:00
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_1614892.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-08-24T06:11:26.872351+08:00
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_1614892.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-08-24T06:11:26.872412+08:00
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_1614892.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-08-24T06:11:26.872455+08:00
Error 704 happened during db open, shutting down database
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_1614892.trc  (incident=124866):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/diag/rdbms/xff/xff/incident/incdir_124866/xff_ora_1614892_i124866.trc
opiodr aborting process unknown ospid (1614892) as a result of ORA-603
2024-08-24T06:11:27.498146+08:00
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_1614892.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-08-24T06:11:27.501122+08:00
ORA-603 : opitsk aborting process
License high water mark = 8
USER(prelim) (ospid: 1614892): terminating the instance due to ORA error 704
2024-08-24T06:11:28.526358+08:00
Instance terminated by USER(prelim), pid = 1614892

官方关于kcbzib_kcrsds_1从解释只有:Bug 31887074 – sr21.1bigscn_hipu3 – trc – ksfdopn2 – ORA-600 [kcbzib_kcrsds_1] (Doc ID 31887074.8)

虽然关于ORA-600 [kcbzib_kcrsds_1],oracle官方没有给出来解决方案,其实通过以往大量的恢复案例和经验中已经知道,这个错误解决方案就是修改oracle scn的方法可以绕过去,以前有过一些类似恢复案例:
ORA-600 kcbzib_kcrsds_1报错
12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1
redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理

Oracle 23ai rm redo*.log恢复

惜分飞 — Sat, 04 May 2024 04:18:47 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 23ai rm redo*.log恢复

在oracle 23ai的pdb中创建用户和表,并且插入数据(不提交),在另外一个会话中abort库,并从os层面rm删除掉redo文件,模拟数据库当前redo丢失,数据库恢复
创建用户和表并插入数据

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:40:55 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05

SQL> 
SQL> 
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 FREEPDB1                       READ WRITE NO

SQL> alter session set container=FREEPDB1;

Session altered.

SQL> create user xff identified by oracle;

User created.

SQL> grant dba to xff ;

Grant succeeded.

SQL> conn xff/oracle@FREEPDB1
Connected.
SQL> create table t1 as select * from dba_objects;

Table created.


SQL> insert into t1 select *from t1;

75877 rows created.

SQL> /

151754 rows created.

另外一个会话中abort库

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:43:30 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.


Connected to:
Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05

SQL> shutdown abort;
ORACLE instance shut down.
SQL> 
SQL> 
SQL> 
SQL> exit
Disconnected from Oracle Database 23ai Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.4.0.24.05

操作系统层面rm -rf 删除redo

[oracle@xifenfei ~]$ cd $ORACLE_BASE/oradata
[oracle@xifenfei oradata]$ ls
FREE
[oracle@xifenfei oradata]$ cd FREE/
[oracle@192 FREE]$ ls
control01.ctl  FREEPDB1  redo01.log  redo03.log    system01.dbf  undotbs2.dbf
control02.ctl  pdbseed   redo02.log  sysaux01.dbf  temp01.dbf    users01.dbf
[oracle@xifenfei FREE]$ ls -ltr
total 2441036
drwxr-x---. 2 oracle     1000         85 May  1 16:49 pdbseed
-rw-r-----. 1 oracle oinstall   20979712 May  1 16:51 temp01.dbf
drwxr-x---. 2 oracle     1000        104 May  1 16:55 FREEPDB1
-rw-r-----. 1 oracle oinstall  209715712 May  3 15:23 redo01.log
-rw-r-----. 1 oracle oinstall  209715712 May  3 15:23 redo02.log
-rw-r-----. 1 oracle oinstall    7348224 May  3 15:23 users01.dbf
-rw-r-----. 1 oracle oinstall 1080041472 May  3 15:43 system01.dbf
-rw-r-----. 1 oracle oinstall  692068352 May  3 15:43 sysaux01.dbf
-rw-rw----. 1 oracle oinstall   52436992 May  3 15:43 undotbs2.dbf
-rw-r-----. 1 oracle oinstall  209715712 May  3 15:43 redo03.log
-rw-r-----. 1 oracle oinstall   18759680 May  3 15:43 control01.ctl
-rw-r-----. 1 oracle oinstall   18759680 May  3 15:43 control02.ctl
[oracle@xifenfei FREE]$ rm -rf redo0*
[oracle@192 FREE]$ ls -l redo*
ls: cannot access 'redo*': No such file or directory
[oracle@xifenfei FREE]$

尝试启动数据库，报ora-00313，ora-00312，ora-27037等错误

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:44:17 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount;
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
Database mounted.
SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/opt/oracle/oradata/FREE/redo03.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

由于所有redo均被删除(包含当前redo),因此只能强制resetlogs方式打开库,尝试打开数据库

SQL> shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
Database mounted.
SQL> recover database until cancel;
ORA-00279: change 4244588 generated at 05/03/2024 15:23:22 needed for thread 1
ORA-00289: suggestion :
/opt/oracle/product/23ai/dbhomeFree/dbs/arch1_6_1167842962.dbf
ORA-00280: change 4244588 for thread 1 is in sequence #6


Specify log: {=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/opt/oracle/oradata/FREE/system01.dbf'


ORA-01112: media recovery not started

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by irrecoverable error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
Process ID: 5596
Session ID: 29 Serial number: 63204

数据库遇到常见的ORA-600 kcbzib_kcrsds_1错误,这类错误类似oracle在12c之前版本中的ORA-600 2662错误
ORA-600 kcbzib_kcrsds_1报错
 存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
 redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
 ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1
这种一般就是scn问题,通过修改数据库scn,数据库启动报ORA-16433错误

[oracle@xifenfei ~]$ sqlplus sys/oracle@free as sysdba

SQL*Plus: Release 23.0.0.0.0 - Production on Fri May 3 15:49:00 2024
Version 23.4.0.24.05

Copyright (c) 1982, 2024, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
SQL> alter database mount;

Database altered.

SQL> recover database;
ORA-00283: recovery session canceled due to errors
ORA-16433: The database or pluggable database must be opened in read/write mode.


SQL> shutdown abort;
ORACLE instance shut down.

处理ctl,open数据库成功

SQL> startup nomount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 1603726344 bytes
Fixed Size                  5360648 bytes
Variable Size             671088640 bytes
Database Buffers          922746880 bytes
Redo Buffers                4530176 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "FREE" NORESETLOGS  NOARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 1024
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 292
  7  LOGFILE
  8    GROUP 1 '/opt/oracle/oradata/FREE/redo01.log'  SIZE 200M BLOCKSIZE 512,
  9    GROUP 2 '/opt/oracle/oradata/FREE/redo02.log'  SIZE 200M BLOCKSIZE 512,
 10    GROUP 3 '/opt/oracle/oradata/FREE/redo03.log'  SIZE 200M BLOCKSIZE 512
 11  -- STANDBY LOGFILE
 12  DATAFILE
 13    '/opt/oracle/oradata/FREE/system01.dbf',
 14    '/opt/oracle/oradata/FREE/pdbseed/system01.dbf',
 15    '/opt/oracle/oradata/FREE/sysaux01.dbf',
 16    '/opt/oracle/oradata/FREE/pdbseed/sysaux01.dbf',
 17    '/opt/oracle/oradata/FREE/users01.dbf',
 18    '/opt/oracle/oradata/FREE/pdbseed/undotbs01.dbf',
 19    '/opt/oracle/oradata/FREE/FREEPDB1/system01.dbf',
 20    '/opt/oracle/oradata/FREE/FREEPDB1/sysaux01.dbf',
 21    '/opt/oracle/oradata/FREE/FREEPDB1/undotbs01.dbf',
 22    '/opt/oracle/oradata/FREE/FREEPDB1/users01.dbf',
 23    '/opt/oracle/oradata/FREE/undotbs2.dbf'
 24  CHARACTER SET AL32UTF8
 25  ;

Control file created.

SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

SQL> show pdbs;

          CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------------- ------------------------------ ---------- ----------
               2 PDB$SEED                       READ ONLY  NO
               3 FREEPDB1                       READ WRITE NO

至此当前数据库redo故障模拟和恢复基本完成

存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理

惜分飞 — Fri, 19 Jan 2024 04:33:22 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理

硬件故障,客户自行强制resetlogs库,报ORA-600 kcbzib_kcrsds_1错误

2024-01-17T17:30:33.094367+08:00
alter database open resetlogs
2024-01-17T17:30:33.105461+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 16456823130015 time 
.... (PID:1584): Clearing online redo logfile 1 /u01/app/oracle/oradata/XFF/redo01.log
.... (PID:1584): Clearing online redo logfile 2 /u01/app/oracle/oradata/XFF/redo02.log
.... (PID:1584): Clearing online redo logfile 3 /u01/app/oracle/oradata/XFF/redo03.log
Clearing online log 1 of thread 1 sequence number 1345
Clearing online log 2 of thread 1 sequence number 1346
Clearing online log 3 of thread 1 sequence number 1347
2024-01-17T17:30:43.054150+08:00
.... (PID:1584): Clearing online redo logfile 1 complete
.... (PID:1584): Clearing online redo logfile 2 complete
.... (PID:1584): Clearing online redo logfile 3 complete
Resetting resetlogs activation ID 259120030 (0xf71db9e)
Online log /u01/app/oracle/oradata/XFF/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u01/app/oracle/oradata/XFF/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u01/app/oracle/oradata/XFF/redo03.log: Thread 1 Group 3 was previously cleared
2024-01-17T17:30:43.201042+08:00
Setting recovery target incarnation to 3
2024-01-17T17:30:43.267669+08:00
Smart fusion block transfer is disabled:
  instance mounted in exclusive mode.
2024-01-17T17:30:43.282033+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
Endian type of dictionary set to little
2024-01-17T17:30:43.396061+08:00
Assigning activation ID 280673168 (0x10babb90)
Redo log for group 1, sequence 1 is not located on DAX storage
2024-01-17T17:30:43.433441+08:00
TT00 (PID:1652): Gap Manager starting
2024-01-17T17:30:43.526972+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u01/app/oracle/oradata/XFF/redo01.log
Successful open of redo thread 1
2024-01-17T17:30:43.528058+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Stopping change tracking
2024-01-17T17:30:44.097557+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc(incident=263984)(PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_263984/XFF_ora_1584_i263984.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-17T17:30:47.913013+08:00
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2024-01-17T17:30:48.228934+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-01-17T17:30:48.229250+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-01-17T17:30:48.229572+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_1584.trc  (incident=263985) (PDBNAME=CDB$ROOT):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_263985/XFF_ora_1584_i263985.trc
2024-01-17T17:30:50.982998+08:00
opiodr aborting process unknown ospid (1584) as a result of ORA-603
2024-01-17T17:30:50.989089+08:00
ORA-603 : opitsk aborting process

这个错误处理过多次一般是由于scn异常导致,以前部分类似文章
ORA-600 kcbzib_kcrsds_1报错
 12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
 redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
 Patch SCN工具一键恢复ORA-600 kcbzib_kcrsds_1
处理好该错误之后,数据库在open pdb的时候报ORA-600 4193错误

2024-01-18T09:59:18.211131+08:00
alter pluggable database all open
2024-01-18T09:59:18.213569+08:00
XFFPDB(4):Pluggable database XFFPDB opening in read write
2024-01-18T09:59:18.590332+08:00
QPI: opatch file present, opatch
QPI: qopiprep.bat file present
2024-01-18T09:59:18.701197+08:00
XFFPDB(4):SUPLOG: Initialize PDB SUPLOG SGA, old value 0x0, new value 0x18
XFFPDB(4):Autotune of undo retention is turned on. 
XFFPDB(4):Endian type of dictionary set to little
2024-01-18T09:59:18.842283+08:00
Buffer Cache Full DB Caching mode changing from FULL CACHING ENABLED to FULL CACHING DISABLED 
Full DB Caching disabled: DEFAULT_CACHE_SIZE should be at least 153531 MBs bigger than current size. 
2024-01-18T09:59:19.017859+08:00
XFFPDB(4):Undo initialization recovery: Parallel FPTR failed: start:233023254 end:233023260 diff:6 ms(0.0 seconds)
XFFPDB(4):Undo initialization recovery: err:0 start: 233023254 end: 233023286 diff: 32 ms (0.0 seconds)
2024-01-18T09:59:19.256942+08:00
XFFPDB(4):[2475] Successfully onlined Undo Tablespace 40.
XFFPDB(4):Undo initialization online undo segments: err:0 start: 233023286 end: 233023500 diff:214 ms(0.2 seconds)
XFFPDB(4):Undo initialization finished serial:0 start:233023228 end:233023508 diff:280 ms (0.3 seconds)
XFFPDB(4):Database Characterset for XFFPDB is ZHS16GBK
XFFPDB(4):*********************************************************************
XFFPDB(4):WARNING: The following temporary tablespaces in container(XFFPDB)
XFFPDB(4):         contain no files.
XFFPDB(4):         This condition can occur when a backup controlfile has
XFFPDB(4):         been restored.  It may be necessary to add files to these
XFFPDB(4):         tablespaces.  That can be done using the SQL statement:
XFFPDB(4): 
XFFPDB(4):         ALTER TABLESPACE  ADD TEMPFILE
XFFPDB(4): 
XFFPDB(4):         Alternatively, if these temporary tablespaces are no longer
XFFPDB(4):         needed, then they can be dropped.
XFFPDB(4):           Empty temporary tablespace: TEMP
XFFPDB(4):*********************************************************************
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=392032) (PDBNAME=XFFPDB):
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
XFFPDB(4):Incident details in:/u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392032/XFF_ora_2475_i392032.trc
XFFPDB(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-18T09:59:20.079597+08:00
XFFPDB(4):*****************************************************************
XFFPDB(4):An internal routine has requested a dump of selected redo.
XFFPDB(4):This usually happens following a specific internal error, when
XFFPDB(4):analysis of the redo logs will help Oracle Support with the
XFFPDB(4):diagnosis.
XFFPDB(4):It is recommended that you retain all the redo logs generated (by
XFFPDB(4):all the instances) during the past 12 hours, in case additional
XFFPDB(4):redo dumps are required to help with the diagnosis.
XFFPDB(4):*****************************************************************
===========================================================
Dumping current patch information
===========================================================
Unable to obtain current patch information due to error: 20001
===========================================================
2024-01-18T09:59:31.793666+08:00
XFFPDB(4):Flush retried for xcb 0x179400620, pmd 0x174a2f6c0
XFFPDB(4):Doing block recovery for file 31 block 1239
2024-01-18T09:59:31.805351+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=392034)(PDBNAME=XFFPDB):
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
XFFPDB(4):Incident details in:/u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392034/XFF_ora_2475_i392034.trc
XFFPDB(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2024-01-18T09:59:32.009996+08:00
XFFPDB(4):*****************************************************************
XFFPDB(4):An internal routine has requested a dump of selected redo.
XFFPDB(4):This usually happens following a specific internal error, when
XFFPDB(4):analysis of the redo logs will help Oracle Support with the
XFFPDB(4):diagnosis.
XFFPDB(4):It is recommended that you retain all the redo logs generated (by
XFFPDB(4):all the instances) during the past 12 hours, in case additional
XFFPDB(4):redo dumps are required to help with the diagnosis.
XFFPDB(4):*****************************************************************
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_392384/XFF_mz00_2879_i392384.trc
2024-01-18T09:59:36.918296+08:00
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
Reread of blocknum=142165, file=/u01/app/oracle/oradata/XFF/system01.dbf. found same corrupt data
2024-01-18T09:59:41.146850+08:00
XFFPDB(4):Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc:
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
2024-01-18T09:59:45.434605+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/trace/XFF_ora_2475.trc  (incident=407714):
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_407714/XFF_ora_2475_i407714.trc
2024-01-18T09:59:45.439083+08:00
Errors in file /u01/app/oracle/diag/rdbms/XFF/XFF/incident/incdir_407714/XFF_ora_2475_i407714.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [4193], [14648], [14652], [], [], [], [], [], [], [], [], []
2024-01-18T09:59:48.575971+08:00
XFFPDB(4):pdb open recovery: pdbid=4 log=255 acquired_ip=1

这个相对比较简单,对异常undo进行处理,cdb和pdb都open成功,使用数据泵把数据迁移到新库,完成本次恢复(比较幸运,硬件故障的库直接迁移成功,没有报其他错误)

Oracle 19c 断电异常恢复

惜分飞 — Thu, 05 Jan 2023 14:37:27 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：Oracle 19c 断电异常恢复

19.3数据库由于异常断电,导致数据库无法启动，报ORA-600 ktbair2: illegal inheritance,ORA-600 6101等错误

2023-01-02T22:01:37.310225+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr0l_3788.trc:
ORA-10562: Error occurred while applying redo to data block (file# 10, block# 399386)
ORA-10564: tablespace DATA
ORA-01110: data file 10: 'D:\ORADATA\DATA04.ORA'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 81882
ORA-00600: internal error code, arguments: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2023-01-02T22:01:37.544630+08:00
Slave exiting with ORA-10562 exception
2023-01-02T22:01:37.560258+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr03_4364.trc:
ORA-10562: Error occurred while applying redo to data block (file# 2, block# 716430)
ORA-10564: tablespace DATA
ORA-01110: data file 2: 'D:\ORADATA\DATA01.ORA'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 81743
ORA-00600: internal error code, arguments: [6101], [0], [16], [0], [0], [0], [], [], [], [], [], []
2023-01-02T22:01:38.294726+08:00
Slave exiting with ORA-10562 exception
2023-01-02T22:01:38.310354+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr0e_5980.trc:
ORA-10562: Error occurred while applying redo to data block (file# 2, block# 714911)
ORA-10564: tablespace DATA
ORA-01110: data file 2: 'D:\ORADATA\DATA01.ORA'
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 74014
ORA-00600: internal error code, arguments: [ktbair2: illegal  inheritance], [], [], [], [], [], [], [], [], [], [], []
2023-01-02T22:01:48.921092+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_pr00_4760.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
2023-01-02T22:01:49.171125+08:00
ORA-756 signalled during: ALTER DATABASE RECOVER  database  ...

报错比较明显由于redo和datafile不匹配导致recover 不成功,尝试强制拉库

SQL> alter database open resetlogs ;
alter database open resetlogs 
*
第 1 行出现错误:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [],
[], [], [], [], [], [], []
进程 ID: 6068
会话 ID: 3631 序列号: 54960

Undo initialization recovery: err:600 start: 20760593 end: 20762484 diff: 1891 ms (1.9 seconds)
2023-01-02T22:09:05.539709+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6068.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2023-01-02T22:09:05.555336+08:00
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6068.trc:
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_6068.trc  (incident=293955):
ORA-00603: ORACLE 服务器会话因致命错误而终止
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_293955\orcl_ora_6068_i293955.trc
2023-01-02T22:09:06.805497+08:00
opiodr aborting process unknown ospid (6068) as a result of ORA-603
2023-01-02T22:09:06.961768+08:00
ORA-603 : opitsk aborting process

参考类似处理open数据库成功:
ORA-600 kcbzib_kcrsds_1报错
 12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
 redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复
 ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1

redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复

惜分飞 — Thu, 27 Jan 2022 15:50:06 +0000

联系：手机/微信(+86 17813235971) QQ(107644445)

标题：redo异常强制拉库报ORA-600 kcbzib_kcrsds_1修复

节点2 asm dismount导致redo写报错(ORA-00340,ORA-00345)，经过分析asm和系统日志,确认是由于多路径异常导致io异常

2022-01-24T23:44:39.966602+08:00
WARNING: group 4 is being dismounted.
WARNING: ASMB force dismounting group 4 (REDO) due to ASM server dismount
SUCCESS: diskgroup REDO was dismounted
2022-01-24T23:44:41.103783+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc:
ORA-00345: redo log write error block 11961764 count 6
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
2022-01-24T23:44:41.156809+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc:
ORA-00340: IO error processing online log 10 of thread 2
ORA-00345: redo log write error block 11961764 count 6
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF2/trace/XFF2_lgwr_228507.trc  (incident=1341402):
ORA-340 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF2/incident/incdir_1341402/XFF2_lgwr_228507_i1341402.trc
2022-01-24T23:44:41.505251+08:00
USER (ospid: 133928): terminating the instance due to error 340

由于节点2是突然crash，节点1做实例恢复失败，由于节点2的redo发生了写丢失,导致节点1实例恢复后库crash,进而是的该集群的相关数据库节点全部crash

2022-01-24T23:46:08.440519+08:00
Slave encountered ORA-10388 exception during crash recovery
2022-01-24T23:46:08.442854+08:00
Slave encountered ORA-10388 exception during crash recovery
Abort recovery for domain 0, flags 4
2022-01-24T23:46:08.444531+08:00
Aborting crash recovery due to error 742
2022-01-24T23:46:08.444695+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_450481.trc:
ORA-00742: Log read detects lost write in thread 2 sequence 63507 block 11941482
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
Abort recovery for domain 0, flags 4
2022-01-24T23:46:08.771108+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_450481.trc:
ORA-00742: Log read detects lost write inthread 2 sequence 63507 block 11941482
ORA-00312: online log 10 thread 2: '+REDO/XFF/ONLINELOG/group_10.261.1074690685'
ORA-742 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:17:165} */...
2022-01-24T23:46:10.143155+08:00
License high water mark = 33
2022-01-24T23:46:10.143752+08:00
USER (ospid: 451049): terminating the instance
2022-01-24T23:46:11.167337+08:00
Instance terminated by USER, pid = 451049

经过第三方强制拉库之后,数据库报ORA-600 kcbzib_kcrsds_1

2022-01-25T10:13:37.922332+08:00
Completed crash recovery at
 Thread 2: RBA 5.3.16, nab 3, scn 0x00000a348a032122
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
2022-01-25T10:13:38.071326+08:00
Thread 2 advanced to log sequence 6 (thread recovery)
validate pdb 0, flags x4, valid 0, pdb flags x204 
* validated domain 0, flags = 0x200
CRASH recovery complete: pdb 0 valid 1 (flags x4, pdb flags x200) 
Picked broadcast on commit scheme to generate SCNs
Endian type of dictionary set to little
2022-01-25T10:13:38.389741+08:00
TT00: Gap Manager starting (PID:220505)
2022-01-25T10:13:38.646484+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: +DATA/XFF/ONLINELOG/group_1.536.1094875107
Successful open of redo thread 1
2022-01-25T10:13:38.647243+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc  (incident=1879556):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF1/incident/incdir_1879556/XFF1_ora_216590_i1879556.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
2022-01-25T10:13:39.734366+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2022-01-25T10:13:39.734424+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2022-01-25T10:13:39.734499+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2022-01-25T10:13:39.734536+08:00
Error 704 happened during db open, shutting down database
Errors in file /u01/app/oracle/diag/rdbms/xff/XFF1/trace/XFF1_ora_216590.trc  (incident=1879557):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xff/XFF1/incident/incdir_1879557/XFF1_ora_216590_i1879557.trc
2022-01-25T10:13:39.894464+08:00
2022-01-25T10:13:40.446888+08:00
opiodr aborting process unknown ospid (216590) as a result of ORA-603
2022-01-25T10:13:40.470643+08:00
ORA-603 : opitsk aborting process
License high water mark = 36
2022-01-25T10:13:40.471453+08:00
USER (ospid: 216590): terminating the instance due to error 704
2022-01-25T10:13:41.436133+08:00
opiodr aborting process unknown ospid (189796) as a result of ORA-1092
2022-01-25T10:13:41.439011+08:00
ORA-1092 : opitsk aborting process
2022-01-25T10:13:41.472060+08:00
PMON (ospid: 189585): terminating the instance due to error 704

该错误是12c之后才有的报错,由于文件异常导致,通过以前的解决经验,接手这个问题之后快速调整数据库文件头信息,顺利open库
参考以前相关blog内容：
Oracle 12c redo 丢失恢复
 模拟19c数据库redo异常恢复
 ORA-600 kcbzib_kcrsds_1报错
 12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
 ORA-00603 ORA-01092 ORA-600 kcbzib_kcrsds_1

[oracle@xifenfei02 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed Jan 26 00:31:50 2022

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile'
ORACLE instance started.

Total System Global Area 3.2320E+11 bytes
Fixed Size                 29879248 bytes
Variable Size            4.5634E+10 bytes
Database Buffers         1.9059E+11 bytes
Redo Buffers             1043861504 bytes
In-Memory Area           8.5899E+10 bytes
Database mounted.
SQL> alter database open ;

Database altered.