数据库open成功后报ORA-00353 ORA-00354错误引起的一系列问题(本质ntfs文件系统异常)

发表于 2024 年 4 月 3 日 由 惜分飞

服务器异常断电之后，开机启动数据库启动成功，但是报ORA-00353 ORA-00354以及ORA-600 kdsgrp1错误

Sun Mar 31 01:19:51 2024
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Mar 31 01:19:51 2024
SMON: enabling cache recovery
[4528] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:64250 end:64859 diff:609 (6 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Sun Mar 31 01:19:57 2024
QMNC started with pid=40, OS id=4912 
Sun Mar 31 01:20:03 2024
Completed: alter database open
Sun Mar 31 01:20:04 2024
Starting background process CJQ0
Sun Mar 31 01:20:04 2024
CJQ0 started with pid=20, OS id=5104 
Setting Resource Manager plan SCHEDULER[0x32DE]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sun Mar 31 01:20:07 2024
Starting background process VKRM
Sun Mar 31 01:20:07 2024
VKRM started with pid=48, OS id=4160 
Sun Mar 31 01:20:32 2024
Incomplete read from log member 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'. Trying next member.
Incomplete read from log member 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'. Trying next member.
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_j000_4180.trc(incident=132393):
ORA-00353: 日志损坏接近块 13792 更改 33686946 时间 03/31/2024 00:17:54
ORA-00334: 归档日志: 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'
Incomplete read from log member 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'. Trying next member.
Incomplete read from log member 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'. Trying next member.
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_j000_4180.trc(incident=132394):
ORA-00353: 日志损坏接近块 13792 更改 33686946 时间 03/31/2024 00:17:54
ORA-00334: 归档日志: 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'
ORA-00353: 日志损坏接近块 13792 更改 33686946 时间 03/31/2024 00:17:54
ORA-00334: 归档日志: 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\xifenfei\xifenfei\incident\incdir_132393\xifenfei_j000_4180_i132393.trc:
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 13792 更改 33686946 时间 03/31/2024 00:17:54
ORA-00334: 归档日志: 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'
ORA-00353: 日志损坏接近块 13792 更改 33686946 时间 03/31/2024 00:17:54
ORA-00334: 归档日志: 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'
Sun Mar 31 01:20:35 2024
Sweep [inc][132394]: completed
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_j000_4180.trc:
ORA-00354: 损坏重做日志块标头
ORA-00353: 日志损坏接近块 13792 更改 33686946 时间 03/31/2024 00:17:54
ORA-00334: 归档日志: 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_j000_4180.trc(incident=132395):
ORA-00600: 内部错误代码, 参数: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incomplete read from log member 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'. Trying next member.
Incomplete read from log member 'E:\APP\ARCHIVELOG\ARC0000005281_1157968160.0001'. Trying next member.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

再次重启数据库报ORA-600 2131错误,数据库无法正常mount

Sun Mar 31 08:59:20 2024
alter database mount exclusive
Errors in file E:\APP\ADMINISTRATOR\diag\rdbms\xifenfei\xifenfei\trace\xifenfei_ora_3232.trc  (incident=144175):
ORA-00600: ??????, ??: [2131], [9], [8], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: alter database mount exclusive...

因为这个库有ctl备份,通过rman还原ctl备份,然后尝试recover库,结果报ORA-00310 ORA-00334（由于需要的redo无法正常应用导致）
20240403223225

对于这类情况,只能通过屏蔽一致性强制打开库
20240403223432
数据报ORA-600 2662错误,此类错误比较简单,使用patch scn工具一键搞定(ORA-600 2662快速恢复之Patch scn工具),数据库open成功,导出数据完整恢复
对于正常open的库,出现此类问题属于反常现象,通过分析系统事件确定是由于ntfs文件系统本身有问题导致
20240403223855
发表在 Oracle备份恢复 | 标签为 , | 留下评论

ORA-600 ktsiseginfo1故障

发表于 2024 年 3 月 31 日 由 惜分飞

oracle 9i的库在运行途中突然报ORA-600 kcbnew_3错误

Sun Mar 31 14:25:11 2024
Undo Segment 69 Onlined
Sun Mar 31 14:25:11 2024
Created Undo Segment _SYSSMU69$
Sun Mar 31 14:25:11 2024
Created Undo Segment _SYSSMU70$
Undo Segment 70 Onlined
Sun Mar 31 14:28:41 2024
Starting control autobackup
Control autobackup written to SBT_TAPE device
	comment 'API Version 2.0,MMS Version 1.2.0.0',
	media 'A90063L4'
	handle 'c-1757743655-20240331-01'
Sun Mar 31 14:31:07 2024
Starting control autobackup
Control autobackup written to SBT_TAPE device
	comment 'API Version 2.0,MMS Version 1.2.0.0',
	media 'A90063L4'
	handle 'c-1757743655-20240331-02'
Sun Mar 31 14:34:31 2024
SMON offlining US=70
Sun Mar 31 14:34:31 2024
Created Undo Segment _SYSSMU73$
Undo Segment 73 Onlined
Sun Mar 31 14:34:31 2024
Created Undo Segment _SYSSMU74$
Undo Segment 74 Onlined
Sun Mar 31 14:34:31 2024
Created Undo Segment _SYSSMU81$
Undo Segment 81 Onlined
Sun Mar 31 14:37:36 2024
Errors in file /u2/oradb/admin/xifenfei/udump/xifenfei_ora_3741.trc:
ORA-00600: 内部错误代码，参数: [kcbnew_3], [156], [], [], [], [], [], []
Sun Mar 31 14:39:22 2024
Errors in file /u2/oradb/admin/xifenfei/bdump/xifenfei_smon_6890.trc:
ORA-01595: error freeing extent (6) of rollback segment (70))
ORA-00600: internal error code, arguments: [kcbnew_3], [156], [], [], [], [], [], []
Sun Mar 31 14:45:05 2024
Errors in file /u2/oradb/admin/xifenfei/udump/xifenfei_ora_4046.trc:
ORA-00600: 内部错误代码，参数: [kcbnew_3], [156], [], [], [], [], [], []
Sun Mar 31 14:47:47 2024
Failure to extend rollback segment 73 because of 3113 condition
FULL status of rollback segment 73 set.
Sun Mar 31 14:47:47 2024
FULL status of rollback segment 73 cleared.
Sun Mar 31 14:48:03 2024
Errors in file /u2/oradb/admin/xifenfei/udump/xifenfei_ora_4074.trc:
ORA-00600: 内部错误代码，参数: [kcbnew_3], [156], [], [], [], [], [], []
Sun Mar 31 14:48:35 2024
Errors in file /u2/oradb/admin/xifenfei/bdump/xifenfei_qmn0_4044.trc:
ORA-00600: internal error code, arguments: [kcbnew_3], [156], [], [], [], [], [], []
Sun Mar 31 14:50:01 2024
Failure to extend rollback segment 73 because of 3113 condition
FULL status of rollback segment 73 set.
Sun Mar 31 14:50:02 2024
FULL status of rollback segment 73 cleared.
Sun Mar 31 14:50:41 2024
Restarting dead background process QMN0
QMN0 started with pid=53
Sun Mar 31 14:51:09 2024
Failure to extend rollback segment 73 because of 3113 condition
FULL status of rollback segment 73 set.
Sun Mar 31 14:51:09 2024
FULL status of rollback segment 73 cleared.
Sun Mar 31 14:54:36 2024
Failure to extend rollback segment 73 because of 3113 condition
FULL status of rollback segment 73 set.
Sun Mar 31 14:54:36 2024
FULL status of rollback segment 73 cleared.
Sun Mar 31 14:54:56 2024
Errors in file /u2/oradb/admin/xifenfei/bdump/xifenfei_smon_6890.trc:
ORA-01595: error freeing extent (6) of rollback segment (70))
ORA-00600: internal error code, arguments: [kcbnew_3], [156], [], [], [], [], [], []

从上述信息看,由于ora-600 kcbnew_3错误导致70/73号回滚段异常,smon无法正常对其进行扩展,关于ora-600 kcbnew_3报错描述
20240331201941

客户尝试关闭数据库重启,数据库报ora-600 ktsiseginfo1错误，无法正常启动

Sun Mar 31 15:13:03 2024
Shutting down instance: further logons disabled
Sun Mar 31 15:15:59 2024
Shutting down instance (immediate)
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
Sun Mar 31 15:15:59 2024
FULL status of rollback segment 73 set.
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
FULL status of rollback segment 73 set.
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
Sun Mar 31 15:15:59 2024
FULL status of rollback segment 73 cleared.
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
Sun Mar 31 15:15:59 2024
FULL status of rollback segment 73 set.
Sun Mar 31 15:15:59 2024
Failure to extend rollback segment 73 because of 1089 condition
Sun Mar 31 15:15:59 2024
FULL status of rollback segment 73 cleared.
Sun Mar 31 15:15:59 2024
FULL status of rollback segment 73 cleared.
Sun Mar 31 15:15:59 2024
License high water mark = 161
Sun Mar 31 15:21:04 2024
Active call for process 2878 user 'oracle' program 'oracle@ttperp (TNS V1-V3)'
SHUTDOWN: waiting for active calls to complete.
Sun Mar 31 15:21:07 2024
Failure to extend rollback segment 73 because of 1089 condition
FULL status of rollback segment 73 set.
Sun Mar 31 15:21:07 2024
SMON: FULL status of rollback segment 73 cleared.
Sun Mar 31 15:21:08 2024
ALTER DATABASE CLOSE NORMAL
SMON: disabling tx recovery
SMON: disabling cache recovery
Sun Mar 31 15:21:16 2024
Shutting down archive processes
Archiving is disabled
Sun Mar 31 15:21:16 2024
ARCH shutting down
Sun Mar 31 15:21:16 2024
ARCH shutting down
Sun Mar 31 15:21:16 2024
ARC1: Archival stopped
Sun Mar 31 15:21:16 2024
ARC0: Archival stopped
Sun Mar 31 15:21:17 2024
Thread 1 closed at log sequence 313828
Successful close of redo thread 1.
Sun Mar 31 15:21:18 2024
Completed: ALTER DATABASE CLOSE NORMAL
Sun Mar 31 15:21:18 2024
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archiving is disabled
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Sun Mar 31 15:23:10 2024
Starting ORACLE instance (normal)
Sun Mar 31 15:23:10 2024
Running with 1 strand for Non-Enterprise Edition
Sun Mar 31 15:23:10 2024
WARNING: EINVAL creating segment of size 0x0000000065400000
fix shm parameters in /etc/system or equivalent
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
SCN scheme 2
Running with 1 strand for Non-Enterprise Edition
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 9.2.0.4.0.
System parameters with non-default values:
  processes                = 1000
  timed_statistics         = TRUE
  shared_pool_size         = 218103808
  sga_max_size             = 1679366944
  large_pool_size          = 67108864
  java_pool_size           = 33554432
  control_files            = /u2/oradb/oradata/xifenfei/control01.ctl
  db_block_size            = 8192
  db_cache_size            = 838860800
  compatible               = 9.2.0.0.0
  log_archive_start        = TRUE
  log_archive_dest         = /u6/archive_u6
  log_archive_format       = %s_%t.dbf
  db_file_multiblock_read_count= 16
  fast_start_mttr_target   = 0
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS7
  undo_retention           = 10800
  remote_login_passwordfile= EXCLUSIVE
  db_domain                = 
  instance_name            = xifenfei
  utl_file_dir             = /u1/usr/oracle/utl
  job_queue_processes      = 10
  hash_join_enabled        = TRUE
  background_dump_dest     = /u2/oradb/admin/xifenfei/bdump
  user_dump_dest           = /u2/oradb/admin/xifenfei/udump
  core_dump_dest           = /u2/oradb/admin/xifenfei/cdump
  sort_area_size           = 104857600
  db_name                  = xifenfei
  open_cursors             = 300
  star_transformation_enabled= FALSE
  query_rewrite_enabled    = FALSE
  pga_aggregate_target     = 209715200
  aq_tm_processes          = 1
PMON started with pid=2
DBW0 started with pid=3
LGWR started with pid=4
CKPT started with pid=5
SMON started with pid=6
RECO started with pid=7
CJQ0 started with pid=8
QMN0 started with pid=9
Sun Mar 31 15:23:11 2024
ARCH: STARTING ARCH PROCESSES
ARC0 started with pid=10
ARC0: Archival started
ARC1 started with pid=11
Sun Mar 31 15:23:11 2024
ARCH: STARTING ARCH PROCESSES COMPLETE
Oracle Data Guard is not available in this edition of Oracle.
Sun Mar 31 15:23:11 2024
ARC1: Archival started
Sun Mar 31 15:23:11 2024
ARC1: Thread not mounted
Sun Mar 31 15:23:11 2024
ARC0: Thread not mounted
Sun Mar 31 15:23:11 2024
ALTER DATABASE   MOUNT
Sun Mar 31 15:23:15 2024
Successful mount of redo thread 1, with mount id 2392685535.
Sun Mar 31 15:23:15 2024
Database mounted in Exclusive Mode.
Completed: ALTER DATABASE   MOUNT
Sun Mar 31 15:23:15 2024
ALTER DATABASE OPEN
Sun Mar 31 15:23:15 2024
LGWR: Primary database is in CLUSTER CONSISTENT mode
Thread 1 opened at log sequence 313828
  Current log# 5 seq# 313828 mem# 0: /u3/oracle/xifenfei/redolog/redo05a.log
  Current log# 5 seq# 313828 mem# 1: /u4/oracle/xifenfei/redolog/redo05b.log
Successful open of redo thread 1.
Sun Mar 31 15:23:17 2024
SMON: enabling cache recovery
Sun Mar 31 15:23:17 2024
Undo Segment 69 Onlined
Undo Segment 70 Onlined
Undo Segment 71 Onlined
Undo Segment 72 Onlined
Sun Mar 31 15:23:21 2024
Errors in file /u2/oradb/admin/xifenfei/udump/xifenfei_ora_4384.trc:
ORA-00600: 内部错误代码，参数: [ktsiseginfo1], [20], [65], [73], [], [], [], []
Sun Mar 31 15:23:22 2024
Errors in file /u2/oradb/admin/xifenfei/udump/xifenfei_ora_4384.trc:
ORA-00600: 内部错误代码，参数: [ktsiseginfo1], [20], [65], [73], [], [], [], []
Error 600 happened during db open, shutting down database
USER: terminating instance due to error 600
Instance terminated by USER, pid = 4384
ORA-1092 signalled during: ALTER DATABASE OPEN...

启动的时候由于回滚段异常导致该问题,临时解决问题可以通过规避掉异常回滚段open库,但是这个问题的本质是由于seg$和undo$中记录不匹配导致,建议进一步通过hcheck进行检查分析,并进行后续处理,参考mos相关描述:
20240331201653

发表在 Oracle备份恢复 | 标签为 , , | 留下评论

ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因

发表于 2024 年 3 月 21 日 由 惜分飞

最近接触过几个ORA-600 16703的恢复case,报错日志类似为：

SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2071.trc  (incident=7357):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_7357/orcl_ora_2071_i7357.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2071.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_2071.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 2071): terminating the instance due to error 704
Instance terminated by USER, pid = 2071
ORA-1092 signalled during: ALTER DATABASE OPEN...

查看客户的日志,发现客户的库被人使用bbed按照网上的思路进行过恢复（使用block替换的方式尝试open库）,结果报出来ORA-00600: internal error code, arguments: [16703], [1403], [4]之类的错误

Wed Mar 20 22:00:39 2024
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1840752.trc  (incident=62557):
ORA-00600: internal error code, arguments: [16703], [1403], [4], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_62557/orcl_ora_1840752_i62557.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1840752.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [4], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1840752.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [4], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 1840752): terminating the instance due to error 704
Instance terminated by USER, pid = 1840752
ORA-1092 signalled during: alter database open...

其实ORA-600 16703 1403 N的错误,表示数据库在检索tab$表的时候,N为发现异常的obj#的值,这里的报错表示数据库在启动过程中无法检索到tab$中的tab$对象,应该是恢复过程中缺少该记录导致,解决问题的思路就是把该记录回写到tab$表中即可
以前写过类似文章:ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理

发表在 Oracle, Oracle备份恢复 | 标签为 | 留下评论