标签归档:kccpb_sanity_check_2

又一例存储cache丢失oracle数据库恢复

10.2.0.5 hp unix rac,由于存储掉电导致cache丢失,数据库无法正常启动,客户要求我们介入处理
数据库mount报ORA-00600 kccpb_sanity_check_2错误

Thu Jul 22 14:52:06 EAT 2021
alter database mount
Thu Jul 22 14:52:10 EAT 2021
Errors in file /oracle/admin/xff/udump/xff1_ora_4611.trc:
ORA-00600: internal error code, arguments: [kccpb_sanity_check_2], [4697564], [4697561], [0x000000000], [], [], [], []

该错误是由于控制文件损坏,尝试重建控制文件报ORA-01163,ORA-01517

'/dev/oradata/rxff_ls94'
CHARACTER SET ZHS16GBK
WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command
Default Temporary Tablespace will be necessary for a locally managed database in future release
Thu Jul 22 14:54:02 EAT 2021
Errors in file /oracle/admin/xff/udump/xff1_ora_7283.trc:
ORA-01163: SIZE clause indicates 262144 (blocks), but should match header 204800
ORA-01517: log member: '/dev/oradata/rxff_redo1_1'
ORA-1503 signalled during: CREATE CONTROLFILE REUSE DATABASE "xff" NORESETLOGS  NOARCHIVELOG

由于redo大小错误导致该问题,设置正确的redo大小继续重建

'/dev/oradata/rxff_ls94'
CHARACTER SET ZHS16GBK
WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command
Default Temporary Tablespace will be necessary for a locally managed database in future release
Thu Jul 22 15:01:00 EAT 2021
Errors in file /oracle/admin/xff/udump/xff1_ora_14737.trc:
ORA-00600: internal error code, arguments: [kccsga_update_ckpt_4], [32], [8], [], [], [], [], []
Thu Jul 22 15:01:01 EAT 2021
Errors in file /oracle/admin/xff/udump/xff1_ora_14737.trc:
ORA-00600: internal error code, arguments: [kccsga_update_ckpt_4], [32], [8], [], [], [], [], []
ORA-1503 signalled during: CREATE CONTROLFILE REUSE DATABASE "xff" NORESETLOGS  NOARCHIVELOG

报ORA-00600 kccsga_update_ckpt_4错误,导致控制文件失败,处理该错误之后,重建控制文件成功,分析文件头信息和redo信息,确认只能强制库,尝试强制open库

Thu Jul 22 16:02:05 EAT 2021
SMON: enabling cache recovery
Thu Jul 22 16:02:05 EAT 2021
ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0002.cdad19ed):
Thu Jul 22 16:02:05 EAT 2021
select ctime, mtime, stime from obj$ where obj# = :1
Thu Jul 22 16:02:05 EAT 2021
Errors in file /oracle/admin/xff/udump/xff1_ora_23219.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 19 with name "_SYSSMU19$" too small
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 23219
ORA-1092 signalled during: alter database open resetlogs...

这个问题比较常见:ORA-00704 ORA-00604 ORA-01555,参考类似文章:
在数据库open过程中常遇到ORA-01555汇总
数据库open过程遭遇ORA-1555对应sql语句补充
数据库open成功但是报ORA-00600 4137

Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan 
Thu Jul 22 16:08:48 EAT 2021
Errors in file /oracle/admin/xff/bdump/xff1_smon_27436.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=30, OS id=997
Thu Jul 22 16:08:49 EAT 2021
LOGSTDBY: Validating controlfile with logical metadata
Thu Jul 22 16:08:49 EAT 2021
ORACLE Instance xff1 (pid = 11) - Error 600 encountered while recovering transaction (1, 43).
Thu Jul 22 16:08:49 EAT 2021
Errors in file /oracle/admin/xff/bdump/xff1_smon_27436.trc:
ORA-00600: internal error code, arguments: [4137], [], [], [], [], [], [], []
Thu Jul 22 16:08:49 EAT 2021
Trace dumping is performing id=[cdmp_20210722160849]
Thu Jul 22 16:08:49 EAT 2021
LOGSTDBY: Validation complete
Completed: alter database open

该问题是由于undo异常,对undo进行处理,数据库无明显报错,安排导出数据

发表在 Oracle备份恢复 | 标签为 , , , | 评论关闭

ORA-600 2131故障说明

oracle 12c数据库启动报ORA-600 2131错误

Mon Nov 26 09:43:57 2018
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
ORACLE_BASE from environment = D:\app\Administrator
alter database mount exclusive
Mon Nov 26 09:44:00 2018
Using default pga_aggregate_limit of 2048 MB
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\trace\orcl12c_ora_3040.trc  (incident=375524):
ORA-00600: ??????, ??: [2131], [9], [8], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\orcl12c\orcl12c\incident\incdir_375524\orcl12c_ora_3040_i375524.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: alter database mount exclusive...

这个日志比较明显,数据库无法mount,在mount操作的时候报ORA-600 2131错误.
trace文件报错

Error: kccpb_sanity_check_2
Control file sequence number mismatch!
fhcsq: 497844 bhcsq: 497849 cfn 0 rpbn 16

----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
ksedst1()+92         CALL???  skdstdst()           000000001 000004000 000030000
                                                   016301338
kccpb_sanity_check(  CALL???  ksedst1()            1492761E0 0000798B4 0000798B9
)+834                                              000000000
kccbmp_get()+275     CALL???  kccpb_sanity_check(  000000000 000000000 000000000
                              )                    000004000
kccsed_rbl()+174     CALL???  kccbmp_get()         000017E28 015A67E14 015592200
                                                   000000001
kccocx()+1399        CALL???  kccsed_rbl()         100000010 100000001 0000354D8
                                                   000035508
kccocf()+167         CALL???  kccocx()+528         016303990 000000000
                                                   7FF00000001 000000000
kcfcmb()+1254        CALL???  kccocf()             000000000 000000000 000000000
                                                   000000000
kcfmdb()+69          CALL???  kcfcmb()             000000000 7FF59FFF856
                                                   000000007 7FE00000000
adbdrv_options()+43  CALL???  kcfmdb()             0163083E0 14903FF2C 000000005
724                                                000000000
adbdrv()+149         CALL???  adbdrv_options()     000000000 000000000 0163084A0
                                                   851F2CC90B75
opiexe()+22668       CALL???  adbdrv()             7FF00000023 000000003
                                                   000000000 016309380
opiosq0()+6009       CALL???  opiexe()             000000000 000000000 016309990
                                                   000000000
kpooprx()+410        CALL???  opiosq0()            000000003 000000000 000000000
                                                   0000000A4
kpoal8()+994         CALL???  kpooprx()            0146A57FC 000000001 0146A5820
                                                   000000001
opiodr()+1601        CALL???  kpoal8()             000000000 015523288 015523270
                                                   0159FCDD0
ttcpip()+1223        CALL???  opiodr()             7FE0000005E 00000001F
                                                   01630DA20 7FE00000000
opitsk()+2160        CALL???  ttcpip()             0146C0690 000000000 000000000
                                                   000000000
opiino()+1079        CALL???  opitsk()             000000007 000000000 01630F200
                                                   01630E970
opiodr()+1601        CALL???  opiino()             00000003C 000000000 01630F470
                                                   000000000
opidrv()+842         CALL???  opiodr()             00000003C 000000004 01630F470
                                                   000000000
sou2o()+94           CALL???  opidrv()+156         10000003C 7FE00000004
                                                   01630F470 0154E6A30
opimai_real()+276    CALL???  sou2o()              1D4851F4C467583 00A9D55E0
                                                   8001A000B07E2 1004B0039001E
opimai()+170         CALL???  opimai_real()        000000000 851F2CB1B179
                                                   00A9D55E0 01630F628
OracleThreadStart()  CALL???  opimai()             000000000 149031F90 000000050
+713                                               0000005C8
00000000775259CD     CALL???  OracleThreadStart()  000000000 000000000 000000000
                                                   000000000
000000007765A561     CALL???  00000000775259C0     000000000 000000000 000000000
                                                   000000000
 

--------------------- Binary Stack Dump ---------------------

这个错误和以往版本中的kccpb_sanity_check_2比较类似,由于数据库异常关闭导致ctl写丢失导致
ORA-600 2131/kccpb_sanity_check_2解释

DESCRIPTION:

  This internal error is raised when the sequence number (seq#) of the
  current block of the controlfile is greater than the seq# in the controlfile header.

  The header value should always be equal to, or greater than the value
  held in the control file block(s).

  This extra check was introduced in Oracle 10gR2 to detect lost writes
  or stale reads to the header.

ARGUMENTS:
  Arg [a] seq# in control block header.
  Arg [b] seq# in the control file header.
  Arg 1 

FUNCTIONALITY:
  Kernel Cache layer Control file component.
发表在 ORA-xxxxx, Oracle备份恢复 | 标签为 , , | 评论关闭

数据库恢复的敏感性—重建控制文件使用不合适数据文件

有客户数据库由于某种原因无法open,请求我们技术支持.通过检查alert日志发现数据库是由于ORA-600 kccpb_sanity_check_2错误.并且他们已经重建控制文件,通过我们的Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)
datafile 6 异常
wrong-file1
wrong-file
wrong-file-redo


通过这里我们发现datafile 6 数据文件头是干净的,而且对应的redo seq为5270,而其他文件头都是fuzzy,而且对应的redo为295902613.这里怀疑datafile 6 可能是错误的.当然对于这样scn相距比较大的情况,我们可以通过隐含参数,修改scn等方法强制让该库起来(或者该文件online),但是从现在看到的情况,文件很可能异常,这样强制恢复,可能没有实际意义.
分析alert日志
alert-log

这个里面可以发现是先删除了sde表空间,然后创建了同一个表空间,只是数据文件路径不一样了.而且正好在seq为5270的地方操作的.现在出现datafile 6异常的原因已经清楚,就是创建数据控制文件的时候,使用了错误的数据文件导致.

完美恢复数据库

D:\>sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 - Production on 星期六 7月 30 16:31:01 2016

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


连接到:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options

SQL> shutdown immediate;
ORA-01109: 数据库未打开


已经卸载数据库。
ORACLE 例程已经关闭。
SQL> STARTUP NOMOUNT
ORACLE 例程已经启动。

Total System Global Area 5133828096 bytes
Fixed Size                  2011360 bytes
Variable Size            2382368544 bytes
Database Buffers         2734686208 bytes
Redo Buffers               14761984 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "LANDDB" NORESETLOGS  ARCHIVELOG
  2      MAXLOGFILES 16
  3      MAXLOGMEMBERS 3
  4      MAXDATAFILES 100
  5      MAXINSTANCES 8
  6      MAXLOGHISTORY 292
  7  LOGFILE
  8    GROUP 1 'F:\ORADATA\LANDDB\ONLINELOG\O1_MF_1_4JCM05KL_.LOG'  SIZE 50M,
  9    GROUP 2 'F:\ORADATA\LANDDB\ONLINELOG\O1_MF_2_4JCM064D_.LOG'  SIZE 50M,
 10    GROUP 3 'F:\ORADATA\LANDDB\ONLINELOG\O1_MF_3_4JCM06PG_.LOG'  SIZE 50M
 11  DATAFILE
 12    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_SYSTEM_4JCLYY6T_.DBF',
 13    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_UNDOTBS1_4JCLYY8S_.DBF',
 14    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_SYSAUX_4JCLYY7B_.DBF',
 15    'F:\ORADATA\LANDDB\DATAFILE\O1_MF_USERS_4JCLYY98_.DBF',
 16    'F:\ORADATA\LANDDB\DATAFILE\FUJIAN',
 17    'D:\data\sde.dbf'
 18  CHARACTER SET ZHS16GBK
 19  ;

控制文件已创建。

SQL> recover database;
完成介质恢复。
SQL> alter database open;

数据库已更改。

SQL>

这个库比较幸运,客户发现异常之后,里面停止了有风险性的操作(比如使用_allow_resetlogs_corruption参数,resetlogs库等),使得数据完美恢复0丢失.如果条件允许最好使用老的控制文件来重建新控制文件,而不要通过人工去系统中找数据文件来实现恢复,这样很可能有遗落或者使用错误的数据文件

发表在 Oracle备份恢复 | 标签为 , | 评论关闭