标签云
asm 恢复 asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (100)
- 数据库 (1,589)
- DB2 (22)
- MySQL (70)
- Oracle (1,459)
- Data Guard (49)
- EXADATA (7)
- GoldenGate (21)
- ORA-xxxxx (158)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (13)
- ORACLE 21C (3)
- Oracle ASM (65)
- Oracle Bug (7)
- Oracle RAC (47)
- Oracle 安全 (6)
- Oracle 开发 (27)
- Oracle 监听 (27)
- Oracle备份恢复 (526)
- Oracle安装升级 (83)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (75)
- PostgreSQL (13)
- PostgreSQL恢复 (3)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (36)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (19)
-
最近发表
- ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
- 最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
- ORA-600 2662快速恢复之Patch scn工具
- TNS-12518: TNS:listener could not hand off client connection
- ora.storage无法启动报ORA-12514故障处理
- 断电引起文件scn异常数据库恢复
- ORA-16188: LOG_ARCHIVE_CONFIG settings inconsistent with previously started instance
- .[hudsonL@cock.li].mkp勒索加密数据库完美恢复
- 模拟带库实现rman远程备份
- 又一例:ORA-600 kclchkblk_4和2662故障
- Oracle误删除数据文件恢复
- Oracle 19C 备库DML重定向—DML Redirection
- ORA-01595/ORA-600 4194处理
- 从ORA-00283 ORA-16433报错开始恢复
- 近期又遇到ORA-600 16703和ORA-702故障
- RECOVER_YOUR_DATA勒索恢复
- ORA-01033: ORACLE initialization or shutdown in progress 故障处理
- Oracle 19c/21c最新patch信息-202401
- 存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
- ORA-600 kcrf_resilver_log_1故障处理
月归档:七月 2015
分享一例由于主库逻辑坏块导致dataguard容灾失效
有朋友数据库配置了物理dataguard,但是由于触发了某种Oracle bug/或者其他原因导致block出现大量逻辑坏块,结果是主备库(主库启动后就crash,备库failover后也启动后就crash)都无法正常工作,请求给予技术支持。
数据库配置了物理dataguard,但是主库在进行了某些操作之后,主库直接crash.重启主库发现,数据库启动之后,稍后数据库继续CRASH
Wed Jul 08 16:32:22 2015 Thread 1 advanced to log sequence 401531 (LGWR switch) Current log# 6 seq# 401531 mem# 0: /opt/oracle/database/fast_recovery_area/xifenfei/onlinelog/o1_mf_6_b9p7qhrm_.log Archived Log entry 605194 added for thread 1 sequence 401530 ID 0xfbe74bc9 dest 1: ARC3: Standby redo logfile selected for thread 1 sequence 401530 for destination LOG_ARCHIVE_DEST_2 ****************************************************************** LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2 ****************************************************************** Wed Jul 08 16:33:02 2015 Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_p032_413611.trc (incident=201910): ORA-00600: internal error code, arguments: [17114], [0x7FB03BB6DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FB03BB6DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201910/xifenfei_p032_413611_i201910.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_p032_413611.trc (incident=201911): ORA-00600: internal error code, arguments: [kghfrempty:ds], [0x7FB03BB6DBE8], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FB03BB6DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FB03BB6DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201911/xifenfei_p032_413611_i201911.trc Wed Jul 08 16:34:23 2015 SMON: slave died unexpectedly, downgrading to serial recovery Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc (incident=201547): ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201547/xifenfei_smon_413207_i201547.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 401530, block 119 to scn 73226510040 Recovery of Online Redo Log: Thread 1 Group 5 Seq 401530 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/xifenfei/onlinelog/o1_mf_5_b9p7qhl0_.log Recovery of Online Redo Log: Thread 1 Group 6 Seq 401531 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/xifenfei/onlinelog/o1_mf_6_b9p7qhrm_.log Block recovery completed at rba 401531.1882.16, scn 17.212066009 ORACLE Instance xifenfei (pid = 16) - Error 600 encountered while recovering transaction (13, 14) on object 135520. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc: ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] Wed Jul 08 16:34:24 2015 Dumping diagnostic data in directory=[cdmp_20150708163424], requested by (instance=1, osid=413207 (SMON)), summary=[incident=201547]. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc (incident=201548): ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F24D680D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/incident/incdir_201548/xifenfei_smon_413207_i201548.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Fatal internal error happened while SMON was doing active transaction recovery. Errors in file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_smon_413207.trc: ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F24D680D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F24D680D7A0], [], [], [], [], [], [], [], [], [], [] SMON (ospid: 413207): terminating the instance due to error 474 System state dump requested by (instance=1, osid=413207 (SMON)), summary=[abnormal instance termination]. System State dumped to trace file /opt/oracle/database/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_diag_413167_20150708163426.trc Dumping diagnostic data in directory=[cdmp_20150708163426], requested by (instance=1, osid=413207 (SMON)), summary=[abnormal instance termination]. Instance terminated by SMON, pid = 413207
由于主库不能正常open,备库直接failover方式激活
Wed Jul 08 17:56:41 2015 alter database recover managed standby database finish Terminal Recovery: request posted (xffdb) Wed Jul 08 17:56:45 2015 Begin: Standby Redo Logfile archival End: Standby Redo Logfile archival Terminal Recovery timestamp is '07/08/2015 17:56:45' Terminal Recovery: applying standby redo logs. Terminal Recovery: thread 1 seq# 401533 redo required Terminal Recovery: Recovery of Online Redo Log: Thread 1 Group 7 Seq 401533 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_7_bb7x31lk_.log Identified End-Of-Redo (failover) for thread 1 sequence 401533 at SCN 0xffff.ffffffff Incomplete Recovery applied until change 73226530800 time 07/08/2015 16:56:40 Terminal Recovery: successful completion Wed Jul 08 17:56:45 2015 ARCH: Archival stopped, error occurred. Will continue retrying Forcing ARSCN to IRSCN for TR 17:212086768 ORACLE Instance xffdb - Archival Error Attempt to set limbo arscn 17:212086768 irscn 17:212086768 ORA-16014: log 7 sequence# 401533 not archived, no available destinations ORA-00312: online log 7 thread 1: '/opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_7_bb7x31lk_.log' Resetting standby activation ID 4226239433 (0xfbe74bc9) Wed Jul 08 17:56:45 2015 MRP0: Media Recovery Complete (xffdb) MRP0: Background Media Recovery process shutdown (xffdb) Terminal Recovery: completion detected (xffdb) Completed: alter database recover managed standby database finish Wed Jul 08 17:56:58 2015 alter database commit to switchover to primary ALTER DATABASE SWITCHOVER TO PRIMARY (xffdb) Maximum wait for role transition is 15 minutes. Backup controlfile written to trace file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_ora_485094.trc Standby terminal recovery start SCN: 73226530482 RESETLOGS after incomplete recovery UNTIL CHANGE 73226530800 Online logfile pre-clearing operation disabled by switchover Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_1_bb7x30gw_.log: Thread 1 Group 1 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log: Thread 1 Group 2 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log: Thread 1 Group 3 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log: Thread 1 Group 4 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log: Thread 1 Group 5 was previously cleared Online log /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_6_bb7x31cz_.log: Thread 1 Group 6 was previously cleared Standby became primary SCN: 73226530481 Wed Jul 08 17:56:58 2015 Setting recovery target incarnation to 3 Switchover: Complete - Database mounted as primary Completed: alter database commit to switchover to primary
数据库激活成功后,重启激活之后数据库发现和主库出现类似情况
Wed Jul 08 17:57:25 2015 ALTER DATABASE MOUNT Successful mount of redo thread 1, with mount id 4243462021 Database mounted in Exclusive Mode Lost write protection disabled Completed: ALTER DATABASE MOUNT Wed Jul 08 17:57:30 2015 ALTER DATABASE OPEN Assigning activation ID 4243462021 (0xfcee1785) LGWR: STARTING ARCH PROCESSES Wed Jul 08 17:57:30 2015 ARC0 started with pid=23, OS id=485230 ARC0: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE ARC0: STARTING ARCH PROCESSES Thread 1 advanced to log sequence 2 (thread open) Wed Jul 08 17:57:31 2015 ARC1 started with pid=24, OS id=485236 Wed Jul 08 17:57:31 2015 ARC2 started with pid=25, OS id=485240 Wed Jul 08 17:57:31 2015 ARC3 started with pid=26, OS id=485244 ARC1: Archival started ARC2: Archival started ARC1: Becoming the 'no FAL' ARCH ARC1: Becoming the 'no SRL' ARCH ARC2: Becoming the heartbeat ARCH Thread 1 opened at log sequence 2 Current log# 2 seq# 2 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set SMON: enabling cache recovery Wed Jul 08 17:57:31 2015 NSA2 started with pid=27, OS id=485248 [485226] Successfully onlined Undo Tablespace 2. Undo initialization finished serial:0 start:3453451748 end:3453452018 diff:270 (2 seconds) Dictionary check beginning Dictionary check complete Verifying file header compatibility for 11g tablespace encryption.. Verifying 11g file header compatibility for tablespace encryption completed SMON: enabling tx recovery Database Characterset is AL32UTF8 No Resource Manager plan active replication_dependency_tracking turned off (no async multimaster replication found) Wed Jul 08 17:57:32 2015 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64481): ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64481/xffdb_p032_485380_i64481.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Starting background process QMNC Wed Jul 08 17:57:32 2015 QMNC started with pid=92, OS id=485512 LOGSTDBY: Validating controlfile with logical metadata LOGSTDBY: Validation complete Dumping diagnostic data in directory=[cdmp_20150708175733], requested by (instance=1, osid=485380 (P032)), summary=[incident=64481]. Thread 1 advanced to log sequence 3 (LGWR switch) Current log# 3 seq# 3 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log ARC3: STARTING ARCH PROCESSES Wed Jul 08 17:57:34 2015 ARC4 started with pid=93, OS id=485516 Wed Jul 08 17:57:35 2015 db_recovery_file_dest_size of 204800 MB is 0.41% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup. ARC4: Archival started ARC3: STARTING ARCH PROCESSES COMPLETE krsk_srl_archive_int: Enabling archival of deferred physical standby SRLs Archived Log entry 273963 added for thread 1 sequence 2 ID 0xfcee1785 dest 1: Archived Log entry 273964 added for thread 1 sequence 401533 ID 0xfbe74bc9 dest 1: Completed: ALTER DATABASE OPEN Wed Jul 08 17:57:36 2015 Starting background process CJQ0 Wed Jul 08 17:57:36 2015 CJQ0 started with pid=95, OS id=485554 Shutting down archive processes ARCH shutting down ARC4: Archival stopped Wed Jul 08 17:57:41 2015 Thread 1 advanced to log sequence 4 (LGWR switch) Current log# 4 seq# 4 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log Wed Jul 08 17:57:41 2015 Archived Log entry 273965 added for thread 1 sequence 3 ID 0xfcee1785 dest 1: Wed Jul 08 17:58:30 2015 Sweep [inc][64481]: completed Sweep [inc2][64481]: completed Wed Jul 08 17:58:31 2015 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64482): ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64482/xffdb_p032_485380_i64482.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Jul 08 17:58:32 2015 Dumping diagnostic data in directory=[cdmp_20150708175832], requested by (instance=1, osid=485380 (P032)), summary=[incident=64482]. Wed Jul 08 17:58:36 2015 Thread 1 advanced to log sequence 5 (LGWR switch) Current log# 5 seq# 5 mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log Wed Jul 08 17:59:02 2015 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64483): ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64483/xffdb_p032_485380_i64483.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Jul 08 17:59:04 2015 Dumping diagnostic data in directory=[cdmp_20150708175904], requested by (instance=1, osid=485380 (P032)), summary=[incident=64483]. Wed Jul 08 17:59:29 2015 Sweep [inc][64483]: completed Sweep [inc][64482]: completed Sweep [inc2][64483]: completed Sweep [inc2][64482]: completed Wed Jul 08 17:59:30 2015 Block recovery from logseq 2, block 104 to scn 73226531646 Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log Recovery of Online Redo Log: Thread 1 Group 4 Seq 4 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log Recovery of Online Redo Log: Thread 1 Group 5 Seq 5 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log Block recovery stopped at EOT rba 5.765.16 Block recovery completed at rba 5.765.16, scn 17.212087614 Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_p032_485380.trc (incident=64484): ORA-00600: internal error code, arguments: [KSMFPG2], [0x7FE96B50D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17114], [0x7FE96B50DBD0], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7FE96B50DBF8], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64484/xffdb_p032_485380_i64484.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Dumping diagnostic data in directory=[cdmp_20150708175934], requested by (instance=1, osid=485380 (P032)), summary=[incident=64487]. Wed Jul 08 17:59:36 2015 SMON: slave died unexpectedly, downgrading to serial recovery Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc (incident=64129): ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64129/xffdb_smon_485190_i64129.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Block recovery from logseq 2, block 104 to scn 73226531646 Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_2_bb7x30js_.log Recovery of Online Redo Log: Thread 1 Group 3 Seq 3 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_3_bb7x310q_.log Recovery of Online Redo Log: Thread 1 Group 4 Seq 4 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_4_bb7x312r_.log Recovery of Online Redo Log: Thread 1 Group 5 Seq 5 Reading mem 0 Mem# 0: /opt/oracle/database/fast_recovery_area/DBHLDB/onlinelog/o1_mf_5_bb7x317f_.log Block recovery completed at rba 5.765.16, scn 17.212087615 ORACLE Instance xffdb (pid = 16) - Error 600 encountered while recovering transaction (13, 14) on object 135520. Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc: ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] Dumping diagnostic data in directory=[cdmp_20150708175937], requested by (instance=1, osid=485190 (SMON)), summary=[incident=64129]. Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc (incident=64130): ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F5AED10D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] Incident details in: /opt/oracle/database/diag/rdbms/xffdb/xffdb/incident/incdir_64130/xffdb_smon_485190_i64130.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Fatal internal error happened while SMON was doing active transaction recovery. Errors in file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_smon_485190.trc: ORA-00600: internal error code, arguments: [KSMFPG2], [0x7F5AED10D000], [], [], [], [], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17182], [0x7F5AED10D7A0], [], [], [], [], [], [], [], [], [], [] SMON (ospid: 485190): terminating the instance due to error 474 System state dump requested by (instance=1, osid=485190 (SMON)), summary=[abnormal instance termination]. System State dumped to trace file /opt/oracle/database/diag/rdbms/xffdb/xffdb/trace/xffdb_diag_485150_20150708175939.trc Dumping diagnostic data in directory=[cdmp_20150708175939], requested by (instance=1, osid=485190 (SMON)), summary=[abnormal instance termination]. Instance terminated by SMON, pid = 485190
通过进一步分析确定是67号文件有异常,使用dbv检查该文件发现
DBVERIFY: Release 11.2.0.4.0 - Production on Thu Jul 9 11:08:27 2015 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. DBVERIFY - Verification starting : FILE = /opt/oracle/database/oradata/xffdb/dk_adv_ts_main_25.dbf Block Checking: DBA = 283035270, Block Type = KTB-managed data block data header at 0x7f87813ea064 kdbchk: the amount of space used is not equal to block size used=7383 fsc=0 avsp=873 dtl=8088 Page 2016902 failed with check code 6110 Block Checking: DBA = 283035271, Block Type = KTB-managed data block data header at 0x7f87813ec064 kdbchk: the amount of space used is not equal to block size used=8676 fsc=0 avsp=832 dtl=8088 Page 2016903 failed with check code 6110 Block Checking: DBA = 283035272, Block Type = KTB-managed data block data header at 0x7f87813ee064 kdbchk: avsp(1306) > tosp(894) Page 2016904 failed with check code 6128 Block Checking: DBA = 283035273, Block Type = KTB-managed data block data header at 0x7f87813f0064 kdbchk: the amount of space used is not equal to block size used=7506 fsc=0 avsp=815 dtl=8088 Page 2016905 failed with check code 6110 Block Checking: DBA = 283035274, Block Type = KTB-managed data block data header at 0x7f87813f2064 kdbchk: the amount of space used is not equal to block size used=7892 fsc=0 avsp=884 dtl=8088 Page 2016906 failed with check code 6110 ………… Block Checking: DBA = 283035368, Block Type = KTB-managed data block data header at 0x7f87814ae064 kdbchk: the amount of space used is not equal to block size used=7934 fsc=0 avsp=837 dtl=8088 Page 2017000 failed with check code 6110 Block Checking: DBA = 283035369, Block Type = KTB-managed data block data header at 0x7f87814b0064 kdbchk: the amount of space used is not equal to block size used=7683 fsc=0 avsp=883 dtl=8088 Page 2017001 failed with check code 6110 Block Checking: DBA = 283035370, Block Type = KTB-managed data block data header at 0x7f87814b2064 kdbchk: the amount of space used is not equal to block size used=8556 fsc=0 avsp=841 dtl=8088 Page 2017002 failed with check code 6110 Block Checking: DBA = 283035371, Block Type = KTB-managed data block data header at 0x7f87814b4064 kdbchk: the amount of space used is not equal to block size used=7460 fsc=0 avsp=822 dtl=8088 Page 2017003 failed with check code 6110 DBVERIFY - Verification complete Total Pages Examined : 4063232 Total Pages Processed (Data) : 2724435 Total Pages Failing (Data) : 102 Total Pages Processed (Index): 1064839 Total Pages Failing (Index): 0 Total Pages Processed (Other): 273957 Total Pages Processed (Seg) : 0 Total Pages Failing (Seg) : 0 Total Pages Empty : 1 Total Pages Marked Corrupt : 0 Total Pages Influx : 0 Total Pages Encrypted : 0 Highest block SCN : 212065913 (17.212065913)
把datafile 67给offline之后,数据库open正常,也不再crash。这里比较明显,导致主库和被动都出现异常,主库直接crash,然后重启后,一会儿就crash;备库激活后,重启后一会儿也就crash;两个库现象相同。最后通过dbv定位到是由于某个文件出现大量逻辑坏块,导致数据库open之后,进行回滚之时crash.对于此类故障,可以通过屏蔽事务回滚,并且通过alert日志和trace文件定位到异常对象,可以删除异常对象可以暂时解决该问题。而导致数据库出现类似问题。
通过这个案例,可以的出来,由于oracle某种bug或者其他原因,导致block 逻辑损坏,而且这个损坏会传输到备库,导致备库也异常,oracle的备份,容灾不能全部依赖物理dataguard容灾。因此在条件允许情况下,建议增加物理备份和逻辑容灾(类似OGG)
在数据库open过程中常遇到ORA-01555汇总
在数据库open的过程中,select ctime, mtime, stime from obj$ where obj# = :1语句报ORA-01555错误,数据库无法正常open
一般情况下会报某个回滚段,但是这里ORA-01555: snapshot too old: rollback segment number 0 with name “SYSTEM” too small这里直接报了system(系统回滚段),属于少见情况
Fri Jun 26 11:47:31 2015 SMON: enabling cache recovery Fri Jun 26 11:47:31 2015 ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0b41.37629378): Fri Jun 26 11:47:31 2015 select ctime, mtime, stime from obj$ where obj# = :1 Fri Jun 26 11:47:31 2015 Errors in file /orabin/admin/doocrm/udump/doocrm_ora_5046722.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 1 ORA-01555: snapshot too old: rollback segment number 0 with name "SYSTEM" too small Error 704 happened during db open, shutting down database USER: terminating instance due to error 704 Instance terminated by USER, pid = 5046722 ORA-1092 signalled during: alter database open resetlogs...
在数据库open的过程中,select increment$,minvalue,maxvalue,cycle#,order$,cache,highwater,audit$,flags from seq$ where obj#=:1语句上报ORA-01555,导致数据库open失败
在数据库open过程中,select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1语句导致数据库open失败.
ARC0: Becoming the 'no SRL' ARCH Sun Jun 28 16:08:22 2015 ARC1: Becoming the heartbeat ARCH Sun Jun 28 16:08:22 2015 SMON: enabling cache recovery Sun Jun 28 16:08:22 2015 ORA-01555 caused by SQL statement below (SQL ID: 7bd391hat42zk, Query Duration=0 sec, SCN: 0x0d27.0a1ce29d): Sun Jun 28 16:08:22 2015 select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp, DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1 Sun Jun 28 16:08:22 2015 Errors in file /oracle/app/oracle/admin/ibsscrm/udump/xxxx_ora_30212428.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-01555: snapshot too old: rollback segment number 0 with name "SYSTEM" too small Error 604 happened during db open, shutting down database USER: terminating instance due to error 604 Instance terminated by USER, pid = 30212428 ORA-1092 signalled during: alter database open...
出现这类问题,一般是由于obj$,seq$,undo$等基表上对象scn大于数据库当前scn或者是由于这些表上有事务未提交,出现上述两种情况,数据库需要找对应的undo的回滚段中记录,而此时对应的回滚段异常(或者是由于redo未进行正常前滚,导致上述对象或者回滚段记录不正常),从而出现类似情况,一般出现此类情况,可以通过10046定位到block,然后故障原因采用bbed修改scn或者bbed提交事务来解决此类问题.
最近两年的恢复中又遇到一些ora-00704 ora-00604 ora-01555的错误,导致数据库无法正常open,对其进行补充,具体参见:数据库open过程遭遇ORA-1555对应sql语句补充
误修改/u01权限/所有者的故障恢复
有朋友找到我,说他对生产库做了误操作,导致数据库异常,请我帮忙处理,对/u01目录修改了用户和权限,导致数据库无法登录,但是业务还在继续
误操作命令
mkdir -p /u01/app/grid mkdir -p /u01/app/11.2.0/grid chown -R grid:oinstall /u01 mkdir -p /u01/app/oracle chown oracle:oinstall /u01/app/oracle chmod -R 775 /u01
尝试sqlplus登录数据库报ORA-12547
[oracle@www.xifenfei.com admin]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Wed Jul 1 17:40:42 2015 Copyright (c) 1982, 2011, Oracle. All rights reserved. ERROR: ORA-12547: TNS:lost contact Enter user-name: ERROR: ORA-12547: TNS:lost contact Enter user-name: ERROR: ORA-12547: TNS:lost contact SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
数据库alert日志报错
Wed Jul 01 18:03:22 2015 Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_28977.trc (incident=129553): ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129553/xifenfei_ora_28977_i129553.trc Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129553/xifenfei_ora_28977_i129553.trc: ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] Wed Jul 01 18:03:22 2015 Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_28979.trc (incident=129561): ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129561/xifenfei_ora_28979_i129561.trc Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129561/xifenfei_ora_28979_i129561.trc: ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] Wed Jul 01 18:03:22 2015 Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_28985.trc (incident=129569): ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129569/xifenfei_ora_28985_i129569.trc Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129569/xifenfei_ora_28985_i129569.trc: ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] Wed Jul 01 18:03:22 2015 Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_28983.trc (incident=129577): ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129577/xifenfei_ora_28983_i129577.trc Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_129577/xifenfei_ora_28983_i129577.trc: ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], []
trace文件信息
Dump file /u01/app/oracle/diag/rdbms/hybris01/hybris01/incident/incdir_129577/hybris01_ora_28983_i129577.trc Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1 System name: Linux Node name: dpppdridbo00 Release: 2.6.32-431.17.1.el6.x86_64 Version: #1 SMP Wed May 7 23:32:49 UTC 2014 Machine: x86_64 Instance name: hybris01 Redo thread mounted by this instance: 0 <none> Oracle process number: 0 Unix process pid: 28983, image: oracle@dpppdridbo00 *** 2015-07-01 18:03:22.296 Dump continued from file: /u01/app/oracle/diag/rdbms/hybris01/hybris01/trace/hybris01_ora_28983.trc ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] ========= Dump for incident 129577 (ORA 600 [spstp: ORACLE_HOME uid does not match euid]) ======== *** 2015-07-01 18:03:22.297 dbkedDefDump(): Starting incident default dumps (flags=0x0, level=3, mask=0x0) ----- Error Stack Dump ----- ORA-00600: internal error code, arguments: [spstp: ORACLE_HOME uid does not match euid], [1100], [1101], [], [], [], [], [], [], [], [], [] ----- SQL Statement (None) ----- Current SQL information unavailable - no SGA.
相关用户名相关信息
[oracle@dpppdridbo00 incdir_129577]$ id grid uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1100(asmadmin),1300(asmdba),1301(asmoper) [oracle@dpppdridbo00 incdir_129577]$ id oracle uid=1101(oracle) gid=1000(oinstall) groups=1000(oinstall),1200(dba),1300(asmdba)
这里比较明显,数据库的oracle_home,的所有者id为1101,但是被修改为了1100
查看相关目录文件权限
[oracle@www.xifenfei.com ~]$ env|grep ORA ORACLE_SID=xifenfei ORACLE_BASE=/u01/app/oracle ORACLE_TERM=xterm ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 [oracle@www.xifenfei.com ~]$ cd /u01/app/oracle/product/11.2.0/db_1 [oracle@www.xifenfei.com db_1]$ ls -ltr total 308 -rwxrwxr-x. 1 grid oinstall 63 Mar 1 16:39 oraInst.loc drwxrwxr-x. 8 grid oinstall 4096 Mar 1 16:40 assistants drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:40 crs drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 csmig drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:40 cv drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 demo drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 diagnostics drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 has drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 emcli drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:40 dv drwxrwxr-x. 8 grid oinstall 4096 Mar 1 16:40 ide drwxrwxr-x. 8 grid oinstall 4096 Mar 1 16:40 javavm drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:40 jdev drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:40 md drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:40 nls drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:40 odbc drwxrwxr-x. 5 grid oinstall 4096 Mar 1 16:40 olap drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:40 oracore drwxrwxr-x. 8 grid oinstall 4096 Mar 1 16:40 ord drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:40 owm drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:40 precomp drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:40 scheduler drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:40 relnotes drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 slax drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 sqlj drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 wwg drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:40 usm drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:40 uix drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:40 plsql drwxrwxr-x. 2 grid oinstall 4096 Mar 1 16:40 utl drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:40 xdk drwxrwxr-x. 2 grid oinstall 4096 Mar 1 16:40 instantclient drwxrwxr-x. 20 grid oinstall 4096 Mar 1 16:40 oc4j drwxrwxr-x. 2 grid oinstall 4096 Mar 1 16:40 timingframework drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:41 clone drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:41 j2ee drwxrwxr-x. 8 grid oinstall 4096 Mar 1 16:41 apex drwxrwxr-x. 13 grid oinstall 4096 Mar 1 16:41 sqldeveloper drwxrwxr-x. 2 grid oinstall 4096 Mar 1 16:41 jlib drwxrwxr-x. 2 grid oinstall 4096 Mar 1 16:41 dc_ocm drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:41 jdk drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:41 jdbc drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:41 ucp drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:41 OPatch drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:41 ccr drwxrwxr-x. 26 grid oinstall 4096 Mar 1 16:41 owb drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:41 mgw drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:41 opmn drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:41 cdata drwxrwxr-x. 6 grid oinstall 4096 Mar 1 16:41 css drwxrwxr-x. 2 grid oinstall 4096 Mar 1 16:41 mesg drwxrwxr-x. 2 grid oinstall 4096 Mar 1 16:41 config drwxrwxr-x. 3 grid oinstall 4096 Mar 1 16:41 EMStage drwxrwxr-x. 12 grid oinstall 4096 Mar 1 16:41 ldap drwxrwxr-x. 15 grid oinstall 4096 Mar 1 16:41 sysman drwxrwxr-x. 9 grid oinstall 4096 Mar 1 16:41 srvm drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:41 racg drwxrwxr-x. 10 grid oinstall 4096 Mar 1 16:41 ctx drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:42 sqlplus drwxrwxr-x. 8 grid oinstall 4096 Mar 1 16:42 oui drwxrwxr-x. 4 grid oinstall 12288 Mar 1 16:42 lib drwxrwxr-x. 5 grid oinstall 4096 Mar 1 16:42 perl drwxrwxr-x. 11 grid oinstall 4096 Mar 1 16:42 network drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:42 deinstall drwxrwxr-x. 5 grid oinstall 4096 Mar 1 16:42 hs -rwxrwxr-x. 1 grid oinstall 494 Mar 1 16:42 root.sh drwxrwxr-x. 13 grid oinstall 4096 Mar 1 16:42 rdbms drwxrwxr-x. 13 grid oinstall 4096 Mar 1 16:42 inventory drwxrwxr-x. 4 grid oinstall 4096 Mar 1 16:42 cfgtoollogs drwxrwxr-x. 7 grid oinstall 4096 Mar 1 16:44 install drwxrwxr-x. 2 grid oinstall 12288 Mar 1 16:44 bin drwxrwxr-x. 4 grid oinstall 4096 Mar 4 18:49 log drwxrwxr-x. 2 grid oinstall 4096 Jun 30 22:31 dbs
处理方法
root用户 chown -R oracle:oinstall /u01 chown oracle:oinstall /u01/app/oracle oracle用户 chmod 6751 $ORACLE_HOME/bin/oracle
这里的修改权限,为了保证业务运行正常,是尽量往大的方向修改的,如果条件允许,在后期有条件的情况下,建议重新安装oracle软件