标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (102)
- 数据库 (1,679)
- DB2 (22)
- MySQL (73)
- Oracle (1,541)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (24)
- ORA-xxxxx (159)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (14)
- ORACLE 21C (3)
- Oracle 23ai (7)
- Oracle ASM (67)
- Oracle Bug (8)
- Oracle RAC (52)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (563)
- Oracle安装升级 (92)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (79)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (37)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (20)
-
最近发表
- 解决oracle数据文件路径有回车故障
- .wstop扩展名勒索数据库恢复
- Oracle Recovery Tools工具一键解决ORA-00376 ORA-01110故障(文件offline)
- OGG-02771 Input trail file format RELEASE 19.1 is different from previous trail file form at RELEASE 11.2.
- OGG-02246 Source redo compatibility level 19.0.0 requires trail FORMAT 12.2 or higher
- GoldenGate 19安装和打patch
- dd破坏asm磁盘头恢复
- 删除asmlib磁盘导致磁盘组故障恢复
- Kylin Linux 安装19c
- ORA-600 krse_arc_complete.4
- Oracle 19c 202410补丁(RUs+OJVM)
- ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复
- .mkp扩展名oracle数据文件加密恢复
- 清空redo,导致ORA-27048: skgfifi: file header information is invalid
- A_H_README_TO_RECOVER勒索恢复
- 通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评
- ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME
- ORA-01092 ORA-00604 ORA-01558故障处理
- ORA-65088: database open should be retried
- Oracle 19c异常恢复—ORA-01209/ORA-65088
标签归档:ORA-600 2037
win强制修改盘符导致oracle异常恢复
有客户反馈,他们在系统没有关闭数据库的情况下,强制修改了win系统盘符,然后导致数据库异常,启动报错
Sat Feb 25 12:50:40 2023 Recovery of Online Redo Log: Thread 1 Group 2 Seq 10440 Reading mem 0 Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO02.LOG Sat Feb 25 12:50:40 2023 Completed redo application Sat Feb 25 12:50:40 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_p001_5604.trc: ORA-00600: 内部错误代码, 参数: [2037], [25801018], [2973409798], [6], [255], [25], [1198764346], [100796692] Sat Feb 25 12:50:40 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_7648.trc: ORA-07445: 出现异常错误: 核心转储 [ACCESS_VIOLATION][_kslwlmod+166][PC:0x469742][ADDR:0x54F8][UNABLE_TO_WRITE][] ORA-04096: 触发器 '' 的 WHEN 子句过大, 限量为 2K Sat Feb 25 12:50:40 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_7252.trc: ORA-00600: internal error code, arguments: [ksuapc2], [258], [0], [2], [1], [2], [], [] Sat Feb 25 12:50:43 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_p001_5604.trc: ORA-00081: 地址范围 [0x77240440, 0x77240444) 不可读 ORA-07445: 出现异常错误: 核心转储 [ACCESS_VIOLATION][_ksl_cleanup+723][PC:0x46E373][ADDR:0x1C][UNABLE_TO_READ][] ORA-00081: 地址范围 [0x77240440, 0x77240444) 不可读 ORA-00600: 内部错误代码, 参数: [2037], [25801018], [2973409798], [6], [255], [25], [1198764346], [100796692] Sat Feb 25 12:50:45 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_dbw0_6332.trc: ORA-07445: ??????: ???? [ACCESS_VIOLATION][_kews_idle_wait+378][PC:0x604AE6][ADDR:0xED30C470][UNABLE_TO_WRITE][] Sat Feb 25 12:50:48 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_7648.trc: ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], [] ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读 ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], [] ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读 ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], [] ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读 ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], [] ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读 ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], [] ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读 ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], [] ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读 ORA-00600: 内部错误代码, 参数: [kslwlflux:1], [0xAB805400], [0x549C], [2], [], [], [], [] ORA-00081: 地址范围 [0x74480443, 0x74480447) 不可读 ORA-00600: 内部错误代码, 参 Sat Feb 25 12:51:34 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_d000_8116.trc: ORA-07445: ??????: ???? [ACCESS_VIOLATION][_kmcgms+121][PC:0x5D6C71][ADDR:0x50][UNABLE_TO_WRITE][] Sat Feb 25 12:52:04 2023 USER: terminating instance due to error 472 Sat Feb 25 12:52:48 2023 USER: terminating instance due to error 472 Sat Feb 25 12:52:48 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_6252.trc: ORA-07445: exception encountered: core dump [ACCESS_VIOLATION][_ksuitm+631][PC:0x410C07][ADDR:0x1][UNABLE_TO_READ][] Sat Feb 25 12:55:35 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_7656.trc: ORA-07445: ??????: ???? [ACCESS_VIOLATION][_kews_idle_wait+378][PC:0x604AE6][ADDR:0xE530C470][UNABLE_TO_WRITE][]
通过恢复一些恢复之后,数据库open之后又挂掉
Sat Feb 25 15:05:49 2023 SMON: enabling tx recovery Sat Feb 25 15:05:49 2023 Database Characterset is ZHS16GBK Sat Feb 25 15:05:50 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_5308.trc: ORA-00600: 内部错误代码, 参数: [4194], [34], [31], [], [], [], [], [] Sat Feb 25 15:05:50 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_7568.trc: ORA-00600: 内部错误代码, 参数: [kcbgtcr_13], [], [], [], [], [], [], [] Sat Feb 25 15:05:51 2023 Non-fatal internal error happenned while SMON was doing logging scn->time mapping. SMON encountered 1 out of maximum 100 non-fatal internal errors. Sat Feb 25 15:05:51 2023 Doing block recovery for file 2 block 2951 Block recovery from logseq 10441, block 78 to scn 109906860017 Sat Feb 25 15:05:51 2023 Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0 Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG Block recovery stopped at EOT rba 10441.81.16 Block recovery completed at rba 10441.81.16, scn 25.2532677517 Doing block recovery for file 2 block 113 Block recovery from logseq 10441, block 78 to scn 109906859718 Sat Feb 25 15:05:52 2023 Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0 Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG Block recovery completed at rba 10441.80.16, scn 25.2532677516 replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=30, OS id=6904 Sat Feb 25 15:05:53 2023 db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup. Sat Feb 25 15:05:53 2023 Completed: alter database open Sat Feb 25 15:05:53 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j000_5400.trc: ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], [] Sat Feb 25 15:05:54 2023 DEBUG: Replaying xcb 0xac458698, pmd 0x8d7e9b9c for failed op 8 Doing block recovery for file 2 block 1515 No block recovery was needed Sat Feb 25 15:05:55 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j000_5400.trc: ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], [] ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], [] Sat Feb 25 15:05:56 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j000_5400.trc: ORA-00603: ORACLE server session terminated by fatal error ORA-00600: internal error code, arguments: [4194], [6], [4], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [6], [4], [], [], [], [], [] Sat Feb 25 15:05:57 2023 Doing block recovery for file 2 block 2951 Block recovery from logseq 10441, block 78 to scn 109906860017 Sat Feb 25 15:05:57 2023 Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0 Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG Block recovery completed at rba 10441.81.16, scn 25.2532677620 Doing block recovery for file 2 block 113 Block recovery from logseq 10441, block 78 to scn 109906860083 Sat Feb 25 15:05:57 2023 Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0 Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG Block recovery completed at rba 10441.138.16, scn 25.2532677684 Sat Feb 25 15:05:57 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j003_1716.trc: ORA-12012: 自动执行作业 42568 出错 ORA-00607: 当更改数据块时出现内部错误 ORA-00607: 当更改数据块时出现内部错误 Sat Feb 25 15:05:59 2023 Flush retried for xcb 0xac4c5a80, pmd 0x8c0cec74 Doing block recovery for file 2 block 2951 Block recovery from logseq 10441, block 78 to scn 109906860017 Sat Feb 25 15:05:59 2023 Recovery of Online Redo Log: Thread 1 Group 3 Seq 10441 Reading mem 0 Mem# 0 errs 0: G:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO03.LOG Sat Feb 25 15:05:59 2023 DEBUG: Replaying xcb 0xac458698, pmd 0x8d7e9b9c for failed op 8 Doing block recovery for file 2 block 1515 No block recovery was needed Sat Feb 25 15:05:59 2023 Block recovery completed at rba 10441.81.16, scn 25.2532677620 Sat Feb 25 15:06:00 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j002_6400.trc: ORA-00600: internal error code, arguments: [4194], [6], [4], [], [], [], [], [] Sat Feb 25 15:06:02 2023 DEBUG: Replaying xcb 0xac458698, pmd 0x8d7e9b9c for failed op 8 Doing block recovery for file 2 block 1515 No block recovery was needed Sat Feb 25 15:06:02 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_pmon_1076.trc: ORA-00600: 内部错误代码, 参数: [4194], [6], [4], [], [], [], [], [] Sat Feb 25 15:06:03 2023 PMON: terminating instance due to error 472 Sat Feb 25 15:06:03 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j007_7188.trc: ORA-00472: PMON 进程因错误而终止 Sat Feb 25 15:06:03 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j006_7624.trc: ORA-00472: PMON 进程因错误而终止 Sat Feb 25 15:06:03 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_j005_5688.trc: ORA-00472: PMON process terminated with error Sat Feb 25 15:06:10 2023 Errors in file g:\oracle\product\10.2.0\admin\orcl\bdump\orcl_smon_7568.trc: ORA-00472: PMON 进程因错误而终止 Instance terminated by PMON, pid = 1076
这个ORA-600 4194错误主要是由于undo异常,从而引起pmon异常,报ORA-00472错误.对undo进行处理,数据库稳定open,逻辑导出数据,完成恢复工作,完美帮忙客户恢复数据.
发表在 Oracle备份恢复
标签为 ksuapc2, ORA-00081, ORA-00472, ORA-600 2037, ORA-600 4194, _kews_idle_wait, _kmcgms, _kslwlmod, _ksuitm
评论关闭
110T oracle故障恢复
有客户一套110T的数据库由于存储控制器故障导致数据库无法正常启动,启动报错如下:
Wed Feb 2 23:16:13 2022 Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0 Mem# 0: /dev/vgredo6/rredo7b Mem# 1: /dev/vgredo4/rredo7a Wed Feb 2 23:16:14 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:15 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:16 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:16 2022 Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc: ORA-00600: internal error code, arguments: [kghstack_free2], [], [], [], [], [], [], [] ORA-00607: Internal error occurred while making a change to a data block ORA-00602: internal programming exception ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] [] ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:26 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc: ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:27 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc: ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] [] ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:16:27 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc: ORA-00602: internal programming exception ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] [] ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] [] Wed Feb 2 23:17:11 2022 PSP0: terminating instance due to error 472 Instance terminated by PSP0, pid = 26724
该错误原因是由于redo信息和数据文件block信息不匹配导致无法正常应用日志,从而出现异常,在后续的recover 中还出现以下错误
Fri Feb 18 16:09:59 2022 ALTER DATABASE RECOVER datafile 609,610,611,612,613,614,615,602,603,604,605,606,607,608 Fri Feb 18 16:09:59 2022 Media Recovery Start parallel recovery started with 16 processes Fri Feb 18 16:10:00 2022 Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0 Mem# 0: /dev/vgredo6/rredo7b Mem# 1: /dev/vgredo4/rredo7a Fri Feb 18 16:12:17 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc: ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], [] Fri Feb 18 16:18:51 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc: ORA-10562: Error occurred while applying redo to data block (file# 602, block# 1693691) ORA-10564: tablespace DBS_DCDL_PT ORA-01110: data file 602: '/dev/vgora12/rdbs_dcdl_pt0155' ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 33682645 ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], [] Fri Feb 18 16:18:55 2022 Media Recovery failed with error 12801
Fri Feb 18 18:23:59 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw1_22483.trc: ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] [] ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error Fri Feb 18 18:24:04 2022 DBW3: terminating instance due to error 472 Fri Feb 18 18:24:04 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc: ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error Fri Feb 18 18:24:04 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc: ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV] [] [] ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error Fri Feb 18 18:24:09 2022 LGWR: terminating instance due to error 472 Fri Feb 18 18:24:09 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_lgwr_22489.trc: ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] [] ORA-00472: PMON process terminated with error
从sqlplus中看到类似一些报错
SQL> recover datafile 601; ORA-03113: end-of-file on communication channel SQL> recover datafile 1066; ORA-00283: recovery session canceled due to errors ORA-12801: error signaled in parallel query server P015 ORA-00600: internal error code, arguments: [2037], [207064103], [207064103], [162], [6], [1], [1833009883], [1130705717] SQL> recover datafile 1065; ORA-00283: recovery session canceled due to errors ORA-12801: error signaled in parallel query server P004 ORA-00600: internal error code, arguments: [kcbzpb_1], [142189139], [3], [0], [], [], [], [] SQL> recover datafile 2042; ORA-00283: recovery session canceled due to errors ORA-12801: error signaled in parallel query server P014 ORA-00600: internal error code, arguments: [3020], [627], [3234156], [2633062764], [], [], [], [] ORA-10567: Redo is inconsistent with data block
通过屏蔽一致性,强制open库成功
Sun Feb 20 21:20:06 2022 SMON: enabling tx recovery Sun Feb 20 21:20:06 2022 Database Characterset is ZHS16GBK Sun Feb 20 21:20:07 2022 ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (74, 17) on object 34131051. Sun Feb 20 21:20:07 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-00376: file 1416 cannot be read at this time ORA-01110: data file 1416: '/dev/vgora14/rdbs_icdl_pt116' Sun Feb 20 21:20:08 2022 Stopping background process MMNL Sun Feb 20 21:20:09 2022 ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (88, 36) on object 33514955. Sun Feb 20 21:20:09 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-00376: file 1264 cannot be read at this time ORA-01110: data file 1264: '/dev/vgora14/rdbs_icdl_pt102' Sun Feb 20 21:20:09 2022 Stopping background process MMON Starting background process MMON Starting background process MMNL MMON started with pid=46, OS id=1482 Sun Feb 20 21:20:10 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767844) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Sun Feb 20 21:20:10 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' replication_dependency_tracking turned off (no async multimaster replication found) Sun Feb 20 21:20:11 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Sun Feb 20 21:20:11 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Sun Feb 20 21:20:11 2022 LOGSTDBY: Validating controlfile with logical metadata Sun Feb 20 21:20:11 2022 LOGSTDBY: Validation complete Sun Feb 20 21:20:11 2022 Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc: ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661) ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205' Completed: alter database open
对于异常undo进行处理,数据库正常open
由于客户短期无法迁移数据,先对于一些坏块进行修复,暂时运行数据库后续有时间窗口进行迁移.
ora-600 2037 ORA-7445 kcbs_dump_adv_state解决
有客户系统断电,导致数据库无法启动,让我们帮忙解决,通过分析主要是ORA-600 2037和ORA-7445 _kcbs_dump_adv_state等错误,通过人工recover解决.
数据库报ORA-03113,无法启动成功
C:\Documents and Settings\Administrator>sqlplus / as sysdba SQL*Plus: Release 10.2.0.1.0 - Production on 星期五 5月 12 09:50:36 2017 Copyright (c) 1982, 2005, Oracle. All rights reserved. 已连接到空闲例程。 SQL> startup ORACLE 例程已经启动。 Total System Global Area 1258291200 bytes Fixed Size 1250548 bytes Variable Size 218106636 bytes Database Buffers 1031798784 bytes Redo Buffers 7135232 bytes 数据库装载完毕。 ORA-03113: 通信通道的文件结束
分析alert日志
Fri May 12 09:50:43 2017 ALTER DATABASE OPEN Fri May 12 09:50:43 2017 Beginning crash recovery of 1 threads parallel recovery started with 15 processes Fri May 12 09:50:43 2017 Started redo scan Fri May 12 09:50:43 2017 Completed redo scan 1240 redo blocks read, 277 data blocks need recovery Fri May 12 09:50:44 2017 Started redo application at Thread 1: logseq 5881, block 41179 Fri May 12 09:50:44 2017 Recovery of Online Redo Log: Thread 1 Group 1 Seq 5881 Reading mem 0 Mem# 0 errs 0: E:\ORACLE\PRODUCT\10.2.0\ORADATA\xff\REDO01.LOG Fri May 12 09:50:44 2017 Completed redo application Fri May 12 09:50:44 2017 Errors in file e:\oracle\product\10.2.0\admin\xff\bdump\xff_p006_6072.trc: ORA-00600: internal error code, arguments: [6110], [193], [3], [], [], [], [], [] Fri May 12 09:50:44 2017 Hex dump of (file 3, block 14004) in trace file e:\oracle\product\10.2.0\admin\xff\bdump\xff_p000_6024.trc Corrupt block relative dba: 0x00c036b4 (file 3, block 14004) Bad header found during crash/instance recovery Data in bad block: type: 255 format: 7 rdba: 0x06010601 last change scn: 0xa206.a2060601 seq: 0xb4 flg: 0x36 spare1: 0x1 spare2: 0x6 spare3: 0x673 consistency value in tail: 0x1b0a0708 check value in block header: 0x36b4 computed block checksum: 0xe4f5 Fri May 12 09:50:44 2017 Hex dump of (file 9, block 65507) in trace file e:\oracle\product\10.2.0\admin\xff\bdump\xff_p003_6056.trc Corrupt block relative dba: 0x0240ffe3 (file 9, block 65507) Bad header found during crash/instance recovery Data in bad block: type: 3 format: 6 rdba: 0x06020601 last change scn: 0xa206.a2060602 seq: 0xe3 flg: 0xff spare1: 0x1 spare2: 0x6 spare3: 0x6dc consistency value in tail: 0xc1028001 check value in block header: 0xffe3 computed block checksum: 0xff01 Fri May 12 09:50:44 2017 Reread of rdba: 0x00c036b4 (file 3, block 14004) found different data Fri May 12 09:50:44 2017 Reread of rdba: 0x0240ffe3 (file 9, block 65507) found different data Fri May 12 09:50:44 2017 Errors in file e:\oracle\product\10.2.0\admin\xff\bdump\xff_p005_6060.trc: ORA-00600: internal error code,arguments:[2037],[17442602],[2718302723],[255],[9],[203],[657105414],[2147549568] Fri May 12 09:50:44 2017 Errors in file e:\oracle\product\10.2.0\admin\xff\bdump\xff_p000_6024.trc: ORA-07445:exception encountered:core dump[ACCESS_VIOLATION][_kclcomplete+79][PC:0x72B0C7][ADDR:0x220][UNABLE_TO_READ][] Fri May 12 09:50:44 2017 Errors in file e:\oracle\product\10.2.0\admin\xff\bdump\xff_p006_6072.trc: ORA-07445: exception encountered:core dump[ACCESS_VIOLATION][_kcbzdh+2496][PC:0x4A4928][ADDR:0xB][UNABLE_TO_READ][] ORA-00600: internal error code, arguments: [6110], [193], [3], [], [], [], [], [] Errors in file e:\oracle\product\10.2.0\admin\xff\bdump\xff_p012_6128.trc: ORA-07445: exception encountered: core dump [ACCESS_VIOLATION] [_kcbs_dump_adv_state+723] [PC:0x5975A3] [ADDR:0xCBC0CBB2] [UNABLE_TO_READ] [] ORA-00600:internal error code,arguments:[2037],[17430318],[2718303745],[128],[1],[203],[4147028486],[2147549568]
错误比较明显由于坏块导致应用日志恢复异常,主要错误集中在ORA-600 2037,ORA-7445 _kcbs_dump_adv_state,ORA-7445_kcbzdh,ORA-7445 _kclcomplete等
dbv检查数据文件
E:\>dbv file=E:\ORACLE\PRODUCT\10.2.0\ORADATA\xff\SYSAUX01.DBF DBVERIFY: Release 10.2.0.1.0 - Production on 星期五 5月 12 09:57:39 2017 Copyright (c) 1982, 2005, Oracle. All rights reserved. DBVERIFY - 开始验证: FILE = E:\ORACLE\PRODUCT\10.2.0\ORADATA\xff\SYSAUX01.DBF 页 13353 标记为损坏 Corrupt block relative dba: 0x00c03429 (file 3, block 13353) Bad header found during dbv: Data in bad block: type: 1 format: 6 rdba: 0x3429a206 last change scn: 0x066f.066f3429 seq: 0x0 flg: 0x00 spare1: 0x6 spare2: 0xa2 spare3: 0x8c96 consistency value in tail: 0x06018001 check value in block header: 0x0 block checksum disabled 页 14004 标记为损坏 Corrupt block relative dba: 0x00c036b4 (file 3, block 14004) Bad header found during dbv: Data in bad block: type: 1 format: 6 rdba: 0x36b4a206 last change scn: 0x0673.067336b4 seq: 0x0 flg: 0x00 spare1: 0x6 spare2: 0xa2 spare3: 0xfb97 consistency value in tail: 0x06010210 check value in block header: 0x0 block checksum disabled 页 15261 标记为损坏 Corrupt block relative dba: 0x00c03b9d (file 3, block 15261) Bad header found during dbv: Data in bad block: type: 2 format: 6 rdba: 0x3b9da206 last change scn: 0x0673.06733b9d seq: 0x0 flg: 0x00 spare1: 0x6 spare2: 0xa2 spare3: 0x0 consistency value in tail: 0x06018001 check value in block header: 0x5549 block checksum disabled DBVERIFY - 验证完成 检查的页总数: 58880 处理的页总数 (数据): 19318 失败的页总数 (数据): 0 处理的页总数 (索引): 18610 失败的页总数 (索引): 0 处理的页总数 (其它): 13747 处理的总页数 (段) : 0 失败的总页数 (段) : 0 空的页总数: 7202 标记为损坏的总页数: 3 流入的页总数: 0 最高块 SCN : 178325323 (0.178325323) E:\>dbv file=E:\ORACLE\PRODUCT\10.2.0\ORADATA\xff\xff_BSE02 DBVERIFY: Release 10.2.0.1.0 - Production on 星期五 5月 12 10:10:24 2017 Copyright (c) 1982, 2005, Oracle. All rights reserved. DBVERIFY - 开始验证: FILE = E:\ORACLE\PRODUCT\10.2.0\ORADATA\xff\xff_BSE02 页 65507 标记为损坏 Corrupt block relative dba: 0x0240ffe3 (file 9, block 65507) Bad header found during dbv: Data in bad block: type: 2 format: 6 rdba: 0xffe3a206 last change scn: 0x06dc.06dcffe3 seq: 0x0 flg: 0x00 spare1: 0x6 spare2: 0xa2 spare3: 0xb32 consistency value in tail: 0x060102ff check value in block header: 0x0 block checksum disabled DBVERIFY - 验证完成 检查的页总数: 1310720 处理的页总数 (数据): 34102 失败的页总数 (数据): 0 处理的页总数 (索引): 30270 失败的页总数 (索引): 0 处理的页总数 (其它): 10850 处理的总页数 (段) : 0 失败的总页数 (段) : 0 空的页总数: 1235497 标记为损坏的总页数: 1 流入的页总数: 0 最高块 SCN : 178325221 (0.178325221)
确实如alert日志报错,file 3和9 都出现坏块导致实例恢复无法进行。根据错误ORA-600 2037和ORA-7445 _kcbs_dump_adv_state,初步判断和During Startup (Open Database) Alert Log Shows ORA-600[2037] and ORA-7445[kcbs_dump_adv_state] (Doc ID 551993.1)文章描述相符(而且版本也相符)
尝试recover datafile部分file
E:\>sqlplus / as sysdba SQL*Plus: Release 10.2.0.1.0 - Production on 星期五 5月 12 10:16:00 2017 Copyright (c) 1982, 2005, Oracle. All rights reserved. 连接到: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production With the Partitioning, OLAP and Data Mining options SQL> recover datafile 1; 完成介质恢复。 SQL> recover datafile 2; 完成介质恢复。 SQL> recover datafile 3; 完成介质恢复。 SQL> recover datafile 4; 完成介质恢复。 SQL> recover datafile 9; 完成介质恢复。 SQL> alter database open; alter database open * 第 1 行出现错误: ORA-00600: 内部错误代码, 参数: [kcratr1_lastbwr], [], [], [], [], [], [], []
ORA-00600 kcratr1_lastbwr错误比较明显,见ORA-00600:[Kcratr1_lastbwr] During Database Startup after a Crash (Doc ID 393984.1)
通过recover database处理
SQL> recover database; 完成介质恢复。 SQL> alter database open; 数据库已更改。
然后通过查询dba_extents 处理坏块对象
补充ORA-600 2037错误
Format: ORA-600 [2037] [a] [b] 1 [d] [e] [f] [g] VERSIONS: versions 8.0 and above DESCRIPTION: During recovery we are examining a block to ensure that it is not corrupt prior to applying any change vectors. The block has failed this check and this exception is raised. ARGUMENTS: Arg [a] Relative Data Block Address (RDBA) that the redo vector is for Arg [b] The Block format Arg {c} RDBA in the block itself Arg [d] The block type Arg [e] The sequence number Arg [f] Flags, if set Arg [g] The return value from the block head/tail checker.