标签云
asm 恢复 asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (100)
- 数据库 (1,589)
- DB2 (22)
- MySQL (70)
- Oracle (1,459)
- Data Guard (49)
- EXADATA (7)
- GoldenGate (21)
- ORA-xxxxx (158)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (13)
- ORACLE 21C (3)
- Oracle ASM (65)
- Oracle Bug (7)
- Oracle RAC (47)
- Oracle 安全 (6)
- Oracle 开发 (27)
- Oracle 监听 (27)
- Oracle备份恢复 (526)
- Oracle安装升级 (83)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (75)
- PostgreSQL (13)
- PostgreSQL恢复 (3)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (36)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (19)
-
最近发表
- ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
- 最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
- ORA-600 2662快速恢复之Patch scn工具
- TNS-12518: TNS:listener could not hand off client connection
- ora.storage无法启动报ORA-12514故障处理
- 断电引起文件scn异常数据库恢复
- ORA-16188: LOG_ARCHIVE_CONFIG settings inconsistent with previously started instance
- .[hudsonL@cock.li].mkp勒索加密数据库完美恢复
- 模拟带库实现rman远程备份
- 又一例:ORA-600 kclchkblk_4和2662故障
- Oracle误删除数据文件恢复
- Oracle 19C 备库DML重定向—DML Redirection
- ORA-01595/ORA-600 4194处理
- 从ORA-00283 ORA-16433报错开始恢复
- 近期又遇到ORA-600 16703和ORA-702故障
- RECOVER_YOUR_DATA勒索恢复
- ORA-01033: ORACLE initialization or shutdown in progress 故障处理
- Oracle 19c/21c最新patch信息-202401
- 存储故障,强制拉库报ORA-600 kcbzib_kcrsds_1处理
- ORA-600 kcrf_resilver_log_1故障处理
月归档:十二月 2018
正常open库报ORA-600 2662,ORA-600 4097错误
又一起数据库正常open之后继续报ORA-600 2662
Sat Dec 29 04:30:40 2018 QMNC started with pid=36, OS id=12985 LOGSTDBY: Validating controlfile with logical metadata LOGSTDBY: Validation complete Completed: alter database open Sat Dec 29 04:30:42 2018 Starting background process CJQ0 Sat Dec 29 04:30:42 2018 CJQ0 started with pid=41, OS id=12999 Sat Dec 29 04:30:43 2018 Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m000_12989.trc (incident=365101): ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], [] Sat Dec 29 04:30:43 2018 Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m002_12993.trc (incident=365117): ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], [] Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365101/orcl_m000_12989_i365101.trc Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365117/orcl_m002_12993_i365117.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m000_12989.trc (incident=365102): ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], [] Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365102/orcl_m000_12989_i365102.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_m002_12993.trc (incident=365118): ORA-00600: internal error code, arguments: [2662], [0], [193646286], [0], [193662077], [50331650], [], [], [], [], [], [] Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_365118/orcl_m002_12993_i365118.trc
在open状态处理掉ORA-00600 2662错误之后,又出现了ORA-600 4097 以及ORA-00353 ORA-00312错误,这个也是一种不常见的现象,一般都是数据库open之前出现类似错误,但是这里在open之后出现(以前类似处理参考:ORA-600 4097故障处理和ORA-00322 ORA-00312 ,ORA-00354 ORA-00353 ORA-00312异常处理)
Sat Dec 29 05:55:22 2018 Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j000_19136.trc (incident=432567): ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], [] Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432567/orcl_j000_19136_i432567.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j000_19136.trc (incident=432568): ORA-00353: log corruption near block 15682 change 193647101 time 12/29/2018 04:40:31 ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log' ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log' ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], [] Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432568/orcl_j000_19136_i432568.trc Errors in file /home/app/diag/rdbms/orcl/orcl/incident/incdir_432567/orcl_j000_19136_i432567.trc: ORA-00399: corrupt change description in redo log ORA-00353: log corruption near block 15682 change 193647101 time 12/29/2018 04:40:31 ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log' ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log' ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], [] Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j000_19136.trc (incident=432569): ORA-00353: log corruption near block 15682 change 193647101 time 12/29/2018 04:40:31 ORA-00334: archived log: '/home/app/oradata/orcl/redo03.log' ORA-00312: online log 3 thread 1: '/home/app/oradata/orcl/redo03.log' ORA-00600: internal error code, arguments: [4097], [3], [15], [76480], [], [], [], [], [], [], [], [] Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432569/orcl_j000_19136_i432569.trc
处理完成上面报错之后,继续出现比较少见的ORA-600 ktecgsc:kcbz_objdchk错误,具体参考:ORA-00600 [ktecgsc:kcbz_objdchk] on 11.2 (Doc ID 1562473.1)
Sat Dec 29 06:00:17 2018 DBMS_STATS: GATHER_STATS_JOB encountered errors. Check the trace file. Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j001_19436.trc: ORA-20011: Approximate NDV failed: ORA-08103: object no longer exists Sat Dec 29 06:00:21 2018 Errors in file /home/app/diag/rdbms/orcl/orcl/trace/orcl_j002_19479.trc (incident=432631): ORA-00600: internal error code, arguments: [ktecgsc:kcbz_objdchk], [0], [0], [1], [], [], [], [], [], [], [], [] Incident details in: /home/app/diag/rdbms/orcl/orcl/incident/incdir_432631/orcl_j002_19479_i432631.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details.
Oracle Exadata坏盘导致磁盘组无法mount恢复
接到朋友求救有客户oracle exadata一体机 的 asm磁盘组无法mount,希望我们提供恢复支持服务
经过分析和了解,大致问题是:磁盘空间已经超容量使用(部分数据不能完成ASM镜像),最近又损坏一块盘,导致asm 磁盘组无法mount。我们分析后,通过重构exadata celldisk数据,将asm 磁盘组 mount成功后,实现五套数据库全部open成功(由于底层磁盘部分数据损坏,导致部分数据访问报错,需要在oracle层面进行处理)。
本次问题的具体分析和处理如下:
存放数据库文件的磁盘组不能mount
Wed Dec 12 21:29:04 2018 SQL> alter diskgroup DATA_XFF mount force NOTE: cache registered group DATA_XFF number=1 incarn=0x5fe882cb NOTE: cache began mount (first) of group DATA_XFF number=1 incarn=0x5fe882cb NOTE: Assigning number (1,36) to disk (o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03) NOTE: Assigning number (1,34) to disk (o/192.168.10.5/DATA_XFF_CD_10_XFFCEL03) NOTE: Assigning number (1,37) to disk (o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03) NOTE: Assigning number (1,38) to disk (o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03) NOTE: Assigning number (1,39) to disk (o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03) NOTE: Assigning number (1,40) to disk (o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03) NOTE: Assigning number (1,41) to disk (o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03) NOTE: Assigning number (1,42) to disk (o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03) NOTE: Assigning number (1,43) to disk (o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03) NOTE: Assigning number (1,44) to disk (o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03) NOTE: Assigning number (1,45) to disk (o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03) NOTE: Assigning number (1,46) to disk (o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03) NOTE: Assigning number (1,22) to disk (o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02) NOTE: Assigning number (1,18) to disk (o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02) NOTE: Assigning number (1,19) to disk (o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02) NOTE: Assigning number (1,15) to disk (o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02) NOTE: Assigning number (1,20) to disk (o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02) NOTE: Assigning number (1,17) to disk (o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02) NOTE: Assigning number (1,16) to disk (o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02) NOTE: Assigning number (1,23) to disk (o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02) NOTE: Assigning number (1,12) to disk (o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02) NOTE: Assigning number (1,21) to disk (o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02) NOTE: Assigning number (1,13) to disk (o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02) NOTE: Assigning number (1,14) to disk (o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02) NOTE: Assigning number (1,1) to disk (o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01) NOTE: Assigning number (1,2) to disk (o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01) NOTE: Assigning number (1,3) to disk (o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01) NOTE: Assigning number (1,4) to disk (o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01) NOTE: Assigning number (1,5) to disk (o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01) NOTE: Assigning number (1,6) to disk (o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01) NOTE: Assigning number (1,7) to disk (o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01) NOTE: Assigning number (1,8) to disk (o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01) NOTE: Assigning number (1,9) to disk (o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01) NOTE: Assigning number (1,10) to disk (o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01) NOTE: Assigning number (1,11) to disk (o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01) Wed Dec 12 21:29:10 2018 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 101 for pid 27, osid 62541 NOTE: Assigning number (1,0) to disk () GMON querying group 1 at 102 for pid 27, osid 62541 NOTE: process _user62541_+asm2 (62541) initiating offline of disk 0.3915937355 () with mask 0x7e[0x7f] in group 1 NOTE: initiating PST update: grp = 1, dsk = 0/0xe968764b, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 103 for pid 27, osid 62541 ERROR: Disk 0 cannot be offlined, since all the disks [0, 25] with mirrored data would be offline. ERROR: too many offline disks in PST (grp 1) WARNING: Offline of disk 0 () in group 1 and mode 0x7f failed on ASM inst 2 NOTE: cache dismounting (not clean) group 1/0x5FE882CB (DATA_XFF) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0x5FE882CB (DATA_XFF) NOTE: cache ending mount (fail) of group DATA_XFF number=1 incarn=0x5fe882cb NOTE: cache deleting context for group DATA_XFF 1/0x5fe882cb GMON dismounting group 1 at 104 for pid 27, osid 62541 ERROR: diskgroup DATA_XFF was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15066: offlining disk "0" in group "DATA_XFF" may result in a data loss ORA-15042: ASM disk "0" is missing from group number "1" ERROR: alter diskgroup DATA_XFF mount force
检查底层损坏情况
CellCLI> list physicaldisk 20:0 KN3VZL normal 20:1 KNAWLL normal 20:2 KN4E4L warning - predictive failure, poor performance 20:3 KNAN5L normal 20:4 KMJKYL normal 20:5 KN5DGL normal 20:6 KMDLWL normal 20:7 KMDKPL normal 20:8 KMDA7L normal 20:9 KN1YJL normal 20:10 KMH1YL normal 20:11 KMVHAL normal CellCLI> list griddisk DATA_XFF_CD_00_XFFCEL01 active DATA_XFF_CD_01_XFFCEL01 active DATA_XFF_CD_02_XFFCEL01 proactive failure DATA_XFF_CD_03_XFFCEL01 active DATA_XFF_CD_04_XFFCEL01 active DATA_XFF_CD_05_XFFCEL01 active DATA_XFF_CD_06_XFFCEL01 active DATA_XFF_CD_07_XFFCEL01 active DATA_XFF_CD_08_XFFCEL01 active DATA_XFF_CD_09_XFFCEL01 active DATA_XFF_CD_10_XFFCEL01 active DATA_XFF_CD_11_XFFCEL01 active
在db节点无法发现异常磁盘的asm disk
[grid@ycdwdb01 grid]$ kfod disk=all -------------------------------------------------------------------------------- Disk Size Path User Group ============================================================ 1: 433152 Mb o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 <unknown> <unknown> 2: 433152 Mb o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 <unknown> <unknown> 3: 433152 Mb o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 <unknown> <unknown> 4: 433152 Mb o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 <unknown> <unknown> 5: 433152 Mb o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 <unknown> <unknown> 6: 433152 Mb o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 <unknown> <unknown> 7: 433152 Mb o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 <unknown> <unknown> 8: 433152 Mb o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 <unknown> <unknown> 9: 433152 Mb o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 <unknown> <unknown> 10: 433152 Mb o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 <unknown> <unknown> 11: 433152 Mb o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 <unknown> <unknown>
根据客户的反馈该磁盘组几乎全部被使用,asmcmd lsdg看到Usable_file_MB已经出现负值.证明该磁盘组本身的normal没有完全存储两份数据,在这样的情况下,继续坏盘会导致部分数据只有一份,因此也就出现了这里的磁盘组无法正常mount成功.
通过底层修复celldisk之后
CellCLI> list griddisk DATA_XFF_CD_00_XFFCEL01 active DATA_XFF_CD_01_XFFCEL01 active DATA_XFF_CD_02_XFFCEL01 active DATA_XFF_CD_03_XFFCEL01 active DATA_XFF_CD_04_XFFCEL01 active DATA_XFF_CD_05_XFFCEL01 active DATA_XFF_CD_06_XFFCEL01 active DATA_XFF_CD_07_XFFCEL01 active DATA_XFF_CD_08_XFFCEL01 active DATA_XFF_CD_09_XFFCEL01 active DATA_XFF_CD_10_XFFCEL01 active DATA_XFF_CD_11_XFFCEL01 active [grid@ycdwdb01 grid]$ kfod disk=all -------------------------------------------------------------------------------- Disk Size Path User Group ============================================================ 1: 433152 Mb o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 <unknown> <unknown> 2: 433152 Mb o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 <unknown> <unknown> 3: 433152 Mb o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01 <unknown> <unknown> 4: 433152 Mb o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 <unknown> <unknown> 5: 433152 Mb o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 <unknown> <unknown> 6: 433152 Mb o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 <unknown> <unknown> 7: 433152 Mb o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 <unknown> <unknown> 8: 433152 Mb o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 <unknown> <unknown> 9: 433152 Mb o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 <unknown> <unknown> 10: 433152 Mb o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 <unknown> <unknown> 11: 433152 Mb o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 <unknown> <unknown> 12: 433152 Mb o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 <unknown> <unknown>
data磁盘组直接mount成功
Fri Dec 14 14:04:59 2018 SQL> alter diskgroup DATA_XFF mount NOTE: cache registered group DATA_XFF number=1 incarn=0x78a886e7 NOTE: cache began mount (not first) of group DATA_XFF number=1 incarn=0x78a886e7 NOTE: Assigning number (1,36) to disk (o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03) NOTE: Assigning number (1,34) to disk (o/192.168.10.5/DATA_XFF_CD_10_XFFCEL03) NOTE: Assigning number (1,37) to disk (o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03) NOTE: Assigning number (1,38) to disk (o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03) NOTE: Assigning number (1,39) to disk (o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03) NOTE: Assigning number (1,40) to disk (o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03) NOTE: Assigning number (1,41) to disk (o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03) NOTE: Assigning number (1,42) to disk (o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03) NOTE: Assigning number (1,43) to disk (o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03) NOTE: Assigning number (1,44) to disk (o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03) NOTE: Assigning number (1,45) to disk (o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03) NOTE: Assigning number (1,46) to disk (o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03) NOTE: Assigning number (1,22) to disk (o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02) NOTE: Assigning number (1,18) to disk (o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02) NOTE: Assigning number (1,19) to disk (o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02) NOTE: Assigning number (1,15) to disk (o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02) NOTE: Assigning number (1,20) to disk (o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02) NOTE: Assigning number (1,17) to disk (o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02) NOTE: Assigning number (1,16) to disk (o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02) NOTE: Assigning number (1,23) to disk (o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02) NOTE: Assigning number (1,12) to disk (o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02) NOTE: Assigning number (1,21) to disk (o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02) NOTE: Assigning number (1,13) to disk (o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02) NOTE: Assigning number (1,14) to disk (o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02) NOTE: Assigning number (1,1) to disk (o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01) NOTE: Assigning number (1,2) to disk (o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01) NOTE: Assigning number (1,3) to disk (o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01) NOTE: Assigning number (1,4) to disk (o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01) NOTE: Assigning number (1,5) to disk (o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01) NOTE: Assigning number (1,6) to disk (o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01) NOTE: Assigning number (1,7) to disk (o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01) NOTE: Assigning number (1,8) to disk (o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01) NOTE: Assigning number (1,9) to disk (o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01) NOTE: Assigning number (1,10) to disk (o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01) NOTE: Assigning number (1,11) to disk (o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01) NOTE: Assigning number (1,0) to disk (o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01) Fri Dec 14 14:04:59 2018 GMON querying group 1 at 78 for pid 28, osid 76016 NOTE: Assigning number (1,24) to disk () NOTE: Assigning number (1,25) to disk () NOTE: Assigning number (1,26) to disk () NOTE: Assigning number (1,27) to disk () NOTE: Assigning number (1,28) to disk () NOTE: Assigning number (1,29) to disk () NOTE: Assigning number (1,30) to disk () NOTE: Assigning number (1,31) to disk () NOTE: Assigning number (1,32) to disk () NOTE: Assigning number (1,33) to disk () NOTE: Assigning number (1,35) to disk () GMON querying group 1 at 79 for pid 28, osid 76016 NOTE: cache opening disk 0 of grp 1: DATA_XFF_CD_02_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01 NOTE: cache opening disk 1 of grp 1: DATA_XFF_CD_05_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 NOTE: cache opening disk 2 of grp 1: DATA_XFF_CD_03_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 NOTE: F1X0 found on disk 2 au 5 fcn 0.15948262 NOTE: cache opening disk 3 of grp 1: DATA_XFF_CD_06_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 NOTE: cache opening disk 4 of grp 1: DATA_XFF_CD_09_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 NOTE: cache opening disk 5 of grp 1: DATA_XFF_CD_04_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 NOTE: cache opening disk 6 of grp 1: DATA_XFF_CD_07_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 NOTE: cache opening disk 7 of grp 1: DATA_XFF_CD_11_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 NOTE: cache opening disk 8 of grp 1: DATA_XFF_CD_01_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 NOTE: cache opening disk 9 of grp 1: DATA_XFF_CD_00_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 NOTE: cache opening disk 10 of grp 1: DATA_XFF_CD_10_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 NOTE: cache opening disk 11 of grp 1: DATA_XFF_CD_08_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 NOTE: cache opening disk 12 of grp 1: DATA_XFF_CD_00_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02 NOTE: cache opening disk 13 of grp 1: DATA_XFF_CD_01_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02 NOTE: cache opening disk 14 of grp 1: DATA_XFF_CD_02_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02 NOTE: cache opening disk 15 of grp 1: DATA_XFF_CD_03_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02 NOTE: cache opening disk 16 of grp 1: DATA_XFF_CD_04_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02 NOTE: cache opening disk 17 of grp 1: DATA_XFF_CD_05_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02 NOTE: cache opening disk 18 of grp 1: DATA_XFF_CD_06_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02 NOTE: cache opening disk 19 of grp 1: DATA_XFF_CD_07_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02 NOTE: cache opening disk 20 of grp 1: DATA_XFF_CD_08_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02 NOTE: cache opening disk 21 of grp 1: DATA_XFF_CD_09_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02 NOTE: F1X0 found on disk 21 au 2 fcn 0.15948262 NOTE: cache opening disk 22 of grp 1: DATA_XFF_CD_10_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02 NOTE: cache opening disk 23 of grp 1: DATA_XFF_CD_11_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02 NOTE: cache opening disk 36 of grp 1: DATA_XFF_CD_11_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03 NOTE: cache opening disk 37 of grp 1: DATA_XFF_CD_04_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03 NOTE: cache opening disk 38 of grp 1: DATA_XFF_CD_00_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03 NOTE: cache opening disk 39 of grp 1: DATA_XFF_CD_03_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03 NOTE: cache opening disk 40 of grp 1: DATA_XFF_CD_05_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03 NOTE: cache opening disk 41 of grp 1: DATA_XFF_CD_08_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03 NOTE: cache opening disk 42 of grp 1: DATA_XFF_CD_01_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03 NOTE: cache opening disk 43 of grp 1: DATA_XFF_CD_09_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03 NOTE: cache opening disk 44 of grp 1: DATA_XFF_CD_06_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03 NOTE: F1X0 found on disk 44 au 2 fcn 0.15948262 NOTE: cache opening disk 45 of grp 1: DATA_XFF_CD_07_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03 NOTE: cache opening disk 46 of grp 1: DATA_XFF_CD_02_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03 NOTE: cache mounting (not first) normal redundancy group 1/0x78A886E7 (DATA_XFF) Fri Dec 14 14:04:59 2018 kjbdomatt send to inst 2 Fri Dec 14 14:04:59 2018 NOTE: attached to recovery domain 1 NOTE: redo buffer size is 512 blocks (2101760 bytes) Fri Dec 14 14:04:59 2018 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 (DATA_XFF) NOTE: LGWR found thread 2 closed at ABA 98.4672 NOTE: LGWR mounted thread 2 for diskgroup 1 (DATA_XFF) NOTE: LGWR opening thread 2 at fcn 0.18931129 ABA 99.4673 NOTE: cache mounting group 1/0x78A886E7 (DATA_XFF) succeeded NOTE: cache ending mount (success) of group DATA_XFF number=1 incarn=0x78a886e7 GMON querying group 1 at 80 for pid 19, osid 9805 Fri Dec 14 14:04:59 2018 NOTE: Instance updated compatible.asm to 11.2.0.3.0 for grp 1 SUCCESS: diskgroup DATA_XFF was mounted SUCCESS: alter diskgroup DATA_XFF mount
恢复后的asm磁盘状态
ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED NORMAL Y 512 4096 4194304 15160320 4776184 5197824 -210820 12 N DATA_XFF/ MOUNTED NORMAL N 512 4096 4194304 864896 863400 298240 282580 0 Y DBFS_DG/ MOUNTED NORMAL N 512 4096 4194304 3787840 2157232 1298688 429272 0 N RECO_XFF/
后续数据库open成功,有部分坏块通过技术手段进行二次处理,至此数据库恢复完成,成功抢救了客户Oracle Exadata中的绝大部分数据.如果有类似xd故障恢复,无法自行解决,需要恢复支持请联系我们
Phone:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
发表在 非常规恢复
标签为 exadata mount, exadata坏盘恢复, exadata恢复, exadata磁盘组恢复, ORA-15040, ORA-15042, ORA-15066, xd坏盘恢复, xd恢复
评论关闭
linux 7安装11.2.0.4集群注意避开特定kernal版本
根据官方认证信息Orace RAC 11.2.0.4通过了Linux 7的认证
但是根据mos上描述对于linux 7的某些Kernel上可能会遭遇到OHASD启动失败,主要参考:ALERT: Grid Infrastructure Fails to Start OHASD With RedHat Linux or Oracle Linux with RedHat Compatible Kernel (RHCK) Version 3.10.0-514.21.2.EL7.X86_64 through 3.10.0-514.26.2.el7 (Doc ID 2282371.1)
具体报错类似:
---In Oracle restart: # crsctl start has CLSU-00100: operating system function: waitpid failed with error data: 0 CLSU-00101: operating system error message: Error 0 CLSU-00103: error location: usrgetgrp12 CLSU-00104: additional error information: child returned 232 CRS-4000: Command Start failed, or completed with errors. CLSRSC-199: Timed out waiting for OHASD to start ---In Grid infrastructure: #crsctl start crs CLSB:1076092480: Oracle Clusterware infrastructure error in CRSCTL (OS PID 22363): GIPC API termination failed with error code 910
关于该问题的详细说明,请见:OHASD Fails to Start With Kernel Version 3.10.0-514.21.2.el7.x86_64 (Doc ID 2281492.1),建议在选择linux 7 安装11.2.0.4的集群最好避开Kernel:3.10.0-514.21.2.el7到3.10.0-514.26.2.el7之间版本