标签云
asm mount asm恢复 asm 恢复 bbed bootstrap$ dul eking In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 sql加密恢复 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币 oracle 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (91)
- 数据库 (1,432)
- DB2 (22)
- MySQL (58)
- Oracle (1,315)
- Data Guard (41)
- EXADATA (7)
- GoldenGate (20)
- ORA-xxxxx (150)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (12)
- ORACLE 21C (3)
- Oracle ASM (60)
- Oracle Bug (7)
- Oracle RAC (42)
- Oracle 安全 (6)
- Oracle 开发 (26)
- Oracle 监听 (26)
- Oracle备份恢复 (446)
- Oracle安装升级 (71)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (63)
- PostgreSQL (13)
- PostgreSQL恢复 (3)
- SQL Server (26)
- SQL Server恢复 (7)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (26)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (9)
-
最近发表
- 云主机快照之后Oracle无法正常启动处理
- ORA-600 2032故障处理
- Oracle Recovery Tools实战批量坏块修复
- ORA-15063: ASM discovered an insufficient number of disks for diskgroup 恢复
- 修改oracle scn小工具(patch scn)
- ORA-01110 ORA-17070 OSD-04006 故障恢复
- fdisk分区导致asm disk破坏数据库恢复
- ORA-600 kcvent_internal_02故障处理
- PostgreSQL恢复系列:pg_filedump基本使用
- Oracle断电故障处理
- PostgreSQL恢复系列:pg_control异常恢复
- PostgreSQL恢复系列:wal日志丢失恢复
- pvcreate asm disk导致asm磁盘组异常恢复
- O/S-Error: (OS 23) 数据错误(循环冗余检查) 数据库恢复
- 数据库open报ORA-00959: tablespace ‘UNDOTBS1′ does not exist分析
- ASM删除表空间恢复
- ORA-12518 ORA-27302: 错误发生在: ssthrddcr
- plsql 插入blob
- 分享运气超级好的一次drop tablespace 数据恢复
- segment header异常对象删除处理
友情链接
标签归档:ORA-15335
ORA-15130: diskgroup “ORADATA” is being dismounted
磁盘组mount之后,立马又dismount
Sat Dec 25 17:48:45 2021 SQL> alter diskgroup ORADATA mount NOTE: cache registered group ORADATA number=5 incarn=0xd4b7ac6a NOTE: cache began mount (first) of group ORADATA number=5 incarn=0xd4b7ac6a NOTE: Assigning number (5,24) to disk (/dev/mapper/data31) NOTE: Assigning number (5,26) to disk (/dev/mapper/data33) NOTE: Assigning number (5,21) to disk (/dev/mapper/data29) NOTE: Assigning number (5,23) to disk (/dev/mapper/data30) NOTE: Assigning number (5,25) to disk (/dev/mapper/data32) NOTE: Assigning number (5,19) to disk (/dev/mapper/data27) NOTE: Assigning number (5,20) to disk (/dev/mapper/data28) NOTE: Assigning number (5,18) to disk (/dev/mapper/data26) NOTE: Assigning number (5,14) to disk (/dev/mapper/data22) NOTE: Assigning number (5,17) to disk (/dev/mapper/data25) NOTE: Assigning number (5,16) to disk (/dev/mapper/data24) NOTE: Assigning number (5,15) to disk (/dev/mapper/data23) NOTE: Assigning number (5,13) to disk (/dev/mapper/data21) NOTE: Assigning number (5,12) to disk (/dev/mapper/data20) NOTE: Assigning number (5,10) to disk (/dev/mapper/data19) NOTE: Assigning number (5,9) to disk (/dev/mapper/data18) NOTE: Assigning number (5,8) to disk (/dev/mapper/data17) NOTE: Assigning number (5,3) to disk (/dev/mapper/data12) NOTE: Assigning number (5,22) to disk (/dev/mapper/data3) NOTE: Assigning number (5,2) to disk (/dev/mapper/data11) NOTE: Assigning number (5,7) to disk (/dev/mapper/data16) NOTE: Assigning number (5,28) to disk (/dev/mapper/data5) NOTE: Assigning number (5,32) to disk (/dev/mapper/data9) NOTE: Assigning number (5,6) to disk (/dev/mapper/data15) NOTE: Assigning number (5,5) to disk (/dev/mapper/data14) NOTE: Assigning number (5,4) to disk (/dev/mapper/data13) NOTE: Assigning number (5,1) to disk (/dev/mapper/data10) NOTE: Assigning number (5,30) to disk (/dev/mapper/data7) NOTE: Assigning number (5,29) to disk (/dev/mapper/data6) NOTE: Assigning number (5,31) to disk (/dev/mapper/data8) NOTE: Assigning number (5,11) to disk (/dev/mapper/data2) NOTE: Assigning number (5,27) to disk (/dev/mapper/data4) NOTE: Assigning number (5,0) to disk (/dev/mapper/data1) Sat Dec 25 17:48:52 2021 NOTE: GMON heartbeating for grp 5 GMON querying group 5 at 153 for pid 32, osid 68608 NOTE: cache opening disk 0 of grp 5: ORADATA_0000 path:/dev/mapper/data1 NOTE: F1X0 found on disk 0 au 2 fcn 0.0 NOTE: cache opening disk 1 of grp 5: ORADATA_0001 path:/dev/mapper/data10 NOTE: cache opening disk 2 of grp 5: ORADATA_0002 path:/dev/mapper/data11 NOTE: cache opening disk 3 of grp 5: ORADATA_0003 path:/dev/mapper/data12 NOTE: cache opening disk 4 of grp 5: ORADATA_0004 path:/dev/mapper/data13 NOTE: cache opening disk 5 of grp 5: ORADATA_0005 path:/dev/mapper/data14 NOTE: cache opening disk 6 of grp 5: ORADATA_0006 path:/dev/mapper/data15 NOTE: cache opening disk 7 of grp 5: ORADATA_0007 path:/dev/mapper/data16 NOTE: cache opening disk 8 of grp 5: ORADATA_0008 path:/dev/mapper/data17 NOTE: cache opening disk 9 of grp 5: ORADATA_0009 path:/dev/mapper/data18 NOTE: cache opening disk 10 of grp 5: ORADATA_0010 path:/dev/mapper/data19 NOTE: cache opening disk 11 of grp 5: ORADATA_0011 path:/dev/mapper/data2 NOTE: cache opening disk 12 of grp 5: ORADATA_0012 path:/dev/mapper/data20 NOTE: cache opening disk 13 of grp 5: ORADATA_0013 path:/dev/mapper/data21 NOTE: cache opening disk 14 of grp 5: ORADATA_0014 path:/dev/mapper/data22 NOTE: cache opening disk 15 of grp 5: ORADATA_0015 path:/dev/mapper/data23 NOTE: cache opening disk 16 of grp 5: ORADATA_0016 path:/dev/mapper/data24 NOTE: cache opening disk 17 of grp 5: ORADATA_0017 path:/dev/mapper/data25 NOTE: cache opening disk 18 of grp 5: ORADATA_0018 path:/dev/mapper/data26 NOTE: cache opening disk 19 of grp 5: ORADATA_0019 path:/dev/mapper/data27 NOTE: cache opening disk 20 of grp 5: ORADATA_0020 path:/dev/mapper/data28 NOTE: cache opening disk 21 of grp 5: ORADATA_0021 path:/dev/mapper/data29 NOTE: cache opening disk 22 of grp 5: ORADATA_0022 path:/dev/mapper/data3 NOTE: cache opening disk 23 of grp 5: ORADATA_0023 path:/dev/mapper/data30 NOTE: cache opening disk 24 of grp 5: ORADATA_0024 path:/dev/mapper/data31 NOTE: cache opening disk 25 of grp 5: ORADATA_0025 path:/dev/mapper/data32 NOTE: cache opening disk 26 of grp 5: ORADATA_0026 path:/dev/mapper/data33 NOTE: cache opening disk 27 of grp 5: ORADATA_0027 path:/dev/mapper/data4 NOTE: cache opening disk 28 of grp 5: ORADATA_0028 path:/dev/mapper/data5 NOTE: cache opening disk 29 of grp 5: ORADATA_0029 path:/dev/mapper/data6 NOTE: cache opening disk 30 of grp 5: ORADATA_0030 path:/dev/mapper/data7 NOTE: cache opening disk 31 of grp 5: ORADATA_0031 path:/dev/mapper/data8 NOTE: cache opening disk 32 of grp 5: ORADATA_0032 path:/dev/mapper/data9 NOTE: cache mounting (first) external redundancy group 5/0xD4B7AC6A (ORADATA) Sat Dec 25 17:48:52 2021 * allocate domain 5, invalid = TRUE kjbdomatt send to inst 2 Sat Dec 25 17:48:52 2021 NOTE: attached to recovery domain 5 NOTE: starting recovery of thread=1 ckpt=92.6417 group=5 (ORADATA) NOTE: advancing ckpt for group 5 (ORADATA) thread=1 ckpt=92.6418 NOTE: cache recovered group 5 to fcn 0.9502919 NOTE: redo buffer size is 256 blocks (1053184 bytes) Sat Dec 25 17:48:52 2021 NOTE: LGWR attempting to mount thread 1 for diskgroup 5 (ORADATA) NOTE: LGWR found thread 1 closed at ABA 92.6417 NOTE: LGWR mounted thread 1 for diskgroup 5 (ORADATA) NOTE: LGWR opening thread 1 at fcn 0.9502919 ABA 93.6418 NOTE: cache mounting group 5/0xD4B7AC6A (ORADATA) succeeded NOTE: cache ending mount (success) of group ORADATA number=5 incarn=0xd4b7ac6a Sat Dec 25 17:48:53 2021 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 5 SUCCESS: diskgroup ORADATA was mounted SUCCESS: alter diskgroup ORADATA mount Sat Dec 25 17:48:53 2021 NOTE: diskgroup resource ora.ORADATA.dg is online WARNING:cache read a corrupt block: group=5(ORADATA)dsk=5 blk=2 disk=5(ORADATA_0005)incarn=2406 au=0 blk=2 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1] NOTE: a corrupted block from group ORADATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc WARNING:cache read(retry)a corrupt block:group=5(ORADATA)dsk=5 blk=2 disk=5(ORADATA_0005)incarn=2406 au=0 blk=2 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1] ERROR: cache failed to read group=5(ORADATA) dsk=5 blk=2 from disk(s): 5(ORADATA_0005) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1] NOTE: cache initiating offline of disk 5 group ORADATA NOTE: process _rbal_+asm1 (48956) initiating offline of disk 5.240607694 (ORADATA_0005) with mask 0x7e in group 5 NOTE: initiating PST update: grp = 5, dsk = 5/0xe5761ce, mask = 0x6a, op = clear GMON updating disk modes for group 5 at 155 for pid 18, osid 48956 ERROR: Disk 5 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 5) Sat Dec 25 17:48:55 2021 NOTE: cache dismounting (not clean) group 5/0xD4B7AC6A (ORADATA) WARNING: Offline for disk ORADATA_0005 in mode 0x7f failed. Sat Dec 25 17:48:55 2021 NOTE: halting all I/Os to diskgroup 5 (ORADATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 22744, image: oracle@wxzldb1 (B000) Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc (incident=1289754): ORA-15335: ASM metadata corruption detected in disk group 'ORADATA' ORA-15130: diskgroup "ORADATA" is being dismounted ORA-15066: offlining disk "ORADATA_0005" in group "ORADATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1] Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_1289754/+ASM1_rbal_48956_i1289754.trc NOTE: LGWR doing non-clean dismount of group 5 (ORADATA) NOTE: LGWR sync ABA=93.6418 last written ABA 93.6418 kjbdomdet send to inst 2 detach from dom 5, sending detach message to inst 2 Sat Dec 25 17:48:56 2021 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4) Sat Dec 25 17:48:56 2021 Sweep [inc][1289754]: completed Global Resource Directory partially frozen for dirty detach * dirty detach - domain 5 invalid = TRUE 41 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete freeing rdom 5 System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_1289754/+ASM1_rbal_48956_i1289754.trc WARNING: dirty detached from domain 5 NOTE: cache dismounted group 5/0xD4B7AC6A (ORADATA)
问题比较明显是由于disk=5 au=0 blk=2有问题导致磁盘组mount之后立马异常.通过kfed分析对应block情况
C:\Users\XFF>kfed read h:\temp\asmdisk\data14.dd|more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483653 ; 0x008: disk=5 kfbh.check: 314993330 ; 0x00c: 0x12c66ab2 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8 kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 186646528 ; 0x020: 0x0b200000 kfdhdb.dsknum: 5 ; 0x024: 0x0005 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: ORADATA_0005 ; 0x028: length=12 kfdhdb.grpname: ORADATA ; 0x048: length=7 kfdhdb.fgname: ORADATA_0005 ; 0x068: length=12 C:\Users\XFF>kfed read h:\temp\asmdisk\data14.dd aun=0 blkn=2|more kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 0066D8200 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
通过kfed分析,该block确实异常,该block主要记录au的分配信息,如果asm 磁盘组的空间不变化,不执行rebalance,一般不会主动访问该block,不访问该block磁盘组也就不会dismount,按照这个解决思路,通过patch解决,让oradata磁盘组不再执行rebalance和分配/回收空间即可一直稳定的mount
数据库直接open成功,实现数据0丢失

发表在 Oracle ASM, Oracle备份恢复
标签为 asm mount, ORA-15066, ORA-15130, ORA-15196, ORA-15335, WARNING: cache read a corrupt block
评论关闭
ORA-15335: ASM metadata corruption detected in disk group ‘DATA’
asm磁盘组增加磁盘进行扩容之后报ORA-15335: ASM metadata corruption detected in disk group ‘DATA’和ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479],磁盘组dismount,然后mount之后立马dismount掉.
Tue Jun 29 09:19:09 2021 SQL> ALTER DISKGROUP DATA ADD DISK '/dev/raw/raw5' SIZE 102400M /* ASMCA */ NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (2,1) to disk (/dev/raw/raw5) NOTE: requesting all-instance membership refresh for group=2 NOTE: initializing header on grp 2 disk DATA_0001 NOTE: requesting all-instance disk validation for group=2 Tue Jun 29 09:19:11 2021 NOTE: skipping rediscovery for group 2/0xb0c845ce (DATA) on local instance. NOTE: requesting all-instance disk validation for group=2 NOTE: skipping rediscovery for group 2/0xb0c845ce (DATA) on local instance. NOTE: initiating PST update: grp = 2 Tue Jun 29 09:19:16 2021 GMON updating group 2 at 7 for pid 27, osid 25020 NOTE: PST update grp = 2 completed successfully NOTE: membership refresh pending for group 2/0xb0c845ce (DATA) GMON querying group 2 at 8 for pid 18, osid 3852 NOTE: cache opening disk 1 of grp 2: DATA_0001 path:/dev/raw/raw5 NOTE: Attempting voting file refresh on diskgroup DATA NOTE: Refresh completed on diskgroup DATA. No voting file found. GMON querying group 2 at 9 for pid 18, osid 3852 SUCCESS: refreshed membership for 2/0xb0c845ce (DATA) Tue Jun 29 09:19:20 2021 SUCCESS: ALTER DISKGROUP DATA ADD DISK '/dev/raw/raw5' SIZE 102400M /* ASMCA */ NOTE: starting rebalance of group 2/0xb0c845ce (DATA) at power 1 Starting background process ARB0 Tue Jun 29 09:19:21 2021 ARB0 started with pid=33, OS id=25176 NOTE: assigning ARB0 to group 2/0xb0c845ce (DATA) with 1 parallel I/O cellip.ora not found. Tue Jun 29 09:19:24 2021 NOTE: Attempting voting file refresh on diskgroup DATA NOTE: Refresh completed on diskgroup DATA. No voting file found. Tue Jun 29 09:19:46 2021 WARNING: cache read a corrupt block: group=2(DATA) dsk=0 blk=7 disk=0 (DATA_0000) incarn=3915953476 au=0 blk=7 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] NOTE: a corrupted block from group DATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ERROR: cache failed to read group=2(DATA) dsk=0 blk=7 from disk(s): 0(DATA_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] NOTE: cache initiating offline of disk 0 group DATA NOTE: process _arb0_+asm1 (25176) initiating offline of disk 0.3915953476 (DATA_0000) with mask 0x7e in group 2 NOTE: initiating PST update: grp = 2, dsk = 0/0xe968b544, mask = 0x6a, op = clear Tue Jun 29 09:19:46 2021 GMON updating disk modes for group 2 at 10 for pid 33, osid 25176 ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 2) Tue Jun 29 09:19:46 2021 NOTE: cache dismounting (not clean) group 2/0xB0C845CE (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 25395, image: oracle@frsrac1 (B000) Tue Jun 29 09:19:46 2021 NOTE: halting all I/Os to diskgroup 2 (DATA) Tue Jun 29 09:19:46 2021 NOTE: LGWR doing non-clean dismount of group 2 (DATA) NOTE: LGWR sync ABA=11.10715 last written ABA 11.10715 WARNING: Offline for disk DATA_0000 in mode 0x7f failed. Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc (incident=54665): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0000" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_54665/+ASM1_arb0_25176_i54665.trc Tue Jun 29 09:19:46 2021 kjbdomdet send to inst 2 detach from dom 2, sending detach message to inst 2 Tue Jun 29 09:19:46 2021 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 1, cluster inc 24) Global Resource Directory partially frozen for dirty detach * dirty detach - domain 2 invalid = TRUE 796 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete Tue Jun 29 09:19:46 2021 WARNING: dirty detached from domain 2 NOTE: cache dismounted group 2/0xB0C845CE (DATA) SQL> alter diskgroup DATA dismount force /* ASM SERVER:2965915086 */ Tue Jun 29 09:19:47 2021 ERROR: ORA-15130 thrown in ARB0 for group number 2 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25176.trc: ORA-15130: diskgroup "DATA" is being dismounted ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0000" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483648] [7] [2183628676 != 686982479] Tue Jun 29 09:19:47 2021 NOTE: stopping process ARB0 Tue Jun 29 09:19:47 2021 Sweep [inc][54665]: completed Tue Jun 29 09:19:47 2021 Sweep [inc2][54665]: completed NOTE: cache deleting context for group DATA 2/0xb0c845ce Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_3852.trc: ORA-15130: diskgroup "DATA" is being dismounted GMON dismounting group 2 at 11 for pid 27, osid 25395 NOTE: Disk DATA_0000 in mode 0x7f marked for de-assignment NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment SUCCESS: diskgroup DATA was dismounted SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER:2965915086 */
通过kfed分析报错block,确认错误
kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 2 ; 0x003: 0x02 kfbh.block.blk: 7 ; 0x004: blk=7 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 2183628676 ; 0x00c: 0x82278784 <<======该值错误,应该为:686982479 kfbh.fcn.base: 3430 ; 0x010: 0x00000d66 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb.aunum: 2240 ; 0x000: 0x000008c0 kfdatb.shrink: 448 ; 0x004: 0x01c0 kfdatb.ub2pad: 0 ; 0x006: 0x0000
通过修复该错误,并且禁止reblance操作[增加磁盘数据需要重新分布],mount磁盘组,然后open库,发现redo已经被覆盖(非归档),强制打开库报错
SQL> alter database open resetlogs; alter database open resetlogs * ERROR at line 1: ORA-00603: ORACLE server session terminated by fatal error ORA-00600: internal error code, arguments: [2662], [0], [2691201882], [0], [2691227745], [12583040], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [2662], [0], [2691201881], [0], [2691227745], [12583040], [], [], [], [], [], [] ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [2662], [0], [2691201879], [0], [2691227745], [12583040], [], [], [], [], [], [] Process ID: 25110 Session ID: 287 Serial number: 3
通过对scn进行处理,数据库顺利open
SQL> startup mount pfile='/tmp/pfile'; ORACLE instance started. Total System Global Area 5044088832 bytes Fixed Size 2261928 bytes Variable Size 1442843736 bytes Database Buffers 3590324224 bytes Redo Buffers 8658944 bytes Database mounted. SQL> alter database open; Database altered.
asm 加磁盘导致磁盘组损坏恢复
接到客户恢复case请求,希望我们接入恢复数据。大概过程是这样的,16年9月份由于硬件问题,导致normal磁盘组(只有2个磁盘)中的一个磁盘丢失,然后在17年3月6日,运维方尝试增加该磁盘进入磁盘组,结果通过force命令加入成功之后,磁盘组dismount,然后再也无法mount成功。
磁盘组创建信息
Fri Jun 24 19:31:38 2016 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2 SUCCESS: diskgroup DATADG was mounted SUCCESS: CREATE DISKGROUP DATADG NORMAL REDUNDANCY DISK '/dev/asm-diskdata01' SIZE 1048576M , '/dev/asm-diskdata02' SIZE 1048576M ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='4M' /* ASMCA */
这里可以看出来datadg是一个normal的au为4M的一个磁盘组
自动drop异常asm disk
Mon Sep 12 11:41:54 2016 WARNING: Waited 15 secs for write IO to PST disk 1 in group 1. WARNING: Waited 15 secs for write IO to PST disk 1 in group 1. Mon Sep 12 11:41:55 2016 NOTE: process _b000_+asm1 (19491) initiating offline of disk 1.3915923833 (DATADG_0001) with mask 0x7e in group 1 NOTE: checking PST: grp = 1 GMON checking disk modes for group 1 at 9 for pid 29, osid 19491 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: checking PST for grp 1 done. NOTE: sending set offline flag message 2870990318 to 1 disk(s) in group 1 WARNING: Disk DATADG_0001 in mode 0x7f is now being offlined NOTE: initiating PST update: grp = 1, dsk = 1/0xe9684179, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 10 for pid 29, osid 19491 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 1 completed successfully NOTE: initiating PST update: grp = 1, dsk = 1/0xe9684179, mask = 0x7e, op = clear GMON updating disk modes for group 1 at 11 for pid 29, osid 19491 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: cache closing disk 1 of grp 1: DATADG_0001 NOTE: PST update grp = 1 completed successfully Mon Sep 12 11:42:55 2016 WARNING: Waited 15 secs for write IO to PST disk 0 in group 1. Mon Sep 12 11:44:58 2016 WARNING: PST-initiated drop of 1 disk(s) in group 1(.1137226115)) SQL> alter diskgroup DATADG drop disk DATADG_0001 force /* ASM SERVER */ NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=1 Mon Sep 12 11:44:59 2016 GMON updating for reconfiguration, group 1 at 12 for pid 29, osid 19491 NOTE: cache closing disk 1 of grp 1: (not open) DATADG_0001 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group 1 PST updated. Mon Sep 12 11:44:59 2016 NOTE: membership refresh pending for group 1/0x43c8b183 (DATADG) Mon Sep 12 11:45:02 2016 NOTE: successfully read ACD block gn=1 blk=0 via retry read Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3526.trc: ORA-15062: ASM disk is globally closed GMON querying group 1 at 13 for pid 18, osid 3532 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x43c8b183 (DATADG) SUCCESS: alter diskgroup DATADG drop disk DATADG_0001 force /* ASM SERVER */ NOTE: starting rebalance of group 1/0x43c8b183 (DATADG) at power 1 SUCCESS: PST-initiated drop disk in group 1(1137226115)) Starting background process ARB0 Mon Sep 12 11:45:03 2016 ARB0 started with pid=35, OS id=19945 NOTE: assigning ARB0 to group 1/0x43c8b183 (DATADG) with 1 parallel I/O cellip.ora not found. NOTE: Rebalance has restored redundancy for any existing control file or redo log in disk group DATADG NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Sep 12 11:46:21 2016 NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=1 Mon Sep 12 11:46:24 2016 GMON updating for reconfiguration, group 1 at 14 for pid 36, osid 20110 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG NOTE: group 1 PST updated. WARNING: offline disk number 1 has references (54679 AUs) Mon Sep 12 11:46:24 2016 NOTE: membership refresh pending for group 1/0x43c8b183 (DATADG) GMON querying group 1 at 15 for pid 18, osid 3532 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x43c8b183 (DATADG) NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. NOTE: stopping process ARB0 SUCCESS: rebalance completed for group 1/0x43c8b183 (DATADG)
这里我们可以看出来磁盘组在2016年9月12日由于disk 1 无法响应,直接被asm 踢出了磁盘组
把被强制删除的磁盘重新加回去
Mon Mar 06 15:36:54 2017 SQL> alter diskgroup DATADG add disk '/dev/asm-diskdata01' name DATADG_0000 NOTE: GroupBlock outside rolling migration privileged region ORA-15032: not all alterations performed ORA-15029: disk '/dev/asm-diskdata01' is already mounted by this instance ERROR: alter diskgroup DATADG add disk '/dev/asm-diskdata01' name DATADG_0000 Mon Mar 06 15:38:27 2017 SQL> alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: Disk DATADG_0001 in mode 0x7f marked for de-assignment ERROR: ORA-15033 signalled during reconfiguration of diskgroup DATADG Mon Mar 06 15:38:28 2017 NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) Mon Mar 06 15:38:31 2017 GMON querying group 1 at 7 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG GMON querying group 1 at 8 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) ORA-15032: not all alterations performed ORA-15033: disk '/dev/asm-diskdata02' belongs to diskgroup "DATADG" ERROR: alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Mar 06 16:04:14 2017 SQL> alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: Disk DATADG_0001 in mode 0x7f marked for de-assignment ERROR: ORA-15033 signalled during reconfiguration of diskgroup DATADG Mon Mar 06 16:04:15 2017 NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) Mon Mar 06 16:04:18 2017 GMON querying group 1 at 9 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG GMON querying group 1 at 10 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) ORA-15032: not all alterations performed ORA-15033: disk '/dev/asm-diskdata02' belongs to diskgroup "DATADG" ERROR: alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Mar 06 16:23:28 2017 SQL> alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/adm-diskdata02' name DATA_0001 NOTE: GroupBlock outside rolling migration privileged region ORA-15032: not all alterations performed ORA-15031: disk specification '/dev/adm-diskdata02' matches no disks ERROR: alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/adm-diskdata02' name DATA_0001 Mon Mar 06 16:24:48 2017 SQL> alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment ERROR: ORA-15033 signalled during reconfiguration of diskgroup DATADG Mon Mar 06 16:24:49 2017 NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) Mon Mar 06 16:24:52 2017 GMON querying group 1 at 11 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG GMON querying group 1 at 12 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) ORA-15032: not all alterations performed ORA-15033: disk '/dev/asm-diskdata02' belongs to diskgroup "DATADG" ERROR: alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Mar 06 16:26:07 2017 SQL> alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 force NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: initializing header on grp 1 disk DATA_0001 NOTE: requesting all-instance disk validation for group=1 Mon Mar 06 16:26:10 2017 NOTE: skipping rediscovery for group 1/0x31584f6b (DATADG) on local instance. NOTE: requesting all-instance disk validation for group=1 NOTE: skipping rediscovery for group 1/0x31584f6b (DATADG) on local instance. Mon Mar 06 16:26:15 2017 GMON updating for reconfiguration, group 1 at 13 for pid 28, osid 12861 NOTE: group 1 PST updated. NOTE: initiating PST update: grp = 1 GMON updating group 1 at 14 for pid 28, osid 12861 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group DATADG: updated PST location: disk 0002 (PST copy 1) NOTE: PST update grp = 1 completed successfully NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) GMON querying group 1 at 15 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG NOTE: cache opening disk 2 of grp 1: DATA_0001 path:/dev/asm-diskdata02 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. GMON querying group 1 at 16 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) Mon Mar 06 16:26:19 2017 SUCCESS: alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 force NOTE: starting rebalance of group 1/0x31584f6b (DATADG) at power 1 Mon Mar 06 16:26:20 2017 Starting background process ARB0 Mon Mar 06 16:26:20 2017 ARB0 started with pid=32, OS id=25833 NOTE: assigning ARB0 to group 1/0x31584f6b (DATADG) with 1 parallel I/O cellip.ora not found. WARNING:cache read a corrupt block: group=1(DATADG) dsk=0 blk=0 disk=0 (DATADG_0000)incarn=3915956130 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] NOTE:a corrupted block from group DATADG was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc WARNING:cache read (retry)a corrupt block:group=1(DATADG) dsk=0 blk=0 disk=0(DATADG_0000)incarn=3915956130 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ERROR: cache failed to read group=1(DATADG) dsk=0 blk=0 from disk(s): 0(DATADG_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] NOTE: cache initiating offline of disk 0 group DATADG NOTE:process _arb0_+asm1 (25833) initiating offline of disk 0.3915956130(DATADG_0000)with mask 0x7e in group 1 NOTE: checking PST: grp = 1 GMON checking disk modes for group 1 at 17 for pid 32, osid 25833 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: too many offline disks in PST (grp 1) NOTE: checking PST for grp 1 done. NOTE: initiating PST update: grp = 1, dsk = 0/0xe968bfa2, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 18 for pid 32, osid 25833 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: Disk 0 cannot be offlined, since all the disks [0, 1] with mirrored data would be offline. ERROR: too many offline disks in PST (grp 1) Mon Mar 06 16:26:23 2017 NOTE: cache dismounting (not clean) group 1/0x31584F6B (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 25889, image: oracle@DBN01 (B000) Mon Mar 06 16:26:23 2017 NOTE: halting all I/Os to diskgroup 1 (DATADG) Mon Mar 06 16:26:23 2017 NOTE: LGWR doing non-clean dismount of group 1 (DATADG) NOTE: LGWR sync ABA=19.2851 last written ABA 19.2851 WARNING: Offline for disk DATADG_0000 in mode 0x7f failed. Mon Mar 06 16:26:23 2017 kjbdomdet send to inst 2 detach from dom 1, sending detach message to inst 2 Mon Mar 06 16:26:23 2017 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 1, cluster inc 8) Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc (incident=65537): ORA-15335: ASM metadata corruption detected in disk group 'DATADG' ORA-15130: diskgroup "DATADG" is being dismounted ORA-15066: offlining disk "DATADG_0000" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] Incident details in:/u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_65537/+ASM1_arb0_25833_i65537.trc Global Resource Directory partially frozen for dirty detach * dirty detach - domain 1 invalid = TRUE 3189 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete ERROR: ORA-15130 in COD recovery for diskgroup 1/0x31584f6b (DATADG) ERROR: ORA-15130 thrown in RBAL for group number 1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_3468.trc: ORA-15130: diskgroup "DATADG" is being dismounted Mon Mar 06 16:26:23 2017 WARNING: dirty detached from domain 1 NOTE: cache dismounted group 1/0x31584F6B (DATADG) ---后续mount报错 SQL> ALTER DISKGROUP DATADG MOUNT /* asm agent *//* {1:18003:2} */ NOTE: cache registered group DATADG number=1 incarn=0xb368408f NOTE: cache began mount (first) of group DATADG number=1 incarn=0xb368408f NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) WARNING:GMON has insufficient disks to maintain consensus. Minimum required is 2:updating 1 PST copies from a total of 2. ERROR: GMON failed to obtain a quorum ofsupporting disks in group 1 NOTE: cache dismounting (clean) group 1/0xB368408F (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 27651, image: oracle@DBN01 (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xB368408F (DATADG) NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xb368408f NOTE: cache deleting context for group DATADG 1/0xb368408f GMON dismounting group 1 at 12 for pid 30, osid 27651 NOTE: Disk DATA_0001 in mode 0x9 marked for de-assignment ERROR: diskgroup DATADG was not mounted ORA-15032: not all alterations performed ORA-15017: diskgroup "DATADG" cannot be mounted ORA-15315: Write errors in disk group DATADG could lead to inconsistent ASM metadata. ERROR: ALTER DISKGROUP DATADG MOUNT /* asm agent *//* {1:18003:2} */
从这里我们可以看出来,前几次加asm disk 由于各种原因都失败了,最后一次通过加force关键字,使得被自动drop的disk重新强制加到datadg里面.可悲的是在加入成功之后,开始做rebalance的时候,发现disk 0出现坏块,从而引起ORA-15196的错误,使得rebalance无法进行下去,进而整个asm 磁盘组datadg自动dismount.后面再次尝试mount datadg的时候,直接提示元数据库不一致,因为disk 0 的磁盘头已经异常.
通过kfed分析disk 0信息
这里是通过dd命令备份的磁盘头到win进行分析的,以前正常的disk 0的磁盘头损坏(全0)
对于这个故障已经比较清楚,恢复思路也基本上确定:依次递进
方案1:通过kfed修改文件头,然后尝试mount磁盘头手工修复ASM DISK HEADER 异常
方案2:直接通过amdu,dul之类的工具拷贝出来数据文件找回ASM中数据文件
方案3:通过底层au重组出来数据文件asm disk header 彻底损坏恢复
在我们的实际恢复中运气比较好,通过方案1就完成了恢复工作,通过kfed修复磁盘头之后,然后报错如下
SQL> alter diskgroup DATADG mount NOTE: cache registered group DATADG number=1 incarn=0x5134d0d4 NOTE: cache began mount (first) of group DATADG number=1 incarn=0x5134d0d4 NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: Assigning number (1,0) to disk (/dev/asm-diskdata01) Tue Mar 07 19:03:40 2017 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 27 for pid 28, osid 13837 NOTE: Assigning number (1,1) to disk () GMON querying group 1 at 28 for pid 28, osid 13837 NOTE: cache closing disk 1 of grp 1: (not open) NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diskdata01 NOTE: F1X0 found on disk 0 au 2 fcn 0.178802 NOTE: cache opening disk 2 of grp 1: DATA_0001 path:/dev/asm-diskdata02 NOTE: cache mounting (first) normal redundancy group 1/0x5134D0D4 (DATADG) Tue Mar 07 19:03:40 2017 * allocate domain 1, invalid = TRUE kjbdomatt send to inst 2 Tue Mar 07 19:03:40 2017 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=19.2851 group=1 (DATADG) NOTE: starting recovery of thread=2 ckpt=13.5327 group=1 (DATADG) NOTE: advancing ckpt for group 1 (DATADG) thread=2 ckpt=13.5327 NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=19.2852 NOTE: cache recovered group 1 to fcn 0.365868 NOTE: redo buffer size is 512 blocks (2101760 bytes) Tue Mar 07 19:03:40 2017 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG) NOTE: LGWR found thread 1 closed at ABA 19.2851 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG) NOTE: LGWR opening thread 1 at fcn 0.365868 ABA 20.2852 NOTE: cache mounting group 1/0x5134D0D4 (DATADG) succeeded NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x5134d0d4 Tue Mar 07 19:03:40 2017 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATADG was mounted SUCCESS: alter diskgroup DATADG mount Tue Mar 07 19:03:40 2017 NOTE: diskgroup resource ora.DATADG.dg is online Tue Mar 07 19:03:41 2017 ASM Health Checker found 1 new failures NOTE: ASM did background COD recovery for group 1/0x5134d0d4 (DATADG) NOTE: starting rebalance of group 1/0x5134d0d4 (DATADG) at power 1 Starting background process ARB0 Tue Mar 07 19:03:42 2017 ARB0 started with pid=30, OS id=13905 NOTE: assigning ARB0 to group 1/0x5134d0d4 (DATADG) with 1 parallel I/O cellip.ora not found. WARNING: cache read a corrupt block: group=1(DATADG) dsk=0 blk=0 disk=0 (DATADG_0000) incarn=2202280062 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] NOTE: a corrupted block from group DATADG was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc WARNING:cache read (retry)a corrupt block:group=1(DATADG) dsk=0 blk=0 disk=0 (DATADG_0000)incarn=2202280062 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ERROR: cache failed to read group=1(DATADG) dsk=0 blk=0 from disk(s): 0(DATADG_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] Tue Mar 07 19:03:52 2017 NOTE: client oradb1:oradb registered, osid 13989, mbr 0x1 NOTE: cache initiating offline of disk 0 group DATADG NOTE:process _arb0_+asm1 (13905) initiating offline of disk 0.2202280062(DATADG_0000)with mask 0x7e in group 1 NOTE: checking PST: grp = 1 Tue Mar 07 19:03:52 2017 GMON checking disk modes for group 1 at 30 for pid 30, osid 13905 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: too many offline disks in PST (grp 1) NOTE: checking PST for grp 1 done. NOTE: initiating PST update: grp = 1, dsk = 0/0x8344207e, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 31 for pid 30, osid 13905 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: Disk 0 cannot be offlined, since all the disks [0, 1] with mirrored data would be offline. ERROR: too many offline disks in PST (grp 1) Tue Mar 07 19:03:52 2017 NOTE: cache dismounting (not clean) group 1/0x5134D0D4 (DATADG) WARNING: Offline for disk DATADG_0000 in mode 0x7f failed. Tue Mar 07 19:03:52 2017 NOTE: halting all I/Os to diskgroup 1 (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 14002, image: oracle@DBN01 (B000) Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc (incident=76402): ORA-15335: ASM metadata corruption detected in disk group 'DATADG' ORA-15130: diskgroup "DATADG" is being dismounted ORA-15066: offlining disk "DATADG_0000" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_76402/+ASM1_arb0_13905_i76402.trc Tue Mar 07 19:03:52 2017 NOTE: LGWR doing non-clean dismount of group 1 (DATADG) NOTE: LGWR sync ABA=20.2857 last written ABA 20.2857
这里比较比较幸运,datadg已经mount成功了,但是由于rab依旧读取到disk header异常信息(没有完全修复成功,而且在日志中不光这个block异常,还有其他block异常,因此不考虑进一步修复),因此直接通过屏蔽asm的acd和cod实现该磁盘组mount,而且不会dismount。
SQL> alter diskgroup DATADG mount NOTE: cache registered group DATADG number=1 incarn=0x9c94d0eb NOTE: cache began mount (first) of group DATADG number=1 incarn=0x9c94d0eb NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: Assigning number (1,0) to disk (/dev/asm-diskdata01) NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639 Tue Mar 07 19:12:45 2017 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 75 for pid 28, osid 15615 NOTE: Assigning number (1,1) to disk () GMON querying group 1 at 76 for pid 28, osid 15615 NOTE: cache closing disk 1 of grp 1: (not open) NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diskdata01 NOTE: F1X0 found on disk 0 au 2 fcn 0.178802 NOTE: cache opening disk 2 of grp 1: DATA_0001 path:/dev/asm-diskdata02 NOTE: cache mounting (first) normal redundancy group 1/0x9C94D0EB (DATADG) Tue Mar 07 19:12:45 2017 * allocate domain 1, invalid = TRUE kjbdomatt send to inst 2 Tue Mar 07 19:12:45 2017 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=25.2870 group=1 (DATADG) NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=25.2873 NOTE: cache recovered group 1 to fcn 0.365897 NOTE: redo buffer size is 512 blocks (2101760 bytes) Tue Mar 07 19:12:45 2017 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG) NOTE: LGWR found thread 1 closed at ABA 25.2872 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG) NOTE: LGWR opening thread 1 at fcn 0.365897 ABA 26.2873 NOTE: cache mounting group 1/0x9C94D0EB (DATADG) succeeded NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x9c94d0eb Tue Mar 07 19:12:45 2017 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATADG was mounted SUCCESS: alter diskgroup DATADG mount Tue Mar 07 19:12:45 2017 NOTE: diskgroup resource ora.DATADG.dg is online NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639
asm的问题解决后,然后登录数据库,发现运气比较好,两个数据库正常open成功,而且alert日志无任何报错,直接通过rman备份出来数据,重建asm磁盘组,还原数据,恢复完成,而且实现数据0丢失。
如果您遇到此类情况,无法解决请联系我们,提供专业ORACLE数据库恢复技术支持
Phone:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com