-
3D Cloud
asm mount asm恢复 bbed bootstrap$ dmp损坏 dul In Memory kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01190 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kccpb_sanity_check_2 ORA-15042 ORACLE 12C oracle dul ORACLE PATCH oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle 比特币 redo异常 undo异常 YOUR FILES ARE ENCRYPTED _ALLOW_RESETLOGS_CORRUPTION 勒索恢复 数据库恢复 比特币 比特币 oracle 比特币加密 比特币勒索
文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (87)
- 数据库 (1,316)
- DB2 (22)
- MySQL (57)
- Oracle (1,202)
- Data Guard (39)
- EXADATA (7)
- GoldenGate (19)
- ORA-xxxxx (145)
- ORACLE 12C (71)
- ORACLE 18C (6)
- ORACLE 19C (8)
- Oracle ASM (56)
- Oracle Bug (7)
- Oracle RAC (40)
- Oracle 安全 (6)
- Oracle 开发 (25)
- Oracle 监听 (26)
- Oracle备份恢复 (393)
- Oracle安装升级 (55)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (43)
- PostgreSQL (10)
- SQL Server (26)
- SQL Server恢复 (7)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (23)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (6)
-
最近发表
- Assistant: Download Reference for Oracle Database/GI PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases-202101
- .sql文件被加密恢复
- ORA-600 3020错误引起ORA-600 2663
- Alpha865qqz.id 加密数据库恢复
- incaseformat 病毒删除文件恢复
- xfs文件系统mysql删库恢复
- 对恢复案例:因对工作调整不满,链家一员工删除公司 9 TB数据:被判7年事件有感
- Oracle Recovery Tools 解决ORA-01190 ORA-01248等故障
- Oracle Recovery Tools 12月份更新
- -bash: /bin/rm: Argument list too long
- ORA-27303: failure occurred at: skgpwinit6
- ORA-600 kffmLoad_1 kffmVerify_4
- ORA-00600 kfrHtAdd01
- ORA-00600 [2662]和ORA-00600 [4194]恢复
- 记录oracle安装的两个小问题(INS-30060和弹出子窗口异常)
- dblink会话引起library cache lock
- asm磁盘类似_DROPPED_0001_DATA名称故障处理
- 12C数据库报ORA-600 kcbzib_kcrsds_1故障处理
- 再次遇到ORA-600 kokasgi1故障恢复
- sysaux表空间不足—WRH$_ACTIVE_SESSION_HISTORY
友情链接
标签归档:ORA-15196
ORA-15196: invalid ASM block header [kfc.c:26368]故障恢复
有客户对asm的data磁盘组增加磁盘进行扩容,在做reblance的过程中重启了主机,结果导致data磁盘组mount之后自动dismount
Fri Oct 09 20:48:06 2020 NOTE: PST enabling heartbeating (grp 1) Fri Oct 09 20:48:06 2020 NOTE: ASM did background COD recovery for group 1/0x739536c (DATA) NOTE: starting rebalance of group 1/0x739536c (DATA) at power 10 Starting background process ARB0 Fri Oct 09 20:48:07 2020 ARB0 started with pid=28, OS id=39278 NOTE: assigning ARB0 to group 1/0x739536c (DATA) with 10 parallel I/Os cellip.ora not found. WARNING:cache read a corrupt block:group=1(DATA) dsk=8 blk=7 disk=8(DATA_0008)incarn=3916014506 au=0 blk=7 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] NOTE: a corrupted block from group DATA was dumped to /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc WARNING:cache read(retry)a corrupt block:group=1(DATA) dsk=8 blk=7 disk=8(DATA_0008)incarn=3916014506 au=0 blk=7 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ERROR: cache failed to read group=1(DATA) dsk=8 blk=7 from disk(s): 8(DATA_0008) Fri Oct 09 20:48:13 2020 NOTE: GroupBlock outside rolling migration privileged region ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] NOTE: requesting all-instance membership refresh for group=1 NOTE: cache initiating offline of disk 8 group DATA NOTE: process _arb0_+asm1 (39278) initiating offline of disk 8.3916014506 (DATA_0008) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 8/0xe969a3aa, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 7 for pid 28, osid 39278 ERROR: Disk 8 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Fri Oct 09 20:48:13 2020 NOTE: cache dismounting (not clean) group 1/0x0739536C (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 39346, image: oracle@rac1 (B000) Fri Oct 09 20:48:13 2020 NOTE: halting all I/Os to diskgroup 1 (DATA) Fri Oct 09 20:48:13 2020 NOTE: LGWR doing non-clean dismount of group 1 (DATA) NOTE: LGWR sync ABA=32.4749 last written ABA 32.4749 WARNING: Offline for disk DATA_0008 in mode 0x7f failed. Fri Oct 09 20:48:13 2020 kjbdomdet send to inst 2 detach from dom 1, sending detach message to inst 2 Fri Oct 09 20:48:13 2020 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 2, cluster inc 4) Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc (incident=337185): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0008" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] Incident details in: /u01/grid/diag/asm/+asm/+ASM1/incident/incdir_337185/+ASM1_arb0_39278_i337185.trc Global Resource Directory partially frozen for dirty detach * dirty detach - domain 1 invalid = TRUE 2341 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete freeing rdom 1 Fri Oct 09 20:48:13 2020 WARNING: dirty detached from domain 1 NOTE: cache dismounted group 1/0x0739536C (DATA)
错误信息比较明显dsk=8 blk=7 au=0 blk=7 的check值不对,本来应该是2190395015现在变为了2182009786,通过kfed分析确实如此
C:\Users\Administrator>kfed read f:/temp/xff/2.dd blkn=7|more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 2 ; 0x003: 0x02 kfbh.block.blk: 7 ; 0x004: blk=7 kfbh.block.obj: 2147483656 ; 0x008: disk=8 kfbh.check: 2182009786 ; 0x00c: 0x820ed3ba kfbh.fcn.base: 2711248 ; 0x010: 0x00295ed0 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb.aunum: 2240 ; 0x000: 0x000008c0 kfdatb.shrink: 448 ; 0x004: 0x01c0 kfdatb.ub2pad: 0 ; 0x006: 0x0000 kfdatb.auinfo[0].link.next: 8 ; 0x008: 0x0008 kfdatb.auinfo[0].link.prev: 8 ; 0x00a: 0x0008 kfdatb.auinfo[1].link.next: 12 ; 0x00c: 0x000c kfdatb.auinfo[1].link.prev: 12 ; 0x00e: 0x000c kfdatb.auinfo[2].link.next: 16 ; 0x010: 0x0010 kfdatb.auinfo[2].link.prev: 16 ; 0x012: 0x0010 kfdatb.auinfo[3].link.next: 20 ; 0x014: 0x0014 kfdatb.auinfo[3].link.prev: 20 ; 0x016: 0x0014 kfdatb.auinfo[4].link.next: 24 ; 0x018: 0x0018 kfdatb.auinfo[4].link.prev: 24 ; 0x01a: 0x0018 kfdatb.auinfo[5].link.next: 28 ; 0x01c: 0x001c kfdatb.auinfo[5].link.prev: 28 ; 0x01e: 0x001c kfdatb.auinfo[6].link.next: 32 ; 0x020: 0x0020 kfdatb.auinfo[6].link.prev: 32 ; 0x022: 0x0020 kfdatb.spare: 0 ; 0x024: 0x00000000
修改该值之后,再次mount data磁盘组,报错如下
Sat Oct 10 13:49:22 2020 ARB0 started with pid=28, OS id=10329 NOTE: assigning ARB0 to group 1/0x3759521c (DATA) with 10 parallel I/Os cellip.ora not found. Sat Oct 10 13:49:26 2020 NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=1 WARNING: cache read a corrupt block: group=1(DATA) dsk=8 blk=8 disk=8 (DATA_0008) incarn=3916014011 au=0 blk=8 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] NOTE: a corrupted block from group DATA was dumped to /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc WARNING:cache read(retry)a corrupt block: group=1(DATA)dsk=8 blk=8 disk=8(DATA_0008)incarn=3916014011 au=0 blk=8 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ERROR: cache failed to read group=1(DATA) dsk=8 blk=8 from disk(s): 8(DATA_0008) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] NOTE: cache initiating offline of disk 8 group DATA NOTE: process _arb0_+asm1 (10329) initiating offline of disk 8.3916014011 (DATA_0008) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 8/0xe969a1bb, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 64 for pid 28, osid 10329 ERROR: Disk 8 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Sat Oct 10 13:49:28 2020 NOTE: cache dismounting (not clean) group 1/0x3759521C (DATA) WARNING: Offline for disk DATA_0008 in mode 0x7f failed. Sat Oct 10 13:49:28 2020 NOTE: halting all I/Os to diskgroup 1 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 10346, image: oracle@rac1 (B000) Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc (incident=363107): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0008" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] Incident details in: /u01/grid/diag/asm/+asm/+ASM1/incident/incdir_363107/+ASM1_arb0_10329_i363107.trc
该报错为:dsk=8 blk=7 au=0 blk=8异常,通过kfed查看发现
C:\Users\Administrator>kfed read f:/temp/xff/2.dd blkn=8 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 006BE8C00 00000000 00000000 00000000 00000000 [................] Repeat 31 times 006BE8E00 012C0000 04AFFC07 003BFFCD 03F15BD0 [..,.......;..[..] 006BE8E10 012BFC30 00000000 00000002 00000002 [0.+.............] 006BE8E20 00008000 00008000 00002000 564F22AF [......... ..."OV] 006BE8E30 5F805293 FFFF0002 0001EF53 00000001 [.R._....S.......] 006BE8E40 545AE384 00000000 00000000 00000001 [..ZT............] 006BE8E50 00000000 0000000B 00000100 0000003C [............<...] 006BE8E60 00000242 0000007B 52438BA0 C44FFA90 [B...{.....CR..O.] 006BE8E70 33B6F381 919E2DBA 00000000 00000000 [...3.-..........] 006BE8E80 00000000 00000000 6361622F 0070756B [......../backup.] 006BE8E90 00000000 00000000 00000000 00000000 [................] Repeat 2 times 006BE8EC0 00000000 00000000 00000000 03ED0000 [................] 006BE8ED0 00000000 00000000 00000000 00000000 [................] 006BE8EE0 00000008 00000000 00000000 AC3C87D6 [..............<.] 006BE8EF0 F1401174 F4F036BD 274FB92F 00000101 [t.@..6../.O'....] 006BE8F00 0000000C 00000000 545AE384 0002F30A [..........ZT....] 006BE8F10 00000004 00000000 00000000 00007FFF [................] 006BE8F20 02508000 00007FFF 00000001 0250FFFF [..P...........P.] 006BE8F30 00000000 00000000 00000000 00000000 [................] 006BE8F40 00000000 00000000 00000000 08000000 [................] 006BE8F50 00000000 00000000 00000000 001C001C [................] 006BE8F60 00000001 00000000 00000000 00000000 [................] 006BE8F70 00000000 00000004 A9AF72B9 0000003B [.........r..;...] 006BE8F80 00000000 00000000 00000000 00000000 [................] Repeat 167 times 006BE9A00 00001CC4 00800101 00001CC9 00800101 [................] 006BE9A10 00001CCD 00800101 00001CD2 00800101 [................] 006BE9A20 00001CD7 00800101 00001CDE 00800101 [................] 006BE9A30 00001CE3 00800101 00001CE8 00800101 [................] 006BE9A40 00001CEC 00800101 00000000 00000000 [................] 006BE9A50 00000000 00000000 00000000 00000000 [................] Repeat 26 times KFED-00322:Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
该block完全损坏,基本上无直接修复的可能,通过对data 磁盘组进行patch操作,让其mount之后不再dismount
NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 76 for pid 27, osid 14466 NOTE: cache opening disk 0 of grp 1: DATA_0000 path:/dev/emcpowere NOTE: F1X0 found on disk 0 au 2 fcn 0.2708382 NOTE: cache opening disk 1 of grp 1: DATA_0001 path:/dev/emcpowerf NOTE: cache opening disk 2 of grp 1: DATA_0002 path:/dev/emcpowerg NOTE: cache opening disk 3 of grp 1: DATA_0003 path:/dev/emcpowerh NOTE: cache opening disk 4 of grp 1: DATA_0004 path:/dev/emcpoweri NOTE: cache opening disk 5 of grp 1: DATA_0005 path:/dev/emcpowerj NOTE: cache opening disk 6 of grp 1: DATA_0006 path:/dev/emcpowerk NOTE: cache opening disk 7 of grp 1: DATA_0007 path:/dev/emcpowerl NOTE: cache opening disk 8 of grp 1: DATA_0008 path:/dev/emcpowerc NOTE: cache mounting (first) external redundancy group 1/0x47495222 (DATA) Sat Oct 10 13:59:38 2020 * allocate domain 1, invalid = TRUE Sat Oct 10 13:59:38 2020 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=53.6778 group=1 (DATA) NOTE: advancing ckpt for group 1 (DATA) thread=1 ckpt=53.6778 NOTE: cache recovered group 1 to fcn 0.2961429 NOTE: redo buffer size is 256 blocks (1053184 bytes) Sat Oct 10 13:59:38 2020 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATA) NOTE: LGWR found thread 1 closed at ABA 53.6777 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATA) NOTE: LGWR opening thread 1 at fcn 0.2961429 ABA 54.6778 NOTE: cache mounting group 1/0x47495222 (DATA) succeeded NOTE: cache ending mount (success) of group DATA number=1 incarn=0x47495222 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATA was mounted SUCCESS: alter diskgroup data mount
然后通过rman备份数据库,删除老磁盘组,创建新磁盘组,恢复数据,实现数据库完美恢复,数据0丢失.
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh]故障处理
客户对asm进行扩容,由于配置不恰当,在使用asmca增加asm disk的时候直接选中了已经被用作文件系统的vg中的磁盘
Tue Nov 19 09:48:48 2019 Non critical error ORA-48180 cFri Nov 22 12:47:48 2019 SQL> ALTER DISKGROUP XIFENFEI ADD DISK '/dev/rhdisk29' SIZE 491520M , '/dev/rhdisk30' SIZE 491520M , '/dev/rhdisk31' SIZE 491520M /* ASMCA */ NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (4,15) to disk (/dev/rhdisk29) NOTE: Assigning number (4,16) to disk (/dev/rhdisk30) NOTE: Assigning number (4,17) to disk (/dev/rhdisk31) NOTE: requesting all-instance membership refresh for group=4 NOTE: initializing header on grp 4 disk XIFENFEI_0015 NOTE: initializing header on grp 4 disk XIFENFEI_0016 NOTE: initializing header on grp 4 disk XIFENFEI_0017 NOTE: requesting all-instance disk validation for group=4 Fri Nov 22 12:47:51 2019 NOTE: skipping rediscovery for group 4/0xb08c40b (XIFENFEI) on local instance. NOTE: requesting all-instance disk validation for group=4 NOTE: skipping rediscovery for group 4/0xb08c40b (XIFENFEI) on local instance. Fri Nov 22 12:47:59 2019 NOTE: initiating PST update: grp = 4 Fri Nov 22 12:47:59 2019 GMON updating group 4 at 12 for pid 27, osid 12649908 NOTE: PST update grp = 4 completed successfully NOTE: membership refresh pending for group 4/0xb08c40b (XIFENFEI) GMON querying group 4 at 13 for pid 18, osid 39912680 Fri Nov 22 12:48:01 2019 NOTE: cache opening disk 15 of grp 4: XIFENFEI_0015 path:/dev/rhdisk29 NOTE: cache opening disk 16 of grp 4: XIFENFEI_0016 path:/dev/rhdisk30 NOTE: cache opening disk 17 of grp 4: XIFENFEI_0017 path:/dev/rhdisk31 NOTE: Attempting voting file refresh on diskgroup XIFENFEI NOTE: Refresh completed on diskgroup XIFENFEI. No voting file found. GMON querying group 4 at 14 for pid 18, osid 39912680 SUCCESS: refreshed membership for 4/0xb08c40b (XIFENFEI) SUCCESS: ALTER DISKGROUP XIFENFEI ADD DISK '/dev/rhdisk29' SIZE 491520M , '/dev/rhdisk30' SIZE 491520M , '/dev/rhdisk31' SIZE 491520M /* ASMCA */
发现增加错磁盘之后,从vg里面强制踢掉被asm使用的磁盘,并且尝试在asm中删除这些磁盘,并加入新磁盘
Fri Nov 22 12:52:03 2019 SQL> ALTER DISKGROUP XIFENFEI DROP DISK 'XIFENFEI_0015','XIFENFEI_0016','XIFENFEI_0017' /* ASMCA */ NOTE: GroupBlock outside rolling migration privileged region Fri Nov 22 12:52:03 2019 NOTE: stopping process ARB0 NOTE: rebalance interrupted for group 4/0xb08c40b (XIFENFEI) NOTE: requesting all-instance membership refresh for group=4 NOTE: membership refresh pending for group 4/0xb08c40b (XIFENFEI) Fri Nov 22 12:52:12 2019 GMON querying group 4 at 15 for pid 18, osid 39912680 SUCCESS: refreshed membership for 4/0xb08c40b (XIFENFEI) SUCCESS: ALTER DISKGROUP XIFENFEI DROP DISK 'XIFENFEI_0015','XIFENFEI_0016','XIFENFEI_0017' /* ASMCA */ NOTE: starting rebalance of group 4/0xb08c40b (XIFENFEI) at power 1 Starting background process ARB0 ………… Fri Nov 22 12:58:26 2019 SQL> ALTER DISKGROUP XIFENFEI ADD DISK '/dev/rhdisk7' SIZE 491520M /* ASMCA */ NOTE: GroupBlock outside rolling migration privileged region Fri Nov 22 12:58:26 2019 NOTE: stopping process ARB0 NOTE: rebalance interrupted for group 4/0xb08c40b (XIFENFEI) NOTE: ASM did background COD recovery for group 4/0xb08c40b (XIFENFEI) NOTE: Assigning number (4,18) to disk (/dev/rhdisk7) NOTE: requesting all-instance membership refresh for group=4 NOTE: initializing header on grp 4 disk XIFENFEI_0018 NOTE: requesting all-instance disk validation for group=4 NOTE: skipping rediscovery for group 4/0xb08c40b (XIFENFEI) on local instance. NOTE: requesting all-instance disk validation for group=4 NOTE: skipping rediscovery for group 4/0xb08c40b (XIFENFEI) on local instance. Fri Nov 22 12:58:41 2019 NOTE: initiating PST update: grp = 4 Fri Nov 22 12:58:41 2019 GMON updating group 4 at 16 for pid 27, osid 12649908 NOTE: PST update grp = 4 completed successfully Fri Nov 22 12:58:41 2019 NOTE: membership refresh pending for group 4/0xb08c40b (XIFENFEI) GMON querying group 4 at 17 for pid 18, osid 39912680 NOTE: cache opening disk 18 of grp 4: XIFENFEI_0018 path:/dev/rhdisk7 NOTE: Attempting voting file refresh on diskgroup XIFENFEI NOTE: Refresh completed on diskgroup XIFENFEI. No voting file found. GMON querying group 4 at 18 for pid 18, osid 39912680 SUCCESS: refreshed membership for 4/0xb08c40b (XIFENFEI) NOTE: starting rebalance of group 4/0xb08c40b (XIFENFEI) at power 1 SUCCESS: ALTER DISKGROUP XIFENFEI ADD DISK '/dev/rhdisk7' SIZE 491520M /* ASMCA */ Starting background process ARB0 Fri Nov 22 12:58:46 2019 ARB0 started with pid=44, OS id=54460432 ………… Fri Nov 22 12:59:57 2019 SQL> ALTER DISKGROUP XIFENFEI ADD DISK '/dev/rhdisk10' SIZE 491520M , '/dev/rhdisk11' SIZE 491520M , '/dev/rhdisk8' SIZE 491520M , '/dev/rhdisk9' SIZE 491520M /* ASMCA */ NOTE: GroupBlock outside rolling migration privileged region Fri Nov 22 12:59:57 2019 NOTE: stopping process ARB0 NOTE: rebalance interrupted for group 4/0xb08c40b (XIFENFEI) NOTE: ASM did background COD recovery for group 4/0xb08c40b (XIFENFEI) NOTE: Assigning number (4,19) to disk (/dev/rhdisk10) NOTE: Assigning number (4,20) to disk (/dev/rhdisk11) NOTE: Assigning number (4,21) to disk (/dev/rhdisk8) NOTE: Assigning number (4,22) to disk (/dev/rhdisk9) NOTE: requesting all-instance membership refresh for group=4 NOTE: initializing header on grp 4 disk XIFENFEI_0019 NOTE: initializing header on grp 4 disk XIFENFEI_0020 NOTE: initializing header on grp 4 disk XIFENFEI_0021 NOTE: initializing header on grp 4 disk XIFENFEI_0022 NOTE: requesting all-instance disk validation for group=4 NOTE: skipping rediscovery for group 4/0xb08c40b (XIFENFEI) on local instance. Fri Nov 22 13:00:08 2019 NOTE: requesting all-instance disk validation for group=4 Fri Nov 22 13:00:08 2019 NOTE: skipping rediscovery for group 4/0xb08c40b (XIFENFEI) on local instance. NOTE: initiating PST update: grp = 4 Fri Nov 22 13:00:13 2019 GMON updating group 4 at 19 for pid 27, osid 12649908 NOTE: PST update grp = 4 completed successfully NOTE: membership refresh pending for group 4/0xb08c40b (XIFENFEI) GMON querying group 4 at 20 for pid 18, osid 39912680 NOTE: cache opening disk 19 of grp 4: XIFENFEI_0019 path:/dev/rhdisk10 NOTE: cache opening disk 20 of grp 4: XIFENFEI_0020 path:/dev/rhdisk11 NOTE: cache opening disk 21 of grp 4: XIFENFEI_0021 path:/dev/rhdisk8 NOTE: cache opening disk 22 of grp 4: XIFENFEI_0022 path:/dev/rhdisk9 NOTE: Attempting voting file refresh on diskgroup XIFENFEI NOTE: Refresh completed on diskgroup XIFENFEI. No voting file found. GMON querying group 4 at 21 for pid 18, osid 39912680 SUCCESS: refreshed membership for 4/0xb08c40b (XIFENFEI) SUCCESS: ALTER DISKGROUP XIFENFEI ADD DISK '/dev/rhdisk10' SIZE 491520M , '/dev/rhdisk11' SIZE 491520M , '/dev/rhdisk8' SIZE 491520M , '/dev/rhdisk9' SIZE 491520M /* ASMCA */ NOTE: starting rebalance of group 4/0xb08c40b (XIFENFEI) at power 1 Starting background process ARB0
asm在做着reblance的过程中遭遇到坏块,直接导致磁盘组dismount
Sun Nov 24 04:42:27 2019 NOTE: group 4 PST updated. WARNING: cache read a corrupt block: group=4(XIFENFEI) dsk=15 blk=258 disk=15 (XIFENFEI_0015) incarn=1717056824 au=113792 blk=2 count=254 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_x000_28639240.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483663] [258] [56 != 0] NOTE: a corrupted block from group XIFENFEI was dumped to /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_x000_28639240.trc WARNING: cache read (retry) a corrupt block: group=4(XIFENFEI) dsk=15 blk=258 disk=15 (XIFENFEI_0015) incarn=1717056824 au=113792 blk=2 count=1 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_x000_28639240.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483663] [258] [56 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483663] [258] [56 != 0] ERROR: cache failed to read group=4(XIFENFEI) dsk=15 blk=258 from disk(s): 15(XIFENFEI_0015) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483663] [258] [56 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483663] [258] [56 != 0] NOTE: cache initiating offline of disk 15 group XIFENFEI NOTE: process _x000_+asm2 (28639240) initiating offline of disk 15.1717056824 (XIFENFEI_0015) with mask 0x7e in group 4 NOTE: initiating PST update: grp = 4, dsk = 15/0x66583538, mask = 0x6a, op = clear GMON updating disk modes for group 4 at 23 for pid 28, osid 28639240 ERROR: Disk 15 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 4) Sun Nov 24 04:42:27 2019 NOTE: cache dismounting (not clean) group 4/0x0B08C40B (XIFENFEI) WARNING: Offline for disk XIFENFEI_0015 in mode 0x7f failed. Sun Nov 24 04:42:27 2019 NOTE: halting all I/Os to diskgroup 4 (XIFENFEI) NOTE: messaging CKPT to quiesce pins Unix process pid: 59441780, image: oracle@xifenfei2 (B000) Sun Nov 24 04:42:27 2019 ERROR: ORA-15130 thrown in ARB0 for group number 4 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_arb0_50856926.trc: ORA-15130: diskgroup "XIFENFEI" is being dismounted
至此两个节点的该磁盘组就陷入了不停的mount,然后dismount的轮流循环中.这里我们可以大概的分析出来,由于vg的磁盘组被写入了数据或者强制剔除的时候导致asm写入该文件的数据被破坏,导致后续的asm reblance遭遇坏块,然后直接dismount.对于该问题的解决方案,通过对对该磁盘组的acd和cod进行patch,让其不进行reblance,保持该磁盘组现在,稳定的mount状态,然后对其数据进行备份和重建该磁盘组.这个客户运气不错,vg中的asm disk磁盘写入较少,数据库运行正常.
对于这种情况,如果发生极端损坏,比如asm磁盘组无法mount,可以参考:找回ASM中数据文件
如果是asm的元数据大量损坏,无法通过asm字典级别恢复,可以通过参考:asm disk header 彻底损坏恢复
发表在 Oracle ASM
标签为 asm vg异常, endian_kfbh, kfc.c:26368, ORA-15196, ORA-15196: invalid ASM block header
评论关闭
asm 加磁盘导致磁盘组损坏恢复
接到客户恢复case请求,希望我们接入恢复数据。大概过程是这样的,16年9月份由于硬件问题,导致normal磁盘组(只有2个磁盘)中的一个磁盘丢失,然后在17年3月6日,运维方尝试增加该磁盘进入磁盘组,结果通过force命令加入成功之后,磁盘组dismount,然后再也无法mount成功。
磁盘组创建信息
Fri Jun 24 19:31:38 2016 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2 SUCCESS: diskgroup DATADG was mounted SUCCESS: CREATE DISKGROUP DATADG NORMAL REDUNDANCY DISK '/dev/asm-diskdata01' SIZE 1048576M , '/dev/asm-diskdata02' SIZE 1048576M ATTRIBUTE 'compatible.asm'='11.2.0.0.0','au_size'='4M' /* ASMCA */
这里可以看出来datadg是一个normal的au为4M的一个磁盘组
自动drop异常asm disk
Mon Sep 12 11:41:54 2016 WARNING: Waited 15 secs for write IO to PST disk 1 in group 1. WARNING: Waited 15 secs for write IO to PST disk 1 in group 1. Mon Sep 12 11:41:55 2016 NOTE: process _b000_+asm1 (19491) initiating offline of disk 1.3915923833 (DATADG_0001) with mask 0x7e in group 1 NOTE: checking PST: grp = 1 GMON checking disk modes for group 1 at 9 for pid 29, osid 19491 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: checking PST for grp 1 done. NOTE: sending set offline flag message 2870990318 to 1 disk(s) in group 1 WARNING: Disk DATADG_0001 in mode 0x7f is now being offlined NOTE: initiating PST update: grp = 1, dsk = 1/0xe9684179, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 10 for pid 29, osid 19491 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 1 completed successfully NOTE: initiating PST update: grp = 1, dsk = 1/0xe9684179, mask = 0x7e, op = clear GMON updating disk modes for group 1 at 11 for pid 29, osid 19491 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: cache closing disk 1 of grp 1: DATADG_0001 NOTE: PST update grp = 1 completed successfully Mon Sep 12 11:42:55 2016 WARNING: Waited 15 secs for write IO to PST disk 0 in group 1. Mon Sep 12 11:44:58 2016 WARNING: PST-initiated drop of 1 disk(s) in group 1(.1137226115)) SQL> alter diskgroup DATADG drop disk DATADG_0001 force /* ASM SERVER */ NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=1 Mon Sep 12 11:44:59 2016 GMON updating for reconfiguration, group 1 at 12 for pid 29, osid 19491 NOTE: cache closing disk 1 of grp 1: (not open) DATADG_0001 NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group 1 PST updated. Mon Sep 12 11:44:59 2016 NOTE: membership refresh pending for group 1/0x43c8b183 (DATADG) Mon Sep 12 11:45:02 2016 NOTE: successfully read ACD block gn=1 blk=0 via retry read Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3526.trc: ORA-15062: ASM disk is globally closed GMON querying group 1 at 13 for pid 18, osid 3532 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x43c8b183 (DATADG) SUCCESS: alter diskgroup DATADG drop disk DATADG_0001 force /* ASM SERVER */ NOTE: starting rebalance of group 1/0x43c8b183 (DATADG) at power 1 SUCCESS: PST-initiated drop disk in group 1(1137226115)) Starting background process ARB0 Mon Sep 12 11:45:03 2016 ARB0 started with pid=35, OS id=19945 NOTE: assigning ARB0 to group 1/0x43c8b183 (DATADG) with 1 parallel I/O cellip.ora not found. NOTE: Rebalance has restored redundancy for any existing control file or redo log in disk group DATADG NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Sep 12 11:46:21 2016 NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=1 Mon Sep 12 11:46:24 2016 GMON updating for reconfiguration, group 1 at 14 for pid 36, osid 20110 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG NOTE: group 1 PST updated. WARNING: offline disk number 1 has references (54679 AUs) Mon Sep 12 11:46:24 2016 NOTE: membership refresh pending for group 1/0x43c8b183 (DATADG) GMON querying group 1 at 15 for pid 18, osid 3532 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x43c8b183 (DATADG) NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. NOTE: stopping process ARB0 SUCCESS: rebalance completed for group 1/0x43c8b183 (DATADG)
这里我们可以看出来磁盘组在2016年9月12日由于disk 1 无法响应,直接被asm 踢出了磁盘组
把被强制删除的磁盘重新加回去
Mon Mar 06 15:36:54 2017 SQL> alter diskgroup DATADG add disk '/dev/asm-diskdata01' name DATADG_0000 NOTE: GroupBlock outside rolling migration privileged region ORA-15032: not all alterations performed ORA-15029: disk '/dev/asm-diskdata01' is already mounted by this instance ERROR: alter diskgroup DATADG add disk '/dev/asm-diskdata01' name DATADG_0000 Mon Mar 06 15:38:27 2017 SQL> alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: Disk DATADG_0001 in mode 0x7f marked for de-assignment ERROR: ORA-15033 signalled during reconfiguration of diskgroup DATADG Mon Mar 06 15:38:28 2017 NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) Mon Mar 06 15:38:31 2017 GMON querying group 1 at 7 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG GMON querying group 1 at 8 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) ORA-15032: not all alterations performed ORA-15033: disk '/dev/asm-diskdata02' belongs to diskgroup "DATADG" ERROR: alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Mar 06 16:04:14 2017 SQL> alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: Disk DATADG_0001 in mode 0x7f marked for de-assignment ERROR: ORA-15033 signalled during reconfiguration of diskgroup DATADG Mon Mar 06 16:04:15 2017 NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) Mon Mar 06 16:04:18 2017 GMON querying group 1 at 9 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG GMON querying group 1 at 10 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) ORA-15032: not all alterations performed ORA-15033: disk '/dev/asm-diskdata02' belongs to diskgroup "DATADG" ERROR: alter diskgroup DATADG add disk '/dev/asm-diskdata02' name DATADG_0001 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Mar 06 16:23:28 2017 SQL> alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/adm-diskdata02' name DATA_0001 NOTE: GroupBlock outside rolling migration privileged region ORA-15032: not all alterations performed ORA-15031: disk specification '/dev/adm-diskdata02' matches no disks ERROR: alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/adm-diskdata02' name DATA_0001 Mon Mar 06 16:24:48 2017 SQL> alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment ERROR: ORA-15033 signalled during reconfiguration of diskgroup DATADG Mon Mar 06 16:24:49 2017 NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) Mon Mar 06 16:24:52 2017 GMON querying group 1 at 11 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG GMON querying group 1 at 12 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) ORA-15032: not all alterations performed ORA-15033: disk '/dev/asm-diskdata02' belongs to diskgroup "DATADG" ERROR: alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. Mon Mar 06 16:26:07 2017 SQL> alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 force NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: requesting all-instance membership refresh for group=1 NOTE: initializing header on grp 1 disk DATA_0001 NOTE: requesting all-instance disk validation for group=1 Mon Mar 06 16:26:10 2017 NOTE: skipping rediscovery for group 1/0x31584f6b (DATADG) on local instance. NOTE: requesting all-instance disk validation for group=1 NOTE: skipping rediscovery for group 1/0x31584f6b (DATADG) on local instance. Mon Mar 06 16:26:15 2017 GMON updating for reconfiguration, group 1 at 13 for pid 28, osid 12861 NOTE: group 1 PST updated. NOTE: initiating PST update: grp = 1 GMON updating group 1 at 14 for pid 28, osid 12861 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG NOTE: group DATADG: updated PST location: disk 0000 (PST copy 0) NOTE: group DATADG: updated PST location: disk 0002 (PST copy 1) NOTE: PST update grp = 1 completed successfully NOTE: membership refresh pending for group 1/0x31584f6b (DATADG) GMON querying group 1 at 15 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG NOTE: cache opening disk 2 of grp 1: DATA_0001 path:/dev/asm-diskdata02 NOTE: Attempting voting file refresh on diskgroup DATADG NOTE: Refresh completed on diskgroup DATADG. No voting file found. GMON querying group 1 at 16 for pid 18, osid 3468 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG SUCCESS: refreshed membership for 1/0x31584f6b (DATADG) Mon Mar 06 16:26:19 2017 SUCCESS: alter diskgroup DATADG add FAILGROUP DATA_0001 disk '/dev/asm-diskdata02' name DATA_0001 force NOTE: starting rebalance of group 1/0x31584f6b (DATADG) at power 1 Mon Mar 06 16:26:20 2017 Starting background process ARB0 Mon Mar 06 16:26:20 2017 ARB0 started with pid=32, OS id=25833 NOTE: assigning ARB0 to group 1/0x31584f6b (DATADG) with 1 parallel I/O cellip.ora not found. WARNING:cache read a corrupt block: group=1(DATADG) dsk=0 blk=0 disk=0 (DATADG_0000)incarn=3915956130 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] NOTE:a corrupted block from group DATADG was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc WARNING:cache read (retry)a corrupt block:group=1(DATADG) dsk=0 blk=0 disk=0(DATADG_0000)incarn=3915956130 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ERROR: cache failed to read group=1(DATADG) dsk=0 blk=0 from disk(s): 0(DATADG_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] NOTE: cache initiating offline of disk 0 group DATADG NOTE:process _arb0_+asm1 (25833) initiating offline of disk 0.3915956130(DATADG_0000)with mask 0x7e in group 1 NOTE: checking PST: grp = 1 GMON checking disk modes for group 1 at 17 for pid 32, osid 25833 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: too many offline disks in PST (grp 1) NOTE: checking PST for grp 1 done. NOTE: initiating PST update: grp = 1, dsk = 0/0xe968bfa2, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 18 for pid 32, osid 25833 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: Disk 0 cannot be offlined, since all the disks [0, 1] with mirrored data would be offline. ERROR: too many offline disks in PST (grp 1) Mon Mar 06 16:26:23 2017 NOTE: cache dismounting (not clean) group 1/0x31584F6B (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 25889, image: oracle@DBN01 (B000) Mon Mar 06 16:26:23 2017 NOTE: halting all I/Os to diskgroup 1 (DATADG) Mon Mar 06 16:26:23 2017 NOTE: LGWR doing non-clean dismount of group 1 (DATADG) NOTE: LGWR sync ABA=19.2851 last written ABA 19.2851 WARNING: Offline for disk DATADG_0000 in mode 0x7f failed. Mon Mar 06 16:26:23 2017 kjbdomdet send to inst 2 detach from dom 1, sending detach message to inst 2 Mon Mar 06 16:26:23 2017 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 1, cluster inc 8) Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_25833.trc (incident=65537): ORA-15335: ASM metadata corruption detected in disk group 'DATADG' ORA-15130: diskgroup "DATADG" is being dismounted ORA-15066: offlining disk "DATADG_0000" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483648] [0] [0 != 1] Incident details in:/u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_65537/+ASM1_arb0_25833_i65537.trc Global Resource Directory partially frozen for dirty detach * dirty detach - domain 1 invalid = TRUE 3189 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete ERROR: ORA-15130 in COD recovery for diskgroup 1/0x31584f6b (DATADG) ERROR: ORA-15130 thrown in RBAL for group number 1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_3468.trc: ORA-15130: diskgroup "DATADG" is being dismounted Mon Mar 06 16:26:23 2017 WARNING: dirty detached from domain 1 NOTE: cache dismounted group 1/0x31584F6B (DATADG) ---后续mount报错 SQL> ALTER DISKGROUP DATADG MOUNT /* asm agent *//* {1:18003:2} */ NOTE: cache registered group DATADG number=1 incarn=0xb368408f NOTE: cache began mount (first) of group DATADG number=1 incarn=0xb368408f NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) WARNING:GMON has insufficient disks to maintain consensus. Minimum required is 2:updating 1 PST copies from a total of 2. ERROR: GMON failed to obtain a quorum ofsupporting disks in group 1 NOTE: cache dismounting (clean) group 1/0xB368408F (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 27651, image: oracle@DBN01 (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xB368408F (DATADG) NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xb368408f NOTE: cache deleting context for group DATADG 1/0xb368408f GMON dismounting group 1 at 12 for pid 30, osid 27651 NOTE: Disk DATA_0001 in mode 0x9 marked for de-assignment ERROR: diskgroup DATADG was not mounted ORA-15032: not all alterations performed ORA-15017: diskgroup "DATADG" cannot be mounted ORA-15315: Write errors in disk group DATADG could lead to inconsistent ASM metadata. ERROR: ALTER DISKGROUP DATADG MOUNT /* asm agent *//* {1:18003:2} */
从这里我们可以看出来,前几次加asm disk 由于各种原因都失败了,最后一次通过加force关键字,使得被自动drop的disk重新强制加到datadg里面.可悲的是在加入成功之后,开始做rebalance的时候,发现disk 0出现坏块,从而引起ORA-15196的错误,使得rebalance无法进行下去,进而整个asm 磁盘组datadg自动dismount.后面再次尝试mount datadg的时候,直接提示元数据库不一致,因为disk 0 的磁盘头已经异常.
通过kfed分析disk 0信息
这里是通过dd命令备份的磁盘头到win进行分析的,以前正常的disk 0的磁盘头损坏(全0)
对于这个故障已经比较清楚,恢复思路也基本上确定:依次递进
方案1:通过kfed修改文件头,然后尝试mount磁盘头手工修复ASM DISK HEADER 异常
方案2:直接通过amdu,dul之类的工具拷贝出来数据文件找回ASM中数据文件
方案3:通过底层au重组出来数据文件asm disk header 彻底损坏恢复
在我们的实际恢复中运气比较好,通过方案1就完成了恢复工作,通过kfed修复磁盘头之后,然后报错如下
SQL> alter diskgroup DATADG mount NOTE: cache registered group DATADG number=1 incarn=0x5134d0d4 NOTE: cache began mount (first) of group DATADG number=1 incarn=0x5134d0d4 NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: Assigning number (1,0) to disk (/dev/asm-diskdata01) Tue Mar 07 19:03:40 2017 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 27 for pid 28, osid 13837 NOTE: Assigning number (1,1) to disk () GMON querying group 1 at 28 for pid 28, osid 13837 NOTE: cache closing disk 1 of grp 1: (not open) NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diskdata01 NOTE: F1X0 found on disk 0 au 2 fcn 0.178802 NOTE: cache opening disk 2 of grp 1: DATA_0001 path:/dev/asm-diskdata02 NOTE: cache mounting (first) normal redundancy group 1/0x5134D0D4 (DATADG) Tue Mar 07 19:03:40 2017 * allocate domain 1, invalid = TRUE kjbdomatt send to inst 2 Tue Mar 07 19:03:40 2017 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=19.2851 group=1 (DATADG) NOTE: starting recovery of thread=2 ckpt=13.5327 group=1 (DATADG) NOTE: advancing ckpt for group 1 (DATADG) thread=2 ckpt=13.5327 NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=19.2852 NOTE: cache recovered group 1 to fcn 0.365868 NOTE: redo buffer size is 512 blocks (2101760 bytes) Tue Mar 07 19:03:40 2017 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG) NOTE: LGWR found thread 1 closed at ABA 19.2851 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG) NOTE: LGWR opening thread 1 at fcn 0.365868 ABA 20.2852 NOTE: cache mounting group 1/0x5134D0D4 (DATADG) succeeded NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x5134d0d4 Tue Mar 07 19:03:40 2017 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATADG was mounted SUCCESS: alter diskgroup DATADG mount Tue Mar 07 19:03:40 2017 NOTE: diskgroup resource ora.DATADG.dg is online Tue Mar 07 19:03:41 2017 ASM Health Checker found 1 new failures NOTE: ASM did background COD recovery for group 1/0x5134d0d4 (DATADG) NOTE: starting rebalance of group 1/0x5134d0d4 (DATADG) at power 1 Starting background process ARB0 Tue Mar 07 19:03:42 2017 ARB0 started with pid=30, OS id=13905 NOTE: assigning ARB0 to group 1/0x5134d0d4 (DATADG) with 1 parallel I/O cellip.ora not found. WARNING: cache read a corrupt block: group=1(DATADG) dsk=0 blk=0 disk=0 (DATADG_0000) incarn=2202280062 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] NOTE: a corrupted block from group DATADG was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc WARNING:cache read (retry)a corrupt block:group=1(DATADG) dsk=0 blk=0 disk=0 (DATADG_0000)incarn=2202280062 au=0 blk=0 count=1 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ERROR: cache failed to read group=1(DATADG) dsk=0 blk=0 from disk(s): 0(DATADG_0000) ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] Tue Mar 07 19:03:52 2017 NOTE: client oradb1:oradb registered, osid 13989, mbr 0x1 NOTE: cache initiating offline of disk 0 group DATADG NOTE:process _arb0_+asm1 (13905) initiating offline of disk 0.2202280062(DATADG_0000)with mask 0x7e in group 1 NOTE: checking PST: grp = 1 Tue Mar 07 19:03:52 2017 GMON checking disk modes for group 1 at 30 for pid 30, osid 13905 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: too many offline disks in PST (grp 1) NOTE: checking PST for grp 1 done. NOTE: initiating PST update: grp = 1, dsk = 0/0x8344207e, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 31 for pid 30, osid 13905 NOTE: cache closing disk 1 of grp 1: (not open) _DROPPED_0001_DATADG ERROR: Disk 0 cannot be offlined, since all the disks [0, 1] with mirrored data would be offline. ERROR: too many offline disks in PST (grp 1) Tue Mar 07 19:03:52 2017 NOTE: cache dismounting (not clean) group 1/0x5134D0D4 (DATADG) WARNING: Offline for disk DATADG_0000 in mode 0x7f failed. Tue Mar 07 19:03:52 2017 NOTE: halting all I/Os to diskgroup 1 (DATADG) NOTE: messaging CKPT to quiesce pins Unix process pid: 14002, image: oracle@DBN01 (B000) Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_13905.trc (incident=76402): ORA-15335: ASM metadata corruption detected in disk group 'DATADG' ORA-15130: diskgroup "DATADG" is being dismounted ORA-15066: offlining disk "DATADG_0000" in group "DATADG" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] ORA-15196: invalid ASM block header [kfc.c:26368] [blk_kfbl] [2147483648] [0] [1022 != 0] Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_76402/+ASM1_arb0_13905_i76402.trc Tue Mar 07 19:03:52 2017 NOTE: LGWR doing non-clean dismount of group 1 (DATADG) NOTE: LGWR sync ABA=20.2857 last written ABA 20.2857
这里比较比较幸运,datadg已经mount成功了,但是由于rab依旧读取到disk header异常信息(没有完全修复成功,而且在日志中不光这个block异常,还有其他block异常,因此不考虑进一步修复),因此直接通过屏蔽asm的acd和cod实现该磁盘组mount,而且不会dismount。
SQL> alter diskgroup DATADG mount NOTE: cache registered group DATADG number=1 incarn=0x9c94d0eb NOTE: cache began mount (first) of group DATADG number=1 incarn=0x9c94d0eb NOTE: Assigning number (1,2) to disk (/dev/asm-diskdata02) NOTE: Assigning number (1,0) to disk (/dev/asm-diskdata01) NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639 Tue Mar 07 19:12:45 2017 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 75 for pid 28, osid 15615 NOTE: Assigning number (1,1) to disk () GMON querying group 1 at 76 for pid 28, osid 15615 NOTE: cache closing disk 1 of grp 1: (not open) NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diskdata01 NOTE: F1X0 found on disk 0 au 2 fcn 0.178802 NOTE: cache opening disk 2 of grp 1: DATA_0001 path:/dev/asm-diskdata02 NOTE: cache mounting (first) normal redundancy group 1/0x9C94D0EB (DATADG) Tue Mar 07 19:12:45 2017 * allocate domain 1, invalid = TRUE kjbdomatt send to inst 2 Tue Mar 07 19:12:45 2017 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=25.2870 group=1 (DATADG) NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=25.2873 NOTE: cache recovered group 1 to fcn 0.365897 NOTE: redo buffer size is 512 blocks (2101760 bytes) Tue Mar 07 19:12:45 2017 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG) NOTE: LGWR found thread 1 closed at ABA 25.2872 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG) NOTE: LGWR opening thread 1 at fcn 0.365897 ABA 26.2873 NOTE: cache mounting group 1/0x9C94D0EB (DATADG) succeeded NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x9c94d0eb Tue Mar 07 19:12:45 2017 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATADG was mounted SUCCESS: alter diskgroup DATADG mount Tue Mar 07 19:12:45 2017 NOTE: diskgroup resource ora.DATADG.dg is online NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639 NOTE: skip COD recovery as part of test at kfrc.c:1639
asm的问题解决后,然后登录数据库,发现运气比较好,两个数据库正常open成功,而且alert日志无任何报错,直接通过rman备份出来数据,重建asm磁盘组,还原数据,恢复完成,而且实现数据0丢失。
如果您遇到此类情况,无法解决请联系我们,提供专业ORACLE数据库恢复技术支持
Phone:13429648788 Q Q:107644445 E-Mail:dba@xifenfei.com