分类目录归档：Oracle ASM

ORA-15063: ASM discovered an insufficient number of disks for diskgroup 恢复

发表于 2022 年 6 月 21 日 由 惜分飞

客户反馈三个磁盘组无法正常mount,报错类似ORA-15032 ORA-15017 ORA-15063

SQL> ALTER DISKGROUP ASM_DATA MOUNT  /* asm agent *//* {0:0:2} */ 
NOTE: cache registered group ASM_DATA number=1 incarn=0xffa85ccd
NOTE: cache began mount (first) of group ASM_DATA number=1 incarn=0xffa85ccd
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: cache dismounting (clean) group 1/0xFFA85CCD (ASM_DATA) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 5709, image: oracle@XFF (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 1/0xFFA85CCD (ASM_DATA) 
NOTE: cache ending mount (fail) of group ASM_DATA number=1 incarn=0xffa85ccd
NOTE: cache deleting context for group ASM_DATA 1/0xffa85ccd
Tue Jun 21 12:24:38 2022
NOTE: No asm libraries found in the system
ASM Health Checker found 1 new failures
GMON dismounting group 1 at 16 for pid 19, osid 5709
ERROR: diskgroup ASM_DATA was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "ASM_DATA" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ASM_DATA"
ERROR: ALTER DISKGROUP ASM_DATA MOUNT  /* asm agent *//* {0:0:2} */

初步判断是asm disk异常导致(比如asm disk不能被扫描到,或者丢失,或者磁盘头损坏等),分析客户的asm disk的udev文件配置

KERNEL=="sdd1", NAME="asm_grid", OWNER="grid", GROUP="asmadmin", MODE="0660"          
KERNEL=="sde1", NAME="asm_system", OWNER="grid", GROUP="asmadmin", MODE="0660"    
KERNEL=="sdf1", NAME="asm_data", OWNER="grid", GROUP="asmadmin", MODE="0660"

从udev的配置中可以看出来,客户以前是对3个磁盘进行分析,然后使用udev映射别名给asm使用的.通过对其中一个磁盘进行分析
20220621220634
20220621220728

通过上述winhex查看，可以确认该分区的磁盘头信息异常[该信息属于磁盘刚分区的时候信息,而不是asm disk的信息],和kfed看到的结果一致[磁盘头位置肯定损坏,其他位置目前未知]

H:\TEMP\dd>kfed read sdf_sdf1.dd
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
0064D8400 00000000 00000000 00000000 00000000  [................]
        Repeat 26 times
0064D85B0 00000000 00000000 00000000 02000000  [................]
0064D85C0 FE8E0001 003FFFFF DFFC0000 0000257F  [......?......%..]
0064D85D0 00000000 00000000 00000000 00000000  [................]
        Repeat 1 times
0064D85F0 00000000 00000000 00000000 AA550000  [..............U.]
0064D8600 00000000 00000000 00000000 00000000  [................]
  Repeat 223 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

分析其他位置的block情况,初步看基本上ok[运气还不错]

H:\TEMP\dd>kfed read sdf_sdf1.dd blkn=2|grep kfbh.type
kfbh.type:                            3 ; 0x002: KFBTYP_ALLOCTBL

H:\TEMP\dd>kfed read sdf_sdf1.dd blkn=3|grep kfbh.type
kfbh.type:                            3 ; 0x002: KFBTYP_ALLOCTBL

H:\TEMP\dd>kfed read sdf_sdf1.dd blkn=1 aun=2|grep kfbh.type
kfbh.type:                            3 ; 0x002: KFBTYP_ALLOCTBL

通过检索备份出来的部分磁盘文件,找出来ORCLDISK信息部分(asm disk header)
20220621221843

然后利用这个部分对损坏的磁盘头进行修复,并且dd回生产环境中,并尝试mount磁盘组,数据库open成功
20220621181430
20220621222356

至此这个数据库运气不错,没有过多损坏,算完美恢复,可以进行了逻辑导出和rman备份,全部正常.为了后续安全,建议对其进行迁移

发表在 Oracle ASM, Oracle备份恢复 | 标签为 , | 评论关闭

pvcreate asm disk导致asm磁盘组异常恢复

发表于 2022 年 5 月 29 日 由 惜分飞

一客户asm磁盘组异常，无法正常mount

SQL> alter diskgroup datadg mount 
2022-05-28T19:08:55.114960+08:00
NOTE: cache registered group DATADG 1/0x2B504997
NOTE: cache began mount (first) of group DATADG 1/0x2B504997
NOTE: Assigning number (1,3) to disk (/dev/oracleasm/disks/DATA05)
NOTE: Assigning number (1,2) to disk (/dev/oracleasm/disks/DATA03)
NOTE: Assigning number (1,1) to disk (/dev/oracleasm/disks/DATA02)
2022-05-28T19:08:55.150062+08:00
ERROR: no read quorum in group: required 1, found 0 disks
2022-05-28T19:08:55.150684+08:00
NOTE: cache dismounting (clean) group 1/0x2B504997 (DATADG)
NOTE: messaging CKPT to quiesce pins Unix process pid: 15103, image: oracle@XFF01 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 1/0x2B504997 (DATADG)
NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0x2b504997
NOTE: cache deleting context for group DATADG 1/0x2b504997
2022-05-28T19:08:55.191073+08:00
GMON dismounting group 1 at 36 for pid 37, osid 15103
2022-05-28T19:08:55.191258+08:00
NOTE: Disk DATA02 in mode 0x8 marked for de-assignment
NOTE: Disk DATA03 in mode 0x8 marked for de-assignment
NOTE: Disk DATA05 in mode 0x8 marked for de-assignment
ERROR: diskgroup DATADG was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATADG" cannot be mounted
ORA-15040: diskgroup is incomplete

通过报错信息,初步判断是由于少了asm disk导致（依据:1. ORA-15040,2.asmlib中的DATA01丢失),初步判断由于某种原因导致asmlib的磁盘异常,从而使得asm磁盘组无法正常mount,通过对dd 到本地的asm磁盘进行分析

C:\Users\XFF>kfed read H:\TEMP\asmdd\sdb6-o.dd
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
0066E8200 00000000 00000000 00000000 00000000  [................]
        Repeat 31 times
0066E8400 4542414C 454E4F4C 00000001 00000000  [LABELONE........]
0066E8410 4E06D490 00000020 324D564C 31303020  [...N ...LVM2 001]
0066E8420 34535542 476A7667 42546C48 6D384675  [BUS4gvjGHlTBuF8m]
0066E8430 7A385273 4B495777 73336242 33637449  [sR8zwWIKBb3sItc3]
0066E8440 48001000 000001E8 00100000 00000000  [...H............]
0066E8450 00000000 00000000 00000000 00000000  [................]
0066E8460 00000000 00000000 00001000 00000000  [................]
0066E8470 000FF000 00000000 00000000 00000000  [................]
0066E8480 00000000 00000000 00000002 00000000  [................]
0066E8490 00000000 00000000 00000000 00000000  [................]
  Repeat 214 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

通过这部分信息可以确认，一个asm disk被创建了pv,进一步分析pv信息
pv

对于这样的情况，表示asm disk被创建了pv但是pv没有加入到任何vg中,也就意味着该disk没有太大破坏,通过信息确认
20220529124428
20220529124734
主要是这两个部分信息被损坏,可以通过一些方法对这两个block信息进行重构

C:\Users\XFF>kfed read H:\TEMP\asmdd\sdb6.dd|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:              2147483648 ; 0x008: disk=0
kfbh.check:                  3196491921 ; 0x00c: 0xbe869891
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:   ORCLDISKDATA01 ; 0x000: length=14
kfdhdb.driver.reserved[0]:   1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved[1]:        12592 ; 0x00c: 0x00003130
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                203424000 ; 0x020: 0x0c200100
kfdhdb.dsknum:                        0 ; 0x024: 0x0000
kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:                  DATA01 ; 0x028: length=6
kfdhdb.grpname:                  DATADG ; 0x048: length=6
kfdhdb.fgname:                   DATA01 ; 0x068: length=6
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             33083792 ; 0x0a8: HOUR=0x10 DAYS=0xc MNTH=0x4 YEAR=0x7e3
kfdhdb.crestmp.lo:           2268043264 ; 0x0ac: USEC=0x0 MSEC=0x3e6 SECS=0x32 MINS=0x21
kfdhdb.mntstmp.hi:             33134479 ; 0x0b0: HOUR=0xf DAYS=0x1c MNTH=0x5 YEAR=0x7e6
-- More  --

C:\Users\XFF>kfed read H:\TEMP\asmdd\sdb6.dd blkn=1|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            2 ; 0x002: KFBTYP_FREESPC
kfbh.datfmt:                          2 ; 0x003: 0x02
kfbh.block.blk:                       1 ; 0x004: blk=1
kfbh.block.obj:              2147483648 ; 0x008: disk=0
kfbh.check:                  2177715180 ; 0x00c: 0x81cd4bec
kfbh.fcn.base:                  3721754 ; 0x010: 0x0038ca1a
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdfsb.aunum:                         0 ; 0x000: 0x00000000
kfdfsb.max:                        1014 ; 0x004: 0x03f6
kfdfsb.cnt:                        1014 ; 0x006: 0x03f6
kfdfsb.bound:                         0 ; 0x008: 0x0000
kfdfsb.flag:                          1 ; 0x00a: B=1
kfdfsb.ub1spare:                      0 ; 0x00b: 0x00
kfdfsb.spare[0]:                      0 ; 0x00c: 0x00000000
kfdfsb.spare[1]:                      0 ; 0x010: 0x00000000
kfdfsb.spare[2]:                      0 ; 0x014: 0x00000000
kfdfse[0].fse:                        0 ; 0x018: FREE=0x0 FRAG=0x0
kfdfse[1].fse:                        0 ; 0x019: FREE=0x0 FRAG=0x0
kfdfse[2].fse:                        0 ; 0x01a: FREE=0x0 FRAG=0x0
kfdfse[3].fse:                        0 ; 0x01b: FREE=0x0 FRAG=0x0
kfdfse[4].fse:                        0 ; 0x01c: FREE=0x0 FRAG=0x0
kfdfse[5].fse:                        0 ; 0x01d: FREE=0x0 FRAG=0x0
kfdfse[6].fse:                        0 ; 0x01e: FREE=0x0 FRAG=0x0
kfdfse[7].fse:                        0 ; 0x01f: FREE=0x0 FRAG=0x0
kfdfse[8].fse:                        0 ; 0x020: FREE=0x0 FRAG=0x0

通过dd写入到原磁盘,通过oracleasm scandisks扫描磁盘
scandisks

磁盘组mount成功
mount
数据库顺利open
20220529140558

20220529140813

这个案例能够完美恢复,主要是客户没有做进一步破坏，没有把这个pv加入到vg中并且写入数据,以前有过类似案例因为写入了数据,恢复比这个难多了,效果也没有这个好asm disk被加入vg恢复
如果不幸有类似oracle asm disk被破坏(格式化,dd部分,做成lv等),需要进行恢复支持,可以联系我们,做专业的恢复评估,最大限度,最快速度抢救数据,减少损失
Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com
恢复过部分asm异常案例:
删除分区 oracle asm disk 恢复
asm disk 磁盘部分被清空恢复
又一例asm格式化文件系统恢复
一次完美的asm disk被格式化ntfs恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
oracle asm disk格式化恢复—格式化为ntfs文件系统
分享oracleasm createdisk重新创建asm disk后数据0丢失恢复案例

发表在 Oracle, Oracle ASM | 标签为 , , , , | 评论关闭

ORA-15130: diskgroup “ORADATA” is being dismounted

发表于 2021 年 12 月 27 日 由 惜分飞

磁盘组mount之后,立马又dismount

Sat Dec 25 17:48:45 2021
SQL> alter diskgroup ORADATA mount 
NOTE: cache registered group ORADATA number=5 incarn=0xd4b7ac6a
NOTE: cache began mount (first) of group ORADATA number=5 incarn=0xd4b7ac6a
NOTE: Assigning number (5,24) to disk (/dev/mapper/data31)
NOTE: Assigning number (5,26) to disk (/dev/mapper/data33)
NOTE: Assigning number (5,21) to disk (/dev/mapper/data29)
NOTE: Assigning number (5,23) to disk (/dev/mapper/data30)
NOTE: Assigning number (5,25) to disk (/dev/mapper/data32)
NOTE: Assigning number (5,19) to disk (/dev/mapper/data27)
NOTE: Assigning number (5,20) to disk (/dev/mapper/data28)
NOTE: Assigning number (5,18) to disk (/dev/mapper/data26)
NOTE: Assigning number (5,14) to disk (/dev/mapper/data22)
NOTE: Assigning number (5,17) to disk (/dev/mapper/data25)
NOTE: Assigning number (5,16) to disk (/dev/mapper/data24)
NOTE: Assigning number (5,15) to disk (/dev/mapper/data23)
NOTE: Assigning number (5,13) to disk (/dev/mapper/data21)
NOTE: Assigning number (5,12) to disk (/dev/mapper/data20)
NOTE: Assigning number (5,10) to disk (/dev/mapper/data19)
NOTE: Assigning number (5,9) to disk (/dev/mapper/data18)
NOTE: Assigning number (5,8) to disk (/dev/mapper/data17)
NOTE: Assigning number (5,3) to disk (/dev/mapper/data12)
NOTE: Assigning number (5,22) to disk (/dev/mapper/data3)
NOTE: Assigning number (5,2) to disk (/dev/mapper/data11)
NOTE: Assigning number (5,7) to disk (/dev/mapper/data16)
NOTE: Assigning number (5,28) to disk (/dev/mapper/data5)
NOTE: Assigning number (5,32) to disk (/dev/mapper/data9)
NOTE: Assigning number (5,6) to disk (/dev/mapper/data15)
NOTE: Assigning number (5,5) to disk (/dev/mapper/data14)
NOTE: Assigning number (5,4) to disk (/dev/mapper/data13)
NOTE: Assigning number (5,1) to disk (/dev/mapper/data10)
NOTE: Assigning number (5,30) to disk (/dev/mapper/data7)
NOTE: Assigning number (5,29) to disk (/dev/mapper/data6)
NOTE: Assigning number (5,31) to disk (/dev/mapper/data8)
NOTE: Assigning number (5,11) to disk (/dev/mapper/data2)
NOTE: Assigning number (5,27) to disk (/dev/mapper/data4)
NOTE: Assigning number (5,0) to disk (/dev/mapper/data1)
Sat Dec 25 17:48:52 2021
NOTE: GMON heartbeating for grp 5
GMON querying group 5 at 153 for pid 32, osid 68608
NOTE: cache opening disk 0 of grp 5: ORADATA_0000 path:/dev/mapper/data1
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache opening disk 1 of grp 5: ORADATA_0001 path:/dev/mapper/data10
NOTE: cache opening disk 2 of grp 5: ORADATA_0002 path:/dev/mapper/data11
NOTE: cache opening disk 3 of grp 5: ORADATA_0003 path:/dev/mapper/data12
NOTE: cache opening disk 4 of grp 5: ORADATA_0004 path:/dev/mapper/data13
NOTE: cache opening disk 5 of grp 5: ORADATA_0005 path:/dev/mapper/data14
NOTE: cache opening disk 6 of grp 5: ORADATA_0006 path:/dev/mapper/data15
NOTE: cache opening disk 7 of grp 5: ORADATA_0007 path:/dev/mapper/data16
NOTE: cache opening disk 8 of grp 5: ORADATA_0008 path:/dev/mapper/data17
NOTE: cache opening disk 9 of grp 5: ORADATA_0009 path:/dev/mapper/data18
NOTE: cache opening disk 10 of grp 5: ORADATA_0010 path:/dev/mapper/data19
NOTE: cache opening disk 11 of grp 5: ORADATA_0011 path:/dev/mapper/data2
NOTE: cache opening disk 12 of grp 5: ORADATA_0012 path:/dev/mapper/data20
NOTE: cache opening disk 13 of grp 5: ORADATA_0013 path:/dev/mapper/data21
NOTE: cache opening disk 14 of grp 5: ORADATA_0014 path:/dev/mapper/data22
NOTE: cache opening disk 15 of grp 5: ORADATA_0015 path:/dev/mapper/data23
NOTE: cache opening disk 16 of grp 5: ORADATA_0016 path:/dev/mapper/data24
NOTE: cache opening disk 17 of grp 5: ORADATA_0017 path:/dev/mapper/data25
NOTE: cache opening disk 18 of grp 5: ORADATA_0018 path:/dev/mapper/data26
NOTE: cache opening disk 19 of grp 5: ORADATA_0019 path:/dev/mapper/data27
NOTE: cache opening disk 20 of grp 5: ORADATA_0020 path:/dev/mapper/data28
NOTE: cache opening disk 21 of grp 5: ORADATA_0021 path:/dev/mapper/data29
NOTE: cache opening disk 22 of grp 5: ORADATA_0022 path:/dev/mapper/data3
NOTE: cache opening disk 23 of grp 5: ORADATA_0023 path:/dev/mapper/data30
NOTE: cache opening disk 24 of grp 5: ORADATA_0024 path:/dev/mapper/data31
NOTE: cache opening disk 25 of grp 5: ORADATA_0025 path:/dev/mapper/data32
NOTE: cache opening disk 26 of grp 5: ORADATA_0026 path:/dev/mapper/data33
NOTE: cache opening disk 27 of grp 5: ORADATA_0027 path:/dev/mapper/data4
NOTE: cache opening disk 28 of grp 5: ORADATA_0028 path:/dev/mapper/data5
NOTE: cache opening disk 29 of grp 5: ORADATA_0029 path:/dev/mapper/data6
NOTE: cache opening disk 30 of grp 5: ORADATA_0030 path:/dev/mapper/data7
NOTE: cache opening disk 31 of grp 5: ORADATA_0031 path:/dev/mapper/data8
NOTE: cache opening disk 32 of grp 5: ORADATA_0032 path:/dev/mapper/data9
NOTE: cache mounting (first) external redundancy group 5/0xD4B7AC6A (ORADATA)
Sat Dec 25 17:48:52 2021
* allocate domain 5, invalid = TRUE 
kjbdomatt send to inst 2
Sat Dec 25 17:48:52 2021
NOTE: attached to recovery domain 5
NOTE: starting recovery of thread=1 ckpt=92.6417 group=5 (ORADATA)
NOTE: advancing ckpt for group 5 (ORADATA) thread=1 ckpt=92.6418
NOTE: cache recovered group 5 to fcn 0.9502919
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Sat Dec 25 17:48:52 2021
NOTE: LGWR attempting to mount thread 1 for diskgroup 5 (ORADATA)
NOTE: LGWR found thread 1 closed at ABA 92.6417
NOTE: LGWR mounted thread 1 for diskgroup 5 (ORADATA)
NOTE: LGWR opening thread 1 at fcn 0.9502919 ABA 93.6418
NOTE: cache mounting group 5/0xD4B7AC6A (ORADATA) succeeded
NOTE: cache ending mount (success) of group ORADATA number=5 incarn=0xd4b7ac6a
Sat Dec 25 17:48:53 2021
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 5
SUCCESS: diskgroup ORADATA was mounted
SUCCESS: alter diskgroup ORADATA mount
Sat Dec 25 17:48:53 2021
NOTE: diskgroup resource ora.ORADATA.dg is online
WARNING:cache read  a corrupt block: group=5(ORADATA)dsk=5 blk=2 disk=5(ORADATA_0005)incarn=2406 au=0 blk=2 count=1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1]
NOTE: a corrupted block from group ORADATA was dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc
WARNING:cache read(retry)a corrupt block:group=5(ORADATA)dsk=5 blk=2 disk=5(ORADATA_0005)incarn=2406 au=0 blk=2 count=1
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1]
ERROR: cache failed to read group=5(ORADATA) dsk=5 blk=2 from disk(s): 5(ORADATA_0005)
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1]
NOTE: cache initiating offline of disk 5 group ORADATA
NOTE: process _rbal_+asm1 (48956) initiating offline of disk 5.240607694 (ORADATA_0005) with mask 0x7e in group 5
NOTE: initiating PST update: grp = 5, dsk = 5/0xe5761ce, mask = 0x6a, op = clear
GMON updating disk modes for group 5 at 155 for pid 18, osid 48956
ERROR: Disk 5 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 5)
Sat Dec 25 17:48:55 2021
NOTE: cache dismounting (not clean) group 5/0xD4B7AC6A (ORADATA) 
WARNING: Offline for disk ORADATA_0005 in mode 0x7f failed.
Sat Dec 25 17:48:55 2021
NOTE: halting all I/Os to diskgroup 5 (ORADATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 22744, image: oracle@wxzldb1 (B000)
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_48956.trc  (incident=1289754):
ORA-15335: ASM metadata corruption detected in disk group 'ORADATA'
ORA-15130: diskgroup "ORADATA" is being dismounted
ORA-15066: offlining disk "ORADATA_0005" in group "ORADATA" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483653] [2] [0 != 1]
Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_1289754/+ASM1_rbal_48956_i1289754.trc
NOTE: LGWR doing non-clean dismount of group 5 (ORADATA)
NOTE: LGWR sync ABA=93.6418 last written ABA 93.6418
kjbdomdet send to inst 2
detach from dom 5, sending detach message to inst 2
Sat Dec 25 17:48:56 2021
List of instances:
 1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
Sat Dec 25 17:48:56 2021
Sweep [inc][1289754]: completed
 Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 5 invalid = TRUE 
 41 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
freeing rdom 5
System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_1289754/+ASM1_rbal_48956_i1289754.trc
WARNING: dirty detached from domain 5
NOTE: cache dismounted group 5/0xD4B7AC6A (ORADATA)

问题比较明显是由于disk=5 au=0 blk=2有问题导致磁盘组mount之后立马异常.通过kfed分析对应block情况

C:\Users\XFF>kfed read h:\temp\asmdisk\data14.dd|more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:              2147483653 ; 0x008: disk=5
kfbh.check:                   314993330 ; 0x00c: 0x12c66ab2
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum:                        5 ; 0x024: 0x0005
kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:            ORADATA_0005 ; 0x028: length=12
kfdhdb.grpname:                 ORADATA ; 0x048: length=7
kfdhdb.fgname:             ORADATA_0005 ; 0x068: length=12

C:\Users\XFF>kfed read h:\temp\asmdisk\data14.dd aun=0 blkn=2|more
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
0066D8200 00000000 00000000 00000000 00000000  [................]
  Repeat 255 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

通过kfed分析,该block确实异常,该block主要记录au的分配信息,如果asm 磁盘组的空间不变化,不执行rebalance,一般不会主动访问该block,不访问该block磁盘组也就不会dismount,按照这个解决思路,通过patch解决,让oradata磁盘组不再执行rebalance和分配/回收空间即可一直稳定的mount
20211227210314

数据库直接open成功,实现数据0丢失
20211227210411
发表在 Oracle ASM, Oracle备份恢复 | 标签为 , , , , , | 评论关闭