标签归档:asm ntfs

一次完美的asm disk被格式化ntfs恢复

又一起win rac的asm disk被格式化为ntfs,导致数据库异常恢复的请求,客户描述有一个500G的data磁盘组(只有一个磁盘,被误操作进行了格式化).格式化asm disk之后
asm的alert日志报错

Tue Jul 09 17:09:45 2019
NOTE: ASM client orcl1:ORCL disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file D:\APP\ADMINISTRATOR\diag\asm\+asm\+asm1\trace\+asm1_ora_5376.trc
Tue Jul 09 17:10:19 2019
Errors in file D:\APP\ADMINISTRATOR\diag\asm\+asm\+asm1\trace\+asm1_lgwr_1448.trc:
ORA-27070: async read/write failed
OSD-04008: WriteFile() 失败, 无法写入文件
O/S-Error: (OS 21) 设备未就绪。
WARNING: Write Failed. group:1 disk:0 AU:15 offset:876544 size:4096
NOTE: unable to write any mirror side for diskgroup DATA
NOTE: cache initiating offline of disk 0 group DATA
NOTE: process _lgwr_+asm1 (3764:1448) initiating offline of disk 0.4042281525 (DATA_0000) with mask 0x7e in group 1
Tue Jul 09 17:10:19 2019
WARNING: Disk 0 (DATA_0000) in group 1 mode 0x7f is now being offlined
WARNING: Disk 0 (DATA_0000) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 0/0xf0f05235, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 10 for pid 15, osid 1448
ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 1)
WARNING: Disk 0 (DATA_0000) in group 1 mode 0x7f offline is being aborted
WARNING: Offline of disk 0 (DATA_0000) in group 1 and mode 0x7f failed on ASM inst 1
Tue Jul 09 17:10:20 2019
NOTE: halting all I/Os to diskgroup 1 (DATA)
NOTE: unable to offline disks after getting write error for diskgroup DATA
Tue Jul 09 17:10:20 2019
NOTE: cache dismounting (not clean) group 1/0xBDB0A2C0 (DATA) 
NOTE: disk 0 had IO error
NOTE: messaging CKPT to quiesce pins Windows thread id: 520528, image: ORACLE.EXE (B000)
Tue Jul 09 17:10:20 2019
NOTE: Deferred communication with ASM instance
Errors in file D:\APP\ADMINISTRATOR\diag\asm\+asm\+asm1\trace\+asm1_ora_6140.trc:
ORA-15130: diskgroup "DATA" is being dismounted
NOTE: deferred map free for map id 4
NOTE: LGWR doing non-clean dismount of group 1 (DATA)
NOTE: LGWR sync ABA=38.3028 last written ABA 38.3029

数据库的alert日志报错

Tue Jul 09 17:09:12 2019
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-27072: 文件 I/O 错误
WARNING: IO Failed. group:1 disk(number.incarnation):0.0xf0f05235 disk_path:\\.\ORCLDISKDATA0
	 AU:1305 disk_offset(bytes):1368456704 io_size:512 operation:Write type:asynchronous
	 result:I/O error process_id:5396
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-15080: 与磁盘的同步 I/O 操作失败
WARNING: failed to write mirror side 1 of virtual extent 23 logical extent 0 
   of file 261 in group 1 on disk 0 allocation unit 1305 
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-00345: 重做日志写入块 47231 计数 1 出错
ORA-00312: 联机日志 1 线程 1: '+DATA/orcl/onlinelog/group_1.261.909498607'
ORA-15081: 无法将 I/O 操作提交到磁盘
ORA-15081: 无法将 I/O 操作提交到磁盘
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl1\trace\orcl1_lgwr_5396.trc:
ORA-27070: 异步读取/写入失败
OSD-04006: ReadFile() 失败, 无法读取文件
O/S-Error: (OS 21) 设备未就绪。

机器上查看磁盘信息
format_ntfs2
format_ntfs


这个客户比较幸运,asm disk被格式化之后,没有进行任何写操作,理解对现场进行了保护,没有任何的二次破坏.因为处理过多起类似故障(oracle asm disk格式化恢复—格式化为ntfs文件系统,oracle asm disk格式化恢复—格式化为ext4文件系统又一例asm格式化文件系统恢复),有一定的经验,一般三种方案恢复数据文件:
1)运气好直接通过kfed进行修复asm disk,然后mount起来,然后把数据文件拷贝到文件系统中
2)通过相关工具把asm disk中相关的数据文件拷贝到文件系统中
3)如果损坏的严重,通过底层碎片,把相关数据文件恢复到文件系统中
拷贝完成数据文件之后,然后根据文件的情况有几种可能性:
1)直接open数据库,处理可能的其他坏块
2)通过一些方法强制拉库,然后对其进行导出导入新库
3)通过工具直接恢复表数据,甚至恢复部分核心数据
这次的恢复运气不错,对asm disk进行一系列修复之后,asm 磁盘组mount成功

C:\Users\Administrator>asmtool -list
NTFS                             \Device\Harddisk0\Partition1            80000M
NTFS                             \Device\Harddisk0\Partition2           491133M
ORCLDISKOCR0                     \Device\Harddisk1\Partition1            34132M
ORCLDISKOCR1                     \Device\Harddisk1\Partition2            34132M
ORCLDISKOCR2                     \Device\Harddisk1\Partition3            34133M
ORCLDISKDATA0                    \Device\Harddisk2\Partition1           511997M
ORCLDISKFRA0                     \Device\Harddisk3\Partition2          1225000M
NTFS                             \Device\Harddisk3\Partition3          1225800M
                                 \Device\Harddisk3\Partition4          1314861M
C:\Users\Administrator>sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on 星期六 7月 13 11:20:02 2019

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup data mount;

Diskgroup altered.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64
bit Production
With the Real Application Clusters and Automatic Storage Management options

mount数据库拷贝文件

C:\Users\Administrator>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期六 7月 13 16:02:08 2019

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup  mount;
ORACLE 例程已经启动。

Total System Global Area 3.4206E+10 bytes
Fixed Size                  2192864 bytes
Variable Size            7516195360 bytes
Database Buffers         2.6642E+10 bytes
Redo Buffers               45727744 bytes
数据库已装载

RMAN>  backup as copy   database format 'I:\rmanback\df_%U.dbf';

启动 backup 于 13-7月 -19
使用通道 ORA_DISK_1
使用通道 ORA_DISK_2
使用通道 ORA_DISK_3
使用通道 ORA_DISK_4
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00006 名称=+DATA/orcl/datafile/XFF5.dmp
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00016 名称=+DATA/orcl/datafile/XFF53
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00017 名称=+DATA/orcl/datafile/XFF54
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00018 名称=+DATA/orcl/datafile/XFF55
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-16_0VU6I1FO.DBF
标记=TAG20190713T120927 RECID=2 STAMP=1013517954
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:36:29
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00019 名称=+DATA/orcl/datafile/XFF5.281.988042407
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-17_10U6I1FO.DBF
标记=TAG20190713T120927 RECID=1 STAMP=1013517951
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:36:29
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00020 名称=+DATA/orcl/datafile/XFF5.282.988042943
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-18_11U6I1FO.DBF
标记=TAG20190713T120927 RECID=3 STAMP=1013517962
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:36:37
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00021 名称=+DATA/orcl/datafile/XFF5.283.988043279
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-6_0UU6I1FO.DBF
标记=TAG20190713T120927 RECID=4 STAMP=1013518052
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:38:12
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00022 名称=+DATA/orcl/datafile/XFF5.284.988043721
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-20_13U6I3K5.DBF
标记=TAG20190713T120927 RECID=5 STAMP=1013519584
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:27:10
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00009 名称=+DATA/orcl/datafile/XFF51.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-21_14U6I3KD.DBF
标记=TAG20190713T120927 RECID=6 STAMP=1013519586
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:27:03
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00010 名称=+DATA/orcl/datafile/XFF52.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-19_12U6I3K5.DBF
标记=TAG20190713T120927 RECID=7 STAMP=1013519599
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:27:26
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00003 名称=+DATA/orcl/datafile/undotbs1.258.909498475
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-22_15U6I3NC.DBF
标记=TAG20190713T120927 RECID=8 STAMP=1013519695
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:27:19
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00005 名称=+DATA/orcl/datafile/undotbs2.264.909498739
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-UNDOTBS1_FNO-3_18U6I57K.DB
F 标记=TAG20190713T120927 RECID=9 STAMP=1013520920
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:22:02
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00012 名称=+DATA/orcl/datafile/XFF5.275.954036129
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-UNDOTBS2_FNO-5_19U6I5AJ.DB
F 标记=TAG20190713T120927 RECID=10 STAMP=1013521015
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:22:03
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00013 名称=+DATA/orcl/datafile/XFF5.276.954036233
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-10_17U6I574.DBF
标记=TAG20190713T120927 RECID=11 STAMP=1013521231
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:27:30
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00002 名称=+DATA/orcl/datafile/sysaux.257.909498475
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-9_16U6I574.DBF
标记=TAG20190713T120927 RECID=12 STAMP=1013521244
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:27:37
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00023 名称=+DATA/orcl/datafile/XFF60.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-12_1AU6I6GU.DBF
标记=TAG20190713T120927 RECID=13 STAMP=1013522267
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:22:25
通道 ORA_DISK_2: 启动数据文件副本
输入数据文件: 文件号=00024 名称=+DATA/orcl/datafile/XFF61.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-13_1BU6I6JU.DBF
标记=TAG20190713T120927 RECID=15 STAMP=1013522367
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:22:26
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00011 名称=+DATA/orcl/datafile/system.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-SYSAUX_FNO-2_1CU6I6QM.DBF
标记=TAG20190713T120927 RECID=14 STAMP=1013522361
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:18:50
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00001 名称=+DATA/orcl/datafile/system.256.909498475
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-23_1DU6I6QT.DBF
标记=TAG20190713T120927 RECID=16 STAMP=1013522400
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:19:19
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00025 名称=+DATA/orcl/datafile/XFF62.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-25_1HU6I7V4.DBF
标记=TAG20190713T120927 RECID=17 STAMP=1013522490
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:01:35
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00026 名称=+DATA/orcl/datafile/XFF63.dmp
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-SYSTEM_FNO-1_1GU6I7U0.DBF
标记=TAG20190713T120927 RECID=18 STAMP=1013522526
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:02:47
通道 ORA_DISK_4: 启动数据文件副本
输入数据文件: 文件号=00007 名称=+DATA/orcl/datafile/precise.dbf
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-SYSTEM_FNO-11_1FU6I7U0.DBF
 标记=TAG20190713T120927 RECID=20 STAMP=1013522576
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:03:33
通道 ORA_DISK_1: 启动数据文件副本
输入数据文件: 文件号=00004 名称=+DATA/orcl/datafile/users.259.909498477
RMAN-03009: backup 命令 (ORA_DISK_3 通道上, 在 07/13/2019 14:03:01 上) 失败
ORA-19566: 超出损坏块限制 0 (文件 +DATA/orcl/datafile/XFF63.dmp)
继续执行其他作业步骤, 将不重新运行失败的作业
通道 ORA_DISK_3: 启动数据文件副本
输入数据文件: 文件号=00014 名称=+DATA/orcl/datafile/test.280.972807149
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-PRECISE_FNO-7_1JU6I837.DBF
 标记=TAG20190713T120927 RECID=19 STAMP=1013522572
通道 ORA_DISK_4: 数据文件复制完毕, 经过时间: 00:00:46
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-USERS_FNO-4_1KU6I84L.DBF
标记=TAG20190713T120927 RECID=21 STAMP=1013522591
通道 ORA_DISK_1: 数据文件复制完毕, 经过时间: 00:00:15
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-TEST_FNO-14_1LU6I84L.DBF
标记=TAG20190713T120927 RECID=22 STAMP=1013522591
通道 ORA_DISK_3: 数据文件复制完毕, 经过时间: 00:00:15
输出文件名=I:\RMANBACK\DF_DATA_D-ORCL_I-1437279340_TS-XFF5_FNO-24_1EU6I7R0.DBF
标记=TAG20190713T120927 RECID=23 STAMP=1013522974
通道 ORA_DISK_2: 数据文件复制完毕, 经过时间: 00:11:44
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================

RMAN-03009: backup 命令 (ORA_DISK_3 通道上, 在 07/13/2019 14:03:01 上) 失败
ORA-19566: 超出损坏块限制 0 (文件 +DATA/orcl/datafile/XFF63.dmp)

运气不错,除+DATA/orcl/datafile/XFF63.dmp文件上面有坏块之外其他文件没有发现坏块,对该文件进行特殊方式拷贝处理
dbv检查该文件

C:\Users\Administrator>dbv file=i:/rmanback/26.dbf

DBVERIFY: Release 11.2.0.1.0 - Production on 星期六 7月 13 14:09:32 2019

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - 开始验证: FILE = I:\RMANBACK\26.DBF
页 132735 流入 - 很可能是介质损坏
Corrupt block relative dba: 0x0682067f (file 26, block 132735)
Fractured block found during dbv:
Data in bad block:
 type: 40 format: 2 rdba: 0x0682067f
 last change scn: 0x0000.333eaa21 seq: 0x2 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0xaa550000
 check value in block header: 0x4242
 computed block checksum: 0xb24f



DBVERIFY - 验证完成

检查的页总数: 153600
处理的页总数 (数据): 9283
失败的页总数 (数据): 0
处理的页总数 (索引): 4789
失败的页总数 (索引): 0
处理的页总数 (其他): 139463
处理的总页数 (段)  : 0
失败的总页数 (段)  : 0
空的页总数: 64
标记为损坏的总页数: 1
流入的页总数: 1
加密的总页数        : 0
最高块 SCN            : 1029666541 (0.1029666541)

这次运气非常好,该文件也只有一个坏块,整体来说,把正在运行的asm disk磁盘格式化为ntfs格式化,整个数据库文件只发现一个坏块.拷贝数据文件,redo,ctl等之后

通过重建控制文件尝试open数据库

SQL> @ctl.sql

控制文件已创建。
SQL> RECOVER DATABASE;
完成介质恢复。
SQL> ALTER DATABASE OPEN;

数据库已更改。

通过dba_extents 定位坏块对象,然后根据实际情况处理(index直接rebuild,表跳过,lob置空等方法),确定数据没有问题,重建磁盘组,数据回迁,恢复完美完成

发表在 非常规恢复 | 标签为 , , , , | 评论关闭

oracle asm disk格式化恢复—格式化为ntfs文件系统

接到网友请求,由于操作人员粗心把asm disk的磁盘映射到另外的机器上,并且格式化为了win ntfs文件系统,导致asm 磁盘组异常,数据库无法使用
asm 日志报ORA-27072错

Mon Nov 30 12:00:13 2015
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27070: async read/write failed
OSD-04008: WriteFile() 失败, 无法写入文件
O/S-Error: (OS 21) 设备未就绪。
WARNING: IO Failed. group:1 disk(number.incarnation):0.0xf0f0bbfb disk_path:\\.\ORCLDISKDATA0
	 AU:1 disk_offset(bytes):2093056 io_size:4096 operation:Write type:synchronous
	 result:I/O error process_id:868
WARNING: disk 0.4042308603 (DATA_0000) not responding to heart beat
ERROR: too many offline disks in PST (grp 1)
WARNING: Disk DATA_0000 in mode 0x7f will be taken offline
Mon Nov 30 12:00:13 2015
NOTE: process 576:37952 initiating offline of disk 0.4042308603 (DATA_0000) with mask 0x7e in group 1
WARNING: Disk DATA_0000 in mode 0x7f is now being taken offline
NOTE: initiating PST update: grp = 1, dsk = 0/0xf0f0bbfb, mode = 0x15
kfdp_updateDsk(): 5 
kfdp_updateDskBg(): 5 
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):1.0xf0f0bbfc disk_path:\\.\ORCLDISKDATA1
	 AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):1.0xf0f0bbfc disk_path:\\.\ORCLDISKDATA1
	 AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):2.0xf0f0bbfd disk_path:\\.\ORCLDISKDATA2
	 AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):2.0xf0f0bbfd disk_path:\\.\ORCLDISKDATA2
	 AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):3.0xf0f0bbfe disk_path:\\.\ORCLDISKDATA3
	 AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):3.0xf0f0bbfe disk_path:\\.\ORCLDISKDATA3
	 AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):4.0xf0f0bbff disk_path:\\.\ORCLDISKDATA4
	 AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):4.0xf0f0bbff disk_path:\\.\ORCLDISKDATA4
	 AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):6.0xf0f0bc01 disk_path:\\.\ORCLDISKDATA6
	 AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):6.0xf0f0bc01 disk_path:\\.\ORCLDISKDATA6
	 AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):7.0xf0f0bc02 disk_path:\\.\ORCLDISKDATA7
	 AU:1 disk_offset(bytes):1048576 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
Errors in file c:\app\administrator\diag\asm\+asm\+asm\trace\+asm_gmon_868.trc:
ORA-27072: File I/O error
WARNING: IO Failed. group:1 disk(number.incarnation):7.0xf0f0bc02 disk_path:\\.\ORCLDISKDATA7
	 AU:1 disk_offset(bytes):1052672 io_size:4096 operation:Read type:synchronous
	 result:I/O error process_id:868
ERROR: no PST quorum in group: required 1, found 0
WARNING: Disk DATA_0000 in mode 0x7f offline aborted
Mon Nov 30 12:00:14 2015
SQL> alter diskgroup DATA dismount force /* ASM SERVER */ 
NOTE: cache dismounting (not clean) group 1/0xBB404B03 (DATA) 
Mon Nov 30 12:00:14 2015
NOTE: halting all I/Os to diskgroup DATA
Mon Nov 30 12:00:14 2015
NOTE: LGWR doing non-clean dismount of group 1 (DATA)
NOTE: LGWR sync ABA=367.7265 last written ABA 367.7265
NOTE: cache dismounted group 1/0xBB404B03 (DATA) 
kfdp_dismount(): 6 
kfdp_dismountBg(): 6 
NOTE: De-assigning number (1,0) from disk (\\.\ORCLDISKDATA0)
NOTE: De-assigning number (1,1) from disk (\\.\ORCLDISKDATA1)
NOTE: De-assigning number (1,2) from disk (\\.\ORCLDISKDATA2)
NOTE: De-assigning number (1,3) from disk (\\.\ORCLDISKDATA3)
NOTE: De-assigning number (1,4) from disk (\\.\ORCLDISKDATA4)
NOTE: De-assigning number (1,5) from disk (\\.\ORCLDISKDATA5)
NOTE: De-assigning number (1,6) from disk (\\.\ORCLDISKDATA6)
NOTE: De-assigning number (1,7) from disk (\\.\ORCLDISKDATA7)
SUCCESS: diskgroup DATA was dismounted
NOTE: cache deleting context for group DATA 1/-1153414397
SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER */
ERROR: PST-initiated MANDATORY DISMOUNT of group DATA

这里的asm日志很明显由于asm disk无法正常访问,报ORA-27072错误,磁盘组强制dismount.

分析磁盘情况
asm-disk1
asm-disk2


通过与客户沟通,确定从I到O本为asm disk 被格式化为了NTFS文件系统的磁盘,结合asmtool分析可以发现还有一个asm disk没有格式化掉,该磁盘组中一个共有8个磁盘格式化掉了7个.

通过kfed分析磁盘信息

C:\Users\Administrator>kfed read '\\.\J:'
kfbh.endian:                        235 ; 0x000: 0xeb
kfbh.hard:                           82 ; 0x001: 0x52
kfbh.type:                          144 ; 0x002: *** Unknown Enum ***
kfbh.datfmt:                         78 ; 0x003: 0x4e
kfbh.block.blk:               542328404 ; 0x004: T=0 NUMB=0x20534654
kfbh.block.obj:                 2105376 ; 0x008: TYPE=0x0 NUMB=0x2020
kfbh.check:                        2050 ; 0x00c: 0x00000802
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                    63488 ; 0x014: 0x0000f800
kfbh.spare1:                   16711743 ; 0x018: 0x00ff003f
kfbh.spare2:                       2048 ; 0x01c: 0x00000800
ERROR!!!, failed to get the oracore error message

C:\Users\Administrator>kfed read '\\.\J:' blkn=2
kfbh.endian:                         70 ; 0x000: 0x46
kfbh.hard:                           73 ; 0x001: 0x49
kfbh.type:                           76 ; 0x002: *** Unknown Enum ***
kfbh.datfmt:                         69 ; 0x003: 0x45
kfbh.block.blk:                  196656 ; 0x004: T=0 NUMB=0x30030
kfbh.block.obj:                33563364 ; 0x008: TYPE=0x0 NUMB=0x22e4
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                    65537 ; 0x010: 0x00010001
kfbh.fcn.wrap:                    65592 ; 0x014: 0x00010038
kfbh.spare1:                        416 ; 0x018: 0x000001a0
kfbh.spare2:                       1024 ; 0x01c: 0x00000400
ERROR!!!, failed to get the oracore error message

C:\Users\Administrator>kfed read '\\.\J:' blkn=256
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                           13 ; 0x002: KFBTYP_PST_NONE
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:              2147483648 ; 0x004: T=1 NUMB=0x0
kfbh.block.obj:              2147483654 ; 0x008: TYPE=0x8 NUMB=0x6
kfbh.check:                    17662471 ; 0x00c: 0x010d8207
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
ERROR!!!, failed to get the oracore error message

C:\Users\Administrator>kfed read '\\.\J:' blkn=510
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                     254 ; 0x004: T=0 NUMB=0xfe
kfbh.block.obj:              2147483654 ; 0x008: TYPE=0x8 NUMB=0x6
kfbh.check:                   717599272 ; 0x00c: 0x2ac5b228
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:    ORCLDISKDATA6 ; 0x000: length=13
kfdhdb.driver.reserved[0]:   1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved[1]:           54 ; 0x00c: 0x00000036
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
…………

通过分析,可以确定asm disk的备份block没有被覆盖,原则上可以通过备份block实现磁盘组恢复,从而减小了恢复难度

kfed恢复磁盘头

C:\Users\Administrator> kfed repair '\\.\J:'
C:\Users\Administrator>kfed read '\\.\J:' 
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                     254 ; 0x004: T=0 NUMB=0xfe
kfbh.block.obj:              2147483654 ; 0x008: TYPE=0x8 NUMB=0x6
kfbh.check:                   717599272 ; 0x00c: 0x2ac5b228
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:    ORCLDISKDATA6 ; 0x000: length=13
kfdhdb.driver.reserved[0]:   1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved[1]:           54 ; 0x00c: 0x00000036
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
…………

确定asm disk相关信息
对于7个被格式化的磁盘都进行类似处理之后,通过工具看到相关磁盘信息如下
asm-disk3


恢复处理
根据ntfs的文件系统分布,我们可以知道,虽然asm disk header备份block正常,但是asm disk中间部分依旧有不少au会被破坏
ntfs


这样的情况,不合适直接使用工具拷贝出来datafile(由于可能记录block的字典正好被覆盖,导致拷贝出来的文件异常,在恢复过程中我们也做了试验小文件拷贝ok,大文件拷贝然后使用dbv检测有很多坏块),我们采用工具(asm disk header 彻底损坏恢复)从底层扫描直接重组出来asm disk中的数据文件,然后结合拷贝出来的控制文件,redo文件,参数文件,然后通过重命名相关路径,然后直接open数据库

Q:\>sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on 星期三 1月 22 16:08:18 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


连接到:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options


SQL> set pages 1000
SQL> col name for a100
SQL> set lines 150
SQL> select file#,name from v$datafile;

     FILE# NAME
---------- --------------------------------------------------------------------
         1 +DATA/vspdb/datafile/system.256.778520603
         2 +DATA/vspdb/datafile/sysaux.257.778520603
         3 +DATA/vspdb/datafile/undotbs1.258.778520603
         4 +DATA/vspdb/datafile/users.259.778520603
         5 +DATA/vspdb/datafile/vsp_tbs.293.779926097
        …………
       147 +DATA/vspdb/datafile/index_dg.418.864665747
       148 +DATA/vspdb/datafile/data_dg.419.864667053
       149 +DATA/vspdb/datafile/vsp_mm_tbs.420.890410367
       150 +DATA/vspdb/datafile/vsp_mm_tbs.421.890410457

SQL> select member from v$logfile;

MEMBER
-------------------------------------------------------------------------------------
+DATA/vspdb/onlinelog/group_7.263.862676593
+DATA/vspdb/onlinelog/group_7.262.862676601
+DATA/vspdb/onlinelog/group_4.410.862652291
+DATA/vspdb/onlinelog/group_4.411.862652307
+DATA/vspdb/onlinelog/group_5.412.862653715
+DATA/vspdb/onlinelog/group_5.413.862653727
+DATA/vspdb/onlinelog/group_6.414.862676425
+DATA/vspdb/onlinelog/group_6.415.862676433

重命名数据文件和redo文件,open数据库

SQL> recover database;
完成介质恢复。
SQL> alter database open;

数据库已更改。

已用时间:  00: 00: 04.51

由于部分block被覆盖,使用空块代替,导致数据访问到该block就会出现ora-8103(模拟普通ORA-08103并解决,模拟极端ORA-08103并解决)错误,对于该种对象,最简单处理方法就是直接通过dul抽出来数据然后truncate table重新导入数据,当然如果你想彻底安全逻辑方式重建库最靠谱
如果您遇到此类情况,无法解决请联系我们,提供专业ORACLE数据库恢复技术支持
Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

发表在 Oracle ASM, 非常规恢复 | 标签为 , , , , | 评论关闭