标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-00742 ORA-01110 ORA-01555 ORA-01578 ORA-01595 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (103)
- 数据库 (1,762)
- DB2 (22)
- MySQL (76)
- Oracle (1,604)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (24)
- ORA-xxxxx (166)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (15)
- ORACLE 21C (3)
- Oracle 23ai (8)
- Oracle ASM (69)
- Oracle Bug (8)
- Oracle RAC (54)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (588)
- Oracle安装升级 (97)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (85)
- PostgreSQL (30)
- pdu工具 (6)
- PostgreSQL恢复 (9)
- SQL Server (32)
- SQL Server恢复 (13)
- TimesTen (7)
- 达梦数据库 (3)
- 达梦恢复 (1)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (39)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (22)
-
最近发表
- 解决一次硬件恢复之后数据文件0kb的故障恢复case
- Error in invoking target ‘libasmclntsh19.ohso libasmperl19.ohso client_sharedlib’问题处理
- ORA-01171: datafile N going offline due to error advancing checkpoint
- linux环境oracle数据库被文件系统勒索加密为.babyk扩展名溯源
- ORA-600 ksvworkmsgalloc: bad reaper
- ORA-600 krccfl_chunk故障处理
- Oracle Recovery Tools恢复案例总结—202505
- ORA-600 kddummy_blkchk 数据库循环重启
- 记录一次asm disk加入到vg通过恢复直接open库的案例
- CHECKDB 发现了 N 个分配错误和 M 个一致性错误
- 达梦数据库dm.ctl文件异常恢复
- Oracle Recovery Tools修复ORA-00742、ORA-600 ktbair2: illegal inheritance故障
- 可能是 tempdb 空间用尽或某个系统表不一致故障处理
- 11.2.0.4库中遇到ORA-600 kcratr_nab_less_than_odr报错
- [MY-013183] [InnoDB] Assertion failure故障处理
- Oracle 19c 202504补丁(RUs+OJVM)-19.27
- Oracle Recovery Tools修复ORA-600 6101/kdxlin:psno out of range故障
- pdu完美支持金仓数据库恢复(KingbaseES)
- 虚拟机故障引起ORA-00310 ORA-00334故障处理
- pg创建gbk字符集库
标签归档:asm disk lvm
记录一次asm disk加入到vg通过恢复直接open库的案例
客户在不清楚磁盘被asm disk使用的情况下,直接分区做pv,加入到vg中并且分配给了lv,导致数据库异常
通过操作系统层面分析,确认客户把data磁盘组的一个磁盘给处理掉了,导致数据库报错
WARNING: ASMB force dismounting group 2 (DATA) due to failover SUCCESS: diskgroup DATA was dismounted 2025-05-04T07:03:19.910082+08:00 KCF: read, write or open error, block=0x201544 online=1 file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf' error=15078 txt: '' 2025-05-04T07:03:19.918972+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwc_18507.trc: 2025-05-04T07:03:19.952045+08:00 KCF: read, write or open error, block=0x2013e7 online=1 file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf' error=15078 txt: '' 2025-05-04T07:03:19.964538+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw7_18486.trc: 2025-05-04T07:03:19.967133+08:00 KCF: read, write or open error, block=0x230e71 online=1 file=105 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_64.dbf' error=15078 txt: '' 2025-05-04T07:03:19.973289+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw2_18466.trc: 2025-05-04T07:03:19.978514+08:00 KCF: read, write or open error, block=0x1f6e91 online=1 file=86 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_52.dbf' error=15078 txt: '' 2025-05-04T07:03:19.991060+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwd_18511.trc: 2025-05-04T07:03:19.995762+08:00 KCF: read, write or open error, block=0x7f8 online=1 file=15 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/undotbs01.dbf' error=15078 txt: '' 2025-05-04T07:03:20.006862+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwa_18498.trc: 2025-05-04T07:03:20.020739+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_imr0_18937.trc:
这个客户比较幸运,处理该磁盘之后,没有往对应的lv中写入太多数据,导致覆盖部分很少
[root@rac01 rules.d]# df -h 文件系统 容量 已用 可用 已用% 挂载点 /dev/mapper/nlas-root 800G 272G 528G 34% / devtmpfs 284G 0 284G 0% /dev tmpfs 284G 637M 283G 1% /dev/shm tmpfs 284G 4.0G 280G 2% /run tmpfs 284G 0 284G 0% /sys/fs/cgroup /dev/mapper/nlas-home 200G 64M 200G 1% /home /dev/sda1 197M 158M 40M 80% /boot tmpfs 57G 40K 57G 1% /run/user/0 tmpfs 57G 48K 57G 1% /run/user/1000 [root@rac01 rules.d]# pvs PV VG Fmt Attr PSize PFree /dev/sda2 nlas lvm2 a-- 564.00g 0 /dev/sdb1 nlas lvm2 a-- <2.00t 1.51t [root@rac01 rules.d]# vgs VG #PV #LV #SN Attr VSize VFree nlas 2 3 0 wz--n- 2.55t 1.51t [root@rac01 rules.d]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home nlas -wi-ao---- 200.00g root nlas -wi-ao---- 800.00g swap nlas -wi-ao---- 64.00g
通过底层对磁盘进行分析,发现备份的磁盘头均以损坏,通过深入分析确认f1b1在sdb磁盘的第10个au上,通过相关信息,使用dul工具加载磁盘组,并分析元数据信息,发现恢复数据需要的元数据都可以正常加载
直接使用dul抽取数据到文件系统,然后open数据库成功

然后通过rman 检测坏块(3T多的库只有不到5000个坏块,相对来说效果非常好),对于坏块对象进行处理,完美完成本次恢复工作.对于这次能够有这样好的恢复效果有几个因素:
1)asm disk 加入到vg,并分配给lv之后,立刻停止写入操作,避免了因为写入数据而覆盖asm 磁盘的带来的风险
2)由于是19c库,默认au为4M,使得数据库文件数据相对比较靠后,覆盖几率小了一点
3)由于文件系统是xfs,相对覆盖比ext4会少很多
4)是云环境的ssd磁盘,没有触发trim功能
以前类似asm disk异常恢复的相关case汇总:
asm磁盘加入vg恢复
asm磁盘dd破坏恢复
asm磁盘分区丢失恢复
pvid=yes导致asm无法mount
win asm disk header 异常恢复
又一例asm disk 加入vg故障
pvcreate asm disk导致asm磁盘组异常恢复
asm disk被加入到另外一个磁盘组故障恢复
再一例asm disk被误加入vg并且扩容lv恢复
再一起asm disk被格式化成ext3文件系统故障恢复
一次完美的asm disk被格式化ntfs恢复
asm disk误设置pvid导致asm diskgroup无法mount恢复
asm disk被分区,格式化为ext4恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
分享oracleasm createdisk重新创建asm disk后数据0丢失恢复案例
asm磁盘加入vg恢复
又一个客户把asm disk做成pv,加到vg中,并且对lv进行了扩展(ext4的文件系统)
这个客户做了上述操作之后,没有对lv进行写入其他数据,所以破坏较少(主要的破坏就是ext4的每个一段就会置空一部分block预留给文件系统写入元数据使用),通过winhex查看被破坏磁盘发现lvm信息

对于这种情况,通过对文件头进行修复,结合工具直接拷贝出来数据文件(个别文件元数据损坏通过基于block的方式恢复dbf)

然后直接恢复dbf中数据文件(对于异常的主要是segment header被置空的tab使用dul单独扫描处理),实现客户数据的最大限度恢复
以前类似文章:
asm disk被加入vg恢复
asm disk被分区,格式化为ext4恢复
pvcreate asm disk导致asm磁盘组异常恢复
再一起asm disk被格式化成ext3文件系统故障恢复
一次完美的asm disk被格式化ntfs恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
asm disk被加入vg恢复
接到客户恢复请求:把oracle asm datagroup中的一个磁盘增加到vg中,现在磁盘组无法mount,数据库无法正常启动.远程登录现场进行分析发现情况如下:
操作系统层面分析
history操作记录
这里比较明显把一个磁盘做成pv,并且加入到vg中,然后再分配199G给lv_home,系统层面分析lvm信息
--查看pv信息 [root@xff1 ~]# pvdisplay --- Physical volume --- PV Name /dev/sda2 VG Name VolGroup PV Size 277.98 GiB / not usable 3.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 71161 Free PE 0 Allocated PE 71161 PV UUID F6QO3f-065n-mwTW-Xbq2-Xx2y-c8HD-Tkr7V7 --- Physical volume --- PV Name /dev/sdg <----新加入的磁盘 VG Name VolGroup PV Size 200.00 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 51199 Free PE 255 Allocated PE 50944 PV UUID i69vUG-nCIK-dtxL-FvpD-2WZd-bvLv-n7lwrb [root@xff1 ~]# lvdisplay --- Logical volume --- LV Path /dev/VolGroup/lv_root LV Name lv_root VG Name VolGroup LV UUID JUNnkN-m4zq-D0gh-h42b-cUM1-Wh1q-ZMtQE4 LV Write Access read/write LV Creation host, time localhost.localdomain, 2017-07-19 20:08:47 +0800 LV Status available # open 1 LV Size 50.00 GiB Current LE 12800 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 --- Logical volume --- LV Path /dev/VolGroup/lv_home LV Name lv_home VG Name VolGroup LV UUID eZTkLt-cNGX-371i-m8Bd-VdD9-q6Hz-wYDRIJ LV Write Access read/write LV Creation host, time localhost.localdomain, 2017-07-19 20:08:54 +0800 LV Status available # open 1 LV Size 422.97 GiB <-----lv大小变成422G,应该是被扩了199G后结果 Current LE 108281 Segments 2 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:2 --- Logical volume --- LV Path /dev/VolGroup/lv_swap LV Name lv_swap VG Name VolGroup LV UUID 54P9ok-VpwO-zM68-hvwY-9GBf-89yb-8xQAMn LV Write Access read/write LV Creation host, time localhost.localdomain, 2017-07-19 20:09:23 +0800 LV Status available # open 1 LV Size 4.00 GiB Current LE 1024 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 [root@xff1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 50G 3.9G 43G 9% / tmpfs 63G 509M 63G 1% /dev/shm /dev/sda1 477M 44M 408M 10% /boot /dev/mapper/VolGroup-lv_home 417G 226G 170G 58% /home <----增加了199g空间,剩余只剩170G,证明增加空间之后最少使用了30G以上
基于这样的情况,基本上可以确定sdg盘加入VolGroup中并且被分配给 lv_home中,而且还写入了数据(/home空闲空间只剩余170G,lv_home当时扩了199G).
asm层面分析
asm磁盘组无法mount,提示缺少一块磁盘
SQL> ALTER DISKGROUP DATA MOUNT /* asm agent *//* {1:12056:279} */ NOTE: cache registered group DATA number=1 incarn=0xa1dbff16 NOTE: cache began mount (first) of group DATA number=1 incarn=0xa1dbff16 NOTE: Assigning number (1,2) to disk (/dev/asmdisk3) NOTE: Assigning number (1,1) to disk (/dev/asmdisk2) Sat Apr 25 13:04:58 2020 ERROR: no read quorum in group: required 1, found 0 disks NOTE: cache dismounting (clean) group 1/0xA1DBFF16 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 81552, image: oracle@rac2db1 (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xA1DBFF16 (DATA) NOTE: cache ending mount (fail) of group DATA number=1 incarn=0xa1dbff16 NOTE: cache deleting context for group DATA 1/0xa1dbff16 GMON dismounting group 1 at 19 for pid 30, osid 81552 NOTE: Disk DATA_0001 in mode 0x9 marked for de-assignment NOTE: Disk DATA_0002 in mode 0x9 marked for de-assignment ERROR: diskgroup DATA was not mounted ORA-15032: not all alterations performed ORA-15017: diskgroup "DATA" cannot be mounted ORA-15040: diskgroup is incomplete ERROR: ALTER DISKGROUP DATA MOUNT /* asm agent *//* {1:12056:279} */
报错比较明显asm disk磁盘头被lvm的信息取代(因为asm disk 被加入到vg中),根据前面的分析,该磁盘被写入数据很可能超过30G,使用kfed分析一个随意au,确认被破坏,证明开始判断基本正确
root@xff1:/home/oracle11g$kfed read /dev/asmdisk1 aun=10000 kfbh.endian: 51 ; 0x000: 0x33 kfbh.hard: 55 ; 0x001: 0x37 kfbh.type: 32 ; 0x002: *** Unknown Enum *** kfbh.datfmt: 42 ; 0x003: 0x2a kfbh.block.blk: 1329801248 ; 0x004: blk=1329801248 kfbh.block.obj: 1128615502 ; 0x008: file=347726 kfbh.check: 1094999892 ; 0x00c: 0x41445f54 kfbh.fcn.base: 675103060 ; 0x010: 0x283d4154 kfbh.fcn.wrap: 1448232275 ; 0x014: 0x56524553 kfbh.spare1: 1598374729 ; 0x018: 0x5f454349 kfbh.spare2: 1162690894 ; 0x01c: 0x454d414e 7F7843EAD400 2A203733 4F432820 43454E4E 41445F54 [37 * (CONNECT_DA] 7F7843EAD410 283D4154 56524553 5F454349 454D414E [TA=(SERVICE_NAME] 7F7843EAD420 6361723D 29626432 44494328 5250283D [=rac2db)(CID=(PR] 7F7843EAD430 4152474F 3A443D4D 4341505C DFCF3153 [OGRAM=D:\PACS1..] 7F7843EAD440 B3BEB7BB 6369445C 65536D6F 72657672 [....\DicomServer] 7F7843EAD450 445C524D 6D6F6369 76726553 524D7265 [MR\DicomServerMR] 7F7843EAD460 6578652E 4F482829 573D5453 362D4E49 [.exe)(HOST=WIN-6] 7F7843EAD470 51414C38 54553645 28294A30 52455355 [8LAQE6UT0J)(USER] 7F7843EAD480 6D64413D 73696E69 74617274 2929726F [=Administrator))] 7F7843EAD490 202A2029 44444128 53534552 5250283D [) * (ADDRESS=(PR] 7F7843EAD4A0 434F544F 743D4C4F 28297063 54534F48 [OTOCOL=tcp)(HOST] 7F7843EAD4B0 2E30313D 2E303831 30332E31 4F502829 [=10.180.1.30)(PO] 7F7843EAD4C0 343D5452 37333539 2A202929 74736520 [RT=49537)) * est] 7F7843EAD4D0 696C6261 2A206873 63617220 20626432 [ablish * rac2db ] 7F7843EAD4E0 3231202A 0A343135 2D534E54 31353231 [* 12514.TNS-1251] 7F7843EAD4F0 54203A34 6C3A534E 65747369 2072656E [4: TNS:listener ] 7F7843EAD500 73656F64 746F6E20 72756320 746E6572 [does not current] 7F7843EAD510 6B20796C 20776F6E 7320666F 69767265 [ly know of servi] 7F7843EAD520 72206563 65757165 64657473 206E6920 [ce requested in ] 7F7843EAD530 6E6E6F63 20746365 63736564 74706972 [connect descript] ……………… 7F7843EAE300 6F636944 7265536D 4D726576 69445C52 [DicomServerMR\Di] 7F7843EAE310 536D6F63 65767265 2E524D72 29657865 [comServerMR.exe)] 7F7843EAE320 534F4828 49573D54 4F302D4E 314B304A [(HOST=WIN-0OJ0K1] 7F7843EAE330 4955304E 55282954 3D524553 696D6441 [N0UIT)(USER=Admi] 7F7843EAE340 7473696E 6F746172 29292972 28202A20 [nistrator))) * (] 7F7843EAE350 52444441 3D535345 4F525028 4F434F54 [ADDRESS=(PROTOCO] 7F7843EAE360 63743D4C 48282970 3D54534F 312E3031 [L=tcp)(HOST=10.1] 7F7843EAE370 312E3038 2930332E 524F5028 35353D54 [80.1.30)(PORT=55] 7F7843EAE380 29383632 202A2029 61747365 73696C62 [268)) * establis] 7F7843EAE390 202A2068 32636172 2A206264 35323120 [h * rac2db * 125] 7F7843EAE3A0 540A3431 312D534E 34313532 4E54203A [14.TNS-12514: TN] 7F7843EAE3B0 696C3A53 6E657473 64207265 2073656F [S:listener does ] 7F7843EAE3C0 20746F6E 72727563 6C746E65 6E6B2079 [not currently kn] 7F7843EAE3D0 6F20776F 65732066 63697672 65722065 [ow of service re] 7F7843EAE3E0 73657571 20646574 63206E69 656E6E6F [quested in conne] 7F7843EAE3F0 64207463 72637365 6F747069 34320A72 [ct descriptor.24] KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][32]
通过上述kfed可以看到第10000 au的位置被写入的是数据库异常之后listener.log的信息(该数据库安装在/home目录中),进一步证明覆盖,通过以下信息证明sdg就是asmdisk1
[root@xff1 dev]# ls -l sdg brw-rw---- 1 root disk 8, 96 Apr 25 00:05 sdg [root@xff1 dev]# ls -l asmdisk1 brw-rw---- 1 grid asmadmin 8, 96 Apr 25 00:05 asmdisk1
基于现在的情况,data磁盘组是由三块 200G的磁盘组成,第一块磁盘被意外加入vg,并且写入数据大于30G,无法从asm层面直接通过kfed修复磁盘组,然后直接mount,只能通过oracle asm磁盘数据块重组技术(asm disk header 彻底损坏恢复)实现没有覆盖数据的恢复.
该客户运气还不错,通过仅剩的2019年12月份几天的不成功备份找出来所有的数据文件(无归档),然后强制拉库成功.通过碎片恢复的最新的数据文件数据结合2019年12月份备份,实现绝大部分业务数据恢复,最大限度减少客户损失.对于oracle rac数据库服务器磁盘操作需要谨慎.
如果不幸有类似oracle asm disk被破坏(格式化,dd部分,做成lv等),需要进行恢复支持,可以联系我们,做专业的恢复评估,最大限度,最快速度抢救数据,减少损失
Phone:17813235971 Q Q:107644445

恢复过部分asm被格式化案例:
又一例asm格式化文件系统恢复
一次完美的asm disk被格式化ntfs恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
oracle asm disk格式化恢复—格式化为ntfs文件系统