分类目录归档：非常规恢复

记录一次asm disk加入到vg通过恢复直接open库的案例

发表于 2025 年 5 月 5 日 由 惜分飞

客户在不清楚磁盘被asm disk使用的情况下,直接分区做pv,加入到vg中并且分配给了lv,导致数据库异常
QQ20250504
144809

通过操作系统层面分析,确认客户把data磁盘组的一个磁盘给处理掉了,导致数据库报错

WARNING: ASMB force dismounting group 2 (DATA) due to failover
SUCCESS: diskgroup DATA was dismounted
2025-05-04T07:03:19.910082+08:00
KCF: read, write or open error, block=0x201544 online=1
        file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.918972+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwc_18507.trc:
2025-05-04T07:03:19.952045+08:00
KCF: read, write or open error, block=0x2013e7 online=1
        file=102 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_61.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.964538+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw7_18486.trc:
2025-05-04T07:03:19.967133+08:00
KCF: read, write or open error, block=0x230e71 online=1
        file=105 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_64.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.973289+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbw2_18466.trc:
2025-05-04T07:03:19.978514+08:00
KCF: read, write or open error, block=0x1f6e91 online=1
        file=86 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/xifenfei_52.dbf'
        error=15078 txt: ''
2025-05-04T07:03:19.991060+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwd_18511.trc:
2025-05-04T07:03:19.995762+08:00
KCF: read, write or open error, block=0x7f8 online=1
        file=15 '+DATA/ORCL/F7D939D6DBE06C71E053C30114AC1F10/DATAFILE/undotbs01.dbf'
        error=15078 txt: ''
2025-05-04T07:03:20.006862+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_dbwa_18498.trc:
2025-05-04T07:03:20.020739+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_imr0_18937.trc:

这个客户比较幸运,处理该磁盘之后,没有往对应的lv中写入太多数据,导致覆盖部分很少

[root@rac01 rules.d]# df -h
文件系统               容量  已用  可用 已用% 挂载点
/dev/mapper/nlas-root  800G  272G  528G   34% /
devtmpfs               284G     0  284G    0% /dev
tmpfs                  284G  637M  283G    1% /dev/shm
tmpfs                  284G  4.0G  280G    2% /run
tmpfs                  284G     0  284G    0% /sys/fs/cgroup
/dev/mapper/nlas-home  200G   64M  200G    1% /home
/dev/sda1              197M  158M   40M   80% /boot
tmpfs                   57G   40K   57G    1% /run/user/0
tmpfs                   57G   48K   57G    1% /run/user/1000
[root@rac01 rules.d]# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  nlas lvm2 a--  564.00g    0 
  /dev/sdb1  nlas lvm2 a--   <2.00t 1.51t
[root@rac01 rules.d]# vgs
  VG   #PV #LV #SN Attr   VSize VFree
  nlas   2   3   0 wz--n- 2.55t 1.51t
[root@rac01 rules.d]# lvs
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home nlas -wi-ao---- 200.00g                                                    
  root nlas -wi-ao---- 800.00g                                                    
  swap nlas -wi-ao----  64.00g

通过底层对磁盘进行分析,发现备份的磁盘头均以损坏,通过深入分析确认f1b1在sdb磁盘的第10个au上,通过相关信息,使用dul工具加载磁盘组,并分析元数据信息,发现恢复数据需要的元数据都可以正常加载
asm-dul

直接使用dul抽取数据到文件系统,然后open数据库成功
open-asm
然后通过rman 检测坏块(3T多的库只有不到5000个坏块,相对来说效果非常好),对于坏块对象进行处理,完美完成本次恢复工作.对于这次能够有这样好的恢复效果有几个因素:
1)asm disk 加入到vg,并分配给lv之后,立刻停止写入操作,避免了因为写入数据而覆盖asm 磁盘的带来的风险
2)由于是19c库,默认au为4M,使得数据库文件数据相对比较靠后,覆盖几率小了一点
3)由于文件系统是xfs,相对覆盖比ext4会少很多
4)是云环境的ssd磁盘,没有触发trim功能
以前类似asm disk异常恢复的相关case汇总:
asm磁盘加入vg恢复
asm磁盘dd破坏恢复
asm磁盘分区丢失恢复
pvid=yes导致asm无法mount
win asm disk header 异常恢复
又一例asm disk 加入vg故障
pvcreate asm disk导致asm磁盘组异常恢复
asm disk被加入到另外一个磁盘组故障恢复
再一例asm disk被误加入vg并且扩容lv恢复
再一起asm disk被格式化成ext3文件系统故障恢复
一次完美的asm disk被格式化ntfs恢复
asm disk误设置pvid导致asm diskgroup无法mount恢复
asm disk被分区,格式化为ext4恢复
oracle asm disk格式化恢复—格式化为ext4文件系统
分享oracleasm createdisk重新创建asm disk后数据0丢失恢复案例

发表在 Oracle ASM, 非常规恢复 | 标签为 , , , | 评论关闭

ORA-39773: parse of metadata stream failed故障处理

发表于 2025 年 3 月 22 日 由 惜分飞

客户expdp导出数据,在写入生成SYS_EXPORT_SCHEMA表所在的users表空间不足,导致expdp报部分异常

Export: Release 11.2.0.4.0 - Production on Thu Feb 27 15:12:36 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
;;; 
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
FLASHBACK automatically enabled to preserve database integrity.
Starting "XFF"."SYS_EXPORT_SCHEMA_95":XFF/**** directory=DUMP dumpfile=20250227.dmp logfile=20250227.log schemas=XFF 
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
ORA-39171: Job is experiencing a resumable wait.
ORA-01653: unable to extend table XFF.SYS_EXPORT_SCHEMA_95 by 128 in tablespace USERS
Total estimation using BLOCKS method: 481.9 GB
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TYPE/TYPE_SPEC
Processing object type SCHEMA_EXPORT/SEQUENCE/SEQUENCE
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/COMMENT
Processing object type SCHEMA_EXPORT/PACKAGE/PACKAGE_SPEC
Processing object type SCHEMA_EXPORT/FUNCTION/FUNCTION
Processing object type SCHEMA_EXPORT/PROCEDURE/PROCEDURE
Processing object type SCHEMA_EXPORT/PACKAGE/COMPILE_PACKAGE/PACKAGE_SPEC/ALTER_PACKAGE_SPEC
Processing object type SCHEMA_EXPORT/FUNCTION/ALTER_FUNCTION
Processing object type SCHEMA_EXPORT/PROCEDURE/ALTER_PROCEDURE
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/INDEX/FUNCTIONAL_INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/FUNCTIONAL_INDEX/INDEX_STATISTICS
Processing object type SCHEMA_EXPORT/VIEW/VIEW
Processing object type SCHEMA_EXPORT/PACKAGE/PACKAGE_BODY
Processing object type SCHEMA_EXPORT/TABLE/TRIGGER
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type SCHEMA_EXPORT/MATERIALIZED_VIEW
Processing object type SCHEMA_EXPORT/JOB
. . exported "XFF"."PUB_WORKINGTASKLOG"              88.09 GB  112681 rows
. . exported "XFF"."SM_BUSILOG_DEFAULT"              51.94 GB 3149092 rows
. . exported "XFF"."FFW_DISTRIBUTESUBTASK"           46.47 GB  552214 rows
. . exported "XFF"."GL_DETAIL"                       11.32 GB 16214805 rows
…………
. . exported "XFF"."ZDP_10000000RR8NKX"                  0 KB       0 rows
. . exported "XFF"."ZDP_10000000RTB1GI"                  0 KB       0 rows
. . exported "XFF"."ZDP_10000000RTB1GK"                  0 KB       0 rows
Master table "XFF"."SYS_EXPORT_SCHEMA_95" successfully loaded/unloaded
******************************************************************************
Dump file set for XFF.SYS_EXPORT_SCHEMA_95 is:
  /rman_bak/dump/20200227.dmp
Job "XFF"."SYS_EXPORT_SCHEMA_95" completed with 10 error(s) at Thu Feb 27 19:32:22 2025 elapsed 0 04:16:28

尝试把该dmp导入数据库,报ORA-31694/ORA-02354/ORA-39773错误

Import: Release 11.2.0.4.0 - Production on 星期日 3月 16 02:38:14 2025

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
;;; 
连接到: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORA-31694: 加载/卸载主表 "XFF"."SYS_IMPORT_FULL_01" 失败
ORA-02354: 导出/导入数据时出错
ORA-39773: parse of metadata stream failed

客户需要恢复其中的GL_DETAIL表数据,通过dul实现expdp dump文件转换sqlldr格式方法进行恢复,实现数据完美恢复
QQ20250322-211130

对于dmp(exp/expdp)文件,我们可以实现比较完美的恢复,最后限度抢救数据.如果你有oracle expdp/exp dmp被加密或者破坏,无法正常导入数据库,可以联系我们对其进行恢复处理:提供(ORACLE数据库恢复技术支持)：
Phone:17813235971    Q Q:107644445    E-Mail:dba@xifenfei.com

发表在 非常规恢复 | 标签为 , | 评论关闭

aix磁盘损坏oracle数据库恢复

发表于 2025 年 3 月 7 日 由 惜分飞

客户aix环境硬盘异常导致系统无法启动,初步判断是数据文件存放在本地磁盘的空间中(本地两个盘都异常,系统无法启动),通过硬件恢复厂商镜像出来,但是通过aix文件系统直接挂载提示需要fsck,但是做fsck之后,提示大量文件丢失(最关键的数据文件和备份文件都被自动删除)
dmp-remove
fsck-remove

基于这种情况,采用镜像主机挂载的方式肯定不行,考虑直接采用软件直接解析,能够看到软件,可惜由于大量的文件系统元数据损坏,解析出来的数据文件和dmp也不可用(大量损坏和空块)
QQ20250307-122939
基于上述情况,只能采用碎片级别恢复出来数据文件
QQ20250307-123123
然后使用dul工具把数据恢复到表中,实现最大限度抢救客户数据
QQ20250307-123653
对于数据库级别恢复,这个是理论上的终极恢复方法

发表在 Oracle, 非常规恢复 | 标签为 , | 评论关闭