分类目录归档：数据库

删除数据库文件并部分覆盖情况下Oracle恢复

发表于 2025 年 9 月 1 日 由 惜分飞

有客户由于磁盘空间满了,人工删除了数据库30多个数据文件,导致数据库无法正常工作,然后又被人offline这些文件启动数据库,并运行了一段时间,导致写入了大量的trace和部分数据库归档日志,导致被删除的数据文件发生了覆盖,对于这样情况,通过底层反删除工具对磁盘进行扫描,发现了部分被删除文件,但是大小基本上显示0kb
dbf-0kb

对于这种情况,os层面反删除恢复,肯定无法恢复出来合适的数据文件,只能做底层数据块扫描恢复,参考以前类似case:
rm -rf误删Oracle数据库恢复
win系统删除oracle数据文件恢复
Oracle 数据文件大小为0kb或者文件丢失恢复
解决一次硬件恢复之后数据文件0kb的故障恢复case
这个客户的情况相对复杂一些:
1. 该磁盘分区中有历史库(也就是说单纯的软件直接按照rdba方式无法直接区分出来合适的数据块,儿实现数据重组)
2. 删除的文件比较多(33个数据文件),分区较大(5T+)
3. 删除文件之后,分区还写入了不少数据,会引起一些覆盖和导致碎片数量增加,导致工作量增加和恢复效果变差
通过对客户alert日志分析,发现一个好消息,客户数据每个数据文件是固定大小(没有设置自增长),这种情况,一般来说数据比较连续,碎片相对比较容易区分出来.
add_datafile
通过工具扫描识别出来oracle block,并把结果记录到数据库中,然后通过人工在数据库中对其进行挑选识别,然后生成dd语句恢复出来数据文件,比如这个只是被覆盖了文件头的22号文件,就比较容易恢复
dd-ok
对于一些碎片严重的文件,就需要人工生成大量dd语句来恢复
dd
对于所有恢复出来的文件,使用工具检查坏块情况
check
然后把这些数据文件中的数据恢复到新库中,完成本次数据恢复工作,最大限度抢救客户数据.

发表在 非常规恢复 | 标签为 , , , , | 评论关闭

Oracle dul 最新版(12.2.0.2.10)

发表于 2025 年 8 月 31 日 由 惜分飞

Oracle官方的数据文件离线unload工具Data UnLoader(DUL),依旧保持强盛生命力,虽然开发者Bernard van Duijnen已经退休,但是产品依旧保持更新,这个唯一一款从Oracle 6开始支持的离线unload数据的产品.

[oracle@www.xifenfei.com dul]$ ./dul

Data UnLoader: 12.2.0.2.10 - Internal Only - on Sun Aug 31 21:04:44 2025
with 64-bit io functions and the decompression option

Copyright (c) 1994 2025 Bernard van Duijnen All rights reserved.

 Strictly Oracle Internal Use Only


DUL: Warning: ulimit process stack size is only 33554432
Found db_id = 1780931490
Found db_name = XIFENFEI
DUL> bootstrap;
Probing file = 1, block = 520
. unloading table                BOOTSTRAP$
DUL: Warning: block number is non zero but marked deferred trying to process it anyhow
      60 rows unloaded
Reading BOOTSTRAP.dat 60 entries loaded
Parsing Bootstrap$ contents
Generating dict.ddl for version 11
 OBJ$: segobjno 18, file 1 block 240
 TAB$: segobjno 2, tabno 1, file 1  block 144
 COL$: segobjno 2, tabno 5, file 1  block 144
 USER$: segobjno 10, tabno 1, file 1  block 208
Running generated file "@dict.ddl" to unload the dictionary tables
. unloading table                      OBJ$   86361 rows unloaded
. unloading table                      TAB$    2898 rows unloaded
. unloading table                      COL$   94610 rows unloaded
. unloading table                     USER$      86 rows unloaded
Reading USER.dat 86 entries loaded
Reading OBJ.dat 86361 entries loaded and sorted 86361 entries
Reading TAB.dat 2898 entries loaded
Reading COL.dat 94610 entries loaded and sorted 94610 entries
Reading BOOTSTRAP.dat 60 entries loaded

DUL: Warning: Recreating file "dict.ddl"
Generating dict.ddl for version 11
 OBJ$: segobjno 18, file 1 block 240
 TAB$: segobjno 2, tabno 1, file 1  block 144
 COL$: segobjno 2, tabno 5, file 1  block 144
 USER$: segobjno 10, tabno 1, file 1  block 208
 TABPART$: segobjno 591, file 1 block 4000
 INDPART$: segobjno 596, file 1 block 4040
 TABCOMPART$: segobjno 613, file 1 block 4176
 INDCOMPART$: segobjno 618, file 1 block 4216
 TABSUBPART$: segobjno 603, file 1 block 4096
 INDSUBPART$: segobjno 608, file 1 block 4136
 IND$: segobjno 2, tabno 3, file 1  block 144
 ICOL$: segobjno 2, tabno 4, file 1  block 144
 LOB$: segobjno 2, tabno 6, file 1  block 144
 COLTYPE$: segobjno 2, tabno 7, file 1  block 144
 TYPE$: segobjno 518, tabno 1, file 1  block 3464
 COLLECTION$: segobjno 518, tabno 2, file 1  block 3464
 ATTRIBUTE$: segobjno 518, tabno 3, file 1  block 3464
 LOBFRAG$: segobjno 624, file 1 block 4264
 LOBCOMPPART$: segobjno 627, file 1 block 4288
 UNDO$: segobjno 15, file 1 block 224
 TS$: segobjno 6, tabno 2, file 1  block 176
 PROPS$: segobjno 98, file 1 block 800
Running generated file "@dict.ddl" to unload the dictionary tables
. unloading table                      OBJ$
DUL: Warning: Recreating file "OBJ.ctl"
   86361 rows unloaded
. unloading table                      TAB$
DUL: Warning: Recreating file "TAB.ctl"
    2898 rows unloaded
. unloading table                      COL$
DUL: Warning: Recreating file "COL.ctl"
   94610 rows unloaded
. unloading table                     USER$
DUL: Warning: Recreating file "USER.ctl"
      86 rows unloaded
. unloading table                  TABPART$     104 rows unloaded
. unloading table                  INDPART$     126 rows unloaded
. unloading table               TABCOMPART$       1 row  unloaded
. unloading table               INDCOMPART$       0 rows unloaded
. unloading table               TABSUBPART$      32 rows unloaded
. unloading table               INDSUBPART$       0 rows unloaded
. unloading table                      IND$    4931 rows unloaded
. unloading table                     ICOL$    7644 rows unloaded
. unloading table                      LOB$    1031 rows unloaded
. unloading table                  COLTYPE$    2565 rows unloaded
. unloading table                     TYPE$    2909 rows unloaded
. unloading table               COLLECTION$    1002 rows unloaded
. unloading table                ATTRIBUTE$   11328 rows unloaded
. unloading table                  LOBFRAG$       1 row  unloaded
. unloading table              LOBCOMPPART$       0 rows unloaded
. unloading table                     UNDO$      21 rows unloaded
. unloading table                       TS$       6 rows unloaded
. unloading table                    PROPS$      36 rows unloaded
Reading USER.dat 86 entries loaded
Reading OBJ.dat 86361 entries loaded and sorted 86361 entries
Reading TAB.dat 2898 entries loaded
Reading COL.dat 94610 entries loaded and sorted 94610 entries
Reading TABPART.dat 104 entries loaded and sorted 104 entries
Reading TABCOMPART.dat 1 entries loaded and sorted 1 entries
Reading TABSUBPART.dat 32 entries loaded and sorted 32 entries
Reading INDPART.dat 126 entries loaded and sorted 126 entries
Reading INDCOMPART.dat 0 entries loaded and sorted 0 entries
Reading INDSUBPART.dat 0 entries loaded and sorted 0 entries
Reading IND.dat 4931 entries loaded
Reading LOB.dat
DUL: Notice: Increased the size of DC_LOBS from 1024 to 8192 entries
 1031 entries loaded
Reading ICOL.dat 7644 entries loaded
Reading COLTYPE.dat 2565 entries loaded
Reading TYPE.dat 2909 entries loaded
Reading ATTRIBUTE.dat 11328 entries loaded
Reading COLLECTION.dat 1002 entries loaded
Reading BOOTSTRAP.dat 60 entries loaded
Reading LOBFRAG.dat 1 entries loaded and sorted 1 entries
Reading LOBCOMPPART.dat 0 entries loaded and sorted 0 entries
Reading UNDO.dat 21 entries loaded
Reading TS.dat 6 entries loaded
Reading PROPS.dat 36 entries loaded
Database character set is ZHS16GBK
Database national character set is AL16UTF16
DUL> unload table sys.obj$;
. unloading table                      OBJ$   86361 rows unloaded
DUL>
发表在 Oracle | 标签为 , | 评论关闭

存储掉电强制拉库引起ORA-01555和ORA-01189/ORA-01190故障处理

发表于 2025 年 8 月 30 日 由 惜分飞

机房存储突然掉电导致Oracle数据库访问存储异常,数据库报出大量的ORA-27072: File I/O error,Linux-x86_64 Error: 5: Input/output error,ORA-15081: failed to submit an I/O operation to a disk等错误,实例直接crash

Wed Aug 27 07:11:53 2025
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_m000_17596.trc:
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 6297632
Additional information: -1
WARNING: Read Failed. group:1 disk:0 AU:3075 offset:16384 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
WARNING: failed to read mirror side 1 of virtual extent 0 logical extent 0 of 
  file 284 in group [1.2747812198] from disk DG_0000  allocation unit 3075 reason error; 
  if possible, will try another mirror side
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_m000_17596.trc:
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: Write Failed. group:1 disk:0 AU:3080 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
Additional information: 16384
WARNING: Write Failed. group:1 disk:0 AU:3075 offset:49152 size:16384
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical 
 extent 0 of file 284 in group 1 on disk 0 allocation unit 3075 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 0 logical extent 0
  of file 283 in group 1 on disk 0 allocation unit 3080 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.283.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ckpt_6165.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.283.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-00206: error in writing (block 3, # blocks 1) of control file
ORA-00202: control file: '+DG/xff/controlfile/current.284.918834897'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
CKPT (ospid: 6165): terminating the instance due to error 221
Wed Aug 27 07:11:53 2025
ORA-1092 : opitsk aborting process

存储恢复之后,尝试open数据库报ORA-00333错误（该错误一般是由于redo写丢失导致)

Wed Aug 27 16:36:32 2025
ALTER DATABASE OPEN
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 31 processes
Started redo scan
Incomplete read from log member '+DG/xff/onlinelog/group_2.287.918834905'. Trying next member.
Incomplete read from log member '+DG/xff/onlinelog/group_2.288.918834911'. Trying next member.
Incomplete read from log member '+DG/xff/onlinelog/group_2.287.918834905'. Trying next member.
Abort recovery for domain 0
Aborting crash recovery due to error 333
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_10257.trc:
ORA-00333: redo log read error block 1275904 count 5721
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_10257.trc:
ORA-00333: redo log read error block 1275904 count 5721
ORA-333 signalled during: ALTER DATABASE OPEN...
1
现场人员使用隐含参数,尝试直接拉库操作报ORA-00704 ORA-01555错误,导致拉库失败
1
Wed Aug 27 16:47:11 2025
ALTER DATABASE RECOVER  database until cancel  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 32 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  database until cancel  ...
Wed Aug 27 16:47:56 2025
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log +DG
Wed Aug 27 16:47:56 2025
Errors with log +DG
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-00308: cannot open archived log '+DG'
ORA-17503: ksfdopn:2 Failed to open file +DG
ORA-15045: ASM file name '+DG' is not in reference form
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '+DG/xff/datafile/system.279.918834827'
Slave exiting with ORA-1547 exception
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_pr00_24154.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '+DG/xff/datafile/system.279.918834827'
ORA-1547 signalled during: ALTER DATABASE RECOVER CANCEL ...
Wed Aug 27 16:48:09 2025
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 25330611827626
Resetting resetlogs activation ID 3307041102 (0xc51d714e)
Deleted Oracle managed file +DG/xff/onlinelog/group_1.285.918834899
Deleted Oracle managed file +DG/xff/onlinelog/group_1.286.918834901
Deleted Oracle managed file +DG/xff/onlinelog/group_2.287.918834905
Deleted Oracle managed file +DG/xff/onlinelog/group_2.288.918834911
Wed Aug 27 16:48:28 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_3.289.918834917
Deleted Oracle managed file +DG/xff/onlinelog/group_3.290.918834923
Deleted Oracle managed file +DG/xff/onlinelog/group_4.293.918835035
Deleted Oracle managed file +DG/xff/onlinelog/group_4.294.918835037
Wed Aug 27 16:48:48 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_5.295.918835041
Deleted Oracle managed file +DG/xff/onlinelog/group_5.296.918835047
Deleted Oracle managed file +DG/xff/onlinelog/group_6.297.918835055
Wed Aug 27 16:48:58 2025
Deleted Oracle managed file +DG/xff/onlinelog/group_6.298.918835061
Wed Aug 27 16:49:10 2025
Setting recovery target incarnation to 3
Wed Aug 27 16:49:10 2025
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Wed Aug 27 16:49:10 2025
Assigning activation ID 3598492411 (0xd67ca2fb)
Thread 2 opened at log sequence 1
  Current log# 4 seq# 1 mem# 0: +DG/xff/onlinelog/group_4.294.1210265317
  Current log# 4 seq# 1 mem# 1: +DG/xff/onlinelog/group_4.293.1210265323
Successful open of redo thread 2
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Wed Aug 27 16:49:10 2025
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x1709.be1eb3b1):
select ctime, mtime, stime from obj$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_23787.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 20 with name "_SYSSMU20_1295954159$" too small
Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_ora_23787.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 20 with name "_SYSSMU20_1295954159$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 23787): terminating the instance due to error 704
Instance terminated by USER, pid = 23787
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (23787) as a result of ORA-1092

现场进行了一系列尝试操作,最后我接手数据库之时报错为:ORA-01190 ORA-01110,无法recover,也无法重建controlfile,陷入了死局

Completed: ALTER DATABASE   MOUNT
Sat Aug 30 10:03:20 2025
ALTER DATABASE OPEN
This instance was first to open
Abort recovery for domain 0
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_6435.trc:
ORA-01190: control file or data file 1 is from before the last RESETLOGS
ORA-01110: data file 1: '+DG/xff/datafile/system0829.dbf'
ORA-1190 signalled during: ALTER DATABASE OPEN...

Sat Aug 30 00:56:32 2025
NOTE: Loaded library: System 
SUCCESS: diskgroup DG was mounted
Errors in file /u01/app/oracle/diag/rdbms/xff/xff1/trace/xff1_ora_17302.trc:
ORA-01189: file is from a different RESETLOGS than previous files
ORA-01110: data file 2: '+DG/xff/datafile/sysaux.280.918834827'
ORA-1503 signalled during: create controlfile reuse database xff noarchivelog noresetlogs

对于这种情况,通过Oracle recovery check脚本可以直接发现异常(WRONG RESETLOGS)
wrong-resetlogs

使用Oracle Recovery Tools小工具实现快速恢复
orarecovery
再尝试重建ctl成功
rectl
然后修改数据库scn信息,顺利open数据库
open
后续建议客户逻辑迁移该库

发表在 Oracle备份恢复 | 标签为 , , | 评论关闭