标签云
asm恢复 bbed bootstrap$ dul kcbzib_kcrsds_1 kccpb_sanity_check_2 kcratr_nab_less_than_odr MySQL恢复 obet ORA-00312 ORA-00704 ORA-00742 ORA-01110 ORA-01200 ORA-01555 ORA-01578 ORA-01595 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-600 kcratr_nab_less_than_odr ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle碎片 OSD-04016 YOUR FILES ARE ENCRYPTED 比特币加密文章分类
- MongoDB (1)
- MongoDB恢复 (1)
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (112)
- 数据库 (1,880)
- DB2 (22)
- MySQL (82)
- Oracle (1,705)
- Data Guard (53)
- EXADATA (8)
- GoldenGate (24)
- ORA-xxxxx (168)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (15)
- ORACLE 21C (3)
- Oracle 23ai (8)
- Oracle ASM (72)
- Oracle Bug (8)
- Oracle RAC (56)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (29)
- Oracle备份恢复 (659)
- Oracle安装升级 (108)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (91)
- PostgreSQL (37)
- pdu工具 (7)
- PostgreSQL恢复 (13)
- SQL Server (34)
- SQL Server恢复 (14)
- TimesTen (7)
- 达梦数据库 (5)
- 达梦恢复 (3)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (51)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (34)
- MongoDB (1)
-
最近发表
- Firebird 损坏数据库数据抢救工具—firebirdext
- Oracle 19c 202610补丁(RUs+OJVM)-19.33
- 由于con$.con#值达到极限,触发数据库增加约束报ORA-600 12807错误
- MongoDB 离线提取工具—mongoext
- .bres勒索病毒加密数据库dmp文件可恢复
- ORA-600 krsm_mrp_rebuild_lf_list.lno_mismatch
- 记录一次0丢失的ORA-00354: 损坏重做日志块标头故障恢复
- 不当数据库恢复操作导致一个月数据丢失
- obet快速修复oracle 位图损坏块
- 不太常见的10.2.0.1的oracle redo损坏恢复
- obet forcecopy功能抢救硬件故障中的数据文件
- kcratr_nab_less_than_odr和system坏块故障处理
- 通过obet 恢复system坏块,打开数据库
- obet dbv功能完整说明
- 分享一例运行在aix上的sap系统数据库恢复过程
- OBET-Oracle Block Editor Tool使用说明
- 几乎动用了所有手段的Oracle故障恢复
- Oracle Block Edit Tool (obet) 功能增强–2026.07
- Oracle 19c 202607补丁(RUs+OJVM)-19.32
- Patch_SCN快速修复ORA-01555数据库open故障
分类目录归档:Oracle备份恢复
ORA-600 16703故障解析—tab$表被清空
最近连续遇到两次数据库启动报ORA-600 16703错误,而导致数据库无法正常启动的问题
Completed: ALTER DATABASE RECOVER database alter database open Beginning crash recovery of 1 threads parallel recovery started with 32 processes Started redo scan Completed redo scan read 0 KB redo, 0 data blocks need recovery Started redo application at Thread 1: logseq 993752, block 2, scn 14872376881763 Recovery of Online Redo Log: Thread 1 Group 6 Seq 993752 Reading mem 0 Mem# 0: /oracle/oradata/xifenfei/redo06.log Completed redo application of 0.00MB Completed crash recovery at Thread 1: logseq 993752, block 3, scn 14872376901765 0 data blocks read, 0 data blocks written, 0 redo k-bytes read Tue Jul 04 23:13:29 2017 Thread 1 advanced to log sequence 993753 (thread open) Thread 1 opened at log sequence 993753 Current log# 7 seq# 993753 mem# 0: /oracle/oradata/xifenfei/redo07.log Successful open of redo thread 1 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Tue Jul 04 23:13:29 2017 SMON: enabling cache recovery Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc (incident=163595): ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_163595/xifenfei_ora_15886_i163595.trc Tue Jul 04 23:13:30 2017 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Errors in file /oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_15886.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [] Error 704 happened during db open, shutting down database USER (ospid: 15886): terminating the instance due to error 704 Instance terminated by USER, pid = 15886 ORA-1092 signalled during: alter database open... opiodr aborting process unknown ospid (15886) as a result of ORA-1092
这里报错比较明显ORA-600 16703,而且是在启动时bootstrap$中的对象出现该问题.
10046分析启动过程
===================== select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1 END OF STMT PARSE #140048443935120:c=0,e=390,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905161433 ===================== select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: select blevel, leafcnt, distkey, lblkkey, dblkkey, clufac, nvl(degree,1), nvl(instances,1) from ind$ where bo# = :1 and obj# = :2 END OF STMT PARSE #140048443934176:c=1000,e=601,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162088 ===================== PARSING IN CURSOR #140048443933232 len=70 dep=1 uid=0 oct=3 lid=0 tim=1499185905162444 hv=3377894161 ad='84f13d70' sqlid='32d4jrb4pd4sj' select charsetid, charsetform from col$ where obj# = :1 and col# = :2 END OF STMT PARSE #140048443933232:c=0,e=294,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905162443 ===================== PARSING IN CURSOR #140048443932288 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499185905247020 hv=429618617 ad='84f0f2d0' sqlid='4krwuz0ctqxdt' select ctime, mtime, stime from obj$ where obj# = :1 END OF STMT PARSE #140048443932288:c=0,e=549,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499185905247019 BINDS #140048443932288: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=00 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91b87bd0 bln=22 avl=02 flg=05 value=20 EXEC #140048443932288:c=2000,e=2686,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1218588913,tim=1499185905249810 WAIT #140048443932288: nam='db file sequential read' ela= 6205 file#=1 block#=337 blocks=1 obj#=36 tim=1499185905256132 WAIT #140048443932288: nam='db file sequential read' ela= 3739 file#=1 block#=338 blocks=1 obj#=36 tim=1499185905266704 WAIT #140048443932288: nam='db file sequential read' ela= 4966 file#=1 block#=241 blocks=1 obj#=18 tim=1499185905271761 FETCH #140048443932288:c=0,e=21976,p=3,cr=3,cu=0,mis=0,r=1,dep=1,og=4,plh=1218588913,tim=1499185905271820 STAT #140048443932288 id=1 cnt=1 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=3 pr=3 pw=0 time=21993 us)' STAT #140048443932288 id=2 cnt=1 pid=1 pos=1 obj=36 op='INDEX RANGE SCAN I_OBJ1 (cr=2 pr=2 pw=0 time=16923 us)' CLOSE #140048443932288:c=0,e=63,dep=1,type=0,tim=1499185905271941 BINDS #140048443935120: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f5f91c07de8 bln=22 avl=02 flg=05 value=20 EXEC #140048443935120:c=1000,e=795,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=2970138452,tim=1499185905272802 WAIT #140048443935120: nam='db file sequential read' ela= 3197 file#=1 block#=169 blocks=1 obj#=3 tim=1499185905276069 WAIT #140048443935120: nam='db file sequential read' ela= 3459 file#=1 block#=170 blocks=1 obj#=3 tim=1499185905404084 WAIT #140048443935120: nam='db file sequential read' ela= 6358 file#=1 block#=145 blocks=1 obj#=4 tim=1499185905410548 FETCH #140048443935120:c=999,e=137805,p=3,cr=3,cu=0,mis=0,r=0,dep=1,og=4,plh=2970138452,tim=1499185905410635 STAT #140048443935120 id=1 cnt=0 pid=0 pos=1 obj=4 op='TABLE ACCESS CLUSTER TAB$ (cr=3 pr=3 pw=0 time=137810 us)' STAT #140048443935120 id=2 cnt=1 pid=1 pos=1 obj=3 op='INDEX UNIQUE SCAN I_OBJ# (cr=2 pr=2 pw=0 time=131330 us)' *** 2017-07-05 00:31:46.094 Incident 176395 created, dump file: /oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_176395/xifenfei_ora_51261_i176395.trc ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
报错信息明显,由于select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1无法正常执行.从而出现了ORA-600 16703的错误,更加直接一点的解释就是obj#=20的对象在tab$中找不到记录,从而出现此类报错.和官方解释ORA-600 TAB$和obj$不匹配一致.
分析system文件
通过dul等工具分析system文件发现tab$表记录为空
Data UnLoader: 11.2.0.1.5 - Internal Only - on Wed Jul 05 01:28:53 2017 with 64-bit io functions and the decompression option Copyright (c) 1994 2017 Bernard van Duijnen All rights reserved. Strictly Oracle Internal Use Only Found db_id = 1334610369 Found db_name = xifenfei DUL> unload table TAB$( OBJ# number, DATAOBJ# number, 2 TS# number, FILE# number, BLOCK# number, 3 BOBJ# number, TAB# number, COLS number, CLUCOLS number, 4 PCTFREE$ ignore, PCTUSED$ ignore, INITRANS ignore, MAXTRANS ignore, 5 FLAGS ignore, AUDIT$ ignore, ROWCNT ignore, BLKCNT ignore, 6 EMPCNT ignore, AVGSPC ignore, CHNCNT ignore, AVGRLN ignore, 7 AVGSPC_FLB ignore, FLBCNT ignore, 8 ANALYZETIME ignore, SAMPLESIZE ignore, 9 DEGREE ignore, INSTANCES ignore, 10 INTCOLS ignore, KERNELCOLS number, PROPERTY number) 11 cluster C_OBJ#(OBJ#) 12 storage ( tablespace 0 segobjno 2 tabno 1 file 1 block 144); . unloading table TAB$ 0 rows unloaded DUL> unload table OBJ$( OBJ# number, DATAOBJ# number, OWNER# number, 2 NAME clean varchar2(30), NAMESPACE ignore, SUBNAME clean varchar2(30), 3 TYPE# number, CTIME ignore, MTIME ignore, STIME ignore, 4 STATUS ignore, REMOTEOWNER ignore, LINKNAME ignore, 5 FLAGS ignore, OID$ hexraw) 6 storage ( tablespace 0 segobjno 18 file 1 block 240); . unloading table OBJ$ 89804 rows unloaded DUL>
发现在obj$中有创建表ORACHKBEC66CBE055000000000001(ORACHK+16进制24位)的一个表名字

该表用途通过分析数据库日志发现
create table ORACHKBEC66CBE055000000000001 tablespace system as select * from sys.tab$;
也就是说,这个orachk的表是用来备份tab$的,然后进一步发现有delete from tab$.至此基本上分析清楚,tab$表备份到ORACHK表中,然后delete tab$表数据.实现数据库破坏以及难以恢复的效果.有点类似plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案的破坏案例
我们可以通过使用bbed对tab$表数据数据进行恢复实现数据库正常启动,实现数据0丢失,最大限度抢救您的数据和减少业务恢复时间
Phone:17813235971 Q Q:107644445
E-Mail:dba@xifenfei.com
源头已经分析清楚,是有人篡改了oracle安装介质,导致数据库运行300天之后出现该问题,具体见:警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
file$ 删除记录恢复(delete file$ recovery)
最近遭遇几次有人因为对oracle不太理解,由于各种情况下,删除了file$中的部分记录,从而使得该文件之后的文件都丢失,使得数据库出现各种异常情况。这里演示了可以重启数据库的情况下两种常见的删除file$中记录的恢复
创建表空间和表
SQL> startup
ORACLE instance started.
Total System Global Area 2421825536 bytes
Fixed Size 2215744 bytes
Variable Size 1828716736 bytes
Database Buffers 570425344 bytes
Redo Buffers 20467712 bytes
Database mounted.
Database opened.
SQL> create tablespace tbs_delete_file datafile '/home/oracle/oradata/xifenfei/file01.dbf' size 128M;
Tablespace created.
SQL> alter tablespace tbs_delete_file add datafile '/home/oracle/oradata/xifenfei/file02.dbf' size 128M;
Tablespace altered.
SQL> alter tablespace tbs_delete_file add datafile '/home/oracle/oradata/xifenfei/file03.dbf' size 128M;
Tablespace altered.
SQL> create table t_xifenfei tablespace tbs_delete_file
2 as select * from dba_objects;
Table created.
SQL> insert into t_xifenfei select * from dba_objects;
71895 rows created.
SQL> /
71895 rows created.
SQL> /
71895 rows created.
SQL> /
71895 rows created.
SQL> commit;
Commit complete.
SQL> select count(*) from t_xifenfei;
COUNT(*)
----------
359475
删除file$中记录
SQL> select ts#,file# from file$;
TS# FILE#
---------- ----------
0 1
1 2
2 3
4 4
6 5
6 6
6 7
7 rows selected.
[oracle@localhost ~]$ ss
SQL*Plus: Release 11.2.0.1.0 Production on Fri Jun 2 23:30:57 2017
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> delete from file$ where file#=5;
1 row deleted.
SQL> commit;
Commit complete.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
不重建控制文件,重启数据库
[oracle@localhost tmp]$ ss
SQL*Plus: Release 11.2.0.1.0 Production on Fri Jun 2 23:46:33 2017
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2421825536 bytes
Fixed Size 2215744 bytes
Variable Size 1828716736 bytes
Database Buffers 570425344 bytes
Redo Buffers 20467712 bytes
Database mounted.
Database opened.
SQL> select file# from file$;
FILE#
----------
1
2
3
4
6
7
6 rows selected.
SQL> select file# from v$datafile;
FILE#
----------
1
2
3
4
5
6
7
7 rows selected.
SQL> select count(*) from t_xifenfei;
COUNT(*)
----------
359475
数据库启动正常,而且文件也未从控制文件中删除,而且记录查询正常,考虑通过逻辑方式迁移数据。
测试重建控制文件
SQL> startup nomount
ORACLE instance started.
Total System Global Area 2421825536 bytes
Fixed Size 2215744 bytes
Variable Size 1828716736 bytes
Database Buffers 570425344 bytes
Redo Buffers 20467712 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "TEST" NORESETLOGS NOARCHIVELOG
2 MAXLOGFILES 16
3 MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 292
4 5 6 7 LOGFILE
8 GROUP 1 '/home/oracle/oradata/xifenfei/redo01.log' SIZE 50M BLOCKSIZE 512,
GROUP 2 '/home/oracle/oradata/xifenfei/redo02.log' SIZE 50M BLOCKSIZE 512,
GROUP 3 '/home/oracle/oradata/xifenfei/redo03.log' SIZE 50M BLOCKSIZE 512
9 10 11 DATAFILE
12 '/home/oracle/oradata/xifenfei/system01.dbf',
'/home/oracle/oradata/xifenfei/sysaux01.dbf',
'/home/oracle/oradata/xifenfei/undotbs01.dbf',
13 14 15 '/home/oracle/oradata/xifenfei/users01.dbf',
'/home/oracle/oradata/xifenfei/file01.dbf',
16 17 '/home/oracle/oradata/xifenfei/file02.dbf',
'/home/oracle/oradata/xifenfei/file03.dbf'
CHARACTER SET AL32UTF8
18 19 20 ;
Control file created.
SQL> alter database open;
Database altered.
SQL> select file# from v$datafile;
FILE#
----------
1
2
3
4
SQL> select file# from file$;
FILE#
----------
1
2
3
4
6
7
6 rows selected.
SQL> select count(*) from t_xifenfei;
select count(*) from t_xifenfei
*
ERROR at line 1:
ORA-00600: internal error code, arguments: [25029], [6], [], [], [], [], [],[], [], [], [], []
删除file$中记录,然后重启库之后,大于删除的file#之后的数据文件全部丢失.
alert日志报错
Fri Jun 02 23:49:42 2017
alter database open
Fri Jun 02 23:49:42 2017
Thread 1 advanced to log sequence 9 (thread open)
Thread 1 opened at log sequence 9
Current log# 3 seq# 9 mem# 0: /home/oracle/oradata/xifenfei/redo03.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jun 02 23:49:42 2017
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
File #5 in the controlfile not found in data dictionary.
Removing file from controlfile.
data file 5: '/home/oracle/oradata/xifenfei/file01.dbf'
File #6 in the controlfile not found in data dictionary.
Removing file from controlfile.
data file 6: '/home/oracle/oradata/xifenfei/file02.dbf'
File #7 in the controlfile not found in data dictionary.
Removing file from controlfile.
data file 7: '/home/oracle/oradata/xifenfei/file03.dbf'
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
This condition can occur when a backup controlfile has
been restored. It may be necessary to add files to these
tablespaces. That can be done using the SQL statement:
ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
Alternatively, if these temporary tablespaces are no longer
needed, then they can be dropped.
Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
No Resource Manager plan active
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
then disabled.
**********************************************************
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Fri Jun 02 23:49:43 2017
QMNC started with pid=20, OS id=11886
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open
Fri Jun 02 23:49:44 2017
db_recovery_file_dest_size of 3882 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Fri Jun 02 23:49:44 2017
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_m001_11890.trc (incident=84344):
ORA-00600: internal error code, arguments: [25029], [6], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/test/test/incident/incdir_84344/test_m001_11890_i84344.trc
Errors in file /opt/oracle/diag/rdbms/test/test/trace/test_m001_11890.trc:
ORA-00600: internal error code, arguments: [25029], [6], [], [], [], [], [], [], [], [], [], []
Fri Jun 02 23:49:45 2017
Trace dumping is performing id=[cdmp_20170602234945]
Fri Jun 02 23:49:46 2017
Starting background process CJQ0
Fri Jun 02 23:49:46 2017
CJQ0 started with pid=21, OS id=11902
这里报错明显,由于file$.file#=5被删除,重建控制文件之后,file#在5之后的数据文件全部从控制文件中删除,类似提示File #5 in the controlfile not found in data dictionary. Removing file from controlfile.
插入file$记录恢复
--通过dump文件头和相关block,找出来对应记录
插入被file$删除记录
SQL> insert into file$ values(5,2,16384,6,5,0,0,0,993135,null,20971522,null,null,null);
1 row created.
SQL> commit;
Commit complete.
SQL> select file# from file$;
FILE#
----------
1
2
3
4
5
6
7
7 rows selected.
--重启数据库,创建控制文件
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup nomount
ORACLE instance started.
Total System Global Area 2421825536 bytes
Fixed Size 2215744 bytes
Variable Size 1828716736 bytes
Database Buffers 570425344 bytes
Redo Buffers 20467712 bytes
SQL> CREATE CONTROLFILE REUSE DATABASE "TEST" NORESETLOGS NOARCHIVELOG
2 MAXLOGFILES 16
3 MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
4 5 6 MAXLOGHISTORY 292
7 LOGFILE
GROUP 1 '/home/oracle/oradata/xifenfei/redo01.log' SIZE 50M BLOCKSIZE 512,
GROUP 2 '/home/oracle/oradata/xifenfei/redo02.log' SIZE 50M BLOCKSIZE 512,
8 9 10 GROUP 3 '/home/oracle/oradata/xifenfei/redo03.log' SIZE 50M BLOCKSIZE 512
DATAFILE
11 12 '/home/oracle/oradata/xifenfei/system01.dbf',
'/home/oracle/oradata/xifenfei/sysaux01.dbf',
13 14 '/home/oracle/oradata/xifenfei/undotbs01.dbf',
15 '/home/oracle/oradata/xifenfei/users01.dbf',
'/home/oracle/oradata/xifenfei/file01.dbf',
'/home/oracle/oradata/xifenfei/file02.dbf',
16 17 18 '/home/oracle/oradata/xifenfei/file03.dbf'
19 CHARACTER SET AL32UTF8
; 20
Control file created.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01113: file 5 needs media recovery
ORA-01110: data file 5: '/home/oracle/oradata/xifenfei/file01.dbf'
SQL> recover database;
Media recovery complete.
SQL> alter database open;
Database altered.
SQL> select file# from file$;
FILE#
----------
1
2
3
4
5
6
7
7 rows selected.
SQL> select file# from v$datafile;
FILE#
----------
1
2
3
4
5
6
7
7 rows selected.
SQL> select count(*) from t_xifenfei;
COUNT(*)
----------
359475
alert日志正常,未提示删除控制文件中数据文件
alter database open
Fri Jun 02 23:56:52 2017
Thread 1 advanced to log sequence 10 (thread open)
Thread 1 opened at log sequence 10
Current log# 1 seq# 10 mem# 0: /home/oracle/oradata/xifenfei/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jun 02 23:56:52 2017
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
This condition can occur when a backup controlfile has
been restored. It may be necessary to add files to these
tablespaces. That can be done using the SQL statement:
ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
Alternatively, if these temporary tablespaces are no longer
needed, then they can be dropped.
Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
No Resource Manager plan active
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
then disabled.
**********************************************************
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Fri Jun 02 23:56:53 2017
QMNC started with pid=20, OS id=12127
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open
通过插入删除记录,重建控制文件,数据库恢复正常,而且相关记录也可以查询。
结论总结
1. 如果删除file$中的记录,而且控制文件未重建,数据库可以正常启动,而且可以查询数据
2. 如果删除file$中的记录,而且控制文件被重建,在数据库启动过程中,从被删除文件之后的所有文件记录从控制文件中删除(类似:File N in the controlfile not found in data dictionary. Removing file from controlfile.).可以在数据库open之后,插入被删除的记录,重建控制文件,数据会被恢复回来.如果数据库无法启动,需要通过其他方式进行恢复被删除记录
asm磁盘分区丢失恢复
有朋友反馈,他们做了xx存储的双活之后,重启主机发现gi无法正常启动,分析发现所有该存储的磁盘分区信息丢失,导致asmlib无法发现磁盘(使用分区做asm disk)
类似如下错误(磁盘分区丢失)
--fdisk -l 显示部分结果 Disk /dev/mapper/datahds1: 1099.5 GB, 1099511627776 bytes 255 heads, 63 sectors/track, 133674 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 --ls -l /dev/mapper/ 显示结果无分区信息 lrwxrwxrwx 1 root root 7 May 6 03:44 datahds1 -> ../dm-1 lrwxrwxrwx 1 root root 7 May 6 03:26 datahds2 -> ../dm-3 lrwxrwxrwx 1 root root 7 May 6 03:26 datahds3 -> ../dm-8 lrwxrwxrwx 1 root root 7 May 6 03:26 ocrhds1 -> ../dm-0 lrwxrwxrwx 1 root root 7 May 6 03:26 ocrhds2 -> ../dm-2 lrwxrwxrwx 1 root root 7 May 6 03:26 ocrhds3 -> ../dm-4
asm日志显示
SUCCESS: diskgroup DATADG was mounted NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 3 SUCCESS: diskgroup OCRHDS was mounted ORA-15032: not all alterations performed ORA-15017: diskgroup "DATA" cannot be mounted ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA"
分析系统日志
May 6 02:23:27 db2 kernel: sdb: unknown partition table May 6 02:23:27 db2 kernel: sde: unknown partition table May 6 02:23:27 db2 kernel: sdc: unknown partition table May 6 02:23:27 db2 kernel: sdf: unknown partition table May 6 02:23:27 db2 kernel: sdd: unknown partition table May 6 02:23:27 db2 kernel: sdj:Dev sdj: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sdi: sdi1 May 6 02:23:27 db2 kernel: sdk: sdk1 May 6 02:23:27 db2 kernel: sdg: unknown partition table May 6 02:23:27 db2 kernel: sdl: sdl1 May 6 02:23:27 db2 kernel: sdm:Dev sdm: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sdo:Dev sdo: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sdn:Dev sdn: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sdp:Dev sdp: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sds:Dev sds: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sdh: May 6 02:23:27 db2 kernel: sdt: sdt1 May 6 02:23:27 db2 kernel: sdv:Dev sdv: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sdq:Dev sdq: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sd 1:0:1:9: [sdr] Very big device. Trying to use READ CAPACITY(16). May 6 02:23:27 db2 kernel: sdr:Dev sdr: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sd 2:0:0:9: [sdab] Very big device. Trying to use READ CAPACITY(16). May 6 02:23:27 db2 kernel: sdab: unknown partition table May 6 02:23:27 db2 kernel: sdac: unknown partition table May 6 02:23:27 db2 kernel: sdw: sdw1 May 6 02:23:27 db2 kernel: sdu:Dev sdu: unable to read RDB block 0 May 6 02:23:27 db2 kernel: unable to read partition table May 6 02:23:27 db2 kernel: sdx: sdx1 May 6 02:23:27 db2 kernel: sdy: sdy1 May 6 02:23:27 db2 kernel: sdaa: sdaa1 May 6 02:23:27 db2 kernel: sdz: sdz1 May 6 02:23:27 db2 kernel: sdae: unknown partition table May 6 02:23:27 db2 kernel: sdaf: unknown partition table May 6 02:23:27 db2 kernel: sdag: unknown partition table May 6 02:23:27 db2 kernel: sdai: May 6 02:23:27 db2 kernel: sdah: unknown partition table May 6 02:23:27 db2 kernel: sdad: unknown partition table May 6 02:23:28 db2 mcelog: failed to prefill DIMM database from DMI data
这里错误比较明显unknown partition table,磁盘的分区信息损坏.使用fdisk无法发现分区
partprobe也无效
[root@db2 oracle]# partprobe /dev/mapper/ocrhds3 [root@db2 oracle]# [root@db2 oracle]# ls -l /dev/mapper/ocrhds3* lrwxrwxrwx 1 root root 7 May 6 07:30 /dev/mapper/ocrhds3 -> ../dm-4
从尚需信息看,磁盘的分区表信息应该已经损坏,现在能够做的,就是希望运气好,磁盘的分区的实际数据没有损坏
分析磁盘实际分区数据
[root@db2 ~]$ dd if=/dev/mapper/datahds1 of=/tmp/datahds1.dd bs=1024k count=50 [root@db2 ~]$ dd if=/tmp/datahds1.dd of=/tmp/xff01.dd bs=3225 skip=1 [grid@db2 ~]$ kfed read /tmp/xff01.dd |more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 3110278718 ; 0x00c: 0xb963163e kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISKHDSDATA1 ; 0x000: length=16 kfdhdb.driver.reserved[0]: 1146307656 ; 0x008: 0x44534448 kfdhdb.driver.reserved[1]: 826364993 ; 0x00c: 0x31415441 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 186646528 ; 0x020: 0x0b200000 kfdhdb.dsknum: 0 ; 0x024: 0x0000 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: DATADG_0000 ; 0x028: length=11 kfdhdb.grpname: DATADG ; 0x048: length=6 kfdhdb.fgname: DATADG_0000 ; 0x068: length=11 kfdhdb.capname: ; 0x088: length=0 kfdhdb.crestmp.hi: 33050696 ; 0x0a8: HOUR=0x8 DAYS=0x2 MNTH=0x4 YEAR=0x7e1 kfdhdb.crestmp.lo: 3813740544 ; 0x0ac: USEC=0x0 MSEC=0x44 SECS=0x35 MINS=0x38 kfdhdb.mntstmp.hi: 33050701 ; 0x0b0: HOUR=0xd DAYS=0x2 MNTH=0x4 YEAR=0x7e1 kfdhdb.mntstmp.lo: 411385856 ; 0x0b4: USEC=0x0 MSEC=0x150 SECS=0x8 MINS=0x6
通过上述分析,我们可以初步判断,分区磁盘的信息很可能是好的(因为asm disk header是好的,根据一般的规则从前往后覆盖,既然header是好的,后面的block被覆盖的概率非常小)
通过准备新磁盘直接把磁盘分区dd到新设备上
dd if=/dev/mapper/ocrhds1 of=/dev/mapper/ocrhdsnew1 skip=1 bs=3225 dd if=/dev/mapper/ocrhds2 of=/dev/mapper/ocrhdsnew2 skip=1 bs=3225 dd if=/dev/mapper/ocrhds3 of=/dev/mapper/ocrhdsnew3 skip=1 bs=3225 dd if=/dev/mapper/datahds1 of=/dev/mapper/datahdsnew1 skip=1 bs=3225 dd if=/dev/mapper/datahds2 of=/dev/mapper/datahdsnew2 skip=1 bs=3225 dd if=/dev/mapper/datahds3 of=/dev/mapper/datahdsnew3 skip=1 bs=3225
asmlib重新扫描磁盘
[root@db1 disks]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... Instantiating disk "HDSOCR3" Instantiating disk "HDSDATA2" Instantiating disk "HDSDATA1" Instantiating disk "HDSDATA3" Instantiating disk "HDSOCR1" Instantiating disk "HDSOCR2" [root@db1 disks]# ls -ltr total 0 brw-rw---- 1 grid asmadmin 8, 160 May 6 13:49 HDSOCR3 brw-rw---- 1 grid asmadmin 8, 192 May 6 13:49 HDSDATA2 brw-rw---- 1 grid asmadmin 8, 176 May 6 13:49 HDSDATA1 brw-rw---- 1 grid asmadmin 8, 208 May 6 13:49 HDSDATA3 brw-rw---- 1 grid asmadmin 8, 128 May 6 13:49 HDSOCR1 brw-rw---- 1 grid asmadmin 8, 144 May 6 13:49 HDSOCR2
kfed验证拷贝的分区
[root@db2 tmp]# /oracle/app/11.2.0/grid_1/bin/kfed read /dev/oracleasm/disks/HDSDATA1 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483648 ; 0x008: disk=0 kfbh.check: 3110278718 ; 0x00c: 0xb963163e kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISKHDSDATA1 ; 0x000: length=16 kfdhdb.driver.reserved[0]: 1146307656 ; 0x008: 0x44534448 kfdhdb.driver.reserved[1]: 826364993 ; 0x00c: 0x31415441 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 186646528 ; 0x020: 0x0b200000 kfdhdb.dsknum: 0 ; 0x024: 0x0000 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: DATADG_0000 ; 0x028: length=11 kfdhdb.grpname: DATADG ; 0x048: length=6 kfdhdb.fgname: DATADG_0000 ; 0x068: length=11 kfdhdb.capname: ; 0x088: length=0
asm和数据库启动正常
[grid@db2 ~]$ asmcmd ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 3145710 2378034 0 2378034 0 N DATADG/ MOUNTED NORMAL N 512 4096 1048576 15342 14416 5114 4651 0 Y OCRHDS/ ASMCMD> [oracle@db2 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Sat May 6 13:54:21 2017 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 3.6077E+10 bytes Fixed Size 2260648 bytes Variable Size 7247757656 bytes Database Buffers 2.8723E+10 bytes Redo Buffers 104382464 bytes Database mounted. Database opened. SQL>
通过上述恢复,实现asm磁盘分区丢失数据0丢失
如果您遇到此类情况,无法解决请联系我们,提供专业ORACLE数据库恢复技术支持
Phone:17813235971 Q Q:107644445
E-Mail:dba@xifenfei.com



加我微信(17813235971)
加我QQ(107644445)

