-
3D Cloud
asm恢复 bbed bootstrap$ dmp损坏 dul In Memory kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00353 ORA-00607 ORA-00704 ORA-01110 ORA-01190 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-15042 ORACLE 12C oracle dul ORACLE PATCH oracle加密恢复 oracle勒索 oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 redo异常 sql加密恢复 undo异常 YOUR FILES ARE ENCRYPTED 勒索恢复 数据库恢复 比特币 oracle 比特币加密
文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (87)
- 数据库 (1,334)
- DB2 (22)
- MySQL (57)
- Oracle (1,220)
- Data Guard (40)
- EXADATA (7)
- GoldenGate (19)
- ORA-xxxxx (145)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (8)
- Oracle ASM (56)
- Oracle Bug (7)
- Oracle RAC (41)
- Oracle 安全 (6)
- Oracle 开发 (25)
- Oracle 监听 (26)
- Oracle备份恢复 (403)
- Oracle安装升级 (55)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (49)
- PostgreSQL (10)
- SQL Server (26)
- SQL Server恢复 (7)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (24)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (7)
-
最近发表
- 记录一次oracle现场故障处理经过
- .Globeimposter-Beta666qqz扩展名数据库加密恢复
- .makop病毒加密数据库恢复
- ORA-600 kcratr_scan_lastbwr 恢复
- ORA-600 16703直接把orachk备份表插入到tab$恢复
- oracle dul 12.2正式版发布
- .eking扩展名数据库恢复
- 一次 CRS-1013: ASM 磁盘组中的 OCR 位置不可访问 故障分析
- Oracle Recovery Tools恢复—ORA-00704 ORA-01555故障
- 使用rman from service 搭建dataguard
- rm -rf 删除数据文件恢复方法—文件系统反删除+oracle碎片重组
- [star-new@email.tg].Devos加密数据库恢复
- 硬件恢复之后,数据库无法open故障恢复
- ORA-00742 ORA-00312 故障恢复
- 硬件故障导致ORA-600 2662错误处理
- ORA-00600: internal error code, arguments: [16513], [1403] 恢复
- .IQ0003后缀名勒索恢复
- .Globeimposter-Alpha666qqz加密恢复
- Assistant: Download Reference for Oracle Database/GI PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases-202101
- .sql文件被加密恢复
友情链接
分类目录归档:非常规恢复
记录一次oracle现场故障处理经过
近期到现场进行了一个数据库恢复,我在恢复之前该库先由于硬件进行恢复,然后由其他人对其进行了一系列数据库恢复,但是未恢复成功,客户希望我们到现场进行处理(因为网络原因无法远程).接手库之后,处理第一个问题,是客户在进行现场备份的时候(把linux数据拷贝到win的过程中)发现有几个文件拷贝异常,这个错误很可能是由于当初的硬件故障修复之后留下的后遗症(由于io设备错误,无法运行此项请求),通过工具进行拷贝,恢复出来
DUL> copy file from /oradata2/xifenfeidata.dbf to /oradata2/xifenfeidata.dbf starting copy datafile '/oradata1/xifenfeidata.dbf' to '/oradata2/xifenfeidata.dbf' read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error read block# error: 560171 read data error from file '/oradata1/xifenfeidata.dbf'.error message:Input/output error read block# error: 560179 datafile copy completed with 2 block error.
[oracle@localhost ~]$ dbv file=/oradata2/xifenfeidata.dbf blocksize=16384 DBVERIFY: Release 11.2.0.3.0 - Production on Mon Mar 29 17:28:17 2021 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. DBVERIFY - Verification starting : FILE = /oradata2/xifenfeidata.dbf Page 560171 is marked corrupt Corrupt block relative dba: 0x3bc88c2b (file 239, block 560171) Completely zero block found during dbv: Page 560179 is marked corrupt Corrupt block relative dba: 0x3bc88c33 (file 239, block 560179) Completely zero block found during dbv: DBVERIFY - Verification complete Total Pages Examined : 4194302 Total Pages Processed (Data) : 2230726 Total Pages Failing (Data) : 0 Total Pages Processed (Index): 1936953 Total Pages Failing (Index): 0 Total Pages Processed (Other): 26618 Total Pages Processed (Seg) : 0 Total Pages Failing (Seg) : 0 Total Pages Empty : 3 Total Pages Marked Corrupt : 2 Total Pages Influx : 0 Total Pages Encrypted : 0 Highest block SCN : 304929867 (106.304929867)
修复完相关无法拷贝文件之后,启动数据库报控制文件异常
Mon Mar 29 15:03:38 2021 alter database mount USER (ospid: 29044): terminating the instance Mon Mar 29 15:03:42 2021 System state dump requested by (instance=1, osid=29044), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff/trace/xff_diag_28961.trc Instance terminated by USER, pid = 29044
尝试重建ctl
[oracle@localhost ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Mon Mar 29 17:40:17 2021 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SQL> startup nomount pfile='/tmp/pfile' ORACLE instance started. Total System Global Area 1.7704E+10 bytes Fixed Size 2235568 bytes Variable Size 2348811088 bytes Database Buffers 1.5301E+10 bytes Redo Buffers 52580352 bytes SQL> @/tmp/ctl.sql CREATE CONTROLFILE REUSE DATABASE xff NORESETLOGS NOARCHIVELOG * ERROR at line 1: ORA-01503: CREATE CONTROLFILE failed ORA-01189: file is from a different RESETLOGS than previous files ORA-01110: data file 249: '/oradata/xff/system03.dbf'
初步判断是由于对方之前恢复导致部分文件resetlogs scn异常,通过bbed进行判断确认
BBED> set file 1 FILE# 1 BBED> p kcvfhrls struct kcvfhrls, 8 bytes @116 ub4 kscnbas @116 0x00000001 ub2 kscnwrp @120 0x0000 BBED> set file 249 FILE# 249 BBED> p kcvfhrls struct kcvfhrls, 8 bytes @116 ub4 kscnbas @116 0x00000001 ub2 kscnwrp @120 0x0000
通过bbed修改相关值,然后重建控制文件成功,尝试resetlogs库,报ORA-01248错误
SQL> alter database open resetlogs; alter database open resetlogs * ERROR at line 1: ORA-01248: file 234 was created in the future of incomplete recovery ORA-01110: data file 234: '/oradata1/xifenfeidata5.DBF'
关于ORA-01248的错误解释
01248, 00000, "file %s was created in the future of incomplete recovery" // *Cause: Attempting to do a RESETLOGS open with a file entry in the // control file that was originally created after the UNTIL time // of the incomplete recovery. // Allowing such an entry may hide the version of the file that // is needed at this time. The file number may be in use for // a different file which would be lost if the RESETLOGS was allowed. // *Action: If more recovery is desired then apply redo until the creation // time of the file is reached. If the file is not wanted and the // same file number is not in use at the stop time of the recovery, // then the file can be taken offline with the FOR DROP option. // Otherwise a different control file is needed to allow the RESETLOGS. // Another backup can be restored and recovered, or a control file can // be created via CREATE CONTROLFILE.
大概的意思是文件的创建时间大于文件当前的scn,通过查询确实如此
SQL> select file#,CREATION_CHANGE#,CREATION_TIME from v$datafile_header where file#=234; FILE# CREATION_CHANGE# CREATION_ ---------------- ---------------- --------- 234 419298664864 02-AUG-19 SQL> SELECT status, 2 to_char(checkpoint_change#,'9999999999999999') "SCN", 3 to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY, 4 count(*) ROW_NUM 5 FROM v$datafile_header 6 GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy 7 ORDER BY status, checkpoint_change#, checkpoint_time; STATUS SCN CHECKPOINT_TIME FUZ ROW_NUM ------- ----------------- ------------------- --- ---------------- ONLINE 417750848223 2021-02-23 23:50:46 YES 7 ONLINE 417750848223 2021-03-21 11:44:25 NO 396
通过对部分scn进行修改(比如减小创建时间的scn),然后尝试resetlogs库
SQL> alter database open resetlogs; alter database open resetlogs * ERROR at line 1: ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00604: error occurred at recursive SQL level 1 ORA-01555: snapshot too old: rollback segment number 5 with name "_SYSSMU5_2708889888$" too small Process ID: 3182 Session ID: 1 Serial number: 3
这个错误比较简单,参考以前的部分文章:在数据库open过程中常遇到ORA-01555汇总数据库open过程遭遇ORA-1555对应sql语句补充,处理之后,数据库open成功
SQL> startup mount; ORACLE instance started. Total System Global Area 1.7704E+10 bytes Fixed Size 2235568 bytes Variable Size 2348811088 bytes Database Buffers 1.5301E+10 bytes Redo Buffers 52580352 bytes Database mounted. SQL> alter database open; Database altered.
本次数据库恢复基本上完成,已经最大限度恢复数据,导出数据到新库,完成恢复任务
ORA-600 16703直接把orachk备份表插入到tab$恢复
有一个朋友和我说,他们数据库出现了以下错误ORA-600 16703 错误
他们是在虚拟化环境中,可以恢复到上一个快照点,但是主机启动之后,数据库依旧异常,让我们进行处理
C:\Users\Administrator>sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Wed Mar 24 17:04:01 2021 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> select open_mode from v$database; OPEN_MODE -------------------- READ WRITE SQL> select count(1) from tab$; COUNT(1) ---------- 0
很明显tab$已经被清空,数据库无法正常使用.因为库没有crash,尝试把备份的orachk表插入进来
SQL> insert into tab$ select * from ORACHKB514061BDCB10EBA9CF58F3; 6318 rows created. SQL> commit; Commit complete. SQL> select 'DROP TRIGGER '||owner||'."'||TRIGGER_NAME||'";' from dba_triggers w here TRIGGER_NAME like 'DBMS_%_INTERNAL% ' 2 union all 3 select 'DROP PROCEDURE '||owner||'."'||a.object_name||'";' from dba_procedu res a where a.object_name like 'DBMS_%_INTERNAL% ' 4 union all 5 select 'drop '||object_type||' '||owner||'.'||object_name||';' from dba_obj ects where object_name in('DBMS_SUPPORT_DBMONITOR','DBMS_SUPPORT_DBMONITORP'); 'DROPTRIGGER'||OWNER||'."'||TRIGGER_NAME||'";' -------------------------------------------------------------------------------- drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP; drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR; SQL> drop PROCEDURE SYS.DBMS_SUPPORT_DBMONITORP; Procedure dropped. SQL> drop TRIGGER SYS.DBMS_SUPPORT_DBMONITOR; Trigger dropped. SQL> commit; Commit complete. SQL>
重启数据库,该故障恢复完成,数据完美恢复0丢失.
硬件恢复之后,数据库无法open故障恢复
由于硬件故障,客户恢复硬件之后,数据库无法正常启动,报ORA-00354 ORA-00353错误
/tmp/> sqlplus / as sysdba SQL*Plus: Release 11.2.0.3.0 Production on Mon Mar 1 17:10:30 2021 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> recover database; ORA-00283: recovery session canceled due to errors ORA-00354: corrupt redo log block header ORA-00353: log corruption near block 86088 change 16135545783340 time 02/23/2021 13:53:24 ORA-00312: online log 2 thread 1: '/oradata02/redo02b.log' ORA-00312: online log 2 thread 1: '/oradata01/redo02a.log'
由于redo损坏,数据库无法继续正常恢复,通过屏蔽一致性,force open库
SQL> alter database open resetlogs; alter database open resetlogs * ERROR at line 1: ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [2662], [3756], [3571444619], [3756], [3648471803], [4194545] Process ID: 5104 Session ID: 576 Serial number: 3
这个错误比较简单,是由于scn问题导致,修改数据库scn启动库
SQL> alter database open; alter database open * ERROR at line 1: ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [] Process ID: 5536 Session ID: 576 Serial number: 1
这个错误比较明显,修改回滚段,尝试启动库
SQL> alter database open; alter database open * ERROR at line 1: ORA-03113: end-of-file on communication channel Process ID: 6033 Session ID: 576 Serial number: 3
数据库依旧无法正常open,alert日志报错如下
ARC3 started with pid=30, OS id=6078 ARC1: Archival started ARC2: Archival started ARC1: Becoming the 'no FAL' ARCH ARC1: Becoming the 'no SRL' ARCH ARC2: Becoming the heartbeat ARCH Exception[type:SIGSEGV Address not mapped to object][ADDR:0x60173487F5][PC:0xC003B1C20,_memcpy()+64][flags:0x0,count:1] Exception[type:SIGSEGV,Address not mapped to object][ADDR:0x60173487F5][PC:0xC003B1C20,_memcpy()+64][flags:0x2,count:2] Exception[type:SIGSEGV,Address not mapped to object][ADDR:0x60173487F5][PC:0xC003B1C20,_memcpy()+64][flags:0x2,count:2] Archived Log entry 2 added for thread 1 sequence 2 ID 0x506cafbb dest 1: ARC3: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE Mon Mar 01 17:44:44 2021 PMON (ospid: 5993): terminating the instance due to error 397 Mon Mar 01 17:44:45 2021 System state dump requested by (instance=1, osid=5993 (PMON)), summary=[abnormal instance termination]. System State dumped to trace file /oracle/diag/rdbms/xff/xff/trace/xff_diag_6001.trc Instance terminated by PMON, pid = 5993
通过其启动过程分析,发现数据库卡在如下对象:
PARSING IN CURSOR #11529215044940435280 len=148 dep=1 uid=0 oct=6 lid=0 tim=223080942765 hv=3540833987 ad='c000000d67a42778' sqlid='5ansr7r9htpq3' update undo$ set name=:2,file#=:3,block#=:4,status$=:5,user#=:6,undosqn=:7,xactsqn=:8, scnbas=:9,scnwrp=:10,inst#=:11,ts#=:12,spare1=:13 where us#=:1 END OF STMT PARSE #11529215044940435280:c=10000,e=8182,p=6,cr=55,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=223080942764 BINDS #11529215044940435280: Bind#0 oacdty=01 mxl=32(20) mxlc=00 mal=00 scl=00 pre=00 oacflg=18 fl2=0001 frm=01 csi=873 siz=32 off=0 kxsbbbfp=c000000d5fd299aa bln=32 avl=20 flg=09 value="_SYSSMU1_3935275865$" Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6e078 bln=24 avl=02 flg=05 value=3 Bind#2 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6e048 bln=24 avl=03 flg=05 value=128 Bind#3 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6e010 bln=24 avl=02 flg=05 value=5 Bind#4 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6dfe0 bln=24 avl=02 flg=05 value=1 Bind#5 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6dfb0 bln=24 avl=04 flg=05 value=28921 Bind#6 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6df80 bln=24 avl=05 flg=05 value=1245262 Bind#7 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6df48 bln=24 avl=06 flg=05 value=1217986655 Bind#8 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6dc90 bln=24 avl=03 flg=05 value=3621 Bind#9 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6dc60 bln=24 avl=01 flg=05 value=0 Bind#10 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6dc30 bln=24 avl=02 flg=05 value=2 Bind#11 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6dc00 bln=24 avl=02 flg=05 value=2 Bind#12 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=9fffffffbcc6e0a8 bln=22 avl=02 flg=05 value=1 WAIT #11529215044940435280: nam='db file sequential read' ela= 21 file#=1 block#=530 blocks=1 obj#=0 tim=223080944352 Incident 528204 created, dump file: /oracle/diag/rdbms/xff/xff/incident/incdir_528204/xff_ora_6593_i528204.trc ORA-00600: internal error code, arguments: [4193], [], [], [], [], [], [], [], [], [], [], []
至此基本上可以确认是由于出现回滚段异常,继续查看日志发现
Error 600 in redo application callback Dump of change vector: TYP:0 CLS:16 AFN:1 DBA:0x00400212 OBJ:4294967295 SCN:0x0ea6.f4f2da14 SEQ:1 OP:5.1 ENC:0 RBL:0 ktudb redo: siz: 320 spc: 5892 flg: 0x0012 seq: 0x0072 rec: 0x08 xid: 0x0000.004.000000bc ktubl redo: slt: 4 rci: 0 opc: 11.1 [objn: 15 objd: 15 tsn: 0] Undo type: Regular undo Begin trans Last buffer split: No Temp Object: No Tablespace Undo: No 0x00000000 prev ctl uba: 0x00400212.0072.07 prev ctl max cmt scn: 0x0eac.d42963be prev tx cmt scn: 0x0eac.d4296f48 txn start scn: 0xffff.ffffffff logon user: 0 prev brb: 4194446 prev bcl: 0 BuExt idx: 0 flg2: 0 KDO undo record: KTB Redo op: 0x04 ver: 0x01 compat bit: 4 (post-11) padding: 1 op: L itl: xid: 0x0000.060.000000bb uba: 0x00400212.0072.04 flg: C--- lkc: 0 scn: 0x0eac.d9736b46 KDO Op code: URP row dependencies Disabled xtype: XA flags: 0x00000000 bdba: 0x004000e1 hdba: 0x004000e0 itli: 4 ispac: 0 maxfr: 4863 tabn: 0 slot: 1(0x1) flag: 0x2c lock: 0 ckix: 0 ncol: 17 nnew: 12 size: 0 col 1: [20] 5f 53 59 53 53 4d 55 31 5f 33 39 33 35 32 37 35 38 36 35 24 col 2: [ 2] c1 02 col 3: [ 2] c1 04 col 4: [ 3] c2 02 1d col 5: [ 6] c5 0d 12 63 43 38 col 6: [ 3] c2 25 16 col 7: [ 5] c4 02 19 35 3f col 8: [ 4] c3 03 5a 16 col 9: [ 1] 80 col 10: [ 2] c1 04 col 11: [ 2] c1 03 col 16: [ 2] c1 03 Block after image is corrupt: buffer tsn: 0 rdba: 0x00400212 (1/530) scn: 0x0ea6.f4f2da14 seq: 0x01 flg: 0x04 tail: 0xda140201 frmt: 0x02 chkval: 0x9dd8 type: 0x02=KTU UNDO BLOCK
使用bbed对file 1 block 530进行处理
struct ktuxcscn, 8 bytes @4148 ub4 kscnbas @4148 0xd42963be ub2 kscnwrp @4152 0x0eac struct ktuxcuba, 8 bytes @4156 ub4 kubadba @4156 0x00400212 ub2 kubaseq @4160 0x0072 ub1 kubarec @4162 0x07 sb2 ktuxcflg @4164 1 (KTUXCFSK) ub2 ktuxcseq @4166 0x0072 sb2 ktuxcnfb @4168 1 ub4 ktuxcinc @4172 0x00000000 sb2 ktuxcchd @4176 4 sb2 ktuxcctl @4178 3 ub2 ktuxcmgc @4180 0x8002 ub4 ktuxcopt @4188 0x7ffffffe
后续发现比较核心报错
SQL> rename FPT_XIFENFEI to FPT_XIFENFEI_0302; rename FPT_XIFENFEI to FPT_XIFENFEI_0302 * ERROR at line 1: ORA-00604: error occurred at recursive SQL level 1 ORA-08102: index key not found, obj# 39, file 1, block 732605 (2)
通过分析是I_OBJ4 记录和obj$不匹配,通过bootstrap$核心index(I_OBJ1,I_USER1,I_FILE#_BLOCK#,I_IND1,I_TS#,I_CDEF1等)异常恢复—ORA-00701错误解决方法重建obj$表处理
发表在 非常规恢复
标签为 obj# 39, ORA-00353, ORA-00354, ORA-08102: index key not found, ORA-600 2662, ORA-600 4194, update undo$
评论关闭