分类目录归档:Oracle

Patch_SCN快速解决ORA-600 2663故障

一个维保客户和我说他们测试库删除了日志文件导致库无法启动,让我帮忙看看
客户现场现况
1. 磁盘空间使用100%

[oracle@We1-db_Test ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  880K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1        40G   38G   92M 100% /
tmpfs           783M     0  783M   0% /run/user/0

2. 数据库redo被删除了部分,而且是active状态的被删除

[oracle@We1-db_Test ~]$ ls -l /opt/app/oracle/oradata/orcl/redo0*
-rw-r----- 1 oracle oinstall 52429312 Jan 15 15:29 /opt/app/oracle/oradata/orcl/redo04.log
-rw-r----- 1 oracle oinstall 52429312 Jan 15 16:26 /opt/app/oracle/oradata/orcl/redo05.log

SQL> select group#,SEQUENCE#,STATUS FROM V$lOG;

    GROUP#  SEQUENCE# STATUS
---------- ---------- ----------------
         1       8989 CURRENT
         2          0 UNUSED
         5          0 UNUSED
         4          0 UNUSED
         3       8988 ACTIVE

SQL> select member from v$logfile;

MEMBER
-----------------------------------------------------
/opt/app/oracle/oradata/orcl/redo03.log
/opt/app/oracle/oradata/orcl/redo02.log
/opt/app/oracle/oradata/orcl/redo01.log
/opt/app/oracle/oradata/orcl/redo04.log
/opt/app/oracle/oradata/orcl/redo05.log

基于当前情况,直接open库无望,但是空间不足问题需要先解决,不然恢复过程中创建redo空间不足依旧会报错卡死,所以先清理了监听和alert等日志,系统空闲了3G多空间,可以进行恢复操作

[oracle@We1-db_Test trace]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  880K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vda1        40G   34G  3.9G  90% /
tmpfs           783M     0  783M   0% /run/user/0

恢复数据库
1. 由于active redo丢失,毫无疑问,直接强制拉库,使用_allow_resetlogs_corruption参数开干

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 2455228416 bytes
Fixed Size                  2255712 bytes
Variable Size             905970848 bytes
Database Buffers         1526726656 bytes
Redo Buffers               20275200 bytes
Database mounted.
SQL> recover database until cancel;
ORA-00283: recovery session canceled due to errors
ORA-01610: recovery using the BACKUP CONTROLFILE option must be done


SQL> recover database using backup controlfile;
ORA-00279: change 311982775 generated at 12/31/2025 17:35:11 needed for thread
1
ORA-00289: suggestion :
/opt/app/oracle/fast_recovery_area/ORCL/archivelog/2026_01_16/o1_mf_1_8988_%u_.a
rc
ORA-00280: change 311982775 for thread 1 is in sequence #8988


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0],
[311982833], [], [], [], [], [], [], []
Process ID: 11917
Session ID: 576 Serial number: 3

alert日志报错

Fri Jan 16 21:25:31 2026
Assigning activation ID 1750515127 (0x6856bdb7)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /opt/app/oracle/oradata/orcl/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 16 21:25:31 2026
SMON: enabling cache recovery
Errors in file /opt/app/oracle/diag/rdbms/orcl/we1db/trace/we1db_ora_11917.trc  (incident=81753):
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0], [311982833], [], []
Incident details in: /opt/app/oracle/diag/rdbms/orcl/we1db/incident/incdir_81753/we1db_ora_11917_i81753.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /opt/app/oracle/diag/rdbms/orcl/we1db/trace/we1db_ora_11917.trc:
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0], [311982833], [], []
Errors in file /opt/app/oracle/diag/rdbms/orcl/we1db/trace/we1db_ora_11917.trc:
ORA-00600: internal error code, arguments: [2663], [0], [311982792], [0], [311982833], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 11917): terminating the instance due to error 600
Instance terminated by USER, pid = 11917
ORA-1092 signalled during: alter database open resetlogs...
opiodr aborting process unknown ospid (11917) as a result of ORA-1092
Fri Jan 16 21:25:33 2026
ORA-1092 : opitsk aborting process

不幸数据库遇到ORA-600 2663错误,这个故障在以前的文章中描述过,基本上和ORA-600 2662的处理思路类似,这里直接使用:Patch_SCN for Linux进行恢复
2. 使用Patch_SCN处理数据库SCN

SQL> startup nomount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 2455228416 bytes
Fixed Size                  2255712 bytes
Variable Size             905970848 bytes
Database Buffers         1526726656 bytes
Redo Buffers               20275200 bytes
SQL>@rectl

Control file created.

SQL> recover database;
Media recovery complete.

patch_scn


SQL> alter database open;

Database altered.
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

     CURRENT_SCN
----------------
       322002903

到这里完成数据库open操作,后续逻辑导出完成恢复任务

发表在 Oracle备份恢复 | 标签为 , | 留下评论

在生产环境错误执行dd命令破坏asm磁盘故障恢复

由于ssh登录错误,客户对生产环境进行了误操作把系统的一块磁盘dd到另外两个磁盘上,由于及时发现立马进行了终止操作,但是还是分别破坏了一点数据(一块盘破坏了2G多,另外一块盘破坏了1G多)
dd


通过分析udev的绑定关系确认被破坏的asm disk名称
udev

再通过asm alert日志确认破坏磁盘在asm disk中情况
asmdisk

通过上述信息基本上可以确认,asmdisk13被分别dd到了asmdisk11和asmdisk26中了部分11
26

基于这种情况,由于asm disk被破坏了1-2G多,直接修复然后正常mount磁盘组基本上没有希望,经过分析以及与客户沟通,确认他们改系统是4节点组成的集群,1/2节点上面跑2套库,3/4节点上跑2套库,数据整体放在data_dg磁盘组中,需要恢复的库是第二个顺序创建的1套库(4套库中只需恢复一套即可),由于破坏的数据本身不多,而且需要恢复的数据不是最初写入asm磁盘组,基于这样的情况,需要恢复的数据库机会比较大.

由于现在三个磁盘头信息一致(一个磁盘被dd到另外两个磁盘上),因此第一步先把损坏的两个磁盘头进行简单修复,为了便于amdu(找回ASM中数据文件)等之类数据可以识别到正确的磁盘头信息,然后进行后续的数据文件提取恢复.使用工具对数据文件进行了批量提取,提取数据完成之后,尝试recover和open库
open

虽然数据库正常打开了,不过很不幸,后台还是有一些坏块报错,通过dbv检查发现有文件有一部分坏块,类似dbv报错信息
1_1

通过分析该文件在磁盘组中各个磁盘的分布情况
map

确认该文件确实有部分block分布在被dd磁盘的破坏的范围内这个部分的数据丢失无法挽回,只能是定位到具体对象然后由业务想办法处理.相对以往的各种dd破坏的案例恢复而言,这个应该是效果比较好的一个,而且也是恢复比较容易的一个,没有使用到asm disk 基于asm au/oracle block 扫描的级别,而且system表空间没有任何损坏,数据库甚至直接open成功了,以往的一些dd案例列举:
asm磁盘dd破坏恢复
dd破坏asm磁盘头恢复
asm disk 磁盘部分被清空恢复

发表在 非常规恢复 | 标签为 , | 留下评论

obet实现对数据文件坏块检测功能

通过一段时间的测试和使用,obet修复了不少bug,关于obet的以往功能和特性的文章:
Oracle数据块编辑工具( Oracle Block Editor Tool)-obet
obet(Oracle Block Editor Tool)第二版发布
并且也在客户的生产环境上进行了实战:obet快速修改scn/resetlogs恢复数据库(缺少归档,ORA-00308).利用周末的时间又对obet的工具进行了功能增强,增加了dbv(数据块校验)功能.

Oracle dbv的不足
1. oracle dbv需要在安装oracle服务端的环境下才能执行
2. oracle dbv对于文件大小不正确(文件头记录block数和实际文件大小不匹配),文件头损坏等情况都可能导致dbv无法执行,类似下面的报错

C:\Users\XFF>dbv file=H:\BaiduNetdisk\kingdee\system01.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 1月 11 17:30:29 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.


DBV-00107: 未知标头格式 (0) (2054913149)

C:\Users\XFF>
C:\Users\XFF>dbv file=H:\BaiduNetdisk\kingdee\users01.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on 星期日 1月 11 20:10:05 2026

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.


DBV-00102: FILE (H:\BAIDUNETDISK\KINGDEE\USERS01.DBF) 在 end read 操作 (-1) 期间出现文件 I/O 错误

C:\Users\XFF>

3. oracle dbv一条命令执行检测一个数据文件,如果数据文件多,检查起来很繁琐
4. oracle dbv么有检查进度,对于io性能较慢,数据文件较大的情况,无法跟踪检查进度.

obet的dbv功能使用
1. 配置listfile.txt文件,格式为file# name

1 H:\BaiduNetdisk\kingdee\system01.dbf
5 H:\BaiduNetdisk\kingdee\EAS_D_EAS_STANDARD.ORA

2. 启动obet,执行open listfile.txt(如果不在obet目录提供完整路径)

OBET> open listfile.txt
Loaded 2 files from config file 'listfile.txt'.

OBET> info

Loaded files (2 total):
----------------------------------------
Number  Path
----------------------------------------
     1  H:\BaiduNetdisk\kingdee\system01.dbf
     5  H:\BaiduNetdisk\kingdee\EAS_D_EAS_STANDARD.ORA
----------------------------------------

3. 执行dbv命令(logfile 部分为可选),默认会记录日志在obet目录下面dbv_年月日时分秒.log的日志

OBET> dbv

===============================================
DBV (Data Block Verification)
Block Size: 8192 bytes
===============================================

Verifying file #1: H:\BaiduNetdisk\kingdee\system01.dbf (131841 blocks) - Started: 2026-01-11 18:03:58
file 1, block 0: checksum error (expected 0xC4DA, got 0xC478), bad block
file 1, block 1: checksum error (expected 0xF835, got 0xB835), bad block
  Progress: 10000 / 131841 blocks checked...
  Progress: 20000 / 131841 blocks checked...
……………….
  Progress: 120000 / 131841 blocks checked...
  Progress: 130000 / 131841 blocks checked...
  File #1 completed: 0 all zero, 0 soft corrupted, 0 tailchk error, 2 checksum error, 0 rdba error
Verifying file #5: H:\BaiduNetdisk\kingdee\EAS_D_EAS_STANDARD.ORA (4194303 blocks) - Started: 2026-01-11 18:04:08
  Progress: 10000 / 4194303 blocks checked...
  Progress: 20000 / 4194303 blocks checked...
……………………
  Progress: 260000 / 4194303 blocks checked...
  Progress: 270000 / 4194303 blocks checked...
file 5, block 277678: tailchk error (expected 0x0228BB21, got 0x0228AD21), bad block
file 5, block 277679: rdba error (expected 277679, got 277615), bad block
file 5, block 277680: rdba error (expected 277680, got 277616), bad block
file 5, block 277681: rdba error (expected 277681, got 277617), bad block
………………
file 5, block 277692: rdba error (expected 277692, got 277628), bad block
file 5, block 277693: rdba error (expected 277693, got 277629), bad block
file 5, block 277694: rdba error (expected 277694, got 277630), bad block
file 5, block 279406: tailchk error (expected 0x02281E22, got 0x00000700), bad block
file 5, block 279407: rdba error (expected 279407, got 448), bad block
file 5, block 279408: rdba error (expected 279408, got 0), bad block
………………
  Progress: 280000 / 4194303 blocks checked...
  Progress: 290000 / 4194303 blocks checked...
  Progress: 300000 / 4194303 blocks checked...
  Progress: 310000 / 4194303 blocks checked...
file 5, block 312932: tailchk error (expected 0x0106C0B8, got 0x010629B8), bad block
file 5, block 312933: rdba error (expected 312933, got 312869), bad block
file 5, block 312934: rdba error (expected 312934, got 312870), bad block
file 5, block 312935: rdba error (expected 312935, got 312871), bad block
file 5, block 312936: rdba error (expected 312936, got 312872), bad block
file 5, block 312937: rdba error (expected 312937, got 312873), bad block
file 5, block 312938: rdba error (expected 312938, got 312874), bad block
file 5, block 312939: rdba error (expected 312939, got 312875), bad block
………………
  Progress: 4180000 / 4194303 blocks checked...
  Progress: 4190000 / 4194303 blocks checked...
  File #5 completed: 1 all zero, 0 soft corrupted, 13 tailchk error, 1 checksum error, 255 rdba error

DBV completed at: 2026-01-11 18:09:30

===============================================
DBV Summary:
Total blocks checked: 4325872
Total all zero blocks found: 1
Total all rdba error blocks found: 255
Total all tailchk error blocks found: 13
Total all soft corrupted blocks found: 0
Total all checksum error blocks found: 3
Total bad blocks found: 272
===============================================

Detailed report saved to: dbv_20260111180358.log

OBET>

检测效果截图
obet-dbv


发表在 Oracle, 小工具 | 标签为 , , , | 留下评论