ORA-600 3716故障处理

数据库由于ORA-600 kcvent_intcic_zero_half错误导致实例重启

Reconfiguration started (old inc 8, new inc 10)
List of instances:
 1 2 (myinst: 2) 
 Nested reconfiguration detected. 
 Global Resource Directory frozen
 Communication channels reestablished
 * domain 0 valid = 1 according to instance 1 
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Fri Jan 30 22:07:08 2026
minact-scn: Master returning as live inst:1 has inc# mismatch instinc:0 cur:10 errcnt:0
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
Fri Jan 30 22:07:17 2026
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ckpt_1748.trc  (incident=228496):
ORA-00600: internal error code, arguments: [kcvent_intcic_zero_half], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle11g/diag/rdbms/hisdb/hisdb2/incident/incdir_228496/hisdb2_ckpt_1748_i228496.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ckpt_1748.trc:
ORA-00600: internal error code, arguments: [kcvent_intcic_zero_half], [], [], [], [], [], [], [], [], [], [], []
CKPT (ospid: 1748): terminating the instance due to error 469
Fri Jan 30 22:07:18 2026
ORA-1092 : opitsk aborting process
Fri Jan 30 22:07:19 2026
System state dump requested by (instance=2, osid=1748 (CKPT)), summary=[abnormal instance termination].

初步分析该错误是由于oracle bug引起,在11.2.0.4版本中比较少见,主要常见于19c版本中
kcvent_intcic_zero_half


正常情况下,实例重启即可,这个客户比较不幸,数据库重启遭遇ORA-600 3716错误

ALTER DATABASE OPEN /* db agent *//* {0:11:28647} */
This instance was first to open
Beginning crash recovery of 2 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 1120 KB redo, 386 data blocks need recovery
Started redo application at
 Thread 2: logseq 847, block 641786
Recovery of Online Redo Log: Thread 2 Group 3 Seq 847 Reading mem 0
  Mem# 0: +DATADISK/hisdb/onlinelog/group_3.267.1109863469
  Mem# 1: +ARCHIVEDISK/hisdb/onlinelog/group_3.261.1109863469
Completed redo application of 0.38MB
Completed crash recovery at
 Thread 2: logseq 847, block 644027, scn 20674490933
 386 data blocks read, 0 data blocks written, 1120 redo k-bytes read
ARCH: STARTING ARCH PROCESSES
Fri Jan 30 22:07:47 2026
ARC0 started with pid=53, OS id=332 
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Fri Jan 30 22:07:48 2026
ARC1 started with pid=54, OS id=364 
Fri Jan 30 22:07:48 2026
ARC2 started with pid=55, OS id=366 
Fri Jan 30 22:07:48 2026
ARC3 started with pid=56, OS id=368 
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ora_29981.trc  (incident=240679):
ORA-00600: ??????, ??: [3716], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle11g/diag/rdbms/hisdb/hisdb2/incident/incdir_240679/hisdb2_ora_29981_i240679.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Abort recovery for domain 0
Aborting crash recovery due to error 600
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ora_29981.trc:
ORA-00600: ??????, ??: [3716], [], [], [], [], [], [], [], [], [], [], []
Abort recovery for domain 0
Errors in file /u01/oracle11g/diag/rdbms/hisdb/hisdb2/trace/hisdb2_ora_29981.trc:
ORA-00600: ??????, ??: [3716], [], [], [], [], [], [], [], [], [], [], []
ORA-600 signalled during: ALTER DATABASE OPEN /* db agent *//* {0:11:28647} */...

接手客户故障尝试启动数据库也报ORA-600 3716错误

SQL> startup
ORACLE instance started.

Total System Global Area 2.0176E+10 bytes
Fixed Size                  2261928 bytes
Variable Size            3288337496 bytes
Database Buffers         1.6844E+10 bytes
Redo Buffers               41463808 bytes
Database mounted.
ORA-00600: internal error code, arguments: [3716], [], [], []

根据官方描述该问题
open


那解决该问题相对比较简单

SQL>@rectl.sql

Control file created.
SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

完美恢复这次故障,实现数据0丢失,业务直接使用

发表在 Oracle备份恢复 | 标签为 | 评论关闭

不当恢复truncate数据导致数据库不能open处理

有客户误truncate操作干掉了数据库中的几张表,然后尝试通过FY_Recover_Data进行恢复,恢复到一半然后终止了,数据库结果就起不来了(具体什么原因不知道,肯定是各种不合适的操作引起的故障),我接手故障的时候,数据库被强制resetlogs,报ORA-600 2662错误

Fri Jan 30 10:24:16 2026
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 49708968810
Clearing online redo logfile 1 /u01/oracle/oradata/orcl/redo01.log
Clearing online log 1 of thread 1 sequence number 1069648
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /u01/oracle/oradata/orcl/redo02.log
Clearing online log 2 of thread 1 sequence number 1069649
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /u01/oracle/oradata/orcl/redo03.log
Clearing online log 3 of thread 1 sequence number 1069647
Clearing online redo logfile 3 complete
Resetting resetlogs activation ID 1738690566 (0x67a25006)
Online log /u01/oracle/oradata/orcl/redo01.log: Thread 1 Group 1 was previously cleared
Online log /u01/oracle/oradata/orcl/redo02.log: Thread 1 Group 2 was previously cleared
Online log /u01/oracle/oradata/orcl/redo03.log: Thread 1 Group 3 was previously cleared
Fri Jan 30 10:24:17 2026
Setting recovery target incarnation to 3
Fri Jan 30 10:24:17 2026
Assigning activation ID 1751706121 (0x6868ea09)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u01/oracle/oradata/orcl/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 30 10:24:17 2026
SMON: enabling cache recovery
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_14707.trc  (incident=123363):
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328561], [11], [2464328917], [12583040], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl/incident/incdir_123363/orcl_ora_14707_i123363.trc
Fri Jan 30 10:24:18 2026
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_14707.trc:
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328561], [11], [2464328917], [12583040], [], [], [], []
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_14707.trc:
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328561], [11], [2464328917], [12583040], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 14707): terminating the instance due to error 600
Instance terminated by USER, pid = 14707
ORA-1092 signalled during: alter database open resetlogs...
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_14707.trc:
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328564], [11], [2464328917], [12583040], [], [], [], []
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328563], [11], [2464328917], [12583040], [], [], [], []
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328561], [11], [2464328917], [12583040], [], [], [], []
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_14707.trc:
ORA-27300: 操作系统系统相关操作: semctl 失败, 状态为: 22
ORA-27301: 操作系统故障消息: Invalid argument
ORA-27302: 错误发生在: sskgpwpost1
ORA-27303: 附加信息: semid = 32779
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328564], [11], [2464328917], [12583040], [], [], [], []
ORA-00600: 内部错误代码, 参数: [2662], [11], [2464328563], [11], [2464328917], [12583040], [], [], [], []
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [11], [2464328564], [11], [2464328917], [12583040], []
ORA-00600: internal error code, arguments: [2662], [11], [2464328563], [11], [2464328917], [12583040], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [11], [2464328561], [11], [2464328917], [12583040], []

通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)脚本检查发现有文件被offline,且resetlogs 信息不对
resetlogs


通过obet工具(Oracle数据块编辑工具( Oracle Block Editor Tool)-obet)对resetlogs相关信息进行修改

OBET> set mode edit
mode set to: edit

OBET> set file 18
filename set to: /tmp/FY_RST_DATA.DAT (file#18)

OBET> copy resetlogscn file 1 to file 18

Confirm Modify resetlogscn:
Source: file#1 (/u01/oracle/oradata/orcl/system01.dbf)
Target: file#18 (/tmp/FY_RST_DATA.DAT)
Proceed? (Y/YES to confirm): y
Successfully copied resetlog SCN information from file#1 to file#18.

OBET> copy chkscn file 1 to file 18

Confirm Modify chkscn:
Source: file#1 (/u01/oracle/oradata/orcl/system01.dbf)
Target: file#18 (/tmp/FY_RST_DATA.DAT)
Proceed? (Y/YES to confirm): y
Successfully copied checkpoint SCN information from file#1 to file#18.

OBET> sum
Check value for File /tmp/FY_RST_DATA.DAT, Block 1:
current = 0xF8EA, required = 0xFCEA

OBET> sum check
Warning: Unknown option 'check', ignored
Check value for File /tmp/FY_RST_DATA.DAT, Block 1:
current = 0xF8EA, required = 0xFCEA

OBET> sum apply

Confirm applying checksum:
File: /tmp/FY_RST_DATA.DAT
Block: 1
Offset in block: 16 (file offset: 0x00002010)
Original value: 0xF8EA
New value:      0xFCEA
Confirm? (Y/YES to proceed): y
Verification successful: Stored checksum matches calculated value (0xFCEA).
Checksum applied successfully.

然后尝试打开库,报ORA-600 kdourp_inorder2错误

Database Characterset is ZHS16GBK
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_9198.trc  (incident=159324):
ORA-00600: internal error code, arguments: [kdourp_inorder2], [16], [0], [146], [108], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl/incident/incdir_159324/orcl_smon_9198_i159324.trc
Stopping background process MMNL
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Block recovery from logseq 2, block 39 to scn 49708988852
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: /u01/oracle/oradata/orcl/redo02.log
Block recovery completed at rba 2.42.16, scn 11.2464348597
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (14, 8) on object 14.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_9198.trc:
ORA-00600: internal error code, arguments: [kdourp_inorder2], [16], [0], [146], [108], [], [], [], [], []
Stopping background process MMON
Fri Jan 30 11:42:26 2026
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_9198.trc  (incident=159325):
ORA-00600: internal error code, arguments: [kdourp_inorder2], [16], [0], [146], [108], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/orcl/orcl/incident/incdir_159325/orcl_smon_9198_i159325.trc
Starting background process MMON
Fri Jan 30 11:42:27 2026
MMON started with pid=49, OS id=10684
Starting background process MMNL
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Block recovery from logseq 2, block 39 to scn 49708988852
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: /u01/oracle/oradata/orcl/redo02.log
Block recovery completed at rba 2.42.16, scn 11.2464348597
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (14, 8) on object 14.
Errors in file /u01/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_9198.trc:
ORA-00600: internal error code, arguments: [kdourp_inorder2], [16], [0], [146], [108], [], [], [], [], []

该报错比较明显是由于undo回滚段异常导致,通过屏蔽回滚段,open库成功.后续对客户truncate的表进行分析,比较悲催由于没有第一时间保护现场而且对所在表空间进行了大量写入操作,导致truncate数据恢复较少.

发表在 Oracle备份恢复 | 标签为 , , , , , | 评论关闭

注意:PostgreSQL库出现readme_to_recover勒索

以前恢复过不少mysql库被删除的勒索case,比如:
read_me_recover_tn勒索恢复
A_H_README_TO_RECOVER勒索恢复
A____Z____RECOVER____DATA勒索恢复
今天在PostgreSQL库中也简单了类似的勒索案例

[postgres@xifenfei database]$ psql 
psql (12.8)
Type "help" for help.

postgres=> \l
                                     List of databases
       Name        |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-------------------+----------+----------+------------+------------+-----------------------
 ldar-test         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 postgres          | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 readme_to_recover | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                   |          |          |            |            | postgres=CTc/postgres
 template1         | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
                   |          |          |            |            | postgres=CTc/postgres
(5 rows)

postgres=> \c readme_to_recover
You are now connected to database "readme_to_recover" as user "postgres".
readme_to_recover=> \d
         List of relations
 Schema |  Name  | Type  |  Owner   
--------+--------+-------+----------
 public | readme | table | postgres
(1 row)

readme_to_recover=> select * from readme;
                                                                                                   
                                 text_field                                              
-------------------------------------------------------------------------------------------------------
-------------------------------------------------
 All your data is backed up. You must pay 0.0051 BTC to bc1q9p333xxxxxxxxxxxxxj2z5fs27pu3u In 48 hours, 
  your data will be publicly disclosed and deleted . (more information: go to http://2info.win/psg)
 After paying send mail to us: rambler+3d2l7@onionmail.org and we will provide a link for 
   you to download your data. Your DBCODE is: xxxxx
(2 rows)

readme_to_recover=> select oid,relname from pg_class;
   oid   |                    relname                    
---------+-----------------------------------------------
 2196735 | readme
    2619 | pg_statistic
    1247 | pg_type
    4159 | pg_toast_2600
    4160 | pg_toast_2600_index
    2830 | pg_toast_2604
    2831 | pg_toast_2604_index
    4161 | pg_toast_3456
    4162 | pg_toast_3456_index
    2832 | pg_toast_2606
    2833 | pg_toast_2606_index
………………
   13373 | _pg_foreign_tables
   13377 | foreign_table_options
   13383 | _pg_user_mappings
   13391 | user_mappings
   13387 | user_mapping_options
(396 rows)

根据这里的readme的oid值,初步怀疑是drop所有业务表,然后创建了readme表,并且插入了勒索信息.

我做的各种数据库恢复比较多,说实话一直对pg这个特点非常不满意:
1. pg的数据存储本身依赖文件系统,通过oid(来表示库,表示表以及其他各种对象,和很多商业化数据库不一样,对于文件系统过于依赖,这样的情况如果发生了drop/truncate表等操作,表从文件系统中被删除,恢复难度较大
2. pg的表对应本身没有在每个数据块的block记录表的id信息,这样使得无法很好的进行底层基于block的扫描恢复数据,目前我知道的pdu一定的这样的功能,但是那是依赖列匹配,原则上来说对于一个稍微复杂一点的系统,这个恢复思路基本上不可行(因为列相似的数据表可能很多,导致恢复的数据过于混乱)

基于上述的两个特点,如果整个业务库被这种勒索破坏,如果没有备份,想比较好的恢复效果基本上很难(除非你运气非常好,os层面反删除恢复效果特别好,这要取决被删除的表在os上没有覆盖,文件系统元数据没有回收,系统没有识别是ssd环境等因素决定)

基于这种故障在没有备份的情况下很难恢复,那我们要做的就是:
1)有有效的备份或者容灾不被破坏;
     1> 数据需要有效的备份,有条件的建议异机或者离线
     2> 如果有条件可以部署延迟同步的备库,避免发生故障备库同步 被破坏

2)尽可能不让这种故障发生:
     1>数据库不要暴露在公网上,减少被攻击和入侵的风险
     2>数据库不要使用弱口令,减少被暴力攻破密码的风险
     3>注意应用漏洞,减少通过sql注入进行数据库进行破坏
     4>注意已知的数据库安全漏洞,防止被利用进行攻击
     5>注意操作系统安全,因为登录了数据库服务器对于数据库来说就没有安全可言

发表在 PostgreSQL | 标签为 , | 评论关闭