标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (102)
- 数据库 (1,670)
- DB2 (22)
- MySQL (73)
- Oracle (1,532)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (21)
- ORA-xxxxx (159)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (14)
- ORACLE 21C (3)
- Oracle 23ai (7)
- Oracle ASM (65)
- Oracle Bug (8)
- Oracle RAC (52)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (560)
- Oracle安装升级 (91)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (78)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (37)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (20)
-
最近发表
- ORA-600 krse_arc_complete.4
- Oracle 19c 202410补丁(RUs+OJVM)
- ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复
- .mkp扩展名oracle数据文件加密恢复
- 清空redo,导致ORA-27048: skgfifi: file header information is invalid
- A_H_README_TO_RECOVER勒索恢复
- 通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评
- ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME
- ORA-01092 ORA-00604 ORA-01558故障处理
- ORA-65088: database open should be retried
- Oracle 19c异常恢复—ORA-01209/ORA-65088
- ORA-600 16703故障再现
- 数据库启动报ORA-27102 OSD-00026 O/S-Error: (OS 1455)
- .[metro777@cock.li].Elbie勒索病毒加密数据库恢复
- 应用连接错误,初始化mysql数据库恢复
- RAC默认服务配置优先节点
- Oracle 19c RAC 替换私网操作
- 监听报TNS-12541 TNS-12560 TNS-00511错误
- drop tablespace xxx including contents恢复
- Linux 8 修改网卡名称
标签归档:kkpo_rcinfo_defstg:delseg
12.2数据库启动报ORA-07445 lmebucp错误
有一个客户找到我们,说他们是数据库启动之时报的错误和数据库不能open 报ORA-7445 lmebucp错类似,让我们对其进行恢复支持,通过分析确定客户数据库版本为12.2.0.1
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Productio PL/SQL Release 12.2.0.1.0 - Production CORE 12.2.0.1.0 Production TNS for Linux: Version 12.2.0.1.0 - Production NLSRTL Version 12.2.0.1.0 - Production
alert日志报错
--最初报错 2020-05-11T03:43:06.787164+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc (incident=639048): ORA-00600: 内部错误代码, 参数: [kkpo_rcinfo_defstg:delseg], [72280], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_639048/orcl_m004_3253_i639048.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2020-05-11T03:43:14.250993+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc (incident=639049): ORA-00600: 内部错误代码, 参数: [kkpo_rcinfo_defstg:delseg], [72280], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_639049/orcl_m004_3253_i639049.trc 2020-05-11T03:43:21.286310+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc (incident=639050): ORA-00600: 内部错误代码, 参数: [kkpo_rcinfo_defstg:delseg], [72280], [], [], [], [], [], [], [], [], [], [] ORA-06512: 在 line 2 Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_639050/orcl_m004_3253_i639050.trc 2020-05-11T03:43:28.059048+08:00 Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2020-05-11T03:43:28.074681+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_m004_3253.trc: ORA-00600: 内部错误代码, 参数: [kkpo_rcinfo_defstg:delseg], [72280], [], [], [], [], [], [], [], [], [], [] ORA-06512: 在 line 2 2020-05-11T08:31:22.416087+08:00 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_j000_16511.trc: ORA-12012: 自动执行作业 141 出错 ORA-30732: 表中不包含用户可见的列 ORA-06512: 在 line 1 ---关闭数据库之后重启报错 2020-05-11T09:42:43.234769+08:00 ALTER DATABASE OPEN 2020-05-11T09:42:44.353085+08:00 Ping without log force is disabled: instance mounted in exclusive mode. Endian type of dictionary set to little 2020-05-11T09:42:44.660388+08:00 TT00: Gap Manager starting (PID:31134) 2020-05-11T09:42:45.180876+08:00 Thread 1 opened at log sequence 78596 Current log# 2 seq# 78596 mem# 0: /u01/app/oracle/oradata/orcl/redo02.log Successful open of redo thread 1 2020-05-11T09:42:45.181357+08:00 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Exception [type: SIGSEGV, Address not mapped to object][ADDR:0x0][PC:0x10F4C112, lmebucp()+34][flags: 0x0, count: 1] Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_31125.trc (incident=646445): ORA-07445: exception encountered:core dump[lmebucp()+34][SIGSEGV][ADDR:0x0][PC:0x10F4C112][Address not mapped to object] Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_646445/orcl_ora_31125_i646445.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. 2020-05-11T09:42:46.070049+08:00 ***************************************************************** An internal routine has requested a dump of selected redo. This usually happens following a specific internal error, when analysis of the redo logs will help Oracle Support with the diagnosis. It is recommended that you retain all the redo logs generated (by all the instances) during the past 12 hours, in case additional redo dumps are required to help with the diagnosis. ***************************************************************** 2020-05-11T09:42:53.344955+08:00 Instance Critical Process (pid: 41, ospid: 31125) died unexpectedly PMON (ospid: 31026): terminating the instance due to error 12752 2020-05-11T09:42:53.377209+08:00 System state dump requested by (instance=1, osid=31026 (PMON)), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_diag_31046_20200511094253.trc 2020-05-11T09:42:56.690224+08:00 Instance terminated by PMON, pid = 31026
对启动过程做10046跟踪
PARSING IN CURSOR #139821999713040 len=65 dep=1 uid=0 oct=3 lid=0 tim=2200910467 hv=1762642493 ad='1bfd79130'sqlid='aps3qh1nhzkjx' select line#, sql_text from bootstrap$ where obj# not in (:1, :2) END OF STMT PARSE #139821999713040:c=378,e=378,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=2200910467 BINDS #139821999713040: Bind#0 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f2ad89fe6c8 bln=22 avl=02 flg=05 value=59 Bind#1 oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00 oacflg=08 fl2=1000001 frm=00 csi=00 siz=24 off=0 kxsbbbfp=7f2ad89fe698 bln=24 avl=06 flg=05 value=4294967295 EXEC #139821999713040:c=219,e=658,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=867914364,tim=2200911218 WAIT #139821999713040: nam='db file sequential read' ela= 9 file#=1 block#=520 blocks=1 obj#=59 tim=2200911300 WAIT #139821999713040: nam='db file scattered read' ela= 19 file#=1 block#=521 blocks=3 obj#=59 tim=2200911559 FETCH #139821999713040:c=404,e=404,p=4,cr=5,cu=0,mis=0,r=0,dep=1,og=4,plh=867914364,tim=2200911646 STAT #139821999713040 id=1 cnt=0 pid=0 pos=1 obj=59 op='TABLE ACCESS FULL BOOTSTRAP$ (cr=5 pr=4 pw=0 str=1 time=406 us)' *** 2020-05-11T12:25:30.977832+08:00 Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x10F4C112, lmebucp()+34] [flags: 0x0, count: 1] 2020-05-11T12:25:31.026914+08:00 Incident 718445 created, dump file:/u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_718445/orcl_ora_14324_i718445.trc ORA-07445: exception encountered:core dump [lmebucp()+34][SIGSEGV][ADDR:0x0][PC:0x10F4C112][Address not mapped to object]
根据该报错,可以大概定位数据库重启之后报ORA-07445 lmebucp()+34错误不能正常启动是由于bootstrap$表异常导致.
BBED> p kcvfhrdb ub4 kcvfhrdb @96 0x00400208 BBED> set block 523 BLOCK# 523 BBED> map File: F:\temp\12.2\1\system01.dbf (0) Block: 523 Dba:0x00000000 ------------------------------------------------------------ KTB Data Block (Table/Cluster) struct kcbh, 20 bytes @0 struct ktbbh, 48 bytes @20 struct kdbh, 14 bytes @68 struct kdbt[1], 4 bytes @82 sb2 kdbr[20] @86 ub1 freespace[1015] @126 ub1 rowdata[7047] @1141 ub4 tailchk @8188 BBED> p *kdbr[1] rowdata[6431] ------------- ub1 rowdata[6431] @7572 0x3c BBED> x /rnnc rowdata[6431] @7572 ------------- flag@7572: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH) lock@7573: 0x01 cols@7574: 0
通过bbed定位rootdba,然后dump相关block,随机找一条记录,确认bootstrap表无后效记录.但是该数据库在重启之前已经报了ORA-600 kkpo_rcinfo_defstg:delseg和ORA-30732错误,很可能还有其他的基表异常.通过先修复bootstrap$记录,然后根据该表中记录分析其他相关表记录,最终确定tab$中记录也异常,通过bbed 批量循环修复方法,对其进行恢复,open数据库,可以验证数据没有问题.至此该问题解决,但是没有找出来故障原因(是人为破坏【直接人工删除】,还是某种工具带入恶意脚本导致,类似:plsql dev引起的数据库被黑勒索比特币实现原理分析和解决方案,亦或者是数据库安装介质带有恶意程序,类似:警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703)
tab$被恶意删除sys用户之外记录
数据库open成功,但是alert日志报大量ORA-00600错误
Sun Apr 14 14:30:46 2019 SMCO started with pid=53, OS id=6761 Completed: ALTER DATABASE OPEN /* db agent *//* {1:65047:2} */ Sun Apr 14 14:30:49 2019 Starting background process CJQ0 Sun Apr 14 14:30:49 2019 CJQ0 started with pid=54, OS id=6776 Setting Resource Manager plan SCHEDULER[0x32DF]:DEFAULT_MAINTENANCE_PLAN via scheduler window Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter Sun Apr 14 14:30:52 2019 Starting background process VKRM Sun Apr 14 14:30:52 2019 VKRM started with pid=37, OS id=6809 Sun Apr 14 14:30:54 2019 Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc (incident=288633): ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288633/xifenfei1_j000_6811_i288633.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc: ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], [] ORA-06512: at "APEX_030200.WWV_FLOW_MAIL", line 695 ORA-06512: at line 1 Sun Apr 14 14:30:57 2019 Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc (incident=288658): ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], [] Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288658/xifenfei1_j000_7491_i288658.trc Sun Apr 14 14:34:10 2019 Dumping diagnostic data in directory=[cdmp_20190414143410], requested by (instance=1, osid=7491 (J000)), summary=[incident=288658]. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc: ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], [] ORA-06512: 在 "WEBCSMS.P_YGERROR", line 3 Sun Apr 14 14:39:08 2019 Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_8515.trc (incident=288593): ORA-00600: 内部错误代码, 参数: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], [] Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288593/xifenfei1_j000_8515_i288593.trc ORA-06512: 在 line 1 Sun Apr 14 14:52:14 2019 Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j001_11062.trc (incident=288705): ORA-00600: internal error code, arguments: [16607], [0x3CFB04C90], [257], [9], [0x000000000], [], [], [], [], [], [], [] Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288705/xifenfei1_j001_11062_i288705.trc Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_16712.trc (incident=288597): ORA-00600: 内部错误代码, 参数: [16607], [0x3C7CEA678], [1281], [9], [0x000000000], [], [], [], [], [], [], [] Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288597/xifenfei1_ora_16712_i288597.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details.
报错比较多,客户还反馈登录数据库之后,发现所有的表都丢失。第一反应可能数据字典损坏了,然后让客户查看备库,现在dg的备库也一样表都丢失了,进一步确认字典可能异常,让客户提供system文件进行本地分析.发现DBMS_SUPPORT_DBMONITOR触发器调用DBMS_SUPPORT_DBMONITORP存储过程,和警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703中的名称非常类似,但是有点不一样,以前的恶意脚本中都是被注入并且触发之后,数据库无法正常启动,这次数据库能够正常open成功.分析恶意脚本,确认原因
确实这次的恶意脚本是在2016年8月份被创建在库中,在600天之后重启被触发,而且是删除非sys的tab$中记录.知道了恶意脚本的源头,那恢复就比较容易,直接通过批量bbed程序对tab$反删除可以实现比较完美恢复.原则上这样的故障可以实现数据库完美恢复,原库继续使用.