分类目录归档：Oracle

CLSRSC-400: A system reboot is required to continue installing.

发表于 2019 年 4 月 15 日由惜分飞

在以前的文章中写过关于12.2.0.1在linux 7.3中安装可能遇到的oracle rac 12.2 执行root.sh报CLSRSC-400问题,最近mos上面提供的相应的patch(25078431)以及处理方法,供参考
12.2.0.1 Cluster Installation Failure – ACFS-9459 With RH 7.3 (文档 ID 2273119.1)文章提供方法

ALERT: root.sh Fails With “CLSRSC-400″ While Installing GI 12.2.0.1 on RHEL or OL with RedHat Compatible Kernel (RHCK) 7.3 (文档 ID 2284463.1)文章提供方法

发表在 Oracle安装升级 | 标签为 25078431, CLSRSC-400 | 评论关闭

12.1人工修改操作系统时间导致数据库异常

发表于 2019 年 4 月 15 日由惜分飞

有客户数据库版本为12.1.0.1 版本RAC,突发发生重启,让协助分析原因
数据库alert日志报ORA-15064错误

Mon Apr 15 15:06:26 2019
WARNING: inbound connection timed out (ORA-3136)
Mon Apr 15 15:41:26 2019
NOTE: ASMB terminating
Mon Apr 15 15:41:26 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_asmb_61426.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 1892 Serial number: 29
Mon Apr 15 15:41:26 2019
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_asmb_61426.trc:
ORA-15064: communication failure with ASM instance
ORA-03113: end-of-file on communication channel
Process ID: 
Session ID: 1892 Serial number: 29
Mon Apr 15 15:41:26 2019
System state dump requested by (instance=1, osid=61426 (ASMB)), summary=[abnormal instance termination].
Mon Apr 15 15:41:26 2019
USER (ospid: 61426): terminating the instance due to error 15064
Mon Apr 15 15:41:26 2019
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_61287.trc
Mon Apr 15 15:41:27 2019
opiodr aborting process unknown ospid (1171) as a result of ORA-1092
Mon Apr 15 15:41:27 2019
ORA-1092 : opitsk aborting process

这里看,明显asmb异常导致数据库无法正常访问asm从而出现数据库crash的问题.

分析asm日志

Mon Apr 15 15:41:26 2019
WARNING: client [+ASM1:+ASM] not responsive for 2069s; state=0x1. pid 23155
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: client [orcl1:orcl] not responsive for 2069s; state=0x1. killing pid 61436
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: fencing client [orcl1:orcl] after 2069 seconds (mbr 2)
WARNING: client [-MGMTDB:_mgmtdb] not responsive for 2070s; state=0x1. killing pid 24026
NOTE: umbilicus traces dumped to /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_gen0_23050.trc
WARNING: fencing client [-MGMTDB:_mgmtdb] after 2070 seconds (mbr 1)
Mon Apr 15 15:41:26 2019
NOTE: cleaned up ASM client -MGMTDB:_mgmtdb
NOTE: cleaned up ASM client orcl1:orcl
Mon Apr 15 15:41:43 2019
NOTE: Standard client -MGMTDB:_mgmtdb registered, osid 183707, mbr 0x1 (reg:1371965153)
Mon Apr 15 15:42:16 2019
NOTE: Standard client orcl1:orcl registered, osid 184063, mbr 0x2 (reg:2088418628)
Mon Apr 15 15:44:30 2019
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.

asm日志中和mos中的GEN0 terminating the ASM instance due to error 15082 (文档 ID 2096988.1)描述比较匹配.根据客户反馈,他们使用ntp进行修改了时间,基本上可以确定是由于oracle的Bug 19032250(在12.1.0.2中修复)在ntp修改时间跨度过大触发的相关问题(人工直接修改时间也可能出现类似问题)

对于rac修改时间建议
1. 如果时间慢了,关闭数据库和集群直接把时间向前调整,启动集群和数据库
2. 如果时间快了,关闭数据库和集群等实际时间过关闭集群和库的时间之后,再往回调整时间,启动集群和数据库

发表在 Oracle RAC | 评论关闭

tab$被恶意删除sys用户之外记录

发表于 2019 年 4 月 14 日由惜分飞

数据库open成功，但是alert日志报大量ORA-00600错误

Sun Apr 14 14:30:46 2019
SMCO started with pid=53, OS id=6761 
Completed: ALTER DATABASE OPEN /* db agent *//* {1:65047:2} */
Sun Apr 14 14:30:49 2019
Starting background process CJQ0
Sun Apr 14 14:30:49 2019
CJQ0 started with pid=54, OS id=6776 
Setting Resource Manager plan SCHEDULER[0x32DF]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sun Apr 14 14:30:52 2019
Starting background process VKRM
Sun Apr 14 14:30:52 2019
VKRM started with pid=37, OS id=6809 
Sun Apr 14 14:30:54 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc  (incident=288633):
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288633/xifenfei1_j000_6811_i288633.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_6811.trc:
ORA-00600: internal error code, arguments: [kkpo_rcinfo_defstg:delseg], [84638], [], [], [], [], [], [], [], [], [], []
ORA-06512: at "APEX_030200.WWV_FLOW_MAIL", line 695
ORA-06512: at line 1
Sun Apr 14 14:30:57 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc  (incident=288658):
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288658/xifenfei1_j000_7491_i288658.trc
Sun Apr 14 14:34:10 2019
Dumping diagnostic data in directory=[cdmp_20190414143410], requested by (instance=1, osid=7491 (J000)), summary=[incident=288658].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_7491.trc:
ORA-00600: 内部错误代码, 参数: [16659], [kqldtu], [INS], [0], [206196], [], [], [], [], [], [], []
ORA-06512: 在 "WEBCSMS.P_YGERROR", line 3
Sun Apr 14 14:39:08 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j000_8515.trc  (incident=288593):
ORA-00600: 内部错误代码, 参数: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288593/xifenfei1_j000_8515_i288593.trc
ORA-06512: 在 line 1
Sun Apr 14 14:52:14 2019
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_j001_11062.trc  (incident=288705):
ORA-00600: internal error code, arguments: [16607], [0x3CFB04C90], [257], [9], [0x000000000], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288705/xifenfei1_j001_11062_i288705.trc
Errors in file /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_16712.trc  (incident=288597):
ORA-00600: 内部错误代码, 参数: [16607], [0x3C7CEA678], [1281], [9], [0x000000000], [], [], [], [], [], [], []
Incident details in: /oracle/oracle/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_288597/xifenfei1_ora_16712_i288597.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

报错比较多，客户还反馈登录数据库之后，发现所有的表都丢失。第一反应可能数据字典损坏了，然后让客户查看备库，现在dg的备库也一样表都丢失了，进一步确认字典可能异常，让客户提供system文件进行本地分析.发现DBMS_SUPPORT_DBMONITOR触发器调用DBMS_SUPPORT_DBMONITORP存储过程,和警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703中的名称非常类似,但是有点不一样,以前的恶意脚本中都是被注入并且触发之后，数据库无法正常启动，这次数据库能够正常open成功.分析恶意脚本,确认原因

确实这次的恶意脚本是在2016年8月份被创建在库中,在600天之后重启被触发,而且是删除非sys的tab$中记录.知道了恶意脚本的源头,那恢复就比较容易,直接通过批量bbed程序对tab$反删除可以实现比较完美恢复.原则上这样的故障可以实现数据库完美恢复,原库继续使用.