分类目录归档：ORACLE 19C

Oracle 19c故障恢复

发表于 2020 年 3 月 10 日由惜分飞

有客户找到我们,他们的oracle 19c数据库由于异常断电,导致启动异常,经过一系列恢复之后,依旧无法解决问题,请求我们给予支持.通过我们的Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check),获取数据库当前信息如下:
数据库版本为19C并且安装了19.5.0.0.191015 (30125133)补丁

数据库使用pdb

数据库启动成功后,一会就crash掉

2020-03-10T01:44:41.018032+08:00
Pluggable database RACBAK opened read write
2020-03-10T01:44:41.018996+08:00
Pluggable database RAC opened read write
2020-03-10T01:44:51.244050+08:00
Completed: ALTER PLUGGABLE DATABASE ALL OPEN
Starting background process CJQ0
Completed: ALTER DATABASE OPEN
2020-03-10T01:44:51.317085+08:00
CJQ0 started with pid=224, OS id=32581 
2020-03-10T01:44:56.067043+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j001_32588.trc  (incident=1095281) (PDBNAME=RAC):
ORA-00600: internal error code, arguments: [4193], [27733], [27754], [], [], [], [], [], [], [], [], []
RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095281/XFF_j001_32588_i1095281.trc
RAC(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-03-10T01:44:56.073112+08:00
RAC(4):*****************************************************************
RAC(4):An internal routine has requested a dump of selected redo.
RAC(4):This usually happens following a specific internal error, when
RAC(4):analysis of the redo logs will help Oracle Support with the
RAC(4):diagnosis.
RAC(4):It is recommended that you retain all the redo logs generated (by
RAC(4):all the instances) during the past 12 hours, in case additional
RAC(4):redo dumps are required to help with the diagnosis.
RAC(4):*****************************************************************
2020-03-10T01:44:56.079228+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j002_32590.trc  (incident=1095289) (PDBNAME=RAC):
ORA-00600: internal error code, arguments: [4193], [2633], [2638], [], [], [], [], [], [], [], [], []
RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095289/XFF_j002_32590_i1095289.trc
RAC(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-03-10T01:44:56.085068+08:00
RAC(4):*****************************************************************
RAC(4):An internal routine has requested a dump of selected redo.
RAC(4):This usually happens following a specific internal error, when
RAC(4):analysis of the redo logs will help Oracle Support with the
RAC(4):diagnosis.
RAC(4):It is recommended that you retain all the redo logs generated (by
RAC(4):all the instances) during the past 12 hours, in case additional
RAC(4):redo dumps are required to help with the diagnosis.
RAC(4):*****************************************************************
2020-03-10T01:44:56.115765+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_j004_32594.trc  (incident=1095305) (PDBNAME=RAC):
ORA-00600: internal error code, arguments: [4193], [63532], [63537], [], [], [], [], [], [], [], [], []
RAC(4):Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1095305/XFF_j004_32594_i1095305.trc
RAC(4):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2020-03-10T01:46:48.202213+08:00
RAC(4):Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
RAC(4):  Mem# 0: /opt/oracle/oradata/XFF/redo02.log
RAC(4):Block recovery completed at rba 0.0.0, scn 0x0000000d3675e48e
RAC(4):DDE: Problem Key 'ORA 600 [4193]' was completely flood controlled (0x6)
Further messages for this problem key will be suppressed for up to 10 minutes
2020-03-10T01:46:48.384040+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_clmn_31741.trc:
ORA-00600: internal error code, arguments: [4193], [27733], [27754], [], [], [], [], [], [], [], [], []
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_clmn_31741.trc  (incident=1093505) (PDBNAME=CDB$ROOT):
ORA-501 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /opt/oracle/diag/rdbms/XFF/XFF/incident/incdir_1093505/XFF_clmn_31741_i1093505.trc
2020-03-10T01:46:49.264624+08:00
USER (ospid: 31741): terminating the instance due to ORA error 501
2020-03-10T01:46:49.280664+08:00
System state dump requested by (instance=1, osid=31741 (CLMN)), summary=[abnormal instance termination].
System State dumped to trace file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_diag_31759.trc
2020-03-10T01:46:53.156926+08:00
ORA-00501: CLMN process terminated with error
2020-03-10T01:46:53.157103+08:00
Errors in file /opt/oracle/diag/rdbms/XFF/XFF/trace/XFF_diag_31759.trc:
ORA-00501: CLMN process terminated with error
2020-03-10T01:46:53.157211+08:00
Dumping diagnostic data in directory=[cdmp_20200310014649], requested by (instance=1, osid=31741 (CLMN)), 
summary=[abnormal instance termination].

通过报错信息判断,数据库open之后(特别是pdb 4 open之后),开始报ORA-600 4193错误.然后由于CLMN进程异常,最后数据库crash.对于这类故障,因为使用的pdb,而且是由于pdb的undo异常导致数据库启动之后crash,可以通过对于pdb进行特殊处理,从而实现数据库启动之后不再crash.

ORA-00700: soft internal error, arguments: [pga physmem limit]

发表于 2020 年 1 月 1 日由惜分飞

19c数据库启动有ORA-00700警告

2019-12-31T20:29:10.905779+08:00
PGA_AGGREGATE_TARGET specified is high
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_25403.trc  (incident=122887):  
ORA-00700: soft internal error, arguments: [pga physmem limit], [3332374528],[2112652492], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_122887/orcl_ora_25403_i122887.trc

分析trace文件

WARNING: PGA_AGGREGATE_TARGET (3178 MB) is too high for  
         the amount of physical memory (31783 MB) and
         SGA size (24576 MB) - it should be less than 2014 MB.

提示PGA_AGGREGATE_TARGET 设置过大.官方对于此问题有说明,分配给数据库的内存(sga+pga)小于80%即可

发表在 ORACLE 19C | 标签为 ORA-00700, ORA-00700 pga physmem limit, pga physmem limit | 评论关闭

fuser命令缺失导致19c打补丁失败

发表于 2019 年 12 月 16 日由惜分飞

Oracle 19.3数据库打最新补丁报错

[oracle@www.xifenfei.com 30125133]$ opatch apply ./
Oracle Interim Patch Installer version 12.2.0.1.18
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/19.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.18
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-00-53AM_1.log

分析日志

Verifying environment and performing prerequisite checks...
Prerequisite check "CheckSystemCommandAvailable" failed.
The details are:
Missing command :fuser
UtilSession failed: 
Prerequisite check "CheckSystemCommandAvailable" failed.
Log file location: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-00-53AM_1.log

OPatch failed with error code 73


[Dec 16, 2019 8:00:57 AM] [INFO]    Following patches can be applied:  30125133
[Dec 16, 2019 8:00:57 AM] [INFO]    Following patches are not required:
[Dec 16, 2019 8:00:57 AM] [INFO]    Following patches are auto rollbackable:
[Dec 16, 2019 8:00:57 AM] [INFO]    Finished checking prereq checkConflictAgainstOHWithDetail
[Dec 16, 2019 8:00:58 AM] [INFO]    Running prerequisite checks...
[Dec 16, 2019 8:00:58 AM] [INFO]    Space Needed : 3052.647MB
[Dec 16, 2019 8:00:58 AM] [INFO]    Missing command :fuser
[Dec 16, 2019 8:00:58 AM] [INFO]    Prerequisite check "CheckSystemCommandAvailable" failed.
                                    The details are:
                                    Missing command :fuser
[Dec 16, 2019 8:00:58 AM] [SEVERE]  OUI-67073:UtilSession failed:
                                    Prerequisite check "CheckSystemCommandAvailable" failed.

基本上可以确定是由于无fuser命令导致CheckSystemCommandAvailable失败.处理办法为安装上相关程序

[root@www.xifenfei.com tmp]# yum search fuser
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
==================================== Matched: fuser ===========================
psmisc.x86_64 : Utilities for managing processes on your system
[root@www.xifenfei.com tmp]# 


[root@www.xifenfei.com tmp]# mount /dev/cdrom /media/ -o loop
[root@www.xifenfei.com tmp]# yum install -y psmisc
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
rhel-yum                       | 4.3 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package psmisc.x86_64 0:22.20-15.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================
 Package                    Arch              Version                    Repository     Size
=============================================================================================
Installing:
 psmisc                     x86_64            22.20-15.el7               rhel-yum      141 k

Transaction Summary
=============================================================================================
Install  1 Package

Total download size: 141 k
Installed size: 475 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : psmisc-22.20-15.el7.x86_64          1/1 
  Verifying  : psmisc-22.20-15.el7.x86_64          1/1 

Installed:
  psmisc.x86_64 0:22.20-15.el7     

Complete!

重新打补丁

[oracle@www.xifenfei.com 30125133]$ opatch apply ./
Oracle Interim Patch Installer version 12.2.0.1.18
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/19.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.18
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-08-44AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30125133  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/19.0/db_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30125133' to OH '/u01/app/oracle/product/19.0/db_1'
ApplySession: Optional component(s) [ oracle.network.gsm, 19.0.0.0.0 ],[ oracle.rdbms.ic, 19.0.0.0.0 ],
[ oracle.network.cman, 19.0.0.0.0 ],[ oracle.net.cman, 19.0.0.0.0 ],[ oracle.options.olap.awm, 19.0.0.0.0],
[ oracle.oraolap.mgmt, 19.0.0.0.0 ],[ oracle.assistants.usm, 19.0.0.0.0 ],[ oracle.assistants.asm, 19.0.0.0.0],
[ oracle.tfa, 19.0.0.0.0 ]  not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 19.0.0.0.0...

Patching component oracle.rdbms.rsf, 19.0.0.0.0...

Patching component oracle.assistants.acf, 19.0.0.0.0...

Patching component oracle.assistants.deconfig, 19.0.0.0.0...

Patching component oracle.assistants.server, 19.0.0.0.0...

Patching component oracle.buildtools.rsf, 19.0.0.0.0...

Patching component oracle.ctx, 19.0.0.0.0...

Patching component oracle.ldap.rsf, 19.0.0.0.0...

Patching component oracle.network.rsf, 19.0.0.0.0...

Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...

Patching component oracle.sdo, 19.0.0.0.0...

Patching component oracle.sqlplus, 19.0.0.0.0...

Patching component oracle.ldap.rsf.ic, 19.0.0.0.0...

Patching component oracle.rdbms.rman, 19.0.0.0.0...

Patching component oracle.ctx.atg, 19.0.0.0.0...

Patching component oracle.rdbms.oci, 19.0.0.0.0...

Patching component oracle.rdbms.util, 19.0.0.0.0...

Patching component oracle.xdk, 19.0.0.0.0...

Patching component oracle.ovm, 19.0.0.0.0...

Patching component oracle.network.listener, 19.0.0.0.0...

Patching component oracle.rdbms.install.plugins, 19.0.0.0.0...

Patching component oracle.dbjava.jdbc, 19.0.0.0.0...

Patching component oracle.dbdev, 19.0.0.0.0...

Patching component oracle.rdbms.deconfig, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf, 19.0.0.0.0...

Patching component oracle.oraolap.dbscripts, 19.0.0.0.0...

Patching component oracle.install.deinstalltool, 19.0.0.0.0...

Patching component oracle.dbjava.ic, 19.0.0.0.0...

Patching component oracle.sdo.locator, 19.0.0.0.0...

Patching component oracle.rdbms.scheduler, 19.0.0.0.0...

Patching component oracle.rdbms.dv, 19.0.0.0.0...

Patching component oracle.ons, 19.0.0.0.0...

Patching component oracle.ldap.security.osdt, 19.0.0.0.0...

Patching component oracle.ctx.rsf, 19.0.0.0.0...

Patching component oracle.duma, 19.0.0.0.0...

Patching component oracle.ldap.owm, 19.0.0.0.0...

Patching component oracle.oracore.rsf, 19.0.0.0.0...

Patching component oracle.rdbms.install.seeddb, 19.0.0.0.0...

Patching component oracle.odbc, 19.0.0.0.0...

Patching component oracle.sdo.locator.jrf, 19.0.0.0.0...

Patching component oracle.network.client, 19.0.0.0.0...

Patching component oracle.sqlplus.ic, 19.0.0.0.0...

Patching component oracle.dbjava.ucp, 19.0.0.0.0...

Patching component oracle.xdk.rsf, 19.0.0.0.0...

Patching component oracle.marvel, 19.0.0.0.0...

Patching component oracle.xdk.parser.java, 19.0.0.0.0...

Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf.core, 19.0.0.0.0...

Patching component oracle.precomp.common, 19.0.0.0.0...

Patching component oracle.precomp.lang, 19.0.0.0.0...
Patch 30125133 successfully applied.
Sub-set patch [29517242] has become inactive due to the application of a super-set patch [30125133].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-08-44AM_1.log

OPatch succeeded.


[oracle@www.xifenfei.com 30128191]$ opatch apply ./
Oracle Interim Patch Installer version 12.2.0.1.18
Copyright (c) 2019, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/app/oracle/product/19.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/19.0/db_1/oraInst.loc
OPatch version    : 12.2.0.1.18
OUI version       : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-12-44AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   30128191  

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/19.0/db_1')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '30128191' to OH '/u01/app/oracle/product/19.0/db_1'

Patching component oracle.javavm.server, 19.0.0.0.0...

Patching component oracle.javavm.server.core, 19.0.0.0.0...

Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...

Patching component oracle.rdbms, 19.0.0.0.0...
Patch 30128191 successfully applied.
Log file location: /u01/app/oracle/product/19.0/db_1/cfgtoollogs/opatch/opatch2019-12-16_08-12-44AM_1.log

OPatch succeeded.

确认补丁安装成功

[oracle@www.xifenfei.com 30128191]$ opatch lspatches
30128191;OJVM RELEASE UPDATE: 19.5.0.0.191015 (30128191)
30125133;Database Release Update : 19.5.0.0.191015 (30125133)
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)

OPatch succeeded.

mos上也有类似文章可参考:Prerequisite check “CheckSystemCommandAvailable” failed (Doc ID 1581604.1)

发表在 ORACLE 19C, Oracle安装升级 | 标签为 Missing command :fuser | 评论关闭

17813235971

QQ 咨询

分类目录归档：ORACLE 19C

Oracle 19c故障恢复

ORA-00700: soft internal error, arguments: [pga physmem limit]

fuser命令缺失导致19c打补丁失败

文章分类

最近发表

分类目录归档：ORACLE 19C

Oracle 19c故障恢复

ORA-00700: soft internal error, arguments: [pga physmem limit]

fuser命令缺失导致19c打补丁失败

标签云

文章分类

最近发表