11gR2 GI打PSU ORACLE_HOME空间不足案例—需要20G以上空闲空间

发表于 2014 年 10 月 15 日 由 惜分飞

GI打auto打patch命令

#/u01/oracle/app/grid/OPatch/opatch auto /u01/soft/18706472 -ocmrf /u01/soft/ocm.rsp
Executing /u01/oracle/app/grid/perl/bin/perl /u01/oracle/app/grid/OPatch/crs/patch11203.pl 
-patchdir /u01/soft -patchn 18706472 -ocmrf /u01/soft/ocm.rsp -paramfile /u01/oracle/app/grid/crs/install/crsconfig_params

This is the main log file: /u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-15-28.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-15-28.report.log

2014-10-15 01:15:28: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/oracle/app/grid/crs/install/crsconfig_params

Stopping CRS...
Stopped CRS successfully

patch /u01/soft/18706472/18522509  apply successful for home  /u01/oracle/app/grid 
patch /u01/soft/18706472/18522515  apply failed  for home  /u01/oracle/app/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

opatch auto succeeded.

这里可以看到,打patch 18522509成功,后续patch都失败,而不是仅仅18522515失败

分析错误日志

Composite patch 18522509 successfully applied.
 OPatch Session completed with warnings.
 Log file location: /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-15_01-19-00AM_1.log

 OPatch completed with warnings.

2014-10-15 01:21:44: patch /u01/soft/18706472/18522509  apply successful for home  /u01/oracle/app/grid
2014-10-15 01:21:44: Executing command /u01/oracle/app/grid/OPatch/opatch napply /u01/soft/18706472/18522515 -local -silent -ocmrf /
u01/soft/ocm.rsp -oh /u01/oracle/app/grid -invPtrLoc /u01/oracle/app/grid/oraInst.loc as grid
2014-10-15 01:21:44: Running as user grid: /u01/oracle/app/grid/OPatch/opatch napply /u01/soft/18706472/18522515 -local -silent -ocm
rf /u01/soft/ocm.rsp -oh /u01/oracle/app/grid -invPtrLoc /u01/oracle/app/grid/oraInst.loc
2014-10-15 01:21:44: s_run_as_user2: Running /bin/su grid -c ' /u01/oracle/app/grid/OPatch/opatch napply /u01/soft/18706472/18522515
 -local -silent -ocmrf /u01/soft/ocm.rsp -oh /u01/oracle/app/grid -invPtrLoc /u01/oracle/app/grid/oraInst.loc '
2014-10-15 01:21:49: Removing file /tmp/uaa7jC7eu
2014-10-15 01:21:49: Successfully removed file: /tmp/uaa7jC7eu
2014-10-15 01:21:49: /bin/su exited with rc=73

2014-10-15 01:21:49: status of apply patch is 18688
2014-10-15 01:21:49: The apply patch output is Oracle Interim Patch Installer version 11.2.0.3.6
 Copyright (c) 2013, Oracle Corporation.  All rights reserved.

 Oracle Home       : /u01/oracle/app/grid
 Central Inventory : /u01/oracle/app/oraInventory
    from           : /u01/oracle/app/grid/oraInst.loc
 OPatch version    : 11.2.0.3.6
 OUI version       : 11.2.0.4.0
 Log file location : /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-15_01-21-44AM_1.log

 Verifying environment and performing prerequisite checks...
 Prerequisite check "CheckSystemSpace" failed.
 The details are:
 Required amount of space(24021.839MB) is not available.
 UtilSession failed:
 Prerequisite check "CheckSystemSpace" failed.
 Log file location: /u01/oracle/app/grid/cfgtoollogs/opatch/opatch2014-10-15_01-21-44AM_1.log

 OPatch failed with error code 73

这里可以看出来,patch 18522509 成功,但是18522515在check的时候失败,由于$ORACLE_HOME(GI)需要可用空间为24021.839MB,现在空闲空间不足

查看空间并腾空间

# df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          10.00      9.56    5%    12443     1% /
/dev/hd2          15.00     11.42   24%    60339     3% /usr
/dev/hd9var       10.00      9.91    1%     3310     1% /var
/dev/hd3          10.00      9.65    4%      364     1% /tmp
/dev/hd1          10.00     10.00    1%       32     1% /home
/dev/hd11admin      0.25      0.25    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt      10.00      8.98   11%    12401     1% /opt
/dev/fslv00       50.00     12.94   75%    59137     2% /u01
/dev/odm           0.00      0.00   -1%        6   100% /dev/odm
/dev/vx/dsk/crs_dg/crs_vol      1.96      1.75   12%        5     1% /crsdata
/dev/vx/dsk/ora_dg/data01_vol   2048.00   2031.32    1%        4     1% /oradata/data01
/dev/vx/dsk/ora_dg/sys_vol    400.00    350.29   13%     3702     1% /oradata/sys
# cd /u01
# ls
lost+found  oracle      soft
# cd soft
# du -sg
17.34   .
# mv /u01/soft /oradata/sys/
# df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          10.00      9.56    5%    12443     1% /
/dev/hd2          15.00     11.42   24%    60339     3% /usr
/dev/hd9var       10.00      9.91    1%     3310     1% /var
/dev/hd3          10.00      9.65    4%      364     1% /tmp
/dev/hd1          10.00     10.00    1%       32     1% /home
/dev/hd11admin      0.25      0.25    1%        5     1% /admin
/proc                 -         -    -         -     -  /proc
/dev/hd10opt      10.00      8.98   11%    12401     1% /opt
/dev/fslv00       50.00     27.56   45%    59137     2% /u01
/dev/odm           0.00      0.00   -1%        6   100% /dev/odm
/dev/vx/dsk/crs_dg/crs_vol      1.96      1.75   12%        5     1% /crsdata
/dev/vx/dsk/ora_dg/data01_vol   2048.00   2031.32    1%        4     1% /oradata/data01
/dev/vx/dsk/ora_dg/sys_vol    400.00    333.54   14%     3702     1% /oradata/sys
# cd soft
# ls -ltr
total 12464202
drwxr-xr-x    5 grid     dba            1024 Jul 07 21:06 18706472
-rw-rw-r--    1 grid     dba            2123 Jul 15 20:06 PatchSearch.xml
-rw-r--r--    1 root     system   1801653734 Oct 14 14:39 p13390677_112040_aix64-5l_1of7.zip
-rw-r--r--    1 root     system   1170882875 Oct 14 14:40 p13390677_112040_aix64-5l_2of7.zip
-rw-r--r--    1 root     system   2127071138 Oct 14 14:41 p13390677_112040_aix64-5l_3of7.zip
-rw-r--r--    1 root     system   1242090126 Oct 14 14:43 p18706472_112040_AIX64-5L.zip
-rw-r--r--    1 root     system     34964928 Oct 14 14:43 p6880880_112000_AIX64-5L.zip
-rwxr-xr-x    1 root     system       127965 Oct 14 14:48 unzip_aix
-rw-r--r--    1 root     system      4871110 Oct 14 14:50 ONEOFF_112043_AIX64.zip
drwxr-xr-x    8 grid     dba            1024 Oct 14 14:55 grid
drwxr-xr-x    8 oracle   dba            1024 Oct 14 14:57 database
-rw-r--r--    1 grid     dba             621 Oct 15 00:11 ocm.rsp

确实/u01空间不足,通过把RAC安装文件迁移到其他目录,确保/u01有足够空间用来打patch

回退命令

# /u01/oracle/app/grid/OPatch/opatch auto /u01/soft/18706472 -rollback -ocmrf /u01/soft/ocm.rsp

重新打patch

#/u01/oracle/app/grid/OPatch/opatch auto /u01/soft/18706472 -ocmrf /u01/soft/ocm.rsp                 <
Executing /u01/oracle/app/grid/perl/bin/perl /u01/oracle/app/grid/OPatch/crs/patch11203.pl -patchdir /oradata/sys/soft 
-patchn 18706472 -ocmrf /oradata/sys/soft/ocm.rsp -paramfile /u01/oracle/app/grid/crs/install/crsconfig_params

This is the main log file: /u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-57-42.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/oracle/app/grid/cfgtoollogs/opatchauto2014-10-15_01-57-42.report.log

2014-10-15 01:57:42: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/oracle/app/grid/crs/install/crsconfig_params

Stopping CRS...
Stopped CRS successfully

patch /oradata/sys/soft/18706472/18522509  apply successful for home  /u01/oracle/app/grid 
patch /oradata/sys/soft/18706472/18522515  apply successful for home  /u01/oracle/app/grid 
patch /oradata/sys/soft/18706472/18522514  apply successful for home  /u01/oracle/app/grid 

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

opatch auto succeeded.

检查确认patch成功

# su - grid
xifenfei1:/home/grid> opatch lspatches
18522514;ACFS PATCH SET UPDATE : 11.2.0.4.3 (18522514)
18522515;OCW Patch Set Update : 11.2.0.4.3 (18522515)
18522509;Database Patch Set Update : 11.2.0.4.3 (18522509)
xifenfei1:/home/grid> crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       xifenfei1                                     
               ONLINE  ONLINE       xifenfei2                                     
ora.asm
               OFFLINE OFFLINE      xifenfei1                                     
               OFFLINE OFFLINE      xifenfei2                                     
ora.gsd
               OFFLINE OFFLINE      xifenfei1                                     
               OFFLINE OFFLINE      xifenfei2                                     
ora.net1.network
               ONLINE  ONLINE       xifenfei1                                     
               ONLINE  ONLINE       xifenfei2                                     
ora.ons
               ONLINE  ONLINE       xifenfei1                                     
               ONLINE  ONLINE       xifenfei2                                     
ora.registry.acfs
               OFFLINE OFFLINE      xifenfei1                                     
               OFFLINE OFFLINE      xifenfei2                                     
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       xifenfei2                                     
ora.cvu
      1        ONLINE  ONLINE       xifenfei2                                     
ora.xifenfei1.vip
      1        ONLINE  ONLINE       xifenfei1                                     
ora.xifenfei2.vip
      1        ONLINE  ONLINE       xifenfei2                                     
ora.oc4j
      1        ONLINE  ONLINE       xifenfei2                                     
ora.scan1.vip
      1        ONLINE  ONLINE       xifenfei2
发表在 Oracle RAC | 评论关闭

通过多次resetlogs规避类似ORA-01248: file N was created in the future of incomplete recovery错误

发表于 2014 年 10 月 14 日 由 惜分飞

数据库现状
控制文件
recover_xifenfei0
控制文件中数据文件信息
recover_xifenfei1
数据文件头信息
recover_xifenfei2
redo信息
recover_xifenfei3
根据当前数据库恢复检查脚本(Oracle Database Recovery Check)收集的信息,数据库的是非归档状态,而且redo已经覆盖,数据库datafile 5 无法直接online.遇到这样情况,可以使用bbed修改文件头scn实现online(使用bbed让rac中的sysaux数据文件online),也可以通过使用_allow_resetlogs_corruption等隐含参数实现online.本恢复案例中有180个数据文件,160个offline,然后open数据库,所以大量数据文件无法正常online,bbed工作量太大.在恢复过程中不幸遇到ORA-01248

数据库resetlogs出现ORA-01248错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01248: file 5 was created in the future of incomplete recovery
ORA-01110: data file 5: 'F:\TTDATA\PUBRTS.DAT'

alert日志记录

Fri Oct 10 15:09:26 2014
alter database open resetlogs
Fri Oct 10 15:09:26 2014
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
ORA-1248 signalled during: alter database open resetlogs...
Fri Oct 10 15:15:22 2014
alter database open
Fri Oct 10 15:15:22 2014
ORA-1589 signalled during: alter database open...
Fri Oct 10 15:15:30 2014
alter database  open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
ORA-1248 signalled during: alter database  open resetlogs...

尝试offline文件然后resetlogs

SQL>ALTER DATABASE DATAFILE 5  OFFLINE;

Database altered.


sql>ALTER DATABASE OPEN RESETLOGS；

ERROR at line 1:
ORA-01245: ffline file 5 will be lost if resetlogs is done
ORA-01110: data file 5: 'F:\TTDATA\PUBRTS.DAT'

alert日志

Fri Oct 10 15:19:37 2014
ALTER DATABASE DATAFILE 5 offline
Fri Oct 10 15:19:37 2014
Completed: ALTER DATABASE DATAFILE 5 offline
Fri Oct 10 15:19:40 2014
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
ORA-1245 signalled during: alter database open resetlogs...

出现该错误原因是由于数据库是非归档模式,offline数据文件需要使用offline drop

Fri Oct 10 15:22:16 2014
alter database datafile 5 offline drop
Fri Oct 10 15:22:17 2014
Completed: alter database datafile 5 offline drop
Fri Oct 10 15:23:13 2014
alter database open resetlogs
Fri Oct 10 15:23:14 2014
Fri Oct 10 15:23:49 2014
RESETLOGS after complete recovery through change 1422423346
Resetting resetlogs activation ID 3503292347 (0xd0cfffbb)
Fri Oct 10 15:24:01 2014
Setting recovery target incarnation to 3
Fri Oct 10 15:24:04 2014
Assigning activation ID 3649065262 (0xd980512e)
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=23, OS id=3772
Fri Oct 10 15:24:04 2014
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=24, OS id=3668
Fri Oct 10 15:24:05 2014
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\CLTTDB\REDO01.LOG
Successful open of redo thread 1
Fri Oct 10 15:24:05 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Oct 10 15:24:05 2014
ARC0: STARTING ARCH PROCESSES
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC2 started with pid=25, OS id=636
Fri Oct 10 15:24:06 2014
ARC0: Becoming the 'no SRL' ARCH
Fri Oct 10 15:24:06 2014
ARC1: Becoming the heartbeat ARCH
Fri Oct 10 15:24:06 2014
SMON: enabling cache recovery
Fri Oct 10 15:24:07 2014
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
File #5 is offline, but is part of an online tablespace.
data file 5: 'F:\TTDATA\PUBRTS.DAT'
Dictionary check complete
Fri Oct 10 15:24:19 2014
SMON: enabling tx recovery
Fri Oct 10 15:24:19 2014
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=26, OS id=868
Fri Oct 10 15:24:21 2014
LOGSTDBY: Validating controlfile with logical metadata
Fri Oct 10 15:24:21 2014
LOGSTDBY: Validation complete
Completed: alter database open resetlogs

open成功后,再次resetlogs库,实现数据文件online

Fri Oct 10 15:28:44 2014
ALTER DATABASE DATAFILE 5 online
Fri Oct 10 15:28:44 2014
Completed: ALTER DATABASE DATAFILE 5 online
Fri Oct 10 15:31:46 2014
alter database open resetlogs
Fri Oct 10 15:31:46 2014
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
Setting recovery target incarnation to 4
Fri Oct 10 15:32:00 2014
Assigning activation ID 3649091231 (0xd980b69f)
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=23, OS id=700
Fri Oct 10 15:32:00 2014
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=24, OS id=3360
Fri Oct 10 15:32:01 2014
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\CLTTDB\REDO01.LOG
Successful open of redo thread 1
Fri Oct 10 15:32:01 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Oct 10 15:32:01 2014
ARC0: STARTING ARCH PROCESSES
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC2 started with pid=25, OS id=2016
Fri Oct 10 15:32:02 2014
ARC0: Becoming the 'no SRL' ARCH
Fri Oct 10 15:32:02 2014
ARC1: Becoming the heartbeat ARCH
Fri Oct 10 15:32:02 2014
SMON: enabling cache recovery
Fri Oct 10 15:32:03 2014
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
Fri Oct 10 15:32:15 2014
Dictionary check complete
Fri Oct 10 15:32:15 2014
SMON: enabling tx recovery
Fri Oct 10 15:32:15 2014
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=26, OS id=256
Fri Oct 10 15:32:17 2014
LOGSTDBY: Validating controlfile with logical metadata
Fri Oct 10 15:32:17 2014
LOGSTDBY: Validation complete
Completed: alter database open resetlogs
发表在 Oracle备份恢复 | 标签为 , , | 评论关闭

MOS又一次不靠谱—ORA-27163: out of memory

发表于 2014 年 9 月 30 日 由 惜分飞

数据库版本

oracle -> 11g @xifenfei:/home/oracle$sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Tue Sep 30 10:28:30 2014

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE    11.2.0.4.0      Production
TNS for HPUX: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

数据库补丁信息

oracle -> 11g @xifenfei:/home/oracle$opatch lspatches
18973907;
18795105;
18701707;
18284357;
18020394;
17564992;
17308789;
17306264;
17259786;
17079301;
16477664;
16188701;
14368995;
14245531;
11744544;
18522515;OCW Patch Set Update : 11.2.0.4.3 (18522515)
18522509;Database Patch Set Update : 11.2.0.4.3 (18522509)

收集spa报告报ORA-27163: out of memory错误

SQL> SELECT dbms_sqlpa.report_analysis_task('SPA_TEST', 'HTML', 'errors','ALL') FROM dual;

ERROR:
ORA-27163: out of memory
ORA-06512: at "SYS.DBMS_SQLTUNE_INTERNAL", line 8211
ORA-06512: at "SYS.DBMS_SQLPA", line 515
ORA-06512: at line 1


no rows selected

设置event后收集

Connected.
SQL> alter session set events '31156 trace name context forever, level 0x400';

Session altered.

SQL> SELECT dbms_sqlpa.report_analysis_task('SPA_TEST', 'HTML', 'errors','ALL') FROM dual;

DBMS_SQLPA.REPORT_ANALYSIS_TASK('SPA_TEST','HTML','ERRORS','ALL')
--------------------------------------------------------------------------------
<html>
    <head>
        <title>
   SQL Performance Impact Analyzer Report

查询MOS发现XML Parser Fails With ORA-27163 (Out Of Memory) (Doc ID 1599434.1),相关描述:
bug-xml
按照文档描述,该问题在11.2.0.4中已经修复,可是在hp unix中依然存在该问题,经验告诉我们,现在的MOS不能完全相信

发表在 Oracle | 标签为 | 评论关闭