分类目录归档：ORA-xxxxx

因使用OEM引起ORA-00600[12761]

发表于 2012 年 6 月 11 日 由 惜分飞

alert日志报ORA-00600[12761]错

Sun Jun 10 13:52:56 2012
Errors in file e:\oracle\product\10.2.0\admin\interlib\udump\interlib_ora_19840.trc:
ORA-04030: 在尝试分配 82444 字节 (pga heap,control file i/o buffer) 时进程内存不足
ORA-00600: 内部错误代码, 参数: [12761], [], [], [], [], [], [], []
ORA-00604: 递归 SQL 级别 2 出现错误
ORA-04030: 在尝试分配 123404 字节 (QERHJ hash-joi,kllcqas:kllsltba) 时进程内存不足

数据库版本信息

Sun Jun 10 13:52:56 2012
ORACLE V10.2.0.1.0 - Production vsnsta=0
vsnsql=14 vsnxtr=3
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
Windows NT Version V5.2 Service Pack 2
CPU                 : 8 - type 586, 2 Physical Cores
Process Affinity    : 0x00000000
Memory (Avail/Total): Ph:1263M/4095M, Ph+PgF:2716M/5976M, VA:19M/2047M
Instance name: interlib

trace信息

*** 2012-06-10 13:52:56.763
ksedmp: internal or fatal error
ORA-00600: 内部错误代码, 参数: [12761], [], [], [], [], [], [], []
ORA-00604: 递归 SQL 级别 2 出现错误
ORA-04030: 在尝试分配 123404 字节 (QERHJ hash-joi,kllcqas:kllsltba) 时进程内存不足
Current SQL statement for this session:
BEGIN EM_PING.RECORD_BATCH_HEARTBEAT(:1, :2, :3); END; 
----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0BC35C44         1  anonymous block
----- Call Stack Trace -----
calling              call     entry                argument values in hex      
location             type     point                (? means dubious value)     
-------------------- -------- -------------------- ----------------------------
00404686             CALLrel  00404694             0 1
0040307E             CALLrel  00404660             0
0043AB6A             CALLrel  00402CFC             3
603A816A             CALLreg  00000000             6190E2E0 3
603A8550             CALLrel  603A80D8             6190E2E0 5E340020 31D9 0
                                                   5ED6CDF8
031B7197             CALLrel  025FA21E             6190E2E0 5E340020 31D9 2 0
02C92859             CALLrel  02C92360             5ED6D2B4 5ED6D3CC 2 61BA71E4
                                                   0 5ED6CEFA
60BAD7C6             CALL???  00000000             5ED6D2B4 5ED6D3CC 3 61BA71E4
                                                   0 5ED6CEFA
60C41C40             CALLrel  60BAD758             5D9356A0 F1 3 5E344888
60C3C780             CALL???  00000000             5D9356A0 951F190 5D9356DC
60C3D1BD             CALLrel  60C3C748             5D9356A0 95A97F0 5D9356DC
60BB0392             CALLrel  60C3CEB0             5D9356A0
60B89393             CALLrel  60BB00B0             5D9356A0 1 0
02600CD9             CALLrel  0260F22C             
0140AF2C             CALLrel  02600B3C             4E8EC08
013CBFEC             CALLrel  01409984             49 3 5ED6DB14
0085174B             CALLreg  00000000             5E 17 5ED6F6F8
60FEFF8D             CALLreg  00000000             5E 17 5ED6F6F8 0
00850A69             CALL???  00000000             
0122134B             CALLrel  00850670             0 0
0085174B             CALLreg  00000000             3C 4 5ED6FC90
00420E53             CALLrel  00851300             3C 4 5ED6FC90 0
00421645             CALLrel  00420B20             3C 4 5ED6FC90
0040116C             CALLrel  00421618             5ED6FC84 3C 4 5ED6FC90
0040105C             CALLrel  004010FC             2 5ED6FCBC
00401900             CALLrel  00401000             
7C82482C             CALLreg  00000000             
 
--------------------- Binary Stack Dump ---------------------

--会话信息
    (session) sid: 525 trans: 00000000, creator: 7AE024D8, flag: (41) USR/- BSY/-/-/-/-/-
              DID: 0001-001B-00000004, short-term DID: 0000-0000-00000000
              txn branch: 00000000
              oct: 47, prv: 0, sql: 7A0F0A38, psql: 7A0A2430, user: 51/SYSMAN
    O/S info: user: , term: , ospid: 1234, machine: tushuguan01
              program: OMS
    client info: tushuguan01_Management_Service
    application name: OEM.SystemPool, hash value=2960518376

通过这里我们可以得到几个信息
1.数据库先发生了ORA-00600[12761],然后引发了ORA-04030
2.引发ORA-00600[12761]错误的原因是因为OEM的某种操作导致
3.未知因某种原因导致Call Stack Trace信息不完善,无法准确评估bug情况
4.查询数据库当前最大使用使用pga为250M,数据库配置pga为500M,原则上讲不是pga消耗完导致4030错误,可能是这个会话在执行某个基表的查询时候的hash-jion运算时pga不足导致.
5.查询dba_users发现EM_PING不是数据库用户,查询dba_source发现RECORD_BATCH_HEARTBEAT不是plsql名称,从这里可以看出OEM调用程序有一定特殊性

对于该问题的解决方案
1.因为OEM功能不太使用,建议直接关闭该进程,并设置为开机不自动启动
2.因为信息不完善,无法确定具体bug,但目前数据库版本为10.2.0.1,强烈建议升级到新版本

发表在 ORA-xxxxx | 标签为 , | 一条评论

记录一次ORA-00600[2252]故障解决

发表于 2012 年 6 月 8 日 由 惜分飞

数据库alert日志报ORA-00600[2252]

Wed Jun 06 08:56:02 2012
Thread 1 cannot allocate new log, sequence 552
Checkpoint not complete
  Current log# 1 seq# 551 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO01.LOG
Thread 1 advanced to log sequence 552
  Current log# 2 seq# 552 mem# 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO02.LOG
Sun Jun 06 09:39:19 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_3344.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076554712], [], [], [], [], []

Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_3344.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076554712], [], [], [], [], []

Sun Jun 06 10:19:49 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_4904.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555573], [], [], [], [], []

Sun Jun 06 10:20:49 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_5984.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555594], [], [], [], [], []

Sun Jun 06 10:21:49 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_4204.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555614], [], [], [], [], []

Sun Jun 06 10:22:49 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_5896.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555634], [], [], [], [], []

Sun Jun 06 10:23:49 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_4612.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555654], [], [], [], [], []

Sun Jun 06 10:24:49 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_4696.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555676], [], [], [], [], []

Sun Jun 06 10:25:50 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_5568.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555696], [], [], [], [], []

Sun Jun 06 10:26:50 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\bdump\orcl_m000_5776.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076555716], [], [], [], [], []

--启动数据库
Mon Jun 07 09:18:39 2010
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Mon Jun 07 09:18:49 2010
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on. 
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 150
  __shared_pool_size       = 100663296
  __large_pool_size        = 12582912
  __java_pool_size         = 4194304
  __streams_pool_size      = 0
  spfile                   = D:\ORACLE\PRODUCT\10.2.0\DB_1\DBS\SPFILEORCL.ORA
  nls_language             = SIMPLIFIED CHINESE
  nls_territory            = CHINA
  sga_target               = 452984832
  control_files            = D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\CONTROL01.CTL
  db_block_size            = 8192
  __db_cache_size          = 327155712
  compatible               = 10.2.0.1.0
  db_file_multiblock_read_count= 16
  db_recovery_file_dest    = D:\oracle\product\10.2.0/flash_recovery_area
  db_recovery_file_dest_size= 2147483648
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  remote_login_passwordfile= EXCLUSIVE
  db_domain                = 
  dispatchers              = (protocol=TCP)
  shared_servers           = 1
  job_queue_processes      = 10
  audit_file_dest          = D:\ORACLE\PRODUCT\10.2.0\ADMIN\ORCL\ADUMP
  background_dump_dest     = D:\ORACLE\PRODUCT\10.2.0\ADMIN\ORCL\BDUMP
  user_dump_dest           = D:\ORACLE\PRODUCT\10.2.0\ADMIN\ORCL\UDUMP
  core_dump_dest           = D:\ORACLE\PRODUCT\10.2.0\ADMIN\ORCL\CDUMP
  db_name                  = orcl
  open_cursors             = 300
  pga_aggregate_target     = 149946368
PSP0 started with pid=3, OS id=3028
MMAN started with pid=4, OS id=3528
PMON started with pid=2, OS id=2772
DBW0 started with pid=5, OS id=816
CKPT started with pid=7, OS id=3372
SMON started with pid=8, OS id=2584
RECO started with pid=9, OS id=3976
CJQ0 started with pid=10, OS id=1912
MMON started with pid=11, OS id=624
Mon Jun 07 09:19:00 2010
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=2696
Mon Jun 07 09:19:00 2010
starting up 1 shared server(s) ...
LGWR started with pid=6, OS id=3128
Mon Jun 07 09:19:04 2010
alter database mount exclusive
Mon Jun 07 09:19:09 2010
Setting recovery target incarnation to 2
Mon Jun 07 09:19:10 2010
Successful mount of redo thread 1, with mount id 1248834568
Mon Jun 07 09:19:10 2010
Database mounted in Exclusive Mode
Completed: alter database mount exclusive
Mon Jun 07 09:19:10 2010
alter database open
Mon Jun 07 09:19:15 2010
Beginning crash recovery of 1 threads
 parallel recovery started with 2 processes
Mon Jun 07 09:19:18 2010
Started redo scan
Mon Jun 07 09:19:19 2010
Completed redo scan
 13 redo blocks read, 7 data blocks need recovery
Mon Jun 07 09:19:20 2010
Started redo application at
 Thread 1: logseq 552, block 28631
Mon Jun 07 09:19:20 2010
Recovery of Online Redo Log: Thread 1 Group 2 Seq 552 Reading mem 0
  Mem# 0 errs 0: D:\ORACLE\PRODUCT\10.2.0\ORADATA\ORCL\REDO02.LOG
Mon Jun 07 09:19:20 2010
Completed redo application
Mon Jun 07 09:19:20 2010
Completed crash recovery at
 Thread 1: logseq 552, block 28644, scn 12176013920948
 7 data blocks read, 7 data blocks written, 13 redo blocks read
Mon Jun 07 09:19:28 2010
Errors in file d:\oracle\product\10.2.0\admin\orcl\udump\orcl_ora_2688.trc:
ORA-00600: 内部错误代码, 参数: [2252], [2834], [4076604085], [], [], [], [], []

通过这些日志可以看出数据库一直在报ORA-00600[2252],在后来因某种原因数据库异常重启后启动不了.仔细观察可以发现系统显示的时间是2010年6月7日和当前时间相差了整整两年.
At any point in time, the Oracle Database calculates a “not to exceed” limit for the number of SCNs a database can have used, based on the number of seconds elapsed since 1988, multiplied by 16,384. This is known as the database’s current maximum SCN limit. Doing this ensures that Oracle Databases will ration SCNs over time, allowing over 500 years of data processing for any Oracle Database.

错误原因
根据错误提示计算scn(2834为现在系统的SCN WRAP，4076604085就是BASE)=2834*2^32+4076604085=12176013920949
根据数据库日志显示系统时间计算最大scn值:

SQL>select to_number( ((to_date('20100607 09:19:28','yyyymmdd hh24:mi:ss')-
  2 to_date('19880101','yyyymmdd'))*24*3600*16*1024),'999999999999999999') max_scn from dual;

   MAX_SCN
--------------
11598377254912

通过这里的计算可以知道数据库当前的SCN大于系统时间点上允许的最大时间的SCN,从而出现ORA-00600[2252]错误.

解决方法
知道了数据库报该错误的原因,那么解决该问题很简单,修改系统时间到正确的时间点即可

SQL> select to_number(((sysdate-to_date('19880101','yyyymmdd'))*24*3600*16*1024),
   2 '999999999999999999') max_scn from dual;

   MAX_SCN
--------------
12634899464192

该SCN大于数据库当前SCN所有数据库不会报ORA-00600[2252]错误可以正常启动.

发表在 ORA-xxxxx | 标签为 , | 一条评论

记录另一起ORA-00600[13013]处理

发表于 2012 年 6 月 8 日 由 惜分飞

发现ORA-00600[13013]错误
During the execution of an UPDATE statement, after several attempts (Arg [a] passcount) we are unable to get a stable set of rows that conform to the WHERE clause.

Fri Jun  1 03:00:33 2012
Errors in file /opt/oracle/admin/oraapp/bdump/oraapp_m000_12104.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12596578], [17], []
Sat Jun  2 03:01:05 2012
Errors in file /opt/oracle/admin/oraapp/bdump/oraapp_m000_1052.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12596578], [17], []
Sun Jun  3 15:00:50 2012
Errors in file /opt/oracle/admin/oraapp/bdump/oraapp_m000_13876.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12603219], [17], []
Mon Jun  4 03:01:05 2012
Errors in file /opt/oracle/admin/oraapp/bdump/oraapp_m000_7704.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12596578], [17], []
Tue Jun  5 03:00:35 2012
Errors in file /opt/oracle/admin/oraapp/bdump/oraapp_m000_27983.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12596578], [17], []
Wed Jun  6 03:01:07 2012
Errors in file /opt/oracle/admin/oraapp/bdump/oraapp_m000_19204.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12596578], [17], []
Thu Jun  7 03:00:37 2012
Errors in file /opt/oracle/admin/oraapp/bdump/oraapp_m000_7273.trc:
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12605556], [17], []

以前处理过一次ORA-600[13013],里面包含了各参数含义,这次也按照常规方法处理,分析如下:
1.通过trace文件找出对应表

*** 2012-06-01 03:00:33.325
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [13013], [5001], [8943], [12596577], [25], [12596578], [17], []
Current SQL statement for this session:
UPDATE WRH$_SQL_BIND_METADATA SET snap_id = :lah_snap_id  WHERE dbid = :dbid    AND (SQL_ID) 
IN (SELECT STR1_KEWRATTR FROM X$KEWRATTRSTALE)

2.通过ORA-600[13013]中表示rdba参数找出表

SQL> select  DBMS_UTILITY.data_block_address_file (12596577) "file#",
  2  DBMS_UTILITY.data_block_address_block (12596577) "block#" 
  3  from dual;

     file#     block#
---------- ----------
         3      13665

SQL> select * from dba_extents where 13665 between block_id and block_id + blocks and file_id=3;

OWNER      SEGMENT_NAME           SEGMENT_TYPE     EXTENT_ID    FILE_ID   BLOCK_ID    BLOCKS
---------- --------------------  --------------- --------------- ---------- -------  ------
SYS        SYS_LOB0000008933C00  LOBSEGMENT           7          3      13657         8    
SYS        WRH$_SQL_BIND_METADA  TABLE                1          3      13665         8

检查对象WRH$_SQL_BIND_METADA是否有坏块或者表和index不一致

SQL> analyze table SYS.WRH$_SQL_BIND_METADATA validate structure cascade online;

Table analyzed.

这里分析WRH$_SQL_BIND_METADA表正常,但是通过上面的查询证明WRH$_SQL_BIND_METADA的第一个extent的第一个数据块上可能出现问题,使得analyze未检查(自己猜猜,未做深入验证).针对这个问题,直接备份WRH$_SQL_BIND_METADATA表,truncate掉该表,然后重新插入数据(注意操作时间避开awr插入数据时间段)

create table SQL_BIND_METADATA_BAK
AS
SELECT * FROM SYS.WRH$_SQL_BIND_METADATA;

TRUNCATE TABLE SYS.WRH$_SQL_BIND_METADATA;

INSERT INTO SYS.WRH$_SQL_BIND_METADATA
SELECT * FROM SQL_BIND_METADATA_BAK;

DROP TABBLE SQL_BIND_METADATA_BAK PURGE;
发表在 ORA-xxxxx | 标签为 , | 评论关闭