分类目录归档：Oracle

19c adg kill进程报ORA-600 ktuGetTemprsp:no tso

发表于 2022 年 10 月 13 日 由 惜分飞

今日一客户19.5的adg环境由于负载过高kill进程之后，导致dg库直接实例crash，报ORA-600 ktuGetTemprsp:no tso错误

2022-10-13T09:54:58.769194+08:00
Recovery of Online Redo Log: Thread 1 Group 12 Seq 84142 Reading mem 0
  Mem# 0: /oracle/oradata/xff1/std_redo12.log
2022-10-13T09:56:51.312075+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl08_145506.trc  (incident=1294241):
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [396], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:51.324562+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl02_145494.trc  (incident=1292913):
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [3447], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:51.324621+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl05_145500.trc  (incident=1294129):
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [204], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:51.336288+08:00
……………………
2022-10-13T09:56:51.708427+08:00
PMON (ospid: 546859): terminating the instance due to ORA error 12752
2022-10-13T09:56:51.764031+08:00
Cause - 'Instance is being terminated due to fatal process death (pid: 1169, ospid: 145502, CL06)'
2022-10-13T09:56:51.841454+08:00
System state dump requested by (instance=1, osid=546859 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_diag_546895.trc
2022-10-13T09:56:52.030755+08:00
opiodr aborting process unknown ospid (61300) as a result of ORA-1092
2022-10-13T09:56:54.073403+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl08_145506.trc:
ORA-12752: An instance-critical process has terminated.
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [396], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:54.093199+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl02_145494.trc:
ORA-12752: An instance-critical process has terminated.
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [3447], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:54.219531+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl07_145504.trc:
ORA-12752: An instance-critical process has terminated.
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [370], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:54.281477+08:00
opidrv aborting process CL08 ospid (145506) as a result of ORA-12752
2022-10-13T09:56:54.288978+08:00
opidrv aborting process CL07 ospid (145504) as a result of ORA-12752
2022-10-13T09:56:54.380794+08:00
opidrv aborting process CL02 ospid (145494) as a result of ORA-12752
2022-10-13T09:57:13.447674+08:00
Instance terminated by PMON, pid = 546859

通过mos分析确认是由于Bug 31443748 – ADG Instance Crashes With ORA-600[ktugettemprsp:no tso] (Doc ID 31443748.8)导致
20221013220330

可以打上patch 31956113进行修复该问题或者直接升级到19.11.0.0.DBRU:210420 (APR 2021) DB Release Update(DB RU)也修复该问题

发表在 Data Guard | 标签为 | 评论关闭

ORA-00257: archiver error的另外一种原因

发表于 2022 年 10 月 12 日 由 惜分飞

今天遇到一个相对特殊的案例,拿出来和大家分享,数据库报错为不能归档ORA-00257

SQL> conn system/xxxxxx
ERROR:
ORA-00257: archiver error. Connect internal only, until freed.

SQL> archive log list;
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /oracle/PRD/oraarch
Oldest online log sequence     11479
Current log sequence           11481

alert日志报错

sapprddb1:oraprd 73> tail -f alert_prd.log
Master archival failure: 19502
Master archival failure: 19502
Master archival failure: 19502
Master archival failure: 19502
Wed Oct 12 09:27:41 2022
Master archival failure: 19502
Master archival failure: 19502
Master archival failure: 19502
Wed Oct 12 09:27:41 2022
Master archival failure: 19502

该错误的含义

sapprddb1:oraprd 74>oerr ora 19502
19502, 00000, "write error on file \"%s\", block number %s (block size=%s)"
// *Cause:  write error on output file
// *Action: check the file

从报错初步看是由于归档目录空间满了导致,查看发现归档空间剩余很多

sapprddb1:/oracle/PRD/sapdata1 # df -h
Filesystem                          Size  Used Avail Use% Mounted on
/dev/sda1                            99G   25G   70G  26% /
udev                                253G  240K  253G   1% /dev
tmpfs                               426G   72K  426G   1% /dev/shm
/dev/sda3                           388G   12G  357G   4% /backup
/dev/mapper/SAPVG-oraclelv          197G   22G  165G  12% /oracle
/dev/mapper/SAPVG-mirrlogA           20G  773M   18G   5% /oracle/PRD/mirrlogA
/dev/mapper/SAPVG-mirrlogBlv         20G  773M   18G   5% /oracle/PRD/mirrlogB
/dev/mapper/SAPVG-oraarchlv         180G  2.2G  169G   2% /oracle/PRD/oraarch    <----剩余很多
/dev/mapper/SAPVG-origlogAlv         20G  894M   18G   5% /oracle/PRD/origlogA
/dev/mapper/SAPVG-origlogBlv         20G  894M   18G   5% /oracle/PRD/origlogB
/dev/mapper/SAPVG-sapdata1lv        591G  561G     0 100% /oracle/PRD/sapdata1
/dev/mapper/SAPVG-sapdata2lv        1.4T  1.4T     0 100% /oracle/PRD/sapdata2
/dev/mapper/SAPVG-sapdata3lv        788G  748G     0 100% /oracle/PRD/sapdata3
/dev/mapper/SAPVG-sapdata4lv        788G  748G     0 100% /oracle/PRD/sapdata4
sapprd:/sapmnt/PRD                   30G   12G   17G  42% /sapmnt/PRD
sapprd:/usr/sap/trans               105G 1005M   99G   1% /usr/sap/trans
/dev/mapper/VGSAP2-lvsapdata5       1.5T  340G  1.1T  25% /oracle/PRD/sapdata5
/dev/mapper/VGSAP2-lvsapdata6       1.5T  706G  696G  51% /oracle/PRD/sapdata6

尝试人工归档

SQL> alter system archive log current
  2  /
alter system archive log current
*
ERROR at line 1:
ORA-19502: write error on file "/oracle/PRD/sapdata1/cntrl/cntrlPRD.dbf", block
number 4837 (block size=16384)
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 28: No space left on device
Additional information: -1
Additional information: 442368

到这里基本上明确了，由于控制文件所在的分区磁盘使用100%，导致归档的时候无法写如记录到控制文件从而导致数据库报ORA-00257错误，对/oracle/PRD/sapdata1中的某个文件进行稍微收缩，数据库恢复正常

发表在 ORA-xxxxx | 标签为 , | 评论关闭

ORA-600 ktubko_1 恢复

发表于 2022 年 10 月 8 日 由 惜分飞

oracle 12.2的rac库,pdb在open成功之后,没过多久会自动crash掉,主要报错ORA-600 ktubko_1

2022-10-08T16:00:17.874444+08:00
XFF(5):Endian type of dictionary set to little
2022-10-08T16:00:18.602483+08:00
XFF(5):[218515] Successfully onlined Undo Tablespace 26.
XFF(5):Undo initialization finished serial:0 start:73483625 end:73484200 diff:575 ms (0.6 seconds)
XFF(5):Database Characterset for XFF is ZHS16GBK
2022-10-08T16:00:19.340271+08:00
Buffer Cache Full DB Caching mode changing from FULL CACHING ENABLED to FULL CACHING DISABLED 
Full DB Caching disabled: DEFAULT_CACHE_SIZE should be at least 1394670 MBs bigger than current size. 
2022-10-08T16:00:21.308122+08:00
XFF(5):Opening pdb with no Resource Manager plan active
2022-10-08T16:00:22.655433+08:00
Pluggable database XFF opened read write
Completed:  ALTER PLUGGABLE DATABASE ALL OPEN
2022-10-08T16:00:36.419719+08:00
XFF(5):Setting Resource Manager plan SCHEDULER[0x4AC8]:DEFAULT_MAINTENANCE_PLAN via scheduler window
XFF(5):Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
2022-10-08T16:00:57.054295+08:00
XFF(5):minact-scn: got error during useg scan e:1555 usn:57
XFF(5):minact-scn: useg scan erroring out with error e:1555
2022-10-08T16:01:41.527943+08:00
Errors in file /u01/app/db/diag/rdbms/orcl/orcl1/trace/orcl1_smon_218039.trc  (incident=737693) (PDBNAME=XFF):
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
XFF(5):Incident details in: /u01/app/db/diag/rdbms/orcl/orcl1/incident/incdir_737693/orcl1_smon_218039_i737693.trc
XFF(5):Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
2022-10-08T16:01:41.530481+08:00
XFF(5):*****************************************************************
XFF(5):An internal routine has requested a dump of selected redo.
XFF(5):This usually happens following a specific internal error, when
XFF(5):analysis of the redo logs will help Oracle Support with the
XFF(5):diagnosis.
XFF(5):It is recommended that you retain all the redo logs generated (by
XFF(5):all the instances) during the past 12 hours, in case additional
XFF(5):redo dumps are required to help with the diagnosis.
XFF(5):*****************************************************************
2022-10-08T16:01:42.611317+08:00
XFF(5):*****************************************************************
XFF(5):An internal routine has requested a dump of selected redo.
XFF(5):This usually happens following a specific internal error, when
XFF(5):analysis of the redo logs will help Oracle Support with the
XFF(5):diagnosis.
XFF(5):It is recommended that you retain all the redo logs generated (by
XFF(5):all the instances) during the past 12 hours, in case additional
XFF(5):redo dumps are required to help with the diagnosis.
XFF(5):*****************************************************************
XFF(5):ORACLE Instance orcl1 (pid = 44) - Error 600 encountered while recovering transaction (12, 1) on object 50.
2022-10-08T16:01:42.611961+08:00
XFF(5):Errors in file /u01/app/db/diag/rdbms/orcl/orcl1/trace/orcl1_smon_218039.trc:
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
2022-10-08T16:01:42.849438+08:00
Errors in file /u01/app/db/diag/rdbms/orcl/orcl1/trace/orcl1_smon_218039.trc  (incident=737694) (PDBNAME=XFF):
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
XFF(5):Incident details in: /u01/app/db/diag/rdbms/orcl/orcl1/incident/incdir_737694/orcl1_smon_218039_i737694.trc
…………
2022-10-08T16:01:55.212368+08:00
Instance Critical Process (pid: 44, ospid: 218039, SMON) died unexpectedly
PMON (ospid: 217933): terminating the instance due to error 474
2022-10-08T16:01:55.379857+08:00
System state dump requested by (instance=1, osid=217933 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/db/diag/rdbms/orcl/orcl1/trace/orcl1_diag_217966_20221008160155.trc
2022-10-08T16:01:56.417514+08:00
ORA-1092 : opitsk aborting process

因为有smon报的ORACLE Instance orcl1 (pid = 44) – Error 600 encountered while recovering transaction (12, 1) on object xxx这种比较明显错误,基本上可以定位是undo问题.对undo异常事务进行处理,数据库顺利open,并且稳定不再crash,然后对异常对象进行处理(当然也可以逻辑迁移)
20221008220645

在oracle 12.2到18.14的rac环境的cdb库中,如果节点sga大小不一致,而且有一个节点sga大于128G,就可能出现该问题,敬请注意
20221008220914
Bug 32347014: ORA-600[4506], ORA-600[KTUBKO_1] OCCUR AND INSTANCE CRASHES

发表在 Oracle备份恢复 | 标签为 , | 评论关闭