分类目录归档：Data Guard

ORA-600 kcbr_apply_change_11

发表于 2023 年 6 月 25 日 由 惜分飞

客户active dataguard应用日志报ORA-600 kcbr_apply_change_11错误

Sun Jun 25 10:31:25 2023
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Sun Jun 25 10:31:25 2023
MRP0 started with pid=32, OS id=6380 
 started logmerger process
Sun Jun 25 10:31:30 2023
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 4 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log D:\APP\ADMINISTRATOR\FAST_RECOVERY_AREA\xffWIN\ARCHIVELOG\2023_06_23\O1_MF_1_196319_L99NQQSN_.ARC
Sun Jun 25 10:31:31 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr03_1540.trc  (incident=180300):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180300\xff_pr03_1540_i180300.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT  LOGFILE DISCONNECT FROM SESSION
Sun Jun 25 10:31:32 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr02_752.trc  (incident=180292):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180292\xff_pr02_752_i180292.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_rfs_6032.trc:
ORA-01265: 无法删除 ARCHIVED LOG /u01/oracle/fast_recovery_area/xffDG/archivelog/2023_05_28/o1_mf_1_192064_l74rj3jz_.arc
ORA-27056: 无法删除文件
OSD-04029: 无法获取文件属性
O/S-Error: (OS 3) 系统找不到指定的路径。
Archived Log entry 35084 added for thread 1 sequence 196424 rlc 1013082900 ID 0xe513780f dest 3:
RFS[7]: Opened log for thread 1 sequence 196430 dbid -451738353 branch 1013082900
Slave exiting with ORA-600 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr02_752.trc:
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Sun Jun 25 10:31:41 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc  (incident=180308):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180308\xff_pr04_3696_i180308.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Sun Jun 25 10:31:41 2023
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_mrp0_6380.trc  (incident=180268):
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\incident\incdir_180268\xff_mrp0_6380_i180268.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Recovery Slave PR02 previously exited with exception 600
Sun Jun 25 10:31:41 2023
MRP0: Background Media Recovery terminated with error 448
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr00_7812.trc:
ORA-00448: 后台进程正常结束
Managed Standby Recovery not using Real Time Apply
Sun Jun 25 10:31:41 2023
Slave exiting with ORA-600 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr03_1540.trc:
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Sun Jun 25 10:31:43 2023
Sweep [inc][180308]: completed
Sweep [inc][180300]: completed
Sweep [inc][180292]: completed
Sweep [inc][180268]: completed
Sweep [inc2][180300]: completed
Sweep [inc2][180292]: completed
Sweep [inc2][180268]: completed
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc:
ORA-00448: 后台进程正常结束
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc:
ORA-00339: 归档日志未包含任何重做
ORA-00334: 归档日志: 'D:\APP\ADMINISTRATOR\ORADATA\xff\REDO02.LOG'
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []
Slave exiting with ORA-600 exception
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xffwin\xff\trace\xff_pr04_3696.trc:
ORA-00600: 内部错误代码, 参数: [kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []

通过分析可能为：Bug 31104809 – ORA-600: [kcbr_apply_change_11] during adg recovery (Doc ID 31104809.8)
官方WORKAROUND:
In ADG, mount the database (do not open it) and restart media recovery; once
the affected redo log sequence is applied, open the ADG in read only mode.

发表在 Data Guard | 标签为 | 评论关闭

login trigger导致ORA-16191问题

发表于 2023 年 4 月 16 日 由 惜分飞

主库突然报ORA-16191错，主库无法传输日志到备库

at Apr 08 19:56:26 2023
Thread 1 advanced to log sequence 32548 (LGWR switch)
  Current log# 1 seq# 32548 mem# 0: /u01/app/oracle/oradata/orcl/redo01_a.rdo
  Current log# 1 seq# 32548 mem# 1: /oracle/fast_recovery_area/orcl/redo01_b.rdo
Sat Apr 08 19:56:26 2023
Archived Log entry 61796 added for thread 1 sequence 32547 ID 0x5fc1b26f dest 1:
Sat Apr 08 19:56:27 2023
LNS: Standby redo logfile selected for thread 1 sequence 32548 for destination LOG_ARCHIVE_DEST_2
Sat Apr 08 20:21:29 2023
NSA:  Error 3135 archiving log 1 to 'orcldg'
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_nsa2_26205.trc:
ORA-03135: connection lost contact
LNS: Failed to archive log 1 thread 1 sequence 32548 (3135)
Sat Apr 08 20:26:26 2023
Thread 1 advanced to log sequence 32549 (LGWR switch)
  Current log# 2 seq# 32549 mem# 0: /u01/app/oracle/oradata/orcl/redo02_a.rdo
  Current log# 2 seq# 32549 mem# 1: /oracle/fast_recovery_area/orcl/redo02_b.rdo
Sat Apr 08 20:26:26 2023
Archived Log entry 61798 added for thread 1 sequence 32548 ID 0x5fc1b26f dest 1:
Sat Apr 08 20:26:55 2023
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
      returning error ORA-16191
------------------------------------------------------------
PING[ARC2]: Heartbeat failed to connect to standby 'orcldg'. Error is 16191.
Sat Apr 08 20:27:57 2023
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
      returning error ORA-16191
------------------------------------------------------------
PING[ARC2]: Heartbeat failed to connect to standby 'orcldg'. Error is 16191.

备库报错

Sat Apr 08 19:29:09 2023
Media Recovery Waiting for thread 1 sequence 32548 (in transit)
Recovery of Online Redo Log: Thread 1 Group 12 Seq 32548 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/orcl/std_redo12.log
Sat Apr 08 21:52:19 2023
RFS[9]: Possible network disconnect with primary database

检查主备库remote_login_passwordfile参数,都是为EXCLUSIVE,确认没有问题

--备库
SQL> show parameter pass;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_login_passwordfile            string      EXCLUSIVE
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ ONLY WITH APPLY

--主库
SQL> select database_role from v$database;

DATABASE_ROLE
----------------
PRIMARY

SQL> show parameter pass;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_login_passwordfile            string      EXCLUSIVE

检查主备库密码文件,证明主备库密码文件一致,也不是密码文件引起不同步

--主库
[oracle@oracle1 trace]$ cd $ORACLE_HOME/dbs
[oracle@oracle1 dbs]$ echo $ORACLE_SID
orcl
[oracle@oracle1 dbs]$ ls -ltr orapw*
-rw-r-----. 1 oracle oinstall 1536 Sep 16  2022 orapworcl
[oracle@oracle1 dbs]$ md5sum orapworcl 
d27c234b5131b4d2e13b1eeb4388f0eb  orapworcl

--备库
[oracle@oracle2 trace]$ cd $ORACLE_HOME/dbs
[oracle@oracle2 dbs]$ echo $ORACLE_SID
orcl
[oracle@oracle2 dbs]$ ls -l orapw*
-rw-r----- 1 oracle oinstall 1536 Sep 16  2022 orapworcl
[oracle@oracle2 dbs]$ md5sum orapworcl 
d27c234b5131b4d2e13b1eeb4388f0eb  orapworcl

查看数据库有logon触发器

SQL> select owner,trigger_name from dba_triggers where triggering_event LIKE '%LOGON%';

OWNER                          TRIGGER_NAME
------------------------------ ------------------------------
DBTOOLS                        TRIGGER_LOGINLIMIT

根据Error 1017 / ORA-16191 In Standby Alertlog (Doc ID 2225190.1)中描述logon触发器可能导致dg传输日志失败.
trigger-ORA-16191

发表在 Data Guard | 标签为 , | 评论关闭

19c adg kill进程报ORA-600 ktuGetTemprsp:no tso

发表于 2022 年 10 月 13 日 由 惜分飞

今日一客户19.5的adg环境由于负载过高kill进程之后，导致dg库直接实例crash，报ORA-600 ktuGetTemprsp:no tso错误

2022-10-13T09:54:58.769194+08:00
Recovery of Online Redo Log: Thread 1 Group 12 Seq 84142 Reading mem 0
  Mem# 0: /oracle/oradata/xff1/std_redo12.log
2022-10-13T09:56:51.312075+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl08_145506.trc  (incident=1294241):
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [396], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:51.324562+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl02_145494.trc  (incident=1292913):
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [3447], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:51.324621+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl05_145500.trc  (incident=1294129):
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [204], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:51.336288+08:00
……………………
2022-10-13T09:56:51.708427+08:00
PMON (ospid: 546859): terminating the instance due to ORA error 12752
2022-10-13T09:56:51.764031+08:00
Cause - 'Instance is being terminated due to fatal process death (pid: 1169, ospid: 145502, CL06)'
2022-10-13T09:56:51.841454+08:00
System state dump requested by (instance=1, osid=546859 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_diag_546895.trc
2022-10-13T09:56:52.030755+08:00
opiodr aborting process unknown ospid (61300) as a result of ORA-1092
2022-10-13T09:56:54.073403+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl08_145506.trc:
ORA-12752: An instance-critical process has terminated.
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [396], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:54.093199+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl02_145494.trc:
ORA-12752: An instance-critical process has terminated.
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [3447], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:54.219531+08:00
Errors in file /u01/app/oracle/diag/rdbms/xff1dg/xff1/trace/xff1_cl07_145504.trc:
ORA-12752: An instance-critical process has terminated.
ORA-00600: internal error code, arguments: [ktuGetTemprsp:no tso], [370], [], [], [], [], [], [], [], [], [], []
2022-10-13T09:56:54.281477+08:00
opidrv aborting process CL08 ospid (145506) as a result of ORA-12752
2022-10-13T09:56:54.288978+08:00
opidrv aborting process CL07 ospid (145504) as a result of ORA-12752
2022-10-13T09:56:54.380794+08:00
opidrv aborting process CL02 ospid (145494) as a result of ORA-12752
2022-10-13T09:57:13.447674+08:00
Instance terminated by PMON, pid = 546859

通过mos分析确认是由于Bug 31443748 – ADG Instance Crashes With ORA-600[ktugettemprsp:no tso] (Doc ID 31443748.8)导致
20221013220330

可以打上patch 31956113进行修复该问题或者直接升级到19.11.0.0.DBRU:210420 (APR 2021) DB Release Update(DB RU)也修复该问题

发表在 Data Guard | 标签为 | 评论关闭