标签云
asm 恢复 asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (100)
- 数据库 (1,597)
- DB2 (22)
- MySQL (70)
- Oracle (1,463)
- Data Guard (49)
- EXADATA (7)
- GoldenGate (21)
- ORA-xxxxx (158)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (13)
- ORACLE 21C (3)
- Oracle ASM (65)
- Oracle Bug (7)
- Oracle RAC (47)
- Oracle 安全 (6)
- Oracle 开发 (27)
- Oracle 监听 (27)
- Oracle备份恢复 (530)
- Oracle安装升级 (84)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (75)
- PostgreSQL (17)
- PostgreSQL恢复 (5)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (36)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (19)
-
最近发表
- Oracle 19c/21c最新patch信息-202404
- PostgreSQL恢复系列:pg_filedump批量处理
- PostgreSQL部分主要字典信息
- PostgreSQL恢复系列:pg_filedump恢复字典构造
- PostgreSQL 16 源码安装
- ORA-00742 ORA-00312 恢复
- 数据库open成功后报ORA-00353 ORA-00354错误引起的一系列问题(本质ntfs文件系统异常)
- ORA-600 ktsiseginfo1故障
- ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
- 最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
- ORA-600 2662快速恢复之Patch scn工具
- TNS-12518: TNS:listener could not hand off client connection
- ora.storage无法启动报ORA-12514故障处理
- 断电引起文件scn异常数据库恢复
- ORA-16188: LOG_ARCHIVE_CONFIG settings inconsistent with previously started instance
- .[hudsonL@cock.li].mkp勒索加密数据库完美恢复
- 模拟带库实现rman远程备份
- 又一例:ORA-600 kclchkblk_4和2662故障
- Oracle误删除数据文件恢复
- Oracle 19C 备库DML重定向—DML Redirection
月归档:九月 2013
误杀进程导致rac hang住
有客户反馈系统hang住,不能归档,需要我们紧急介入分析
节点1日志
出现redo不能归档,redo日志都已经被写满,人工执行了ALTER SYSTEM ARCHIVE LOG CURRENT,数据库就开始把redo全部归档,但是后面产生的redo又不能归档,当redo全部写满之后,数据库有出现大量log file switch (archiving needed)等待
Tue Sep 24 22:05:37 2013 Thread 1 advanced to log sequence 47282 (LGWR switch) Current log# 6 seq# 47282 mem# 0: +DATA/q9db/onlinelog/group_6.1244.818697409 Tue Sep 24 22:07:31 2013 ORACLE Instance q9db1 - Can not allocate log, archival required Thread 1 cannot allocate new log, sequence 47283 All online logs needed archiving Current log# 6 seq# 47282 mem# 0: +DATA/q9db/onlinelog/group_6.1244.818697409 Tue Sep 24 22:28:17 2013 ALTER SYSTEM ARCHIVE LOG Archived Log entry 259646 added for thread 1 sequence 47266 ID 0x354620c2 dest 1: Tue Sep 24 22:28:18 2013 Thread 1 advanced to log sequence 47283 (LGWR switch) Current log# 7 seq# 47283 mem# 0: +DATA/q9db/onlinelog/group_7.1243.818697415 Archived Log entry 259647 added for thread 1 sequence 47267 ID 0x354620c2 dest 1: Archived Log entry 259648 added for thread 1 sequence 47268 ID 0x354620c2 dest 1: Archived Log entry 259649 added for thread 1 sequence 47269 ID 0x354620c2 dest 1: Archived Log entry 259650 added for thread 1 sequence 47270 ID 0x354620c2 dest 1: Archived Log entry 259651 added for thread 1 sequence 47271 ID 0x354620c2 dest 1: Archived Log entry 259652 added for thread 1 sequence 47272 ID 0x354620c2 dest 1: Tue Sep 24 22:28:28 2013 Archived Log entry 259653 added for thread 1 sequence 47273 ID 0x354620c2 dest 1: Archived Log entry 259654 added for thread 1 sequence 47274 ID 0x354620c2 dest 1: Archived Log entry 259655 added for thread 1 sequence 47275 ID 0x354620c2 dest 1: Archived Log entry 259656 added for thread 1 sequence 47276 ID 0x354620c2 dest 1: Archived Log entry 259657 added for thread 1 sequence 47277 ID 0x354620c2 dest 1: Archived Log entry 259658 added for thread 1 sequence 47278 ID 0x354620c2 dest 1: Archived Log entry 259659 added for thread 1 sequence 47279 ID 0x354620c2 dest 1: Tue Sep 24 22:28:39 2013 Archived Log entry 259660 added for thread 1 sequence 47280 ID 0x354620c2 dest 1: Archived Log entry 259661 added for thread 1 sequence 47281 ID 0x354620c2 dest 1: Archived Log entry 259662 added for thread 1 sequence 47282 ID 0x354620c2 dest 1: Tue Sep 24 22:29:39 2013 Thread 1 advanced to log sequence 47284 (LGWR switch) Current log# 8 seq# 47284 mem# 0: +DATA/q9db/onlinelog/group_8.1242.818697417 Tue Sep 24 22:31:18 2013 Thread 1 advanced to log sequence 47285 (LGWR switch) Current log# 16 seq# 47285 mem# 0: +DATA/q9db/onlinelog/group_16.1884.827003545 Thread 1 advanced to log sequence 47286 (LGWR switch) Current log# 17 seq# 47286 mem# 0: +DATA/q9db/onlinelog/group_17.1885.827003587
节点2日志
节点2中出现大量的IPC Send timeout
Tue Sep 24 15:22:19 2013 IPC Send timeout detected. Sender: ospid 4008 [oracle@q9db02.800best.com (PING)] ………… Tue Sep 24 18:51:55 2013 IPC Send timeout detected. Sender: ospid 4008 [oracle@q9db02.800best.com (PING)] Tue Sep 24 18:57:54 2013 IPC Send timeout detected. Sender: ospid 4008 [oracle@q9db02.800best.com (PING)] Receiver: inst 1 binc 464003926 ospid 1566 Tue Sep 24 19:03:57 2013 IPC Send timeout detected. Sender: ospid 4008 [oracle@q9db02.800best.com (PING)] Receiver: inst 1 binc 464003926 ospid 1566 Tue Sep 24 19:09:53 2013 IPC Send timeout detected. Sender: ospid 4008 [oracle@q9db02.800best.com (PING)] ………… Tue Sep 24 20:22:00 2013 IPC Send timeout detected. Sender: ospid 4008 [oracle@q9db02.800best.com (PING)]
节点1因为不能归档hang住,节点2紧接着也就hang住。对节点1hang住之时对两个节点分别做systemstate dump,使用ass进行分析得到节点1和节点2的记录大体如下:
节点1
393:waiting for 'log file switch (archiving needed)' 394:waiting for 'log file switch (archiving needed)' Cmd: Insert 395:waiting for 'log file switch (archiving needed)' Cmd: Insert 397:waiting for 'log file switch (archiving needed)' Cmd: Insert 398:waiting for 'log file switch (archiving needed)' Cmd: Insert 451:waiting for 'SQL*Net message from client' 469:waiting for 'log file switch (archiving needed)' Cmd: Insert 470:waiting for 'log file switch (archiving needed)' Cmd: Insert 471:waiting for 'log file switch (archiving needed)' Cmd: Insert 618:waiting for 'log file switch (archiving needed)' Cmd: Insert 626:waiting for 'log file switch (archiving needed)' Cmd: Insert NO BLOCKING PROCESSES FOUND
节点2
515:waiting for 'gc buffer busy acquire' Cmd: Insert 516:waiting for 'gc buffer busy acquire' Cmd: Insert 517:waiting for 'gc buffer busy acquire' Cmd: Insert 518:waiting for 'gc buffer busy acquire' Cmd: Insert 519:waiting for 'gc buffer busy acquire' Cmd: Insert 520:waiting for 'gc buffer busy acquire' Cmd: Select 521:waiting for 'gc current request' Cmd: Insert 522:waiting for 'enq: TX - row lock contention'[Enq TX-00BA0020-001E3E3C] Cmd: Select 523:waiting for 'gc buffer busy acquire' Cmd: Insert 524:waiting for 'SQL*Net message from client' 525:waiting for 'gc buffer busy acquire' Cmd: Insert 526:waiting for 'gc buffer busy acquire' Cmd: Insert 527:waiting for 'enq: TX - row lock contention'[Enq TX-00BA0020-001E3E3C] Cmd: Select 528:waiting for 'SQL*Net message from client' 529:waiting for 'gc buffer busy acquire' Cmd: Select Resource Holder State Enq TX-0005001E-0022374F 223: waiting for 'gc current request' Enq TX-0047001B-002BCEB2 247: waiting for 'gc current request' Enq TX-015B001E-000041FF 330: waiting for 'gc current request' Enq TX-00010010-002EA7CD 179: waiting for 'gc current request' Enq TX-00BA0020-001E3E3C ??? Blocker Object Names ~~~~~~~~~~~~ Enq TX-0005001E-0022374F Enq TX-0047001B-002BCEB2 Enq TX-015B001E-000041FF Enq TX-00010010-002EA7CD Enq TX-00BA0020-001E3E3C
通过这里,我们可以明白,节点2的很多事务hang住是因为请求gc current request,而该等待是因为节点1无法归档,有些block无法正常传输到节点2,导致节点2一直hang在这里,然后就出现IPC Send timeout;节点1上的事务阻塞甚至hang住是因为无法归档导致.到此需要定位的问题是为什么节点1不能归档
继续分析节点1 alert日志
Tue Sep 24 15:18:20 2013 opidrv aborting process O000 ospid (7332) as a result of ORA-28 Immediate Kill Session#: 1904, Serial#: 1065 Immediate Kill Session: sess: 0x24a2522a38 OS pid: 7338 Immediate Kill Session#: 3597, Serial#: 11107 Immediate Kill Session: sess: 0x24c27cf498 OS pid: 7320 Tue Sep 24 15:18:23 2013 opidrv aborting process W000 ospid (7980) as a result of ORA-28 Tue Sep 24 15:18:23 2013 opidrv aborting process W001 ospid (8560) as a result of ORA-28 Tue Sep 24 15:18:35 2013 LGWR: Detected ARCH process failure LGWR: Detected ARCH process failure LGWR: Detected ARCH process failure LGWR: Detected ARCH process failure LGWR: STARTING ARCH PROCESSES Tue Sep 24 15:18:35 2013 ARC0 started with pid=66, OS id=10793 Tue Sep 24 15:18:35 2013 Errors in file /u01/app/oracle/diag/rdbms/q9db/q9db1/trace/q9db1_nsa2_12635.trc: ORA-00028: your session has been killed LNS: Failed to archive log 8 thread 1 sequence 47156 (28) ARC0: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE Thread 1 advanced to log sequence 47157 (LGWR switch) ARC0: STARTING ARCH PROCESSES Current log# 9 seq# 47157 mem# 0: +DATA/q9db/onlinelog/group_9.1241.818697421 Tue Sep 24 15:18:36 2013 ARC1 started with pid=81, OS id=10805 Tue Sep 24 15:18:36 2013 ARC2 started with pid=84, OS id=10807 Tue Sep 24 15:18:36 2013 ARC3 started with pid=87, OS id=10809 ARC1: Archival started ARC2: Archival started ARC2: Becoming the 'no FAL' ARCH ARC2: Becoming the 'no SRL' ARCH ARC1: Becoming the heartbeat ARCH Error 1031 received logging on to the standby PING[ARC1]: Heartbeat failed to connect to standby 'q9adgdg'. Error is 1031. ARC3: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE Archived Log entry 259135 added for thread 1 sequence 47156 ID 0x354620c2 dest 1: Error 1031 received logging on to the standby FAL[server, ARC3]: Error 1031 creating remote archivelog file 'q9adgdg' FAL[server, ARC3]: FAL archive failed, see trace file. ARCH: FAL archive failed. Archiver continuing ORACLE Instance q9db1 - Archival Error. Archiver continuing. Tue Sep 24 15:18:46 2013 opidrv aborting process O001 ospid (9605) as a result of ORA-28 Tue Sep 24 15:18:46 2013 opidrv aborting process O000 ospid (10813) as a result of ORA-28 Tue Sep 24 15:18:46 2013 Immediate Kill Session#: 2909, Serial#: 369 Immediate Kill Session: sess: 0x24226c7200 OS pid: 9091 Immediate Kill Session#: 3380, Serial#: 30271 Immediate Kill Session: sess: 0x2422782c58 OS pid: 10265 Immediate Kill Session#: 3597, Serial#: 11109 Immediate Kill Session: sess: 0x24c27cf498 OS pid: 10267 Tue Sep 24 15:20:14 2013 Restarting dead background process DIAG Tue Sep 24 15:20:14 2013 DIAG started with pid=64, OS id=20568 Restarting dead background process PING Tue Sep 24 15:20:14 2013 PING started with pid=68, OS id=20570 Restarting dead background process LMHB Tue Sep 24 15:20:14 2013 LMHB started with pid=70, OS id=20572 Restarting dead background process SMCO ………… Tue Sep 24 15:23:13 2013 ARC0: Detected ARCH process failure Tue Sep 24 15:23:13 2013 Thread 1 advanced to log sequence 47158 (LGWR switch) Current log# 10 seq# 47158 mem# 0: +DATA/q9db/onlinelog/group_10.1240.818697423 ARC0: STARTING ARCH PROCESSES ARC0: STARTING ARCH PROCESSES COMPLETE ARC0: Becoming the heartbeat ARCH ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance q9db1 - Archival Error ORA-00028: your session has been killed
查看ARCn进程
[oracle@q9db01 ~]$ ps -ef|grep ora_ar oracle 20718 12870 0 22:07 pts/14 00:00:00 grep ora_ar [oracle@q9db01 ~]$ ps -ef|grep ora_ar oracle 25998 12870 0 22:07 pts/14 00:00:00 grep ora_ar
这里基本上明白了,因为客户的系统从15:15开始由于中间件程序异常,导致大量会话连接数据库,然后dba为了防止其他业务不受影响,然后开始大量通过alter system kill session,误杀了不少系统进程,包括ARCn(0,1,2,3)进程,在后面ARCn进程因为某种原因无法正常启动,导致redo无法归档,所有的redo组写满系统即hang住,该系统由于大量kill session已经导致了实例本身异常(正常情况ARCn进程kill之后会自动重启),处理方案:先增加redo组配合定时人工归档,等待业务低峰重启节点1,解决问题。温馨提醒:kill进程请小心
发表在 Oracle RAC
2 条评论
ALERT: Disable Transparent HugePages on SLES11, RHEL6, OEL6 and UEK2 Kernels
随着去ioe的潮流越来越汹涌,随着内存越来越便宜,使用Liunx大内存跑核心业务的系统越来越多。因为大内存,多会话,使得配置HugePages也成了必然的选择,在以前的文章中介绍过怎么去配置HugePages(Liunx系统中Oracle使用HugePages配置)以及因为没有配置Hugepage导致的故障(因未配置Hugepage会话数添增悲剧案例).但是随着Linux 6(RedHat 6, OEL 6, SLES 11 and UEK2 kernels等)的广泛应用,他们所在HugePages方面所具有的新特性(Transparent HugePages)很多人可能还没有注意到,Oracle强烈建议无论是rac还是单实例都关闭该特性.
检查是否启用Transparent HugePages方法
To check if the Transparent HugePages are enabled in your server execute the following: Default/Enabled setting is [always]: # cat /sys/kernel/mm/transparent_hugepage/enabled [always] never Disabled setting is [never]: # cat /sys/kernel/mm/transparent_hugepage/enabled always [never] If "enabled" is NOT set to "[never]", the Transparent HugePages are being used. You can also issue: # grep AnonHugePages /proc/meminfo If the output contains a line like "AnonHugepages: xxxx kB", with a value > 0kB the kernel is using Transparent HugePages.
禁用Transparent HugePages方法
Add the following to the kernel boot line in /etc/grub.conf (this is the preferred method) and reboot the server: transparent_hugepage=never Once modified the line will read similar to the following example: title Oracle Linux Server (2.6.32-300.25.1.el6uek.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-300.25.1.el6uek.x86_64 ro root=LABEL=/ transparent_hugepage=never initrd /initramfs-2.6.32-300.25.1.el6uek.x86_64.img OR Add the following lines in /etc/rc.local and reboot the server: if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
具体见文档:ALERT Disable Transparent HugePages on SLES11 RHEL6 OEL6 and UEK2 Kernels(Doc ID 1557478.1)
oerr ora code本质探讨
oerr ora code 命令
[oracle@q9-gg02 ~]$ oerr ora 10047 10047, 00000, "trace switching of sessions" // *Cause: // *Action:
unix/linux系统下$ORACLE_HOME/rdbms/mesg/oraus.msg包含ora-记录
[oracle@q9-gg02 ~]$ vi $ORACLE_HOME/rdbms/mesg/oraus.msg ………… 10047, 00000, "trace switching of sessions" // *Cause: // *Action: 10048, 00000, "Undo segment shrink" // *Cause: // *Action: 10049, 00000, "protect library cache memory heaps" // *Cause: // *Action: Use the OS memory protection (if available) to protect library // cache memory heaps that are pinned.
oerr ora code本质
[oracle@q9-gg02 ~]$ strace oerr ora 10047 execve("/u01/oracle/app/product/10.2/db_1/bin/oerr", ["oerr", "ora", "10047"], [/* 34 vars */]) = 0 brk(0) = 0x1c239000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b60db3fe000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b60db3ff000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/u01/oracle/app/product/10.2/db_1/lib/tls/x86_64/libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/u01/oracle/app/product/10.2/db_1/lib/tls/x86_64", 0x7fff7a3cb830) = -1 ENOENT (No such file or directory) open("/u01/oracle/app/product/10.2/db_1/lib/tls/libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/u01/oracle/app/product/10.2/db_1/lib/tls", 0x7fff7a3cb830) = -1 ENOENT (No such file or directory) open("/u01/oracle/app/product/10.2/db_1/lib/x86_64/libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/u01/oracle/app/product/10.2/db_1/lib/x86_64", 0x7fff7a3cb830) = -1 ENOENT (No such file or directory) open("/u01/oracle/app/product/10.2/db_1/lib/libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/u01/oracle/app/product/10.2/db_1/lib", {st_mode=S_IFDIR|0750, st_size=12288, ...}) = 0 open("tls/x86_64/libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("tls/libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("x86_64/libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("libtermcap.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=154281, ...}) = 0 mmap(NULL, 154281, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b60db400000 close(3) = 0 open("/lib64/libtermcap.so.2", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\20 \214>\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=15840, ...}) = 0 mmap(0x3e8c200000, 2108944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e8c200000 mprotect(0x3e8c203000, 2093056, PROT_NONE) = 0 mmap(0x3e8c402000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x3e8c402000 close(3) = 0 open("/u01/oracle/app/product/10.2/db_1/lib/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("tls/x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("tls/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("x86_64/libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("libdl.so.2", O_RDONLY) = -1 ENOENT (No such file or directory) open("/lib64/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\16`\214>\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=23360, ...}) = 0 mmap(0x3e8c600000, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e8c600000 mprotect(0x3e8c602000, 2097152, PROT_NONE) = 0 mmap(0x3e8c802000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x3e8c802000 close(3) = 0 open("/u01/oracle/app/product/10.2/db_1/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) open("tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) open("tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) open("x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) open("libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) open("/lib64/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\332\341\213>\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1722328, ...}) = 0 mmap(0x3e8be00000, 3502424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e8be00000 mprotect(0x3e8bf4f000, 2093056, PROT_NONE) = 0 mmap(0x3e8c14e000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14e000) = 0x3e8c14e000 mmap(0x3e8c153000, 16728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3e8c153000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b60db426000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b60db427000 arch_prctl(ARCH_SET_FS, 0x2b60db426fe0) = 0 mprotect(0x3e8c802000, 4096, PROT_READ) = 0 mprotect(0x3e8c14e000, 16384, PROT_READ) = 0 mprotect(0x3e8bc1c000, 4096, PROT_READ) = 0 munmap(0x2b60db400000, 154281) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 open("/dev/tty", O_RDWR|O_NONBLOCK) = 3 close(3) = 0 brk(0) = 0x1c239000 brk(0x1c25a000) = 0x1c25a000 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=56458208, ...}) = 0 mmap(NULL, 56458208, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b60db428000 close(3) = 0 getuid() = 1000 getgid() = 1000 geteuid() = 1000 getegid() = 1000 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 open("/proc/meminfo", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b60dea00000 read(3, "MemTotal: 132051308 kB\nMemFr"..., 1024) = 781 close(3) = 0 munmap(0x2b60dea00000, 4096) = 0 rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], 0}, 8) = 0 rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigaction(SIGQUIT, {0x1, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 uname({sys="Linux", node="q9-gg02.800best.com", ...}) = 0 stat("/home/oracle", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 getpid() = 25928 open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=25464, ...}) = 0 mmap(NULL, 25464, PROT_READ, MAP_SHARED, 3, 0) = 0x2b60dea00000 close(3) = 0 getppid() = 25927 getpgrp() = 25927 rt_sigaction(SIGCHLD, {0x436360, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 open("/u01/oracle/app/product/10.2/db_1/bin/oerr", O_RDONLY) = 3 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff7a3cc060) = -1 ENOTTY (Inappropriate ioctl for device) lseek(3, 0, SEEK_CUR) = 0 read(3, "#!/bin/sh\n#\n# $Id: oerr 28-aug-2"..., 80) = 80 lseek(3, 0, SEEK_SET) = 0 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=64*1024}) = 0 dup2(3, 255) = 255 close(3) = 0 fcntl(255, F_SETFD, FD_CLOEXEC) = 0 fcntl(255, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) fstat(255, {st_mode=S_IFREG|0755, st_size=2365, ...}) = 0 lseek(255, 0, SEEK_CUR) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 read(255, "#!/bin/sh\n#\n# $Id: oerr 28-aug-2"..., 2365) = 2365 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 pipe([3, 4]) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0 lseek(255, -973, SEEK_CUR) = 1392 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b60db427070) = 25929 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGCHLD, {0x436360, [], SA_RESTORER, 0x3e8be302d0}, {0x436360, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 close(4) = 0 read(3, "ora:rdbms:*:\n", 128) = 13 read(3, "", 128) = 0 --- SIGCHLD (Child exited) @ 0 (0) --- wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 25929 wait4(-1, 0x7fff7a3cb804, WNOHANG, NULL) = -1 ECHILD (No child processes) rt_sigreturn(0) = 0 close(3) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {0x436f40, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 read(255, "if [ $? -ne 0 ]\nthen\n\techo \"oerr"..., 2365) = 973 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 pipe([3, 4]) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0 lseek(255, -645, SEEK_CUR) = 1720 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b60db427070) = 25931 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGCHLD, {0x436360, [], SA_RESTORER, 0x3e8be302d0}, {0x436360, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 close(4) = 0 read(3, "Facility=ora\nComponent=rdbms\n", 128) = 29 read(3, "", 128) = 0 --- SIGCHLD (Child exited) @ 0 (0) --- wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 25931 wait4(-1, 0x7fff7a3cb904, WNOHANG, NULL) = -1 ECHILD (No child processes) rt_sigreturn(0) = 0 close(3) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {0x436f40, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 read(255, "if [ -z \"$Facility\" -o -z \"$Comp"..., 2365) = 645 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 --检查oraus.msg文件是否存在并且有权限访问 stat("/u01/oracle/app/product/10.2/db_1/rdbms/mesg/oraus.msg", {st_mode=S_IFREG|0644, st_size=3789794, ...}) = 0 geteuid() = 1000 getegid() = 1000 getuid() = 1000 getgid() = 1000 --访问oraus.msg文件 access("/u01/oracle/app/product/10.2/db_1/rdbms/mesg/oraus.msg", R_OK) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 --查找awk命令位置 stat("/u01/oracle/app/product/10.2/db_1/jre/1.4.2/bin/awk", 0x7fff7a3cbe80) = -1 ENOENT (No such file or directory) stat("/u01/oracle/app/product/10.2/db_1/bin/awk", 0x7fff7a3cbe80) = -1 ENOENT (No such file or directory) stat("/usr/lib64/qt-3.3/bin/awk", 0x7fff7a3cbe80) = -1 ENOENT (No such file or directory) stat("/usr/kerberos/bin/awk", 0x7fff7a3cbe80) = -1 ENOENT (No such file or directory) stat("/usr/local/bin/awk", 0x7fff7a3cbe80) = -1 ENOENT (No such file or directory) --找到awk命令 stat("/bin/awk", {st_mode=S_IFREG|0755, st_size=338744, ...}) = 0 access("/bin/awk", X_OK) = 0 access("/bin/awk", R_OK) = 0 stat("/bin/awk", {st_mode=S_IFREG|0755, st_size=338744, ...}) = 0 access("/bin/awk", X_OK) = 0 access("/bin/awk", R_OK) = 0 rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [INT CHLD], 8) = 0 rt_sigprocmask(SIG_SETMASK, [INT CHLD], NULL, 8) = 0 lseek(255, -8, SEEK_CUR) = 2357 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b60db427070) = 25934 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 rt_sigaction(SIGINT, {0x436f40, [], SA_RESTORER, 0x3e8be302d0}, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 --通过awk命令获取oraus.msg中相关命令 wait4(-1, 10047, 00000, "trace switching of sessions" // *Cause: // *Action: [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 25934 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 --- SIGCHLD (Child exited) @ 0 (0) --- wait4(-1, 0x7fff7a3cbb54, WNOHANG, NULL) = -1 ECHILD (No child processes) rt_sigreturn(0xffffffffffffffff) = 0 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x3e8be302d0}, {0x436f40, [], SA_RESTORER, 0x3e8be302d0}, 8) = 0 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 read(255, "\nexit 0\n", 2365) = 8 rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 exit_group(0) = ?
通过这里可以知道oerr其实就是通过awk对$ORACLE_HOME/rdbms/mesg/oraus.msg文件进行筛选,显示出来对应ORA-错误号的提示