分类目录归档：Oracle

安装19c 数据库 libclntsh.so: file format not recognized

发表于 2021 年 4 月 20 日由惜分飞

今天在qq群里面,讨论一个19c(19.3)在linux系统(CentOS 7.9)安装的问题,感觉有点意思,给整理出来供参考
安装日志报错

INFO: 
/usr/bin/ld:/home/oracle/app/oracle/product/19.0.0/dbhome_1/lib//libclntsh.so: 
      file format not recognized; treating as linker script
/usr/bin/ld:/home/oracle/app/oracle/product/19.0.0/dbhome_1/lib//libclntsh.so:1: syntax error

INFO: 
make[2]: *** [dlopenlib] Error 1

INFO: 
make[2]: Leaving directory `/home/oracle/app/oracle/product/19.0.0/dbhome_1/rdbms/lib'

INFO: 
make[1]: Leaving directory `/home/oracle/app/oracle/product/19.0.0/dbhome_1/rdbms/lib'

INFO: 
make[1]: *** [/home/oracle/app/oracle/product/19.0.0/dbhome_1/lib/libasmperl19.so] Error 2

INFO: 
make: *** [libasmperl19.ohso] Error 2

拿这个错误在mos上检索,发现匹配文章:19c Installation Fails with error “libclntsh.so: file format not recognized; treating as linker script” (Doc ID 2631283.1),结合文章分析,第一反应怀疑安装介质有问题,让网友进行sha256sum 验证

通过验证确认安装介质没有问题,和官方公布的一致.进一步检查 ls -l $ORACLE_HOME/libcln* 文件

这个问题比较明显,根据官方的描述,这些文件应该是:

这里比较明显对应的ln丢失了.从提供的以前unzip解压日志看

解压有这些文件,但是ln丢失,qq群朋友对unzip进行安装

然后重新解压oracle安装程序,验证libcln,一切正常,安装也成功

只可惜以前的unzip直接被安装新的unzip覆盖了,未做保留现场进行分析,为什么解压后ln丢失

发表在 Oracle安装升级 | 标签为 libclntsh.so | 评论关闭

raid强制上线后数据库无法启动故障处理

发表于 2021 年 4 月 20 日由惜分飞

由于raid掉盘过多,强制raid上线,然后启动数据库报以下错误

Mon Apr 19 23:19:28 2021
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 106750 KB redo, 9080 data blocks need recovery
Mon Apr 19 23:19:45 2021
Slave exiting with ORA-1115 exception
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_p000_3277.trc:
ORA-01115: IO error reading block from file 9 (block # 339)
ORA-01110: data file 9: '/u01/app/oracle/oradata/orcl/dev02.dbf'
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I/O error
Additional information: 4
Additional information: 326
Additional information: 24576
ORA-27072: File I
Mon Apr 19 23:19:45 2021
Aborting crash recovery due to slave death, attempting serial crash recovery
Beginning crash recovery of 1 threads
Started redo scan
Completed redo scan
 read 106750 KB redo, 9080 data blocks need recovery
Aborting crash recovery due to error 1115
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_3275.trc:
ORA-01115: IO error reading block from file 9 (block # 329)
ORA-01110: data file 9: '/u01/app/oracle/oradata/orcl/dev02.dbf'
ORA-1115 signalled during: ALTER DATABASE OPEN...

错误提示比较明显IO error,结合客户强行上线raid的操作,比较明显是由于底层io问题导致该错误,直接对此文件dbv检查

[oracle@database orcl]$ dbv file=dev02.dbf

DBVERIFY: Release 11.2.0.4.0 - Production on Mon Apr 19 23:59:03 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /u01/app/oracle/oradata/orcl/dev02.dbf

DBV-00600: Fatal Error - [28] [27061] [0] [0]

对于此类情况,通过工具进行处理

DUL> copy file from  /u01/app/oracle/oradata/orcl/dev02.dbf to /oradata/dev02.dbf
 
starting copy datafile '/u01/app/oracle/oradata/orcl/dev02.dbf' to '/oradata/dev02.dbf'
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf'.error message:Input/output error
read block# error: 303
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf'.error message:Input/output error
read block# error: 304
read data error from file '/u01/app/oracle/oradata/orcl/dev02.dbf'.error message:Input/output error
read block# error: 329
datafile copy completed with 3 block error.

dbv校验文件

[oracle@database oradata]$ dbv file=dev02.dbf 

DBVERIFY: Release 11.2.0.4.0 - Production on Tue Apr 20 00:28:31 2021

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

DBVERIFY - Verification starting : FILE = /oradata/dev02.dbf
Page 303 is marked corrupt
Corrupt block relative dba: 0x0240012f (file 9, block 303)
Completely zero block found during dbv: 

Page 304 is marked corrupt
Corrupt block relative dba: 0x02400130 (file 9, block 304)
Completely zero block found during dbv: 

Page 329 is marked corrupt
Corrupt block relative dba: 0x02400149 (file 9, block 329)
Completely zero block found during dbv: 



DBVERIFY - Verification complete

Total Pages Examined         : 3932160
Total Pages Processed (Data) : 3213723
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 714294
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 4139
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 1
Total Pages Marked Corrupt   : 3
Total Pages Influx           : 0
Total Pages Encrypted        : 0
Highest block SCN            : 85078875 (6.85078875)

通过对io error的文件进行处理,最终损坏三个block,最大限度抢救数据.使用被恢复出来的文件,尝试open库遭遇以下错误

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [6], [85035771], [6],
[85084136], [12583040], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [6], [85035770], [6],
[85084136], [12583040], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [6], [85035764], [6],
[85084136], [12583040], [], [], [], [], [], []
Process ID: 6733
Session ID: 570 Serial number: 3

ora-600 2662这个错误比较明显,处理文件头scn,继续open库

SQL> alter database open ;
alter database open 
*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 6840
Session ID: 570 Serial number: 3

查看alert日志信息

Tue Apr 20 01:22:27 2021
alter database open upgrade
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Started redo scan
Completed redo scan
 read 1 KB redo, 3 data blocks need recovery
Started redo application at
 Thread 1: logseq 1, block 3
Recovery of Online Redo Log: Thread 1 Group 1 Seq 1 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/orcl/redo01.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 1, block 5, scn 25854859541
 3 data blocks read, 3 data blocks written, 1 redo k-bytes read
Tue Apr 20 01:22:28 2021
Thread 1 advanced to log sequence 2 (thread open)
Thread 1 opened at log sequence 2
  Current log# 2 seq# 2 mem# 0: /u01/app/oracle/oradata/orcl/redo02.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue Apr 20 01:22:28 2021
SMON: enabling cache recovery
[6840] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:5902014 end:5905574 diff:3560 (35 seconds)
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc  (incident=63970):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_63970/orcl_smon_6824_i63970.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (24, 2) on object 89023.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Tue Apr 20 01:22:38 2021
ORACLE Instance orcl (pid = 14) - Error 600 encountered while recovering transaction (63, 3) on object 89023.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc:
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_smon_6824.trc  (incident=63974):
ORA-00600: internal error code, arguments: [6006], [1], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_63974/orcl_smon_6824_i63974.trc
Tue Apr 20 01:22:55 2021
PMON (ospid: 6798): terminating the instance due to error 474

这个错误是比较常见的错误,参考:ORACLE Instance XFF (pid = 18) – Error 600 encountered while recovering transaction ，通过处理之后,数据库open成功

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 1603411968 bytes
Fixed Size                  2253664 bytes
Variable Size            1023413408 bytes
Database Buffers          570425344 bytes
Redo Buffers                7319552 bytes
Database mounted.
SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

后续安排逻辑导出,导入新库

KGL-heap-size-exceeded报错

发表于 2021 年 4 月 19 日由惜分飞

alert日志报错

2021-04-19T15:06:08.208159+08:00
Memory Notification: Library Cache Object loaded into SGA
Heap size 512000K exceeds notification threshold (51200K)
Details in trace file /mnt/app/diag/rdbms/xff/xff/trace/xff_ora_12300.trc
2021-04-19T15:06:08.208212+08:00
KGL object name :select DISTINCT(BEGIN_TIME) beginTime,
    MILE mile,
    END_TIME endTime,
    LAST_TIME lastTime,
    BEGIN_TIME beginTime2,
    END_TIME endTime2,
    TEMPERATURE1 temperature1,
    TEMPERATURE2 temperature2,
    TEMPERATURE3 temperature3,
    TEMPERATURE4 temperature4,
    LATITUDE latitude,
    LONGITUDE longitude
    from T_XIFENFEI
    where VEHICLE_ID = :1 
    and END_TIME is not null
    and BEGIN_TIME BETWEEN TO_DATE(:2 , 'yyyy-MM-dd hh24:mi:ss') and TO_DATE(:3 , 'yyyy-MM-dd hh24
Errors in file /mnt/app/diag/rdbms/xff/xff/trace/xff_ora_12300.trc  (incident=162089):
ORA-00600: internal error code, arguments: [KGL-heap-size-exceeded], 
[0x1C4782708], [0], [524288008], [], [], [], [], [], [], [], []
Incident details in: /mnt/app/diag/rdbms/xff/xff/incident/incdir_162089/xff_ora_12300_i162089.trc
Use ADRCI or Support Workbench to package the incident.

数据库版本信息

Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Build label:    RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME:    /mnt/app/product/12.2.0/dbhome_1
System name:    Linux
Node name:      izbp1399kdj96ra6bjtwyxz
Release:        3.10.0-693.2.2.el7.x86_64
Version:        #1 SMP Tue Sep 12 22:26:13 UTC 2017
Machine:        x86_64

Call Stack信息

----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+119         call     kgdsdst()            7FFD6CB9A258 000000002
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 000000082 ?
dbkedDefDump()+1200  call     ksedst()             000000000 000000002 ?
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
ksedmp()+259         call     dbkedDefDump()       000000003 000000002
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
dbgexPhaseII()+2130  call     ksedmp()             0000003EB 000000002 ?
                                                   7FFD6CB7BCC0 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
dbgexExplicitEndInc  call     dbgexPhaseII()       7F4853DB46C0 7F4853DB77A0
()+602                                             7FFD6CB9E388 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
dbgeEndDDEInvocatio  call     dbgexExplicitEndInc  7F4853DB46C0 7F4853DB77A0
nImpl()+658                   ()                   7FFD6CB9E388 ? 7FFD6CB7BDD8 ?
                                                   000000000 ? 000000082 ?
kglLargeHeapWarning  call     dbgeEndDDEInvocatio  7F4853DB46C0 7F4853DB77A0
()+1544                       nImpl()              7F4853DF49A0 7FFD6CB7BDD8 ?
                                                   7F4853DF49A0 000000082 ?
kglHeapAllocCbk()+2  call     kglLargeHeapWarning  7F4853DF49A0 ? 1C4782708 ?
27                            ()                   000000000 01F400060 ?
                                                   0B6DADD88 000000082 ?
kghalp()+1086        call     kglHeapAllocCbk()    7F4853DF49A0 ? 1C4782708 ?
                                                   2EE171B70 01F400060 ?
                                                   000000058 ? 000000082 ?
kkocsRegBindEqvCtxC  call     kghalp()             7F4853DF49A0 ? 0B6DADD88 ?
ommon()+145                                        2EE171B70 ? 000000001 ?
                                                   000000058 ? 0119E7860 ?
kkocsAddBeElem()+81  call     kkocsRegBindEqvCtxC  7F4853DF49A0 ? 7FFD6CB9F460 ?
1                             ommon()              2EE171B70 ? 000000001 ?
                                                   000000002 ? 0119E7860 ?
kkocsProcBSensPredC  call     kkocsAddBeElem()     7F484E2EF998 ? 7F484E2EFAC8 ?
B()+431                                            0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
qksopVisitPreds()+5  call     kkocsProcBSensPredC  7F484E2EF998 ? 7FFD6CB9F640 ?
39                            B()                  0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkocsBuildBESelCtx(  call     qksopVisitPreds()    7F484E2EF998 ? 7FFD6CB9F640 ?
)+153                                              0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkocsBindSensCheck(  call     kkocsBuildBESelCtx(  7F484E2EF998 ? 7FFD6CB9F640 ?
)+585                         )                    0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkoipt()+901         call     kkocsBindSensCheck(  7F484E2EFA20 ? 7FFD6CB9F640 ?
                              )                    0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 0119E7860 ?
kkoqbc()+4677        call     kkoipt()             7F484E31F1E0 ? 7FFD6CB9F640 ?
                                                   0BBED4798 ? 209DFEA60 ?
                                                   000000002 ? 000000000
apakkoqb()+183       call     kkoqbc()             7F484E31F1E0 ? 7F4853CC2958 ?
                                                   0BBED4798 ? 209DFEA60 ?
                                                   000000001 000000000 ?
apaqbdDescendents()  call     apakkoqb()           7FFD6CBA1870 ? 000000EB5 ?
+503                                               0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
apaqbd()+136         call     apaqbdDescendents()  7FFD6CBA1870 ? 7F4853CC2958 ?
                                                   0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
apadrv()+1228        call     apaqbd()             7FFD6CBA1870 ? 7F4853CC2958 ?
                                                   0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
opitca()+2139        call     apadrv()             0BBED4C18 ? 7F4853CC2958 ?
                                                   0BBED4C18 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
kksFullTypeCheck()+  call     opitca()             7F484E2F2A08 0BBED4C18
86                                                 7FFD6CBA42F0 209DFEA60 ?
                                                   000000001 ? 000000000 ?
rpiswu2()+627        call     kksFullTypeCheck()   7FFD6CBA2FF8 ? 0BBED4C18 ?
                                                   7FFD6CBA42F0 ? 209DFEA60 ?
                                                   000000001 ? 000000000 ?
kksLoadChild()+8003  call     rpiswu2()            7FFD6CBA2FF8 ? 0BBED4C18 ?
                                                   7FFD6CBA42F0 ? 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 00000006A ?
kxsGetRuntimeLock()  call     kksLoadChild()       7F4853DF49A0 0BBED4C18 ?
+2029                                              7FFD6CBA42F0 ? 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 00000006A ?
kksfbc()+15091       call     kxsGetRuntimeLock()  7F4853DF49A0 7F484E2F2A08
                                                   7FFD6CBA4270 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708
opiexe()+2932        call     kksfbc()             7F484E2F2A08 7F484E2F2A08 ?
                                                   7FFD6CBA4270 ? 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708 ?
kpoal8()+2679        call     opiexe()             000000049 7F484E2F2A08 ?
                                                   7FFD6CBA58F0 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708 ?
opiodr()+1229        call     kpoal8()             00000005E 000000026
                                                   7FFD6CBA9300 209DFEA60 ?
                                                   7FFD6CBA24F8 ? 1C4782708 ?
ttcpip()+1257        call     opiodr()             00000005E 000000026
                                                   7FFD6CBA9300 000000000
                                                   7FFD6CBA24F8 ? 1C4782708 ?
opitsk()+1940        call     ttcpip()             7F4853E11750 ? 000000026 ?
                                                   7FFD6CBA9300 00000017D ?
                                                   7FFD6CBA8D50 7FFD6CBA955C
opiino()+941         call     opitsk()             000000000 000000000
                                                   7FFD6CBA9300 ? 00000017D ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
opiodr()+1229        call     opiino()             00000003C 000000004
                                                   7FFD6CBAAA18 00000017D ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
opidrv()+1021        call     opiodr()             00000003C 000000004
                                                   7FFD6CBAAA18 000000000
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
sou2o()+145          call     opidrv()             00000003C 000000004
                                                   7FFD6CBAAA18 000000000 ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
opimai_real()+455    call     sou2o()              7FFD6CBAA9F0 00000003C
                                                   000000004 7FFD6CBAAA18
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
ssthrdmain()+417     call     opimai_real()        000000000 7FFD6CBAAD00
                                                   000000004 ? 7FFD6CBAAA18 ?
                                                   7FFD6CBA8D50 ? 7FFD6CBA955C ?
main()+262           call     ssthrdmain()         000000000 000000002
                                                   7FFD6CBAAD00 000000001
                                                   000000000 7FFD6CBA955C ?
__libc_start_main()  call     main()               000000000 7FFD6CBAAF38
+245                                               7FFD6CBAAD00 ? 000000001 ?
                                                   000000000 ? 7FFD6CBA955C ?
_start()+41          call     __libc_start_main()  000D04D40 000000002
                                                   7FFD6CBAAF38 7F484F241555 ?
                                                   000000000 ? 7FFD6CBA955C ?
--------------------- Binary Stack Dump ---------------------

根据官方描述,该问题是由于Bug 20917487 – CORRUPT KKOCS STRUCTURES AFTER PARTIAL PURGE OF CURSOR HEAP引起
解决方案：
1. 升级数据库到19.1+版本
2. 应用 Patch 20917487
3. Disable adaptive cursor sharing

alter system set "_optimizer_adaptive_cursor_sharing" = false;
alter system set "_optimizer_extended_cursor_sharing_rel" = "none";

4. 通过隐含参数设置一个足够大的值

alter system set "_kgl_large_heap_warning_threshold"=83886080 scope=spfile; --一个足够大的值
--or
alter system set "_kgl_large_heap_warning_threshold"=0 scope=spfile;
alter system set "_kgl_large_heap_assert_threshold"=0 scope=spfile;

shutdown immediate
startup

参考文档：
ORA-600 [KGL-heap-size-exceeded] Errors in the Alert Log (Doc ID 2628072.1)
Memory Notification: Library Cache Object loaded into SGA / ORA-600 [KGL-heap-size-exceeded] (Doc ID 330239.1)

发表在 ORA-xxxxx | 标签为 KGL-heap-size-exceeded | 评论关闭

17813235971

QQ 咨询

分类目录归档：Oracle

安装19c 数据库 libclntsh.so: file format not recognized

raid强制上线后数据库无法启动故障处理

KGL-heap-size-exceeded报错

文章分类

最近发表

分类目录归档：Oracle

安装19c 数据库 libclntsh.so: file format not recognized

raid强制上线后数据库无法启动故障处理

KGL-heap-size-exceeded报错

标签云

文章分类

最近发表