分类目录归档：Oracle RAC

WARNING: detected duplicate paths to the same disk导致crs无法正常启动故障解决

发表于 2026 年 5 月 24 日由惜分飞

最近一周之内,两个客户由于错误修改asm_diskstring=’/dev/dm*’,’/dev/mapper/*’,导致后续集群无法启动成功,查看asm的alert日志发现类似错误(为了说明类似问题，实际中情况比现在复杂)

WARNING: detected duplicate paths to the same disk:
  '/dev/mapper/mpathi' and
  '/dev/dm-3'
ERROR: diskgroup CRS was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "CRS" cannot be mounted
ORA-15040: diskgroup is incomplete

由于asm的vote磁盘组无法mount,从而使得crs无法正常启动,进而导致集群无法正常启动和工作.而在系统中mpathi和dm-3是同一块磁盘

[root@rac1 mapper]# ls -ltr mpathi
lrwxrwxrwx 1 root root       7 May 24 13:09 mpathi -> ../dm-3

GPnP profile中关于asm_diskstring配置

[grid@rac1 trace]$  gpnptool get -o-
……
<orcl:ASM-Profile id="asm" DiscoveryString="/dev/dm*,/dev/mapper/mpath*" 
SPFile="+DATA/cimp-cluster/asmparameterfile/registry.253.889969697"/>
……

对于这样的情况,需要把profile中值修改了才行,通过重建asm spfile来实现

---pfile
large_pool_size= 12M
instance_type= 'asm'
remote_login_passwordfile= 'EXCLUSIVE'
asm_diskstring= '/dev/mapper/*'
asm_power_limit= 1
diagnostic_dest= '/opt/app/grid'

创建spfile(create spfile=’+CRS’ from pfile=’/tmp/pfile’)

NOTE: Attempting voting file refresh on diskgroup CRS
NOTE: Refresh completed on diskgroup CRS. No voting file found.
NOTE: Voting file relocation is required in diskgroup CRS
NOTE: Attempting voting file relocation on diskgroup CRS
NOTE: voting file deletion on grp 1 disk CRS_0000
NOTE: voting file deletion on grp 1 disk CRS_0001
NOTE: voting file deletion on grp 1 disk CRS_0002
NOTE: No voting file found on diskgroup CRS
Sun May 24 13:23:25 2026
NOTE: updated gpnp profile ASM SPFILE to 
NOTE: updated gpnp profile ASM diskstring: /dev/mapper/*
NOTE: updated gpnp profile ASM diskstring: /dev/mapper/*
NOTE: updated gpnp profile ASM SPFILE to +CRS/cluster/asmparameterfile/registry.253.1234099407

主要注意在create spfile的过程中voting file的信息被删除,使用kfed进行验证

[root@rac1 dbs]# kfed read /dev/asm_mpathi|grep vf
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000
[root@rac1 dbs]# kfed read /dev/asm_mpatha|grep vf
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000
[root@rac1 dbs]# kfed read /dev/asm_mpathd|grep vf
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000

需要执行replace votedisk

[root@rac1 dbs]# crsctl replace votedisk +CRS
Successful addition of voting disk 99df870869c64f78bf944b40bfff8644.
Successful addition of voting disk e6db086a74d64f49bf0aff02944bcac5.
Successful addition of voting disk e88a412fe04b4fe6bf1ca067bca779b8.
Successful deletion of voting disk 615579e778684f54bf31c0ce83709f37.
Successful deletion of voting disk 9e48b89f95084fefbfb7648897508684.
Successful deletion of voting disk 274b913b3b874f7cbfb6b025028b4eaa.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced

这样操作完成之后,再次查看GPnP profile中关于asm_diskstring配置

[grid@rac1 trace]$  gpnptool get -o-
……
<orcl:ASM-Profile id="asm" DiscoveryString="/dev/asm_*" 
SPFile="+DATA/cimp-cluster/asmparameterfile/registry.253.1234099407"/>
……

然后再次重启集群,一切恢复正常,解决了asm_diskstring=’/dev/dm*’,’/dev/mapper/*’误操作引起的WARNING: detected duplicate paths to the same disk:故障

发表在 Oracle ASM, Oracle RAC | 留下评论

aix环境rac 私网直连导致haip启动异常

发表于 2026 年 4 月 12 日由惜分飞

以前写过一篇在linux平台rac环境,心跳网络通过网线直连,当其中一台机器关机之后,另外一个节点无法检测到心跳网络是active,导致无法启动的情况:私网直连后遗症:一节点无法启动导致另外节点haip无法启动
昨天晚上在aix环境中遇到类似情况,由于某种原因,需要关闭rac的一个节点,另外一个节点启动crs的过程中,haip始终无法启动,虽然haip起不来,但是过了一会儿,asm服务启动成功,磁盘组mount,数据库正常open(这个和linux环境有一定的区别,linux 下面11.2.0.4的rac,如果haip无法启动,默认情况启动asm服务),业务临时恢复

bash-4.2$ crsctl status res -t -init
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       db2                      Started             
ora.cluster_interconnect.haip
      1        ONLINE  OFFLINE                                                   
ora.crf
      1        ONLINE  ONLINE       db2                                          
ora.crsd
      1        ONLINE  ONLINE       db2                                          
ora.cssd
      1        ONLINE  ONLINE       db2                                          
ora.cssdmonitor
      1        ONLINE  ONLINE       db2                                          
ora.ctssd
      1        ONLINE  ONLINE       db2                      OBSERVER            
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.drivers.acfs
      1        ONLINE  ONLINE       db2                                          
ora.evmd
      1        ONLINE  ONLINE       db2                                          
ora.gipcd
      1        ONLINE  ONLINE       db2                                          
ora.gpnpd
      1        ONLINE  ONLINE       db2                                          
ora.mdnsd
      1        ONLINE  ONLINE       db2

分析haip对应的日志如下

[ USRTHRD][7257]{0:0:221} Starting Probe for ip 169.254.57.103
[ USRTHRD][7257]{0:0:221} Transitioning to Probe State
[ USRTHRD][7257]{0:0:221}  Arp::sProbe { 
[ USRTHRD][7257]{0:0:221} Arp::sSend:  sending type 1
[ USRTHRD][7257]{0:0:221} [NetHAWork] thread hit OSD exception failed to send arp
[ USRTHRD][7257]{0:0:221} (null) category: -2, operation: write, loc: arpsend:1,os, OS error: 69, other: 
[ USRTHRD][7257]{0:0:221} [NetHAWork] thread stopping
[ USRTHRD][7257]{0:0:221} Thread:[NetHAWork]isRunning is reset to false here
[ USRTHRD][5201]{0:0:221} use all detected INF
[ USRTHRD][5201]{0:0:221} Thread:[NetHAWork]thread constructor
[ USRTHRD][5201]{0:0:221} HAIP:  Moving ip '' from inf 'en6' to inf 'en6'
[ USRTHRD][5201]{0:0:221} pausing thread
[ USRTHRD][5201]{0:0:221} posting thread
[ USRTHRD][5201]{0:0:221} Waiting for HAIP work thread to cleanup ARP
[ USRTHRD][5201]{0:0:221} timeout to wait thread to cleanup ARP
[ USRTHRD][5201]{0:0:221} Thread:[NetHAWork]start {
[ USRTHRD][5201]{0:0:221} Thread:[NetHAWork]start }
[ USRTHRD][7514]{0:0:221} [NetHAWork] thread started
[ USRTHRD][7514]{0:0:221}  Arp::sCreateSocket { 
[ USRTHRD][7514]{0:0:221}  Arp::sCreateSocket } 
[ USRTHRD][5201]{0:0:221} use all detected INF
[ USRTHRD][7514]{0:0:221} Failed to check 169.254.57.103 on en6
[ USRTHRD][7514]{0:0:221} (null) category: 0, operation: , loc: , OS error: 0, other:

这里初步看是把169.254.57.103这个ip增加到en6的网卡上,但是由于OS error: 69失败了.通过aix工程师分析,这个错误可能是物理网络不通导致,对网卡状态进行分析

bash-4.2# entstat -d ent6
-------------------------------------------------------------
ETHERNET STATISTICS (ent6) :
Device Type: 2-Port Gigabit Ethernet-SX PCI-Express Adapter (14103f03)
Hardware Address: 40:f2:e9:91:eb:7a
Elapsed Time: 0 days 1 hours 38 minutes 14 seconds

Transmit Statistics:                          Receive Statistics:
--------------------                          -------------------
Packets: 4128                                 Packets: 5077
Bytes: 35215659                               Bytes: 370511
Interrupts: 0                                 Interrupts: 4815
Transmit Errors: 0                            Receive Errors: 0
Packets Dropped: 0                            Packets Dropped: 0
                                              Bad Packets: 0
Max Packets on S/W Transmit Queue: 1         
S/W Transmit Queue Overflow: 0
Current S/W+H/W Transmit Queue Length: 0

Broadcast Packets: 12                         Broadcast Packets: 0
Multicast Packets: 62                         Multicast Packets: 66
No Carrier Sense: 0                           CRC Errors: 0
DMA Underrun: 0                               DMA Overrun: 0
Lost CTS Errors: 0                            Alignment Errors: 0
Max Collision Errors: 0                       No Resource Errors: 0
Late Collision Errors: 0                      Receive Collision Errors: 0
Deferred: 0                                   Packet Too Short Errors: 0
SQE Test: 0                                   Packet Too Long Errors: 0
Timeout Errors: 0                             Packets Discarded by Adapter: 0
Single Collision Count: 0                     Receiver Start Count: 0
Multiple Collision Count: 0
Current HW Transmit Queue Length: 0

General Statistics:
-------------------
No mbuf Errors: 0
Adapter Reset Count: 0
Adapter Data Rate: 2000
Driver Flags: Up Broadcast Simplex 
        Limbo 64BitSupport ChecksumOffload 
        LargeSend DataRateSet 

2-Port Gigabit Ethernet-SX PCI-Express Adapter (14103f03) Specific Statistics:
------------------------------------------------------------------------------
Link Status : Down      <======表示网络链路状态异常(一般就是直连导致，如果通过交换机不会这样)
Media Speed Selected: Auto negotiation
Media Speed Running: Unknown
PCI Mode: PCI-Express X4
    Relaxed Ordering: Enabled
    TLP Size: 256
    MRR Size: 4096
Jumbo Frames: Disabled
TCP Segmentation Offload: Enabled
TCP Segmentation Offload Packets Transmitted: 3625
TCP Segmentation Offload Packet Errors: 0
Transmit and Receive Flow Control Status: Enabled
XON Flow Control Packets Transmitted: 0
XON Flow Control Packets Received: 0
XOFF Flow Control Packets Transmitted: 0
XOFF Flow Control Packets Received: 0
Transmit and Receive Flow Control Threshold (High): 40960
Transmit and Receive Flow Control Threshold (Low): 20480
Transmit and Receive Storage Allocation (TX/RX): 4/44

通过解决掉异常问题,把故障主机启动之后,启动该机器之后,网络链路状态恢复正常,启动haip成功,但是由于该集群在haip异常的时候启动成功,心跳网络使用是直接的私网ip(没有使用haip),因此还是要对集群进行一次重启恢复到正常状态.

发表在 AIX, Oracle RAC | 评论关闭

CSSD signal 11 in thread clssnmRcfgMgrThread故障处理

发表于 2025 年 2 月 21 日由惜分飞

一个客户,集群无法启动,只能启动到如下状态

查看cssd日志有CSSD signal 11 in thread clssnmRcfgMgrThread报错

2025-02-21 18:21:25.500: [    CSSD][2788693760]clssnmDoSyncUpdate: node(2) is transitioning from joining state to active state
2025-02-21 18:21:25.500: [    CSSD][2788693760]clssnmDoSyncUpdate: Wait for 0 vote ack(s)
2025-02-21 18:21:25.500: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:25.700: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:25.901: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:25.995: [    CSSD][2801538816]clssnmvDiskPing: Writing with status 0x2, timestamp 1740133285/5870104
2025-02-21 18:21:25.997: [    CSSD][2799818496]clssnmvDiskKillCheck: not evicted, file /dev/dm-4 flags 0x00000000,
                                                                          kill block unique 0, my unique 1740133265
2025-02-21 18:21:26.000: [    CSSD][2793424640]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 2 waited 500
2025-02-21 18:21:26.101: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.302: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.497: [    CSSD][2801538816]clssnmvDiskPing: Writing with status 0x2, timestamp 1740133286/5870604
2025-02-21 18:21:26.502: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.702: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.902: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.997: [    CSSD][2799818496]clssnmvDiskKillCheck: not evicted, file /dev/dm-4 flags 0x00000000,
                                                                      kill block unique 0, my unique 1740133265
2025-02-21 18:21:26.997: [    CSSD][2801538816]clssnmvDiskPing: Writing with status 0x2, timestamp 1740133286/5871114
2025-02-21 18:21:27.000: [    CSSD][2793424640]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 2 waited 0
2025-02-21 18:21:27.102: [    CSSD][2788693760]clssnmCheckDskInfo: Checking disk info...
2025-02-21 18:21:27.102: [    CSSD][2788693760]clssnmCheckDskInfo: diskTimeout set to (200000)ms
2025-02-21 18:21:27.103: [    CSSD][2788693760]###################################
2025-02-21 18:21:27.103: [    CSSD][2788693760]clssscExit: CSSD signal 11 in thread clssnmRcfgMgrThread
2025-02-21 18:21:27.103: [    CSSD][2788693760]###################################
2025-02-21 18:21:27.103: [    CSSD][2788693760](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2025-02-21 18:21:27.103: [    CSSD][2788693760]

----- Call Stack Trace -----
2025-02-21 18:21:27.103: [    CSSD][2788693760]calling              call     entry                argument values in hex      
2025-02-21 18:21:27.103: [    CSSD][2788693760]location             type     point                (? means dubious value)     
2025-02-21 18:21:27.103: [    CSSD][2788693760]-------------------- -------- -------------------- ----------------------------
2025-02-21 18:21:27.109: [    CSSD][2788693760]clssscExit()+745     call     kgdsdst()            000000000 ? 000000000 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]                                                   7F9EA637A650 ? 7F9EA637A728 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]                                                   7F9EA637F1D0 ? 000000003 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]s0clsssc_sighandler  call     clssscExit()         001FB9FA0 ? 000000002 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]()+616                                             7F9EA637A650 ? 7F9EA637A728 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]                                                   7F9EA637F1D0 ? 000000003 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]__sighandler()       call     s0clsssc_sighandler  00000000B ? 000000002 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                              ()                   7F9EA637A650 ? 7F9EA637A728 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA637F1D0 ? 000000003 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmCheckSplit()+  signal   __sighandler()       001BEE8A8 ? 000000000 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]378                                                002039A80 ? 000000001 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   0004D2B40 ? 7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmCheckDskInfo(  call     clssnmCheckSplit()   001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760])+387                                              000030D40 ? 000000001 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   0004D2B40 ? 7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmDoSyncUpdate(  call     clssnmCheckDskInfo(  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760])+4692                        )                    000000001 ? 000000001 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   0004D2B40 ? 7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmLocalJoinEven  call     clssnmDoSyncUpdate(  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]t()+3992                      )                    FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmRcfgMgrThread  call     clssnmLocalJoinEven  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]()+2290                       t()                  FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssscthrdmain()+25  call     clssnmRcfgMgrThread  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]3                             ()                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]start_thread()+209   call     clssscthrdmain()     001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]clone()+109          call     start_thread()       7F9EA6381700 ? 001DC83F0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]0000000000000000     call     clone()              7F9EA6381700 ? 001DC83F0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760] 
2025-02-21 18:21:27.111: [    CSSD][2788693760]--------------------- Binary Stack Dump ---------------------

这里提示表决盘超时,尝试启动nocrs貌似,在表决盘存在的情况下,启动依旧失败,通过处理让启动过程不读表决盘,启动nocrs模式成功,并mount其他业务磁盘组

确认其他磁盘没有问题,重建crs磁盘组

SQL> create diskgroup OCR  external redundancy disk '/dev/dm-4' force  attribute 'COMPATIBLE.ASM' = '11.2.0';
# ocrconfig -restore /u01/app/11.2.0.3/grid/cdata/scan/backup00.ocr
# crsctl replace votedisk +OCR
SQL> create spfile from pfile='/tmp/pfile.asm';

然后重启crs恢复正常

发表在 Oracle RAC | 标签为 clssnmRcfgMgrThread | 评论关闭

17813235971

QQ 咨询

分类目录归档：Oracle RAC

WARNING: detected duplicate paths to the same disk导致crs无法正常启动故障解决

aix环境rac 私网直连导致haip启动异常

CSSD signal 11 in thread clssnmRcfgMgrThread故障处理

文章分类

最近发表

分类目录归档：Oracle RAC

WARNING: detected duplicate paths to the same disk导致crs无法正常启动故障解决

aix环境rac 私网直连导致haip启动异常

CSSD signal 11 in thread clssnmRcfgMgrThread故障处理

标签云

文章分类

最近发表