标签云
asm 恢复 asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (100)
- 数据库 (1,598)
- DB2 (22)
- MySQL (70)
- Oracle (1,463)
- Data Guard (49)
- EXADATA (7)
- GoldenGate (21)
- ORA-xxxxx (158)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (13)
- ORACLE 21C (3)
- Oracle ASM (65)
- Oracle Bug (7)
- Oracle RAC (47)
- Oracle 安全 (6)
- Oracle 开发 (27)
- Oracle 监听 (27)
- Oracle备份恢复 (530)
- Oracle安装升级 (84)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (75)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (36)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (19)
-
最近发表
- PostgreSQL解析wal日志之—walminer
- Oracle 19c/21c最新patch信息-202404
- PostgreSQL恢复系列:pg_filedump批量处理
- PostgreSQL部分主要字典信息
- PostgreSQL恢复系列:pg_filedump恢复字典构造
- PostgreSQL 16 源码安装
- ORA-00742 ORA-00312 恢复
- 数据库open成功后报ORA-00353 ORA-00354错误引起的一系列问题(本质ntfs文件系统异常)
- ORA-600 ktsiseginfo1故障
- ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
- 最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
- ORA-600 2662快速恢复之Patch scn工具
- TNS-12518: TNS:listener could not hand off client connection
- ora.storage无法启动报ORA-12514故障处理
- 断电引起文件scn异常数据库恢复
- ORA-16188: LOG_ARCHIVE_CONFIG settings inconsistent with previously started instance
- .[hudsonL@cock.li].mkp勒索加密数据库完美恢复
- 模拟带库实现rman远程备份
- 又一例:ORA-600 kclchkblk_4和2662故障
- Oracle误删除数据文件恢复
分类目录归档:Oracle RAC
网卡异常导致数据库实例启动异常
一套集群,一个节点启动正常,另外一个节点无法正常启动实例,启动异常节点alert日志
Tue Mar 07 19:07:29 2023 IPC Send timeout detected. Receiver ospid 6386 [ Tue Mar 07 19:07:29 2023 Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_lms0_6386.trc: IPC Send timeout detected. Receiver ospid 6402 [ Tue Mar 07 19:07:29 2023 Errors in file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_lms4_6402.trc: Tue Mar 07 19:07:29 2023 Received an instance abort message from instance 1 Please check instance 1 alert and LMON trace files for detail. System state dump requested by (instance=2, osid=6384 (LMD0)), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/xff/xff2/trace/xff2_diag_6374_20230307190729.trc LMD0 (ospid: 6384): terminating the instance due to error 481 Dumping diagnostic data in directory=[cdmp_20230307190729], requested by (instance=2, osid=6384 (LMD0)), summary=[abnormal instance termination]. Instance terminated by LMD0, pid = 6384
正常节点alert日志
Tue Mar 07 19:02:07 2023 Reconfiguration started (old inc 20, new inc 22) List of instances: 1 2 (myinst: 1) Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Mar 07 19:02:08 2023 LMS 5: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Tue Mar 07 19:02:08 2023 LMS 2: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Tue Mar 07 19:02:08 2023 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Tue Mar 07 19:02:08 2023 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Tue Mar 07 19:02:08 2023 LMS 3: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Tue Mar 07 19:02:08 2023 LMS 7: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Tue Mar 07 19:02:08 2023 Tue Mar 07 19:02:08 2023 LMS 4: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 6: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Submitted all GCS remote-cache requests Fix write in gcs resources Tue Mar 07 19:02:27 2023 IPC Send timeout detected. Sender: ospid 6936 [oracle@xffnode1.localdomain (PING)] Receiver: inst 2 binc 441249706 ospid 59731 Tue Mar 07 19:07:29 2023 IPC Send timeout detected. Sender: ospid 6946 [oracle@xffnode1.localdomain (LMS0)] Receiver: inst 2 binc 429479852 ospid 6386 Tue Mar 07 19:07:29 2023 IPC Send timeout detected. Sender: ospid 6962 [oracle@xffnode1.localdomain (LMS4)] Receiver: inst 2 binc 429479854 ospid 6402 Tue Mar 07 19:07:29 2023 IPC Send timeout detected. Sender: ospid 6966 [oracle@xffnode1.localdomain (LMS5)]
通过上述日志,可以确认主要由于两个节点之间无法正常通讯,从而使得新节点无法加入到集群(无法完成集群重组),从而使得实例启动异常.一般出现这类情况最检查的就是私网异常,通过分析oswnetstat记录发现packet reassembles failed特别严重
一般出现该问题,考虑是由于ipfrag_*_thresh默认值不足导致,通过设置
net.ipv4.ipfrag_high_thresh = 16777216 net.ipv4.ipfrag_low_thresh = 15728640
packet reassembles failed依旧在增加,通过分析网卡情况发现网卡异常,采用haip(双万兆网卡)的其中一块网卡异常
为了数据库性能不收太大影响,临时禁用异常网卡,重启库正常
后续等网络层面解决之后再启用该网卡
发表在 Oracle RAC
评论关闭
11.2 crs启动超时dd npohasd 处理
客户由于光纤链路故障导致表决盘异常从而使得主机重启,主机重启之后,集群没有正常启动
操作系统和crs版本
[root@rac1 ~]# cat /etc/redhat-release CentOS release 6.9 (Final) [root@rac1 ~]# sqlplus -v SQL*Plus: Release 11.2.0.4.0 Production
人工启动crs hang住一段时间然后报错
[root@rac1 ~]# crsctl start crs CRS-4640: Oracle High Availability Services is already active CRS-4000: Command Start failed, or completed with errors.
查看启动进程
[grid@rac1 ~]$ ps -ef|grep d.bin root 7043 1 0 11:48 ? 00:00:00 /u01/app/grid/product/11.2.0/bin/ohasd.bin reboot root 8311 1 0 11:53 ? 00:00:00 /u01/app/grid/product/11.2.0/bin/ohasd.bin reboot grid 10984 10954 0 12:10 pts/2 00:00:00 grep d.bin
根据经验这个故障很可能就是BUG:17229230 – DURING REBOOT, “OHASD.BIN REBOOT” REMAINS SLEEPING,临时解决方案,一个会话启动crs,然后在另外一个会话发起
/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1
后续crs启动正常
[root@rac1 ~]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@rac1 ~]# crsctl status res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm 1 ONLINE OFFLINE Instance Shutdown ora.cluster_interconnect.haip 1 ONLINE OFFLINE ora.crf 1 ONLINE ONLINE rac1 ora.crsd 1 ONLINE OFFLINE ora.cssd 1 ONLINE OFFLINE STARTING ora.cssdmonitor 1 ONLINE ONLINE rac1 ora.ctssd 1 ONLINE OFFLINE ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE OFFLINE ora.gipcd 1 ONLINE ONLINE rac1 ora.gpnpd 1 ONLINE ONLINE rac1 ora.mdnsd 1 ONLINE ONLINE rac1
终止dd命令,集群启动正常
ocr磁盘组掉盘故障处理
由于某种故障导致crs的OCR_0001盘掉线,votedisk从3个变为了2个
WARNING: Write Failed. group:3 disk:1 AU:1 offset:4190208 size:4096 WARNING: Hbeat write to PST disk 1.3915948466 in group 3 failed. [4] Mon Jun 14 15:31:11 2021 NOTE: process _b000_+asm1 (21889) initiating offline of disk 1.3915948466 (OCR_0001) with mask 0x7e in group 3 NOTE: checking PST: grp = 3 GMON checking disk modes for group 3 at 14 for pid 28, osid 21889 NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: checking PST for grp 3 done. NOTE: sending set offline flag message 1047812201 to 1 disk(s) in group 3 WARNING: Disk OCR_0001 in mode 0x7f is now being offlined INFO: Instance #2 could not find disk 1 in group 3 NOTE: initiating PST update: grp = 3, dsk = 1/0xe968a1b2, mask = 0x6a, op = clear GMON updating disk modes for group 3 at 15 for pid 28, osid 21889 NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: PST update grp = 3 completed successfully NOTE: initiating PST update: grp = 3, dsk = 1/0xe968a1b2, mask = 0x7e, op = clear GMON updating disk modes for group 3 at 16 for pid 28, osid 21889 NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: cache closing disk 1 of grp 3: OCR_0001 NOTE: PST update grp = 3 completed successfully Mon Jun 14 15:31:13 2021 NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 3 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Successful voting file relocation on diskgroup OCR NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 2 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Successful voting file relocation on diskgroup OCR Mon Jun 14 15:34:08 2021 WARNING: PST-initiated drop of 1 disk(s) in group 3(.1918390620)) SQL> alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */ NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=3 Mon Jun 14 15:34:10 2021 GMON updating for reconfiguration, group 3 at 17 for pid 28, osid 21889 NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: cache closing disk 1 of grp 3: (not open) OCR_0001 NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: group 3 PST updated. Mon Jun 14 15:34:10 2021 NOTE: membership refresh pending for group 3/0x7258515c (OCR) NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 2 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Successful voting file relocation on diskgroup OCR GMON querying group 3 at 18 for pid 18, osid 8900 NOTE: group OCR: updated PST location: disk 0000 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 1) NOTE: cache closing disk 1 of grp 3: (not open) _DROPPED_0001_OCR SUCCESS: refreshed membership for 3/0x7258515c (OCR) SUCCESS: alter diskgroup OCR drop disk OCR_0001 force /* ASM SERVER */
在第一次掉盘之后rebalance完成之后,又掉一块盘,ocr磁盘组正常,表决盘因为就只有一个磁盘,无法在ocr磁盘组中refresh到其他磁盘上
Tue Jun 15 04:41:42 2021 WARNING: Waited 15 secs for write IO to PST disk 0 in group 3. WARNING: Waited 15 secs for write IO to PST disk 0 in group 3. Tue Jun 15 04:41:42 2021 NOTE: process _b000_+asm1 (58548) initiating offline of disk 0.3915948465 (OCR_0000) with mask 0x7e in group 3 NOTE: checking PST: grp = 3 GMON checking disk modes for group 3 at 23 for pid 28, osid 58548 NOTE: group OCR: updated PST location: disk 0002 (PST copy 0) NOTE: checking PST for grp 3 done. NOTE: sending set offline flag message 3615961191 to 1 disk(s) in group 3 WARNING: Disk OCR_0000 in mode 0x7f is now being offlined INFO: Instance #2 could not find disk 1 in group 3 NOTE: initiating PST update: grp = 3, dsk = 0/0xe968a1b1, mask = 0x6a, op = clear GMON updating disk modes for group 3 at 24 for pid 28, osid 58548 NOTE: group OCR: updated PST location: disk 0002 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 0) NOTE: PST update grp = 3 completed successfully NOTE: initiating PST update: grp = 3, dsk = 0/0xe968a1b1, mask = 0x7e, op = clear GMON updating disk modes for group 3 at 25 for pid 28, osid 58548 NOTE: group OCR: updated PST location: disk 0002 (PST copy 0) NOTE: group OCR: updated PST location: disk 0002 (PST copy 0) NOTE: cache closing disk 0 of grp 3: OCR_0000 NOTE: PST update grp = 3 completed successfully Tue Jun 15 04:41:44 2021 NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 2 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR WARNING: Waited 18 secs for write IO to PST disk 0 in group 3. WARNING: Waited 18 secs for write IO to PST disk 0 in group 3. NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR Tue Jun 15 04:44:21 2021 NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR Tue Jun 15 04:44:21 2021 WARNING: PST-initiated drop of 1 disk(s) in group 3(.1918390620)) SQL> alter diskgroup OCR drop disk OCR_0000 force /* ASM SERVER */ NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=3 NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR Tue Jun 15 04:44:24 2021 GMON updating for reconfiguration, group 3 at 26 for pid 28, osid 58548 NOTE: cache closing disk 0 of grp 3: (not open) OCR_0000 NOTE: group OCR: updated PST location: disk 0002 (PST copy 0) NOTE: group 3 PST updated. NOTE: membership refresh pending for group 3/0x7258515c (OCR) NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR GMON querying group 3 at 27 for pid 18, osid 8900 NOTE: cache closing disk 0 of grp 3: (not open) _DROPPED_0000_OCR SUCCESS: refreshed membership for 3/0x7258515c (OCR) NOTE: starting rebalance of group 3/0x7258515c (OCR) at power 1 SUCCESS: alter diskgroup OCR drop disk OCR_0000 force /* ASM SERVER */
可以明显的看到,ocr磁盘组只剩余1个disk,查询表决盘信息
node1-> crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 3619aee7c3b04fc1bfa5c4ce659acbf7 (/dev/emcpowerc) [OCR] 2. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR] Located 2 voting disk(s).
可以发现表决盘中的两个disk一个属于ocr磁盘组,一个是被ocr磁盘组drop掉的磁盘,尝试增加以前离线的磁盘到ocr磁盘组
SQL> alter diskgroup OCR add disk '/dev/emcpowerc'; alter diskgroup OCR add disk '/dev/emcpowerc' * ERROR at line 1: ORA-15032: not all alterations performed ORA-15033: disk '/dev/emcpowerc' belongs to diskgroup "OCR" SQL> alter diskgroup OCR add disk '/dev/emcpowerc' force 2 ; alter diskgroup OCR add disk '/dev/emcpowerc' force * ERROR at line 1: ORA-03113: end-of-file on communication channel Process ID: 15191 Session ID: 1613 Serial number: 7
查看alert日志
SQL> alter diskgroup OCR add disk '/dev/emcpowerc' force NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (3,4) to disk (/dev/emcpowerc) NOTE: requesting all-instance membership refresh for group=3 WARNING: ignoring disk /dev/emcpowerd in deep discovery NOTE: initializing header on grp 3 disk OCR_0004 WARNING: ignoring disk /dev/emcpowerd in deep discovery NOTE: requesting all-instance disk validation for group=3 NOTE: skipping rediscovery for group 3/0x725d2390 (OCR) on local instance. NOTE: requesting all-instance disk validation for group=3 NOTE: skipping rediscovery for group 3/0x725d2390 (OCR) on local instance. NOTE: Attempting voting file relocation on diskgroup OCR Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_12207.trc (incident=311185): ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_311185/+ASM1_rbal_12207_i311185.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. ERROR: ORA-600 thrown in RBAL for group number 3 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_12207.trc: ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], [] Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_12207.trc: ORA-00600: internal error code, arguments: [kfdvfGetCurrent_baddsk], [], [], [], [], [], [], [], [], [], [], [] RBAL (ospid: 12207): terminating the instance due to error 488
由于ORA-600 kfdvfGetCurrent_baddsk错误导致增加磁盘失败,通过上面查询的votedisk的信息,可以发现emcpowerc这个盘虽然ocr中离线,但是依旧还是votedisk盘,因此无法增加到该磁盘组中,采用变通方法,先加另外一块盘
SQL> alter diskgroup OCR add failgroup OCR_0001 disk '/dev/emcpowerd' force; Diskgroup altered. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options node1-> crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR] 2. ONLINE 0eef8152df5d4f41bf973ad5dc5a6cb1 (/dev/emcpowerd) [OCR] Located 2 voting disk(s).
增加成功emcpowerd之后,emcpowerc已经不再是表决盘,变为了emcpowerd,再次增加emcpowerc
SQL> alter diskgroup OCR add failgroup OCR_0000 disk '/dev/emcpowerc' force; Diskgroup altered. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options node1-> crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 00bc3e79f7404ff2bf60925a7b8a5a6d (/dev/emcpowere) [OCR] 2. ONLINE 0eef8152df5d4f41bf973ad5dc5a6cb1 (/dev/emcpowerd) [OCR] 3. ONLINE 4f6201f808dc4ff3bf928b14eae0d4a6 (/dev/emcpowerc) [OCR] Located 3 voting disk(s). ASMCMD> lsdsk -G ocr Path /dev/emcpowerc /dev/emcpowerd /dev/emcpowere
SQL> alter diskgroup OCR add failgroup OCR_0000 disk '/dev/emcpowerc' force NOTE: GroupBlock outside rolling migration privileged region NOTE: Assigning number (3,0) to disk (/dev/emcpowerc) NOTE: requesting all-instance membership refresh for group=3 NOTE: initializing header on grp 3 disk OCR_0000 NOTE: requesting all-instance disk validation for group=3 Mon Jan 24 17:47:42 2022 NOTE: skipping rediscovery for group 3/0x725dccb9 (OCR) on local instance. NOTE: requesting all-instance disk validation for group=3 NOTE: skipping rediscovery for group 3/0x725dccb9 (OCR) on local instance. Mon Jan 24 17:47:48 2022 GMON updating for reconfiguration, group 3 at 20 for pid 30, osid 16978 NOTE: group 3 PST updated. NOTE: initiating PST update: grp = 3 GMON updating group 3 at 21 for pid 30, osid 16978 NOTE: group OCR: updated PST location: disk 0002 (PST copy 0) NOTE: group OCR: updated PST location: disk 0005 (PST copy 1) NOTE: group OCR: updated PST location: disk 0000 (PST copy 2) NOTE: PST update grp = 3 completed successfully NOTE: membership refresh pending for group 3/0x725dccb9 (OCR) NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 2 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR GMON querying group 3 at 22 for pid 18, osid 15952 NOTE: cache opening disk 0 of grp 3: OCR_0000 path:/dev/emcpowerc Mon Jan 24 17:47:53 2022 NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 2 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: Failed voting file relocation on diskgroup OCR GMON querying group 3 at 23 for pid 18, osid 15952 SUCCESS: refreshed membership for 3/0x725dccb9 (OCR) Mon Jan 24 17:47:53 2022 SUCCESS: alter diskgroup OCR add failgroup OCR_0000 disk '/dev/emcpowerc' force NOTE: starting rebalance of group 3/0x725dccb9 (OCR) at power 1 Starting background process ARB0 Mon Jan 24 17:47:53 2022 ARB0 started with pid=31, OS id=17092 NOTE: assigning ARB0 to group 3/0x725dccb9 (OCR) with 1 parallel I/O cellip.ora not found. NOTE: F1X0 copy 3 relocating from 65534:4294967294 to 0:2 for diskgroup 3 (OCR) NOTE: stopping process ARB0 SUCCESS: rebalance completed for group 3/0x725dccb9 (OCR) NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 2 voting file(s). NOTE: Voting file relocation is required in diskgroup OCR NOTE: Attempting voting file relocation on diskgroup OCR NOTE: voting file allocation on grp 3 disk OCR_0000 NOTE: Successful voting file relocation on diskgroup OCR Mon Jan 24 17:47:57 2022 NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=3 NOTE: membership refresh pending for group 3/0x725dccb9 (OCR) Mon Jan 24 17:48:03 2022 GMON querying group 3 at 24 for pid 18, osid 15952 SUCCESS: refreshed membership for 3/0x725dccb9 (OCR) Mon Jan 24 17:48:06 2022 NOTE: Attempting voting file refresh on diskgroup OCR NOTE: Refresh completed on diskgroup OCR . Found 3 voting file(s).
表决磁盘组从2个变为了3个,ocr磁盘组也恢复了正常的3个,至此OCR掉盘的故障处理完成