标签归档：ORA-59048

在一套Oracle 19c RAC环境中有ocr/vote所在磁盘组有磁盘offline

ASMCMD> lsdg  
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304  24576000  19047940                0        19047940              0             N  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304   2500608   1043156                0         1043156              0             N  FRA/
MOUNTED  NORMAL  Y         512             512   4096  4194304    102400    101576            51200           25188              1             Y  SYSDG/
ASMCMD> lsdsk -G sysdg
Path

/dev/asm_ocr1
/dev/asm_ocr2

ASMCMD>

在sqlplus中查询具体信息

[grid@dbserver1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Nov 6 19:31:39 2026
Version 19.16.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.16.0.0.0

SQL> col path for a40
SQL> set lines 150
SQL> select disk_number,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,CREATE_DATE,MOUNT_DATE,
   2 name,path from v$asm_disk where group_number=3;

DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    CREATE_DA MOUNT_DAT NAME                  PATH
----------- ------- ------------ ------- -------- --------- --------- --------------------- -------------
          2 MISSING UNKNOWN      OFFLINE FORCING  07-SEP-22 22-OCT-26 _DROPPED_0002_SYSDG
          1 CACHED  MEMBER       ONLINE  NORMAL   07-SEP-22 22-OCT-26 SYSDG_0001           /dev/asm_ocr2
          0 CACHED  MEMBER       ONLINE  NORMAL   07-SEP-22 22-OCT-26 SYSDG_0000           /dev/asm_ocr1

这里显示被丢失的磁盘状态处于missing,而且state=FORCING，磁盘名字变为_DROPPED_0002_SYSDG,这里表现出来的是磁盘应该是不可访问,然后asm 自动踢掉该盘,但是没有正常清除元数据.查询磁盘reblance状态

SQL> SELECT GROUP_NUMBER, OPERATION, STATE, POWER, ERROR_CODE
  2      FROM GV$ASM_OPERATION
  3      WHERE GROUP_NUMBER = 3;

GROUP_NUMBER OPERA STAT      POWER ERROR_CODE
------------ ----- ---- ---------- --------------------------------------------
           3 REBAL WAIT          1
           3 REBAL ERRS          1 ORA-59048
           3 REBAL WAIT          1
           3 REBAL WAIT          1
           3 REBAL WAIT          1
           3 REBAL ERRS          1 ORA-59048
           3 REBAL WAIT          1
           3 REBAL WAIT          1

该磁盘组在reblance的过程中由于遭遇了ORA-59048导致无法清除掉异常磁盘元数据.关于ORA-59048的官方解释

SQL> !oerr ora 59048
59048, 0000, "Redundancy could not be restored in group '%0!s'."
// *Cause:  An attempt to restore redundancy during rebalance failed for some
//          user data or ASM metadata due to the presence of dropping or
//          forcing disks.
// *Action: Add back all the disks in DROPPING or FORCING state. If this is an
//          Oracle appliance, also add back all the missing disks.

出现该问题的原因是由于
In 12.1.0.2 ASM and higher releases, it is not allowed to Normal Drop the Failgroup in a Normal/High Redundancy Diskgroup if it lowers the Failgroup count below 3 for Normal Redundancy Diskgroup and below 5 for High Redundancy Diskgroup.Any attempt to force drop Failgroup will not expel the disks completely and leave the disks in “FORCING” state.
解决该问题官方的建议:

---

查看asm的alert日志确认该磁盘组当时具体状态,最早确实是由于/dev/asm_ocr3这个盘访问报io错误

2026-10-25T10:20:06.228405+08:00
NOTE: SMON did instance recovery for group FRA domain 2
NOTE: SMON detected lock domain 3 invalid at system inc 4 10/25/26 10:20:05
WARNING: Read Failed. group:3 disk:2 AU:0 offset:0 size:4096
path:/dev/asm_ocr3
	 incarnation:0xf0f3546f asynchronous result:'I/O error' ioreason:2818 why:11
	 subsys:System krq:0x7ffff294e3d8 bufp:0x7ffff2910000 osderr1:0x69b5 osderr2:0x0
	 IO elapsed time: 0 usec Time waited on I/O: 0 usec
NOTE: SMON starting instance recovery of group SYSDG domain 3 inc 4 (mounted) at 10/25/26 10:20:05
NOTE: 10/25/26 10:20:05 SYSDG.F1X0 found on disk 0 au 10 fcn 0.0 datfmt 1
NOTE: 10/25/26 10:20:05 SYSDG.F1X0 found on disk 1 au 10 fcn 0.0 datfmt 1
NOTE: SMON will attempt offline of disk 2 - no header
NOTE: cache initiating offline of disk 2 group SYSDG
NOTE: checking PST: grp = 3
2026-10-25T10:20:06.235215+08:00
GMON checking disk modes for group 3 at 20 for pid 23, osid 67867
2026-10-25T10:20:06.236213+08:00
NOTE: checking PST for grp 3 done.
NOTE: initiating PST update: grp 3 (SYSDG), dsk = 2/0xf0f3546f, mask = 0x6a, op = clear mandatory
2026-10-25T10:20:06.236627+08:00
GMON updating disk modes for group 3 at 21 for pid 23, osid 67867
cluster guid (11de9e990b017f7ebf235441d97d8d20) generated for PST Hbeat for instance 1
WARNING: Write Failed. group:3 disk:2 AU:1 offset:4190208 size:4096
path:/dev/asm_ocr3
	 incarnation:0xf0f3546f synchronous result:'I/O error' ioreason:13825 why:54
	 subsys:System krq:0x7ffff2606068 bufp:0x7ffff7e28000 osderr1:0x69c0 osderr2:0x0
	 IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: found another non-responsive disk 2.4042478703 (SYSDG_0002) that will be offlined 

然后asm 自动踢掉该磁盘,并尝试进行reblance

2026-10-25T10:20:22.547474+08:00
NOTE: cache closing disk 2 of grp 3: SYSDG_0002
2026-10-25T10:20:22.555942+08:00
NOTE: cache closing disk 2 of grp 3: (not open) SYSDG_0002
2026-10-25T10:20:22.558591+08:00
NOTE: PST update grp = 3 completed successfully 
2026-10-25T10:20:24.398958+08:00
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_24660.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
2026-10-25T10:20:24.489753+08:00
NOTE: Attempting voting file refresh on diskgroup SYSDG
NOTE: Refresh completed on diskgroup SYSDG. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup SYSDG
NOTE: Attempting voting file relocation on diskgroup SYSDG
NOTE: Successful voting file relocation on diskgroup SYSDG
2026-10-25T10:20:24.970204+08:00
WARNING: Disk 2 (SYSDG_0002) in group 3 will be dropped in: (0) secs on ASM inst 1
NOTE: The rdbms compatibility of group SYSDG is 10.1.0.0.0
WARNING: PST-initiated drop of 1 disk(s) in group 3(.2297668793))
2026-10-25T10:20:24.977298+08:00
SQL> alter diskgroup SYSDG drop disk SYSDG_0002 force /* ASM SERVER */ 
2026-10-25T10:20:24.989168+08:00
NOTE: cache closing disk 2 of grp 3: (not open) SYSDG_0002
NOTE: GroupBlock outside rolling migration privileged region
2026-10-25T10:20:25.020909+08:00
NOTE: client +ASM1:+ASM:dbsvr-cluster no longer has group 1 (DATA) mounted
2026-10-25T10:20:25.028211+08:00
NOTE: requesting all-instance membership refresh for group=3
2026-10-25T10:20:25.033193+08:00
GMON updating for reconfiguration, group 3 at 32 for pid 47, osid 37477
2026-10-25T10:20:25.033508+08:00
NOTE: cache closing disk 2 of grp 3: (not open) SYSDG_0002
2026-10-25T10:20:25.036948+08:00
NOTE: group 3 PST updated.
2026-10-25T10:20:25.037055+08:00
NOTE: cache closing disk 2 of grp 3: (not open) SYSDG_0002
2026-10-25T10:20:25.138731+08:00
NOTE: membership refresh pending for group 3/0x88f3a4b9 (SYSDG)
NOTE: Attempting voting file refresh on diskgroup SYSDG
NOTE: Refresh completed on diskgroup SYSDG. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup SYSDG
NOTE: Attempting voting file relocation on diskgroup SYSDG
NOTE: Successful voting file relocation on diskgroup SYSDG
2026-10-25T10:20:25.243884+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-10-25T10:20:25.244375+08:00
GMON querying group 3 at 33 for pid 26, osid 67882
2026-10-25T10:20:25.244593+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-10-25T10:20:25.264523+08:00
SUCCESS: refreshed membership for 3/0x88f3a4b9 (SYSDG)
2026-10-25T10:20:25.265056+08:00
SUCCESS: alter diskgroup SYSDG drop disk SYSDG_0002 force /* ASM SERVER */
2026-10-25T10:20:25.267949+08:00
NOTE: starting rebalance of group 3/0x88f3a4b9 (SYSDG) at power 1
NOTE: starting process ARBA
Starting background process ARBA
2026-10-25T10:20:25.320353+08:00
ARBA started with pid=34, OS id=38626 
NOTE: starting process ARB0
Starting background process ARB0
2026-10-25T10:20:25.348943+08:00
ARB0 started with pid=39, OS id=38628 
NOTE: assigning ARBA to group 3/0x88f3a4b9 (SYSDG) to compute estimates
NOTE: assigning ARB0 to group 3/0x88f3a4b9 (SYSDG) with 1 parallel I/O
NOTE: starting priority rebalance
2026-10-25T10:20:25.741091+08:00
NOTE: 10/25/26 10:20:24 SYSDG.F1X0 copy 3 relocating from 2:10 to 65534:4294967294 at FCN 0.437476
NOTE: header on disk 0 advanced to format #2 using fcn 0.0
NOTE: F1B1 fcn on disk 0 synced at fcn 0.437476
NOTE: header on disk 1 advanced to format #2 using fcn 0.0
NOTE: F1B1 fcn on disk 1 synced at fcn 0.437476
2026-10-25T10:20:27.284097+08:00
NOTE: starting priority rebalance
NOTE: starting priority rebalance
NOTE: stopping process ARB0
NOTE: stopping process ARBA
NOTE: Attempting voting file refresh on diskgroup SYSDG
NOTE: Refresh completed on diskgroup SYSDG. Found 2 voting file(s).
NOTE: Starting expel slave for group 3/0x88f3a4b9 (SYSDG)
2026-10-25T10:20:27.638396+08:00
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=3
2026-10-25T10:20:27.682346+08:00
GMON updating for reconfiguration, group 3 at 34 for pid 34, osid 38635
2026-10-25T10:20:27.682672+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-10-25T10:20:27.686280+08:00
NOTE: group 3 PST updated.
2026-10-25T10:20:27.686317+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
SUCCESS: grp 3 disk _DROPPED_0002_SYSDG emptied
NOTE: grp 3 disk 2 not expelled because ASM metadata redundancy has 
      not been restored due to an insufficientnumber of failure groups

比较简单的而且安全的做法是客户划分一块新的ocr盘上来,然后udev配置并加入到该磁盘组中即可.但是由于该客户存储没有新空间,只能分析尝试把原盘该加入进来.先确认该盘现在读写是否正常(存储故障已经修复)
1. kfed read 或者dd读取该盘确认正常
2. 由于该盘依旧属于sysdg磁盘组的member,无法直接加入

SQL> alter diskgroup sysdg add disk '/dev/asm_ocr3' ;
alter diskgroup sysdg add disk '/dev/asm_ocr3'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15033: disk 'DATA01' is already part of diskgroup


SQL> ALTER DISKGROUP SYSDG ADD DISK '/dev/asm_ocr3' 
2026-11-06T19:35:15.741021+08:00
NOTE: GroupBlock outside rolling migration privileged region
2026-11-06T19:35:15.909199+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
NOTE: Assigning number (3,3) to disk (/dev/asm_ocr3)
NOTE: requesting all-instance membership refresh for group=3
2026-11-06T19:35:15.998569+08:00
NOTE: membership refresh pending for group 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:16.018117+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:16.034067+08:00
GMON querying group 3 at 43 for pid 26, osid 67882
2026-11-06T19:35:16.035115+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
GMON querying group 3 at 44 for pid 26, osid 67882
2026-11-06T19:35:16.035967+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:16.097552+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
GMON querying group 3 at 45 for pid 26, osid 67882
2026-11-06T19:35:16.098168+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:16.119434+08:00
SUCCESS: refreshed membership for 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:16.263264+08:00
ORA-15032: not all alterations performed
ORA-15033: disk '/dev/asm_ocr3' belongs to diskgroup "SYSDG"

2026-11-06T19:35:16.263630+08:00
ERROR: ALTER DISKGROUP SYSDG ADD DISK '/dev/asm_ocr3'

3. 直接使用force强制加入,也可以先dd掉磁盘头然后加入

SQL> ALTER DISKGROUP SYSDG ADD FAILGROUP SYSDG_0002 DISK '/dev/asm_ocr3' FORCE REBALANCE POWER 1;

Diskgroup altered.

SQL> SELECT GROUP_NUMBER, OPERATION, STATE, POWER, ERROR_CODE
  2      FROM GV$ASM_OPERATION
  3      WHERE GROUP_NUMBER = 3;

GROUP_NUMBER OPERA STAT      POWER ERROR_CODE
------------ ----- ---- ---------- --------------------------------------------
           3 REBAL REAP          1
           3 REBAL DONE          1
           3 REBAL DONE          1
           3 REBAL DONE          1
           3 REBAL WAIT          1
           3 REBAL WAIT          1
           3 REBAL WAIT          1
           3 REBAL WAIT          1

8 rows selected.

SQL> col path for a40
SQL> set lines 150
SQL> select disk_number,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,CREATE_DATE,MOUNT_DATE,
   2 name,path from v$asm_disk where group_number=3;

DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    CREATE_DA MOUNT_DAT NAME         PATH
----------- ------- ------------ ------- -------- --------- --------- ------------ --------------
          1 CACHED  MEMBER       ONLINE  NORMAL   07-SEP-22 22-OCT-26 SYSDG_0001   /dev/asm_ocr2
          3 CACHED  MEMBER       ONLINE  NORMAL   06-NOV-26 06-NOV-26 SYSDG_0003   /dev/asm_ocr3
          0 CACHED  MEMBER       ONLINE  NORMAL   07-SEP-22 22-OCT-26 SYSDG_0000   /dev/asm_ocr1

磁盘成功加入,并且reblance完成,而且_DROPPED_0002_SYSDG磁盘记录也完全清除

[grid@dbserver1 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB   Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304  24576000  19047940                0        19047940              0             N  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304   2500608   1043156                0         1043156              0             N  FRA/
MOUNTED  NORMAL  N         512             512   4096  4194304    153600    152524            51200           50662              0             Y  SYSDG/
ASMCMD> lsdsk -G sysdg
Path
/dev/asm_ocr1
/dev/asm_ocr2
/dev/asm_ocr3
ASMCMD> 

整个加入磁盘asm alert日志记录

2026-11-06T19:35:31.368323+08:00
SQL> ALTER DISKGROUP SYSDG ADD FAILGROUP SYSDG_0002 DISK '/dev/asm_ocr3' FORCE REBALANCE POWER 1
2026-11-06T19:35:31.369612+08:00
NOTE: GroupBlock outside rolling migration privileged region
2026-11-06T19:35:31.480932+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
NOTE: Assigning number (3,3) to disk (/dev/asm_ocr3)
NOTE: requesting all-instance membership refresh for group=3
NOTE: Disk 3 in group 3 is assigned fgnum=3
NOTE: discarding redo for group 3 disk 3
NOTE: initializing header (replicated) on grp 3 disk SYSDG_0003
NOTE: initializing header on grp 3 disk SYSDG_0003
NOTE: requesting all-instance disk validation for group=3
2026-11-06T19:35:31.544284+08:00
NOTE: skipping rediscovery for group 3/0x88f3a4b9 (SYSDG) on local instance.
2026-11-06T19:35:31.685931+08:00
NOTE: running client discovery for group 3 (reqid:9868412322975013714)
NOTE: requesting all-instance disk validation for group=3
2026-11-06T19:35:31.690689+08:00
NOTE: skipping rediscovery for group 3/0x88f3a4b9 (SYSDG) on local instance.
2026-11-06T19:35:31.798128+08:00
NOTE: running client discovery for group 3 (reqid:9868412322975027894)
NOTE: Allocated 1 DD reserve extents for group 3 (SYSDG)
NOTE: Allocated 2 VAT reserve extents for group 3 (SYSDG).
NOTE: Adding disk 3 (SYSDG_0003) to grp 3 (SYSDG) (12800 AUs)
2026-11-06T19:35:32.800462+08:00
GMON updating for reconfiguration, group 3 at 47 for pid 34, osid 32035
2026-11-06T19:35:32.800767+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
NOTE: group SYSDG: updated PST location: disks 0000 0001 0003
2026-11-06T19:35:32.806670+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:32.806867+08:00
NOTE: group 3 PST updated.
2026-11-06T19:35:32.817316+08:00
NOTE: membership refresh pending for group 3/0x88f3a4b9 (SYSDG)
NOTE: Attempting voting file refresh on diskgroup SYSDG
NOTE: Refresh completed on diskgroup SYSDG. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup SYSDG
2026-11-06T19:35:32.870762+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:32.871172+08:00
GMON querying group 3 at 48 for pid 26, osid 67882
2026-11-06T19:35:32.871938+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
NOTE: cache opening disk 3 of grp 3: SYSDG_0003 path:/dev/asm_ocr3
2026-11-06T19:35:33.135859+08:00
NOTE: Attempting voting file refresh on diskgroup SYSDG
NOTE: Refresh completed on diskgroup SYSDG. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup SYSDG
NOTE: Attempting voting file relocation on diskgroup SYSDG
NOTE: Failed voting file relocation on diskgroup SYSDG
2026-11-06T19:35:33.202862+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:33.203321+08:00
GMON querying group 3 at 49 for pid 26, osid 67882
2026-11-06T19:35:33.203546+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:33.248381+08:00
SUCCESS: refreshed membership for 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:33.249044+08:00
SUCCESS: ALTER DISKGROUP SYSDG ADD FAILGROUP SYSDG_0002 DISK '/dev/asm_ocr3' FORCE REBALANCE POWER 1
2026-11-06T19:35:33.249626+08:00
NOTE: starting rebalance of group 3/0x88f3a4b9 (SYSDG) at power 1
NOTE: starting process ARBA
Starting background process ARBA
2026-11-06T19:35:33.300642+08:00
ARBA started with pid=39, OS id=32653
2026-11-06T19:35:33.300642+08:00
ARBA started with pid=39, OS id=32653
NOTE: starting process ARB0
Starting background process ARB0
2026-11-06T19:35:33.330349+08:00
ARB0 started with pid=40, OS id=32655
NOTE: assigning ARBA to group 3/0x88f3a4b9 (SYSDG) to compute estimates
NOTE: assigning ARB0 to group 3/0x88f3a4b9 (SYSDG) with 1 parallel I/O
NOTE: starting priority rebalance
2026-11-06T19:35:33.862973+08:00
NOTE: group 3, file 1, extent 0: successful reread of 256 block(s)
NOTE: group 3, file 1, extent 0: successful reread of 256 block(s)
NOTE: group 3, file 1, extent 0: successful reread of 256 block(s)
NOTE: group 3, file 1, extent 0: successful reread of 256 block(s)
NOTE: header on disk 3 advanced to format #2 using fcn 0.0
NOTE: F1X0 on disk 3 (fmt 2) relocated at fcn 0.442279: AU 0 -> AU 10
NOTE: 11/06/26 19:35:33 SYSDG.F1X0 copy 3 relocating from 65534:4294967294 to 3:10 at FCN 0.442279
NOTE: F1B1 fcn on disk 0 synced at fcn 0.442279
NOTE: F1B1 fcn on disk 1 synced at fcn 0.442279
…………
NOTE: group 3, file 121, extent 7: successful reread of 256 block(s)
NOTE: group 3, file 121, extent 7: successful reread of 256 block(s)
NOTE: group 3, file 255, extent 2147483648: successful reread of 256 block(s)
NOTE: group 3, file 255, extent 2147483648: successful reread of 256 block(s)
NOTE: group 3, file 255, extent 2147483648: successful reread of 256 block(s)
NOTE: group 3, file 255, extent 2147483648: successful reread of 256 block(s)
2026-11-06T19:35:43.100750+08:00
NOTE: starting priority rebalance
NOTE: Starting expel slave for group 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:43.270964+08:00
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=3
2026-11-06T19:35:43.373350+08:00
GMON updating for reconfiguration, group 3 at 50 for pid 45, osid 32714
2026-11-06T19:35:43.373665+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:43.375552+08:00
NOTE: group 3 PST updated.
SUCCESS: grp 3 disk _DROPPED_0002_SYSDG emptied
NOTE: Expelling disk 2 (_DROPPED_0002_SYSDG) from grp 3 (SYSDG)
2026-11-06T19:35:43.414788+08:00
GMON updating for reconfiguration, group 3 at 51 for pid 45, osid 32714
2026-11-06T19:35:43.415757+08:00
NOTE: cache closing disk 2 of grp 3: (not open) _DROPPED_0002_SYSDG
2026-11-06T19:35:43.420906+08:00
NOTE: group 3 PST updated.
NOTE: grp 3 disk 2 expelled from the PST.
2026-11-06T19:35:43.528766+08:00
NOTE: membership refresh pending for group 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:43.537768+08:00
GMON querying group 3 at 52 for pid 26, osid 67882
GMON querying group 3 at 53 for pid 26, osid 67882
2026-11-06T19:35:43.559379+08:00
NOTE: Disk _DROPPED_0002_SYSDG in mode 0x0 marked for de-assignment
SUCCESS: refreshed membership for 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:44.127190+08:00
NOTE: starting priority rebalance
NOTE: stopping process ARB0
NOTE: stopping process ARBA
2026-11-06T19:35:44.568463+08:00
NOTE: Attempting voting file refresh on diskgroup SYSDG
NOTE: Refresh completed on diskgroup SYSDG. Found 3 voting file(s).
NOTE: Starting expel slave for group 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:44.593489+08:00
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=3
2026-11-06T19:35:44.607062+08:00
NOTE: membership refresh pending for group 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:44.610642+08:00
GMON querying group 3 at 54 for pid 26, osid 67882
2026-11-06T19:35:44.654503+08:00
SUCCESS: refreshed membership for 3/0x88f3a4b9 (SYSDG)
2026-11-06T19:35:46.278541+08:00
SUCCESS: rebalance completed for group 3/0x88f3a4b9 (SYSDG)
NOTE: Attempting voting file refresh on diskgroup SYSDG
NOTE: Refresh completed on diskgroup SYSDG. Found 3 voting file(s).