分类目录归档：Linux

又一例TRIM导致asm磁盘数据丢失的故障

发表于 2026 年 4 月 12 日由惜分飞

以前遇到过一个case,存储直连虚拟机,对磁盘误操作之后触发trim,导致数据被清空:ssd trim导致fdisk格式化磁盘之后无法恢复,最近再次遇到类似案例:客户错误对一块asm disk磁盘进行了格式化

该磁盘是由6块磁盘组成了磁盘组

被格式化之后data磁盘组直接dismount

Tue Apr 07 18:22:31 2026
WARNING: cache read  a corrupt block: group=2(DATA) fn=261 indblk=0 disk=0 (DATA_0000) incarn=3958745085 au=605 blk=0 count=1
Errors in file /home/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_639087.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [261] [2147483648] [0 != 1]
NOTE: a corrupted block from group DATA was dumped to /home/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_639087.trc
WARNING: cache read (retry) a corrupt block: group=2(DATA) fn=261 indblk=0 disk=0 (DATA_0000) incarn=3958745085 au=605 blk=0 count=1
Errors in file /home/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_639087.trc:
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [261] [2147483648] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [261] [2147483648] [0 != 1]
ERROR: cache failed to read group=2(DATA) fn=261 indblk=0 from disk(s): 0(DATA_0000)
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [261] [2147483648] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [261] [2147483648] [0 != 1]
NOTE: cache initiating offline of disk 0 group DATA
NOTE: process _user639087_+asm1 (639087) initiating offline of disk 0.3958745085 (DATA_0000) with mask 0x7e in group 2
NOTE: initiating PST update: grp = 2, dsk = 0/0xebf5a7fd, mask = 0x6a, op = clear
Tue Apr 07 18:22:31 2026
GMON updating disk modes for group 2 at 10 for pid 28, osid 639087
ERROR: Disk 0 cannot be offlined, since diskgroup has external redundancy.
ERROR: too many offline disks in PST (grp 2)
Tue Apr 07 18:22:31 2026
NOTE: cache dismounting (not clean) group 2/0xE9E5571F (DATA) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 115720, image: oracle@ajjorcl1 (B000)
Tue Apr 07 18:22:31 2026
NOTE: halting all I/Os to diskgroup 2 (DATA)
WARNING: Offline for disk DATA_0000 in mode 0x7f failed.
Tue Apr 07 18:22:31 2026
NOTE: LGWR doing non-clean dismount of group 2 (DATA)
NOTE: LGWR sync ABA=15.1625 last written ABA 15.1625
Errors in file /home/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_639087.trc  (incident=309345):
ORA-15335: ASM metadata corruption detected in disk group 'DATA'
ORA-15130: diskgroup "DATA" is being dismounted
ORA-15066: offlining disk "DATA_0000" in group "DATA" may result in a data loss
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [261] [2147483648] [0 != 1]
ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [261] [2147483648] [0 != 1]
Incident details in: /home/app/grid/diag/asm/+asm/+ASM1/incident/incdir_309345/+ASM1_ora_639087_i309345.trc
Tue Apr 07 18:22:31 2026
List of instances:
 1
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 30)
 Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE 
 26 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Tue Apr 07 18:22:31 2026
freeing rdom 2
Tue Apr 07 18:22:31 2026
WARNING: dirty detached from domain 2
NOTE: cache dismounted group 2/0xE9E5571F (DATA) 
SQL> alter diskgroup DATA dismount force /* ASM SERVER:3924121375 */ 
Tue Apr 07 18:22:32 2026
Sweep [inc][309345]: completed
System State dumped to trace file /home/app/grid/diag/asm/+asm/+ASM1/incident/incdir_309345/+ASM1_ora_639087_i309345.trc
Tue Apr 07 18:22:32 2026
Dumping diagnostic data in directory=[cdmp_20260407182232], requested by (instance=1, osid=639087), summary=[incident=309345].
Tue Apr 07 18:22:32 2026
NOTE: cache deleting context for group DATA 2/0xe9e5571f
GMON dismounting group 2 at 11 for pid 32, osid 115720
NOTE: Disk DATA_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0002 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0003 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0004 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0005 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 2
Tue Apr 07 18:22:34 2026
Sweep [inc2][309345]: completed
NOTE: AMDU dump of disk group DATA created at /home/app/grid/diag/asm/+asm/+ASM1/incident/incdir_309345
Tue Apr 07 18:22:37 2026
NOTE: ASM client orcl1:orcl disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file /home/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_504268.trc
Tue Apr 07 18:23:02 2026
SUCCESS: diskgroup DATA was dismounted
SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER:3924121375 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group DATA

通过kfed分析被格式化的磁盘,随机找了一些au发现都被置空

使用lsblk查看对应磁盘是否启用了TRIM 特性

基于这样的情况,基本上可以判断,该磁盘大概率已经触发了trim,数据被置空的概率非常大,最后对于镜像磁盘通过winhex查看,确认磁盘中除了基本的分区和文件系统信息之外其他都为空

基于此种情况,最好的结果就是恢复该6个磁盘组磁盘中5个磁盘的数据,这样丢失数据最少1/6以上,但是也是没有办法中的办法,尽可能减少损失了.

发表在 Linux, Oracle | 标签为 asm disk格式化, TRIM asm disk, trim数据丢失 | 评论关闭

oracle linux 8.10注意pmlogger导致空间被大量占用

发表于 2026 年 1 月 10 日由惜分飞

最近在多个客户的oracle linux 8.10的机器上发现/var/log/pcp/pmlogger目录占用空间过大,比如

[root@oracledb lib]# df -h
文件系统             容量  已用  可用 已用% 挂载点
devtmpfs              16G     0   16G    0% /dev
tmpfs                 16G     0   16G    0% /dev/shm
tmpfs                 16G  202M   16G    2% /run
tmpfs                 16G     0   16G    0% /sys/fs/cgroup
/dev/mapper/ol-root   60G   60G   63M  100% /
/dev/mapper/ol-home   30G  241M   29G    1% /home
/dev/vda1           1014M  360M  655M   36% /boot
/dev/vdb1            688G  224G  430G   35% /u01
tmpfs                3.1G   12K  3.1G    1% /run/user/42
tmpfs                 60M     0   60M    0% /var/log/rtlog
tmpfs                3.1G  8.0K  3.1G    1% /run/user/0
[root@oracledb ~]# cd /var/log/pcp/
[root@oracledb pcp]# du -h -d 1
76K	./pmcd
0	./pmfind
476K	./pmie
34G	./pmlogger
0	./pmproxy
0	./sa
34G	.
[root@oracledb pcp]# cd pmlogger/
[root@oracledb pmlogger]# du -h -d 1
34G	./localhost.localdomain
108K	./oracledb
34G	.

目录里面具体类容类似:

操作系统环境为:

[root@oracledb localhost.localdomain]# uname -a
Linux oracledb 5.15.0-206.153.7.1.el8uek.x86_64 #2 SMP Wed May 22 20:49:34 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux
[root@oracledb localhost.localdomain]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.10 (Ootpa)
[root@oracledb localhost.localdomain]# cat /etc/os-release 
NAME="Oracle Linux Server"
VERSION="8.10"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.10"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:10:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.10
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.10

pmlogger 是 Performance Co-Pilot（PCP）的核心组件，用于采集、归档系统 / 应用性能指标，生成可回放的 PCP 归档日志，支撑离线追溯分析与长期性能基线管理。它通过 PMCD（Performance Metrics Collector Daemon）获取指标，按配置策略记录并维护多卷归档，配合 PCP 工具链实现全链路性能分析。
这个本身是一个系统性能指标采集监控的重要工具,但是由于某种原因导致占用磁盘空间过大,而且大部分情况下有另外的监控来取代,引起大部分情况下，可以考虑关闭相关服务，并清空相关日志，避免出现磁盘空间不足的情况

[root@oracledb ~]# systemctl status pmlogger
● pmlogger.service - Performance Metrics Archive Logger
   Loaded: loaded (/usr/lib/systemd/system/pmlogger.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2025-11-03 22:25:48 CST; 1 months 27 days ago
     Docs: man:pmlogger(1)
 Main PID: 5580 (pmlogger)
    Tasks: 1 (limit: 818103)
   Memory: 27.9G
   CGroup: /system.slice/pmlogger.service
           └─5580 /usr/libexec/pcp/bin/pmlogger -N -P -r -T24h10m -c config.default -v 100mb -mreexec %Y%m%d.%H.%M

Nov 03 22:25:46 localhost.localdomain systemd[1]: Starting Performance Metrics Archive Logger...
Nov 03 22:25:48 localhost.localdomain systemd[1]: Started Performance Metrics Archive Logger.
[root@oracledb ~]# systemctl stop pmlogger
[root@oracledb ~]# cd /var
[root@oracledb var]# rm -rf /var/log/pcp/pmlogger/*
[root@oracledb var]# du -sh
665M	.
[root@oracledb var]# systemctl disable pmlogger
Removed /etc/systemd/system/multi-user.target.wants/pmlogger.service.

磁盘空间恢复正常

[root@oracledb var]# df -h
文件系统             容量  已用  可用 已用% 挂载点
devtmpfs              16G     0   16G    0% /dev
tmpfs                 16G     0   16G    0% /dev/shm
tmpfs                 16G  202M   16G    2% /run
tmpfs                 16G     0   16G    0% /sys/fs/cgroup
/dev/mapper/ol-root   60G   27G   34G   44% /
/dev/mapper/ol-home   30G  241M   29G    1% /home
/dev/vda1           1014M  360M  655M   36% /boot
/dev/vdb1            688G  224G  430G   35% /u01
tmpfs                3.1G   12K  3.1G    1% /run/user/42
tmpfs                 60M     0   60M    0% /var/log/rtlog
tmpfs                3.1G  8.0K  3.1G    1% /run/user/0
[root@oracledb var]#

发表在 Linux | 标签为 pmlogger, pmlogger占用大量空间 | 评论关闭

记录一次国产数据库被rm -rf /*删除的救援过程

发表于 2026 年 1 月 3 日由惜分飞

有一套国产数据库运行在arm平台的Kylin Linux上

NAME=&amp;quot;Kylin Linux Advanced Server&amp;quot;
VERSION=&amp;quot;V10 (Lance)&amp;quot;
ID=&amp;quot;kylin&amp;quot;
VERSION_ID=&amp;quot;V10&amp;quot;
PRETTY_NAME=&amp;quot;Kylin Linux Advanced Server V10 (Lance)&amp;quot;
ANSI_COLOR=&amp;quot;0;31&amp;quot;

由于数据库归档日志导致磁盘空间满,在清理归档日志的时候不慎执行了rm -rf /*命令

导致系统无法正常启动,客户尝试通过把磁盘挂载到其他机器上启动,但是无法恢复需要数据.由于客户这个是ssd盘的虚拟化环境,担心可能触发trim和进一步破坏的风险,建议客户尽快把虚拟磁盘下载到本地,然后通过工具分析,发现大量数据已经被删除,特别是数据库目录只剩下部分文件

通过分析data/global/1262文件(pg_database)的出来数据库名字对应关系,然后根据应用提供的需要恢复的数据库名称.在data目录中尝试恢复该目录中oid文件,发现最核心的几个字典文件全部异常(元数据异常)

pg_class	1259	存储数据库中所有关系（表、索引、视图、序列等）的元数据
pg_attribute	1249	存储表的列（属性）信息，每个字段对应一行
pg_namespace	2615	存储模式（Schema）信息
pg_type	        1247	存储数据类型信息（内置类型、自定义类型）

对于这种问题,普通的恢复工具无法解决,协调专业的文件系统恢复专家对其进行处理,比较完美的恢复出来的该目录下面绝大部分文件(包括字典文件)