system表空间丢失部分文件恢复

有客户因为system表空间有一个数据文件放在其他位置,当时没有正常拷贝出来(备份了oradata路径下面文件,遗漏了一个system文件),尝试启动库报ORA-01157 ORA-01147等错误

[oracle@xifenfei check_db]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Oct 5 21:13:28 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> recover datafile 1;
Media recovery complete. 
SQL> recover datafile 2,3,4,5,6,7,8,9,10;   
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01110: data file 11:
'/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'

SQL> alter database datafile 11 offline drop;

Database altered.

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01147: SYSTEM tablespace file 11 is offline
ORA-01110: data file 11:
'/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'

alert日志报错信息

Sun Oct 05 22:35:01 2025
alter database open
Sun Oct 05 22:35:01 2025
Errors in file /data/app/oracle/diag/rdbms/mtxdb1/mtxdb1/trace/mtxdb1_dbw0_5946.trc:
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01110: data file 11: '/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /data/app/oracle/diag/rdbms/mtxdb1/mtxdb1/trace/mtxdb1_ora_11264.trc:
ORA-01157: cannot identify/lock data file 11 - see DBWR trace file
ORA-01110: data file 11: '/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'
ORA-1157 signalled during: alter database open...
Sun Oct 05 22:35:25 2025
alter database datafile 11 offline 
ORA-1145 signalled during: alter database datafile 11 offline ...
alter database datafile 11 offline drop
Completed: alter database datafile 11 offline drop
alter database open
Errors in file /data/app/oracle/diag/rdbms/mtxdb1/mtxdb1/trace/mtxdb1_ora_11264.trc:
ORA-01147: SYSTEM tablespace file 11 is offline
ORA-01110: data file 11: '/u01/app/oracle/product/11.2.0.4/db_1/dbs/path_to_datafile.dbf'
ORA-1147 signalled during: alter database open...

由于11号文件是system表空间的一个数据文件,对于这种数据文件丢失无法offline该数据文件,然后open库(也就是说在open库的时候,system表空间的数据文件必须全部online,如果有部分文件offline就会报ORA-01147).对于这样的情况,以前有过类似恢复经历:bbed打开丢失部分system数据文件库,这次的编写了一个m_scn程序实现快速处理

[oracle@xifenfei  tmp]$ cat 1.txt
1@/data/app/oracle/oradata/mtxdb1/system01.dbf
11@/tmp/11.dbf
[oracle@xifenfei  tmp]$ ./m_scn 1.txt

-------------Is processing datafile:/tmp/11.dbf-------------
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.000835728 s, 1.3 GB/s

[oracle@xifenfei tmp]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 8 11:27:32 2025

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> set numw 16
SQL> col CHECKPOINT_TIME for a40
SQL> set lines 150
SQL> set pages 1000
SQL> SELECT status,
  2  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
  3  count(*) ROW_NUM
  4  FROM v$datafile_header
  5  GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
  6  ORDER BY status, checkpoint_change#, checkpoint_time;

STATUS  CHECKPOINT_TIME                          FUZ CHECKPOINT_CHANGE#          ROW_NUM
------- ---------------------------------------- --- ------------------ ----------------
OFFLINE 2025-10-02 06:50:06                      NO      17328662858685                1
ONLINE  2025-10-02 06:50:06                      NO      17328662858685               10


SQL> alter database datafile 11 online;

Database altered.

然后重建ctl,并尝试打开库
ctl_re


然后查询11号文件中涉及的对象情况

SQL> select distinct owner,segment_name,segment_type from dba_extents where file_id=11;

OWNER                          SEGMENT_NAME                           SEGMENT_TYPE
------------------------------ -------------------------------------- ------------------
SYS                            SYSTEM                                 ROLLBACK
SYS                            I_COL1                                 INDEX
SYS                            AUD$                                   TABLE

SQL> select owner,segment_name from dba_segments where HEADER_FILE=11;

no rows selected

证明丢失的11号文件(system表空间文件),涉及的对象较少,而且不涉及核心字典,比如tab$,obj$,col$等非常核心对象,评估理论上应该不涉业务数据丢失,尝试直接expdp导出数据,但是很不幸,报ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018]错误

. . exported "XFF020"."OTHERBILLDETAIL_DEL"              6.405 MB  126048 rows
. . exported "XFF020"."POSSOLDOUT"                       7.784 MB  281413 rows
ORA-31693: Table data object "XFF020"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=159:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UNLOAD_DATA [TABLE_DATA:"XFF020"."MATERIELTRAN"] 
UPDATE "SYS"."SYS_EXPORT_FULL_01" SET processing_state = :1, processing_status = :2
    WHERE process_order = :3 AND duplicate = 0
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 7866
ORA-31693: Table data object "XFF020"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=159:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 9721

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xef2fc508     21979  package body SYS.KUPW$WORKER
0xef2fc508      9742  package body SYS.KUPW$WORKER
0xef2fc508      3437  package body SYS.KUPW$WORKER
0xef2fc508     10436  package body SYS.KUPW$WORKER
0xef2fc508      1824  package body SYS.KUPW$WORKER
0xef2feb20         2  anonymous block

ORA-39097: Data Pump job encountered unexpected error -607
ORA-39065: unexpected master process exception in DISPATCH
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []

ORA-31693: Table data object "XFF020"."ANALYSEREPORT" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=161:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UNLOAD_DATA [TABLE_DATA:"XFF020"."ANALYSEREPORT"] 
UPDATE "SYS"."SYS_EXPORT_FULL_01" SET processing_state = :1, processing_status = :2
   WHERE process_order = :3 AND duplicate = 0
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 7866
ORA-31693: Table data object "XFF020"."ANALYSEREPORT" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=161:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 9721

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xef2fc508     21979  package body SYS.KUPW$WORKER
0xef2fc508      9742  package body SYS.KUPW$WORKER
0xef2fc508      3437  package body SYS.KUPW$WORKER
0xef2fc508     10436  package body SYS.KUPW$WORKER
0xef2fc508      1824  package body SYS.KUPW$WORKER
0xef2feb20         2  anonymous block

ORA-31693: Table data object "XFF020CW"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=160:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UNLOAD_DATA [TABLE_DATA:"XFF020CW"."MATERIELTRAN"] 
UPDATE "SYS"."SYS_EXPORT_FULL_01" SET processing_state = :1, processing_status = :2
   WHERE process_order = :3 AND duplicate = 0
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPW$WORKER", line 7866
ORA-31693: Table data object "XFF020CW"."MATERIELTRAN" failed to load/unload and is being skipped due to error:
ORA-39068: invalid master table data in row with PROCESS_ORDER=160:1000001
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [11], [3], [18018], [], [], [], [], [], [], [], []
ORA-06512: at "SYS.KUPF$FILE", line 3720
ORA-06512: at line 1

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 9721

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xef2fc508     21979  package body SYS.KUPW$WORKER
0xef2fc508      9742  package body SYS.KUPW$WORKER
0xef2fc508      3437  package body SYS.KUPW$WORKER
0xef2fc508     10436  package body SYS.KUPW$WORKER
0xef2fc508      1824  package body SYS.KUPW$WORKER
0xef2feb20         2  anonymous block

Job "SYS"."SYS_EXPORT_FULL_01" stopped due to fatal error at Wed Oct 8 11:59:29 2025 elapsed 0 00:18:48

对ORA-600 kdBlkCheckError进行分析分析(11表示文件号,3表示block),是由于导出生成的master表写入在system表空间,而system表空间中的file# 11是人工构造出来的,block 3 是位图分配信息(该信息和实际字典中存储信息不匹配),所以导致出现该错误,对于这个问题解决方法为expdp写master表不在system表空间即可,通过该操作,顺利导出数据,完成本次恢复任务
expdp_ok


发表在 非常规恢复 | 标签为 , , , , | 评论关闭

arm环境vg损坏mysql数据库恢复

国庆节期间接到朋友咨询,原先在vg中的磁盘被重新pvcreate了,想恢复原磁盘中的mysql数据库
pvcreate


通过分析系统的history日志,发现操作不是简单的pvcreate,我简单梳理下操作步骤
故障之前磁盘情况

[root@0002 ~]# lsblk
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                    11:0    1 1024M  0 rom  
vda                   253:0    0  200G  0 disk 
├─vda1                253:1    0  600M  0 part /boot/efi
├─vda2                253:2    0    1G  0 part /boot
└─vda3                253:3    0 38.4G  0 part 
  ├─klas-root         252:0    0 34.4G  0 lvm  /
  └─klas-swap         252:1    0    4G  0 lvm  [SWAP]
vdb                   253:16   0 1000G  0 disk 
└─vdb1                253:17   0  500G  0 part 
  └─mysql-mysql--mycg 252:2    0  500G  0 lvm  /mysql

这里可以看到出来vdb磁盘一共1000G,分区vdb1 为500G,然后这500G加入到vg中并分配了lv.

vdb磁盘现状

[root@0002 mysql]# lsblk /dev/vdb
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vdb                   253:16   0 1000G  0 disk 
└─vdb1                253:17   0 1000G  0 part 

Disk /dev/vdb: 1000 GiB, 1073741824000 bytes, 2097152000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5a6aaeee

Device     Boot Start        End    Sectors  Size Id Type
/dev/vdb1        2048 2097151999 2097149952 1000G 8e Linux LVM

这里基本上可以确定,vdb1磁盘分区从以前的500G变成了1000G(也就是说被重新分区了,后续和现场沟通确认进行了重新分区操作)


通过history日志追述大概的操作过程

  898  [2025-09-28 11:55:13][root]fdisk -l
  899  [2025-09-28 11:55:21][root]df -h
  900  [2025-09-28 11:56:41][root]lsblk
  901  [2025-09-28 11:59:44][root]fdisk /dev/vdb
  902  [2025-09-28 12:00:46][root]partprobe /dev/vdb
  903  [2025-09-28 12:00:50][root]pvresize /dev/vdb1
  904  [2025-09-28 12:00:56][root]df -h
  905  [2025-09-28 12:01:25][root]vgdisplay mysql
  906  [2025-09-28 12:01:40][root]lsblk
  907  [2025-09-28 12:02:05][root]sudo partprobe /dev/vdb
  908  [2025-09-28 12:02:10][root]pvresize /dev/vdb1
  909  [2025-09-28 12:02:27][root]sudo pvresize /dev/vdb1
  910  [2025-09-28 12:03:07][root]sudo pvcreate /dev/vdb1
  911  [2025-09-28 12:03:22][root]sudo pvscan
  912  [2025-09-28 12:03:30][root]sudo pvdisplay
  913  [2025-09-28 12:05:37][root]parted /dev/vdb
  914  [2025-09-28 12:06:11][root]pvresize /dev/vdb1
  915  [2025-09-28 12:06:15][root]lsblk
  916  [2025-09-28 12:09:48][root]lvextend -l +100%FREE /dev/mysql/mysql--mycg
  917  [2025-09-28 12:10:00][root]cd /dev/mysql/
  918  [2025-09-28 12:10:01][root]ll
  919  [2025-09-28 12:10:20][root]pwd
  920  [2025-09-28 12:10:32][root]lvextend -l +100%FREE /dev/mysql/mysql-mycg
  921  [2025-09-28 12:10:55][root]lsblk /dev/vdb

基本上可以确定9月28日先进行了fdisk分区操作,然后尝试pvresize 操作[应该不会成功,因为重新分区导致pv信息丢失],然后进行了pvcreate之后再次进行parted分区操作,再pvresize,lvextend操作[同理pv信息丢失应该不会成功],然后10月5日继续进行的部分操作

  956  [2025-10-05 08:29:27][root]umount /mysql
  957  [2025-10-05 08:29:38][root]lsof /mysql
  958  [2025-10-05 08:29:58][root]service mysqld stop
  959  [2025-10-05 08:30:02][root]umount /mysql
  960  [2025-10-05 08:30:05][root]lsof /mysql
  961  [2025-10-05 08:30:23][root]cd /
  962  [2025-10-05 08:30:25][root]umount /mysql
  963  [2025-10-05 08:30:34][root]pvcreate --force /dev/vdb1
  964  [2025-10-05 08:30:47][root]vgextend mysql /dev/vdb1
  965  [2025-10-05 08:31:02][root]df -h
  966  [2025-10-05 08:31:33][root]pvdisplay /dev/vdb1
  967  [2025-10-05 08:31:41][root]pvcreate --force /dev/vdb1
  968  [2025-10-05 08:32:11][root]lvs | grep mysql-mysql--mycg
  969  [2025-10-05 08:32:19][root]dmsetup ls | grep mysql
  970  [2025-10-05 08:32:38][root]fuser /dev/vdb1
  971  [2025-10-05 08:32:41][root]lsof /dev/vdb1
  972  [2025-10-05 08:32:50][root]pvcreate --force /dev/vdb1
  973  [2025-10-05 08:33:14][root]reboot
  974  [2025-10-05 08:36:23][root]pvcreate --force /dev/vdb1
  975  [2025-10-05 08:36:47][root]lvdisplay /dev/mapper/mysql-mysql--mycg
  976  [2025-10-05 08:36:53][root]vgextend mysql /dev/vdb1
  977  [2025-10-05 08:37:10][root]lvextend -l +100%FREE /dev/mysql/mysql--mycg

初步看,应该是先尝试umount /dev/vdb1,但是没有成功,然后直接reboot重启了主机,起来之后,进行了pvcreate[操作成功],vgextend,lvextend等操作[失败,因为vg里面的之前的pv信息已经丢失],而且之前lv无法mount成功,数据库文件/备份均在这个lv里面,而且从库很久之前没有正常同步.基于这样的情况,就一定要对vdb磁盘中数据进行恢复.查看操作系统信息,确认是arm系统
arm


由于arm系统一般工具均无法正常解析,只能让客户把磁盘挂载到x86环境进行处理,通过专业恢复工具解析,运气不错可以直接读取数据
m1

传输数据到客户服务器中,并成功启动mysql,客户测试业务没有任何问题,数据完整恢复
2

发表在 Linux, MySQL恢复 | 标签为 , | 评论关闭

redhat系列7/8进入单用户模式

以前写过一篇文章在linux老版本中,进入单用户模式的方法:linux 4/5/6版本进入单用户模式,今天测试主流的redhat系列(测试使用OEL,没有本质区别)7和8版本中进入单用户.
主要操作步骤:
1)选择linux启动项,输入e
2)根据你的RHEL/CentOS/OEL版本,找到 linux16/linux/linuxefi等类似启动语句语句,按下键盘上的 End 键,跳到行末,添加关键词 rd.break,按下 Ctrl+x 或 F10 来进入单用户模式
3)mount 根文件系统为读写模式:mount -o remount,rw /sysroot
4)指定/sysroot为/挂载点:chroot /sysroot
5)进行需要的系统操作,比如重设root密码,修改不合适的系统配置(fstab,sysctl.conf等),然后sync同步数据
6)重启系统:reboot -f(也可以两次exit实现重启)
linux 7系列进入单用户演示
s1
s2
s3


linux 8系列进入单用户演示
s4
s5
s6

发表在 Linux | 标签为 , | 评论关闭