分类目录归档：Oracle

mount: wrong fs type, bad option, bad superblock恢复Oracle

发表于 2017 年 8 月 18 日由惜分飞

有朋友找到我们,说对lv进行收缩操作之后,导致文件系统无法mount,提示超级块损坏.
尝试mount失败

[root@GZGSDB data]# mount /dev/vg_gzgsdb/lv_home /home
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_gzgsdb-lv_home,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

系统日志报错

Aug 16 21:23:24 GZGSDB kernel: EXT4-fs (dm-5): ext4_check_descriptors: Block bitmap for group 1 not in group (block 0)!
Aug 16 21:23:24 GZGSDB kernel: EXT4-fs (dm-5): group descriptors corrupted!

主要操作

这里可以看出来，明显操作错误，在没有收缩文件系统之前，直接收缩的lv.

这里可以看出来，在发生故障之后，又做了很多操作，包括fsck和testdisk等,最终无法恢复,请求我们协助处理。

找出来备份超级块

[root@GZGSDB dul]# dumpe2fs /dev/vg_gzgsdb/lv_home |grep superblock
dumpe2fs 1.41.12 (17-May-2010)
ext2fs_read_bb_inode: A block group is missing an inode table 
  Primary superblock at 0, Group descriptors at 1-52
  Backup superblock at 32768, Group descriptors at 32769-32820
  Backup superblock at 98304, Group descriptors at 98305-98356
  Backup superblock at 163840, Group descriptors at 163841-163892
  Backup superblock at 229376, Group descriptors at 229377-229428
  Backup superblock at 294912, Group descriptors at 294913-294964
  Backup superblock at 819200, Group descriptors at 819201-819252
  Backup superblock at 884736, Group descriptors at 884737-884788
  Backup superblock at 1605632, Group descriptors at 1605633-1605684
  Backup superblock at 2654208, Group descriptors at 2654209-2654260
  Backup superblock at 4096000, Group descriptors at 4096001-4096052
  Backup superblock at 7962624, Group descriptors at 7962625-7962676
  Backup superblock at 11239424, Group descriptors at 11239425-11239476
  Backup superblock at 20480000, Group descriptors at 20480001-20480052
  Backup superblock at 23887872, Group descriptors at 23887873-23887924
  Backup superblock at 71663616, Group descriptors at 71663617-71663668
  Backup superblock at 78675968, Group descriptors at 78675969-78676020
  Backup superblock at 102400000, Group descriptors at 102400001-102400052
  Backup superblock at 214990848, Group descriptors at 214990849-214990900

使用fsck修复

[root@GZGSDB data]# fsck -y /dev/vg_gzgsdb/lv_home 
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
fsck.ext4: Group descriptors look bad... trying backup blocks...
fsck.ext4: The ext2 superblock is corrupt when using the backup blocks
fsck.ext4: going back to original superblock
fsck.ext4: Device or resource busy while trying to open /dev/mapper/vg_gzgsdb-lv_home
Filesystem mounted or opened exclusively by another program?
--指定超级块恢复
[root@GZGSDB data]# fsck -y -b 102400000 /dev/vg_gzgsdb/lv_home 
…………
Illegal block #0 (1296647770) in inode 354315.  CLEARED.
Illegal block #1 (1398362886) in inode 354315.  CLEARED.
Illegal block #3 (453538936) in inode 354315.  CLEARED.
Illegal block #5 (808333361) in inode 354315.  CLEARED.
Illegal block #6 (775434798) in inode 354315.  CLEARED.
Illegal block #8 (1180306180) in inode 354315.  CLEARED.
Illegal block #9 (1413893971) in inode 354315.  CLEARED.
Illegal block #10 (1229347423) in inode 354315.  CLEARED.
Illegal block #11 (1498613325) in inode 354315.  CLEARED.
Illegal indirect block (1296389203) in inode 354315.  CLEARED.
Inode 354315 is too big.  Truncate? yes

Block #1074301965 (69632) causes directory to be too big.  CLEARED.
Warning... fsck.ext4 for device /dev/mapper/vg_gzgsdb-lv_home exited with signal 11.

[root@GZGSDB data]# mount /dev/vg_gzgsdb/lv_home /home
mount: wrong fs type, bad option, bad superblock on /dev/mapper/vg_gzgsdb-lv_home,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

至此基本上可以判断,直接修复该文件系统,让其正常mount的概率很小.直接通过lun层面来恢复.

通过工具解析lun

比较明显,客户经过一系列操作现在的现象是lv 1.03T,文件系统只有821G,明显和客户给我反馈的操作不符(应该是lv 821G),证明客户做了大量操作,已经导致文件系统损坏

由于文件系统严重异常,工具获取到的文件都是没有名称,直接从inode里面读取的数据.获取到这些数据之后,然后结合oracle的特性,判断出来对应的文件号关系(这里有大量文件重复);另外有个别文件通过inode恢复丢失,通过底层碎片重组进行恢复出来文件,然后恢复出来其中数据(asm disk header 彻底损坏恢复).这个客户比较幸运,system文件在另外一个分区中,不然工作量会大很多.
再次提醒:对lv操作一定要谨慎,特别是lvreduce操作,另外出现发生误操作之后,应该第一时间保护现场,而不是百度着去乱操作,可能导致故障更加悲剧

发表在 Linux, 非常规恢复 | 标签为 bad superblock, superblock | 评论关闭

Quick Reference to Patch Numbers for Database/GI PSU, SPU(CPU), Bundle Patches and Patchsets—201707

发表于 2017 年 8 月 12 日由惜分飞

12.1.0.2 (12.1.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER)	21419221
11.2.0.4 (11.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER)	13390677
11.2.0.3 (11.2.0.3.0 PATCH SET FOR ORACLE DATABASE SERVER)	10404530
11.2.0.2 (11.2.0.2.0 PATCH SET FOR ORACLE DATABASE SERVER)	10098816
11.1.0.7 (11.1.0.7.0 PATCH SET FOR ORACLE DATABASE SERVER)	6890831
^A10.2.0.4 (10.2.0.4.0 PATCH SET FOR ORACLE DATABASE SERVER)	6810189
^B10.2.0.3 (10.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER)	5337014
10.2.0.2 (10.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER)	4547817
10.1.0.5 (10.1.0.5 PATCH SET FOR ORACLE DATABASE SERVER)	4505133
10.1.0.4 (10.1.0.4 PATCH SET FOR ORACLE DATABASE SERVER)	4163362
10.1.0.3 (10.1.0.3 PATCH SET FOR ORACLE DATABASE SERVER)	3761843
9.2.0.8 (9.2.0.8 PATCH SET FOR ORACLE DATABASE SERVER)	4547809
9.2.0.7 (9.2.0.7 PATCH SET FOR ORACLE DATABASE SERVER)	4163445
9.2.0.6 (9.2.0.6 PATCH SET FOR ORACLE DATABASE SERVER)	3948480
9.2.0.5 (ORACLE 9I DATABASE SERVER RELEASE 2 – PATCH SET 4 VERSION 9.2.0.5.0)	3501955
9.2.0.4 (9.2.0.4 PATCH SET FOR ORACLE DATABASE SERVER)	3095277
9.2.0.3 (9.2.0.3 PATCH SET FOR ORACLE DATABASE SERVER)	2761332
9.2.0.2 (9.2.0.2 PATCH SET FOR ORACLE DATABASE SERVER)	2632931
9.0.1.5 (9.0.1.5 PATCHSET)	3301544
9.0.1.4 (9.0.1.4 PATCH SET FOR ORACLE DATABASE SERVER)	2517300
9.0.1.3 (9.0.1.3. PATCH SET FOR ORACLE DATA SERVER)	2271678
8.1.7.4 (8.1.7.4 PATCH SET FOR ORACLE DATA SERVER)	2376472
8.1.7.3 (8.1.7.3 PATCH SET FOR ORACLE DATA SERVER)	2189751
8.1.7.2 (8.1.7.2.1 PATCH SET FOR ORACLE DATA SERVER)	1909158

12.2.0.1
Description	Database RU	GI RU	Windows Bundle Patch
JUL2017 (12.2.0.1.170718)	26123830	26133434	26204212

12.1.0.2
Description	PSU	GI PSU	Proactive Bundle Patch	Bundle Patch (Windows 32bit & 64bit)
JUL2017 (12.1.0.2.170718)	25755742	25901062	26022196	26161724
APR2017 (12.1.0.2.170418)	25171037	25434003	25433352	25632533
JAN2017 (12.1.0.2.170117)	24732082	24917825	24968615	25115951
OCT2016 (12.1.0.2.161018)	24006101	24412235	24448103	24591642
JUL2016 (12.1.0.2.160719)	23054246	23273629	23273686	23530387
APR2016 (12.1.0.2.160419)	22291127	22646084	22899531	22809813
JAN2016 (12.1.0.2.160119)	21948354	22191349	22243551	22310559
OCT2015	21359755 (12.1.0.2.5)	21523234 (12.1.0.2.5)	21744410 (12.1.0.2.13)	21821214 (12.1.0.2.10)
JUL2015	20831110 (12.1.0.2.4)	20996835 (12.1.0.2.4)	21188742 (12.1.0.2.10)	21126814 (12.1.0.2.7)
APR2015	20299023 (12.1.0.2.3)	20485724 (12.1.0.2.3)	20698050 (12.1.0.2.7)	20684004 (12.1.0.2.4)
JAN2015	19769480 (12.1.0.2.2)	19954978 (12.1.0.2.2)	20141343 (12.1.0.2.4)	19720843 (12.1.0.2.1)
OCT2014	19303936 (12.1.0.2.1)	19392646 (12.1.0.2.1)	19404326 (12.1.0.2.1)	N/A

12.1.0.1
Description	PSU	GI PSU		Bundle Patch
Description	PSU	A	B	Windows 64 bit	Windows 32 bit
JUL2016 (12.1.0.1.160719)	23054354	23273935	23273958	23530410
APR2016 (12.1.0.1.160419)	22291141	22654153	22654166	22839614
JAN2016 (12.1.0.1.160119)	21951844	22191492	22191511	22494866
OCT2015	21352619(12.1.0.1.9)	21551666(12.1.0.1.9)	21551685(12.1.0.1.9)	21744907 (12.1.0.1.21)
JUL2015	20831107(12.1.0.1.8)	20996901(12.1.0.1.8)	20996911(12.1.0.1.8)	21076681 (12.1.0.1.20)
APR2015	20299016(12.1.0.1.7)	20485762(12.1.0.1.7)	19971331(12.1.0.1.7)	20558101 (12.1.0.1.18)
JAN2015	19769486(12.1.0.1.6)	19971324(12.1.0.1.6)	19971331(12.1.0.1.6)	20160748 (12.1.0.1.16)
OCT2014	19121550(12.1.0.1.5)	19392372(12.1.0.1.5)	19392451(12.1.0.1.5)	19542943 (12.1.0.1.14)
JUL2014	18522516(12.1.0.1.4)	18705901(12.1.0.1.4)	18705972(12.1.0.1.4)	19062327 (12.1.0.1.11)
APR2014	18031528(12.1.0.1.3)	18139660(12.1.0.1.3)	18413105(12.1.0.1.3)	18448604 (12.1.0.1.7)
JAN2014	17552800(12.1.0.1.2)	17735306 (12.1.0.1.2)		17977915 (12.1.0.1.3)
OCT2013	17027533(12.1.0.1.1)	17272829 (12.1.0.1.1)		17363796(12.1.0.1.1)	17363795(12.1.0.1.1)

11.2.0.4
Description	PSU	SPU(CPU)	GI PSU	Bundle Patch (Windows 32bit & 64bit)
JUL2017 (11.2.0.4.170718)	25869727	25879656	26030799	26194136
APR2017 (11.2.0.4.170418)	24732075	25369547	25476126	25632525
JAN2017	N/A	N/A	N/A	N/A
OCT2016 (11.2.0.4.161018)	24006111	24433711	24436338	24591646
JUL2016 (11.2.0.4.160719)	23054359	23177648	23274134	23530402
APR2016 (11.2.0.4.160419)	22502456	22502493	22646198	22839608
JAN2016 (11.2.0.4.160119)	21948347	21972320	22191577	22310544
OCT2015	21352635 (11.2.0.4.8)	21352646	21523375 (11.2.0.4.8)	21821802 (11.2.0.4.20)
JUL2015	20760982 (11.2.0.4.7)	20803583	20996923 (11.2.0.4.7)	21469106 (11.2.0.4.18)
APR2015	20299013 (11.2.0.4.6)	20299015	20485808 (11.2.0.4.6)	20544696 (11.2.0.4.15)
JAN2015	19769489 (11.2.0.4.5)	19854503	19955028 (11.2.0.4.5)	20127071 (11.2.0.4.12)
OCT2014	19121551 (11.2.0.4.4)	19271443	19380115 (11.2.0.4.4)	19651773 (11.2.0.4.10)
JUL2014	18522509 (11.2.0.4.3)	18681862	18706472 (11.2.0.4.3)	18842982 (11.2.0.4.7)
APR2014	18031668 (11.2.0.4.2)	18139690	18139609 (11.2.0.4.2)	18296644 (11.2.0.4.4)
JAN2014	17478514 (11.2.0.4.1)	17551709	N/A	17987366 (11.2.0.4.1)

Assistant: Download Reference for Oracle Database/GI PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (文档 ID 2118136.2)

发表在 Oracle, Oracle安装升级 | 标签为 ORACLE PATCH | 评论关闭

ORA-12012: error on auto execute of job “SYS”.”ORA$AT_OS_OPT_SY_383″

发表于 2017 年 8 月 6 日由惜分飞

12.2.0.1版本数据库报类似ORA-12012: error on auto execute of job “SYS”.”ORA$AT_OS_OPT_SY_383″错误

Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_j000_10213.trc:
ORA-12012: error on auto execute of job "SYS"."ORA$AT_OS_OPT_SY_383"
ORA-20001: Statistics Advisor: Invalid task name for the current user
ORA-06512: at "SYS.DBMS_STATS", line 47207
ORA-06512: at "SYS.DBMS_STATS_ADVISOR", line 882
ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 20059
ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 22201
ORA-06512: at "SYS.DBMS_STATS", line 47197

trace文件信息

Trace file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_j000_10379.trc
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Build label:    RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME:    /u01/app/oracle/product/12.2.0/db_1
System name:    Linux
Node name:      yoridb2
Release:        3.10.0-514.26.2.el7.x86_64
Version:        #1 SMP Tue Jul 4 15:04:05 UTC 2017
Machine:        x86_64
Instance name: xifenfei
Redo thread mounted by this instance: 1
Oracle process number: 148
Unix process pid: 10379, image: oracle@yoridb2 (J000)


*** 2017-08-04T14:41:26.486692+08:00
*** SESSION ID:(508.52539) 2017-08-04T14:41:26.486716+08:00
*** CLIENT ID:() 2017-08-04T14:41:26.486722+08:00
*** SERVICE NAME:(SYS$USERS) 2017-08-04T14:41:26.486727+08:00
*** MODULE NAME:(DBMS_SCHEDULER) 2017-08-04T14:41:26.486732+08:00
*** ACTION NAME:(ORA$AT_OS_OPT_SY_384) 2017-08-04T14:41:26.486738+08:00
*** CLIENT DRIVER:() 2017-08-04T14:41:26.486743+08:00

ORA-12012: error on auto execute of job "SYS"."ORA$AT_OS_OPT_SY_384"
ORA-20001: Statistics Advisor: Invalid task name for the current user
ORA-06512: at "SYS.DBMS_STATS", line 47207
ORA-06512: at "SYS.DBMS_STATS_ADVISOR", line 882
ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 20059
ORA-06512: at "SYS.DBMS_STATS_INTERNAL", line 22201
ORA-06512: at "SYS.DBMS_STATS", line 47197

错误比较明显，是在执行统计信息分析的时候报错了，报错原因是由于SCHEDULER的task name无效.

解决方法

[oracle@yoridb2 trace]$ ss

SQL*Plus: Release 12.2.0.1.0 Production on Fri Aug 4 23:40:33 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select name, ctime, how_created
  2    from sys.wri$_adv_tasks
  3   where owner_name = 'SYS'
   and name in ('AUTO_STATS_ADVISOR_TASK','INDIVIDUAL_STATS_ADVISOR_TASK');  4  

no rows selected

SQL> EXEC dbms_stats.init_package();

PL/SQL procedure successfully completed.

SQL> select name, ctime, how_created
  2    from sys.wri$_adv_tasks
  3   where owner_name = 'SYS'
   and name in ('AUTO_STATS_ADVISOR_TASK','INDIVIDUAL_STATS_ADVISOR_TASK');  4  

NAME
--------------------------------------------------------------------------------
CTIME     HOW_CREATED
--------- ------------------------------
AUTO_STATS_ADVISOR_TASK
04-AUG-17 CMD

INDIVIDUAL_STATS_ADVISOR_TASK
04-AUG-17 CMD

发表在 ORA-xxxxx | 标签为 ORA$AT_OS_OPT_SY, ORA-12012 | 评论关闭

17813235971

QQ 咨询

分类目录归档：Oracle

mount: wrong fs type, bad option, bad superblock恢复Oracle

Quick Reference to Patch Numbers for Database/GI PSU, SPU(CPU), Bundle Patches and Patchsets—201707

ORA-12012: error on auto execute of job “SYS”.”ORA$AT_OS_OPT_SY_383″

文章分类

最近发表

分类目录归档：Oracle

mount: wrong fs type, bad option, bad superblock恢复Oracle

Quick Reference to Patch Numbers for Database/GI PSU, SPU(CPU), Bundle Patches and Patchsets—201707

ORA-12012: error on auto execute of job “SYS”.”ORA$AT_OS_OPT_SY_383″

标签云

文章分类

最近发表