标签云
asm 恢复 asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 ORACLE恢复 Oracle 恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (100)
- 数据库 (1,597)
- DB2 (22)
- MySQL (70)
- Oracle (1,463)
- Data Guard (49)
- EXADATA (7)
- GoldenGate (21)
- ORA-xxxxx (158)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (13)
- ORACLE 21C (3)
- Oracle ASM (65)
- Oracle Bug (7)
- Oracle RAC (47)
- Oracle 安全 (6)
- Oracle 开发 (27)
- Oracle 监听 (27)
- Oracle备份恢复 (530)
- Oracle安装升级 (84)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (75)
- PostgreSQL (17)
- PostgreSQL恢复 (5)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (36)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (19)
-
最近发表
- Oracle 19c/21c最新patch信息-202404
- PostgreSQL恢复系列:pg_filedump批量处理
- PostgreSQL部分主要字典信息
- PostgreSQL恢复系列:pg_filedump恢复字典构造
- PostgreSQL 16 源码安装
- ORA-00742 ORA-00312 恢复
- 数据库open成功后报ORA-00353 ORA-00354错误引起的一系列问题(本质ntfs文件系统异常)
- ORA-600 ktsiseginfo1故障
- ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
- 最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
- ORA-600 2662快速恢复之Patch scn工具
- TNS-12518: TNS:listener could not hand off client connection
- ora.storage无法启动报ORA-12514故障处理
- 断电引起文件scn异常数据库恢复
- ORA-16188: LOG_ARCHIVE_CONFIG settings inconsistent with previously started instance
- .[hudsonL@cock.li].mkp勒索加密数据库完美恢复
- 模拟带库实现rman远程备份
- 又一例:ORA-600 kclchkblk_4和2662故障
- Oracle误删除数据文件恢复
- Oracle 19C 备库DML重定向—DML Redirection
月归档:六月 2019
ERROR: diskgroup XXXX was not mounted
aix平台10.2.0.5 2节点RAC,由于节点2系统盘故障,通过节点1镜像系统,复制到节点2,结果由于节点2磁盘顺序和节点1不匹配,aix工程师进行了相关操作之后,节点1重启之后datadg磁盘组无法mount
SQL> alter diskgroup datadg mount Mon Jun 10 23:23:46 CST 2019 NOTE: cache registered group DATADG number=1 incarn=0x8cf61164 Mon Jun 10 23:23:46 CST 2019 NOTE: Hbeat: instance first (grp 1) Mon Jun 10 23:23:50 CST 2019 NOTE: start heartbeating (grp 1) Mon Jun 10 23:23:50 CST 2019 NOTE: cache dismounting group 1/0x8CF61164 (DATADG) NOTE: dbwr not being msg'd to dismount ERROR: diskgroup DATADG was not mounted
检查datadg磁盘组相关信息
Tue Jan 29 19:21:45 CST 2019 NOTE: start heartbeating (grp 2) NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/rhdisk6 Tue Jan 29 19:21:45 CST 2019 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/rhdisk7 NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/rhdisk8 NOTE: cache opening disk 3 of grp 2: DATADG_0003 path:/dev/rhdisk9 NOTE: cache mounting (first) group 2/0x60E59155 (DATADG) * allocate domain 2, invalid = TRUE Tue Jan 29 19:21:45 CST 2019 NOTE: attached to recovery domain 2 Tue Jan 29 19:21:45 CST 2019 NOTE: cache recovered group 2 to fcn 0.849668 Tue Jan 29 19:21:45 CST 2019 NOTE: LGWR attempting to mount thread 1 for disk group 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.849668 ABA NOTE: seq=21 blk=5394 Tue Jan 29 19:21:46 CST 2019 NOTE: cache mounting group 2/0x60E59155 (DATADG) succeeded SUCCESS: diskgroup DATADG was mounted
通过这里可以看出来datadg磁盘组是由rhdisk6-9 四块磁盘组成,查询相关磁盘信息发现
这里确定rhdisk7磁盘异常,通过kfed分析磁盘情况
D:\BaiduNetdiskDownload\xifenfei>kfed read rhdisk7.dd kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 34 ; 0x001: 0x22 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 49407 ; 0x004: blk=49407 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 58396 ; 0x010: 0x0000e41c kfbh.fcn.wrap: 131072 ; 0x014: 0x00020000 kfbh.spare1: 4294967064 ; 0x018: 0xffffff18 kfbh.spare2: 2105310074 ; 0x01c: 0x7d7c7b7a 005918A00 00002200 0000C0FF 00000000 00000000 [."..............] 005918A10 0000E41C 00020000 FFFFFF18 7D7C7B7A [............z{|}] 005918A20 00000000 00000000 00000000 00000000 [................] Repeat 253 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] D:\BaiduNetdiskDownload\xifenfei>kfed read rhdisk7.dd blkn=1 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 006EF8A00 00000000 00000000 00000000 00000000 [................] Repeat 255 times KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] D:\BaiduNetdiskDownload\xifenfei>kfed read rhdisk7.dd blkn=2|more kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 33554432 ; 0x004: blk=33554432 kfbh.block.obj: 16777344 ; 0x008: file=128 kfbh.check: 3844041089 ; 0x00c: 0xe51f6981 kfbh.fcn.base: 1297484544 ; 0x010: 0x4d560b00 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb10.aunum: 0 ; 0x000: 0x00000000 kfdatb10.shrink: 49153 ; 0x004: 0xc001 kfdatb10.ub2pad: 20555 ; 0x006: 0x504b kfdatb10.auinfo[0].link.next: 2048 ; 0x008: 0x0800 kfdatb10.auinfo[0].link.prev: 2048 ; 0x00a: 0x0800 kfdatb10.auinfo[0].free: 0 ; 0x00c: 0x0000 kfdatb10.auinfo[0].total: 49153 ; 0x00e: 0xc001 kfdatb10.auinfo[1].link.next: 4096 ; 0x010: 0x1000 kfdatb10.auinfo[1].link.prev: 4096 ; 0x012: 0x1000 kfdatb10.auinfo[1].free: 0 ; 0x014: 0x0000 kfdatb10.auinfo[1].total: 0 ; 0x016: 0x0000 kfdatb10.auinfo[2].link.next: 6144 ; 0x018: 0x1800 kfdatb10.auinfo[2].link.prev: 6144 ; 0x01a: 0x1800 kfdatb10.auinfo[2].free: 0 ; 0x01c: 0x0000 kfdatb10.auinfo[2].total: 0 ; 0x01e: 0x0000 kfdatb10.auinfo[3].link.next: 8192 ; 0x020: 0x2000 kfdatb10.auinfo[3].link.prev: 8192 ; 0x022: 0x2000 kfdatb10.auinfo[3].free: 0 ; 0x024: 0x0000
对比磁盘可能的损坏情况,由于在aix 平台asm disk的block有一个特征一般0082开头,通过工具打开磁盘,检索该标记对比
正常磁盘
异常磁盘
通过上述分析,大概评估rhdisk7 元数据部分损坏的不光是block 0和1,人工修复继续使用的可能性不太大,而且基于客户的数据库不大,采取方案是直接拷贝数据文件、redo、控制文件到文件系统,然后在本地文件系统open库
运气不错,实现完美恢复数据0丢失
WARNING: Read Failed.导致asm磁盘组异常
有客户对asm dg进行扩容,一段时间之后,asm data 磁盘组直接dismount
Wed May 29 18:37:25 2019 SUCCESS: ALTER DISKGROUP DATA ADD DISK '/dev/oracleasm/disks/DATA_0028' SIZE 511993M , '/dev/oracleasm/disks/DATA_0027' SIZE 511993M , '/dev/oracleasm/disks/DATA_0026' SIZE 511993M , '/dev/oracleasm/disks/DATA_0025' SIZE 511993M /* ASMCA */ NOTE: starting rebalance of group 1/0x9e18e2f1 (DATA) at power 1 Wed May 29 18:37:26 2019 Starting background process ARB0 Wed May 29 18:37:26 2019 ARB0 started with pid=34, OS id=96638 NOTE: assigning ARB0 to group 1/0x9e18e2f1 (DATA) with 1 parallel I/O NOTE: Attempting voting file refresh on diskgroup DATA NOTE: Refresh completed on diskgroup DATA. No voting file found. cellip.ora not found. Wed May 29 19:21:43 2019 WARNING: Read Failed. group:1 disk:27 AU:0 offset:360448 size:4096 WARNING: cache failed reading from group=1(DATA) dsk=27 blk=88 count=1 from disk= 27 (DATA_0027) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596 ERROR: cache failed to read group=1(DATA) dsk=27 blk=88 from disk(s): 27(DATA_0027) ORA-15080: synchronous I/O operation to a disk failed ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 704 Additional information: -1 NOTE: cache initiating offline of disk 27 group DATA NOTE: process _user31879_+asm1 (31879) initiating offline of disk 27.3915911747 (DATA_0027) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 27/0xe9681243, mask = 0x6a, op = clear Wed May 29 19:21:43 2019 GMON updating disk modes for group 1 at 10 for pid 35, osid 31879 ERROR: Disk 27 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Wed May 29 19:21:43 2019 NOTE: cache dismounting (not clean) group 1/0x9E18E2F1 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 90256, image: oracle@ftz-db-o1 (B000) Wed May 29 19:21:43 2019 NOTE: halting all I/Os to diskgroup 1 (DATA) WARNING: Offline for disk DATA_0027 in mode 0x7f failed. Wed May 29 19:21:43 2019 NOTE: LGWR doing non-clean dismount of group 1 (DATA) NOTE: LGWR sync ABA=27.3207 last written ABA 27.3207 Wed May 29 19:21:43 2019 ERROR: ORA-15130 thrown in ARB0 for group number 1 Errors in file /oracle/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_96638.trc: ORA-15130: diskgroup "" is being dismounted ORA-15130: diskgroup "DATA" is being dismounted Wed May 29 19:21:43 2019 NOTE: stopping process ARB0
后续继续mount data 磁盘组成功,但是立马又dismount
Wed May 29 18:37:25 2019 SUCCESS: ALTER DISKGROUP DATA ADD DISK '/dev/oracleasm/disks/DATA_0028' SIZE 511993M , '/dev/oracleasm/disks/DATA_0027' SIZE 511993M , '/dev/oracleasm/disks/DATA_0026' SIZE 511993M , '/dev/oracleasm/disks/DATA_0025' SIZE 511993M /* ASMCA */ NOTE: starting rebalance of group 1/0x9e18e2f1 (DATA) at power 1 Wed May 29 18:37:26 2019 Starting background process ARB0 Wed May 29 18:37:26 2019 ARB0 started with pid=34, OS id=96638 NOTE: assigning ARB0 to group 1/0x9e18e2f1 (DATA) with 1 parallel I/O NOTE: Attempting voting file refresh on diskgroup DATA NOTE: Refresh completed on diskgroup DATA. No voting file found. cellip.ora not found. Wed May 29 19:21:43 2019 WARNING: Read Failed. group:1 disk:27 AU:0 offset:360448 size:4096 WARNING: cache failed reading from group=1(DATA) dsk=27 blk=88 count=1 from disk= 27 (DATA_0027) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596 ERROR: cache failed to read group=1(DATA) dsk=27 blk=88 from disk(s): 27(DATA_0027) ORA-15080: synchronous I/O operation to a disk failed ORA-27072: File I/O error Linux-x86_64 Error: 5: Input/output error Additional information: 4 Additional information: 704 Additional information: -1 NOTE: cache initiating offline of disk 27 group DATA NOTE: process _user31879_+asm1 (31879) initiating offline of disk 27.3915911747 (DATA_0027) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 27/0xe9681243, mask = 0x6a, op = clear Wed May 29 19:21:43 2019 GMON updating disk modes for group 1 at 10 for pid 35, osid 31879 ERROR: Disk 27 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Wed May 29 19:21:43 2019 NOTE: cache dismounting (not clean) group 1/0x9E18E2F1 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 90256, image: oracle@ftz-db-o1 (B000) Wed May 29 19:21:43 2019 NOTE: halting all I/Os to diskgroup 1 (DATA) WARNING: Offline for disk DATA_0027 in mode 0x7f failed. Wed May 29 19:21:43 2019 NOTE: LGWR doing non-clean dismount of group 1 (DATA) NOTE: LGWR sync ABA=27.3207 last written ABA 27.3207 Wed May 29 19:21:43 2019 ERROR: ORA-15130 thrown in ARB0 for group number 1 Errors in file /oracle/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_96638.trc: ORA-15130: diskgroup "" is being dismounted ORA-15130: diskgroup "DATA" is being dismounted Wed May 29 19:21:43 2019 NOTE: stopping process ARB0
对于上述的故障现象,本质原因是由于asm 磁盘组增加新磁盘之后,开始做rebalance,但是由于遭遇到 27号盘上有IO读错误,使得asm磁盘组无法正常完成rebalance,因而data磁盘组无法稳定的mount。解决该问题思路,通过patch asm磁盘组,禁止rebalance,从而使得data磁盘组不再dismount,再进行后续恢复
回收站中有大量wri$_rcs表
在对一套Oracle 12.1.0.2的数据库巡检之时发现大量WRI$_RCS_数字_1的表在回收站中,从命名中看该表应该是Oracle某个自动任务处理后,表未被正常处理干净,遗留在回收站中.
查询mos确认是Bug 20114306 – Objects left in recyclebin after upgrade to 12.1.0.2 or with fix for bug 16851194 present – superseded (文档 ID 20114306.8)
可以尝试打上补丁21498770或者23100700然后设置_fix_control
alter system set "_fix_control"='16851194:off' ;
确认该_fix_control是否可以设置,可以查询 v$system_fix_control视图