bbed 恢复 GLOBAL_NAME 为空故障

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:bbed 恢复 GLOBAL_NAME 为空故障

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

看到dbsnake关于UPDATE GLOBAL_NAME为空之后的恢复写的不是很完整,自己通过试验和对bbed的研究,完善他的blog内容(泄露一点内部的东西)
模拟错误

SQL> select * from v$version;

BANNER
------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
PL/SQL Release 11.2.0.3.0 - Production
CORE    11.2.0.3.0      Production
TNS for Linux: Version 11.2.0.3.0 - Production
NLSRTL Version 11.2.0.3.0 - Production

SQL> select * from global_name ;

GLOBAL_NAME
--------------------------------------------------------------------
ORA11G

SQL> update global_name set global_name='';

1 row updated.

SQL> COMMIT;

Commit complete.

SQL> SHUTDOWN IMMEDIATE
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> STARTUP
ORACLE instance started.

Total System Global Area  313860096 bytes
Fixed Size                  1344652 bytes
Variable Size             260049780 bytes
Database Buffers           46137344 bytes
Redo Buffers                6328320 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [18062], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 28306
Session ID: 125 Serial number: 5

alert日志

Errors in file /u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_28306.trc  (incident=20556):
ORA-00600: internal error code, arguments: [18062], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/oracle/diag/rdbms/ora11g/ora11g/incident/incdir_20556/ora11g_ora_28306_i20556.trc
Wed Aug 08 23:21:48 2012
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_28306.trc:
ORA-00600: internal error code, arguments: [18062], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_28306.trc:
ORA-00600: internal error code, arguments: [18062], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 28306): terminating the instance due to error 600
Instance terminated by USER, pid = 28306
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (28306) as a result of ORA-1092
Wed Aug 08 23:21:48 2012
ORA-1092 : opitsk aborting process

找出global_name相关信息(另外库中)

SQL> SET LONG 1000
SQL> select dbms_metadata.get_ddl('VIEW','GLOBAL_NAME','SYS') from dual;

DBMS_METADATA.GET_DDL('VIEW','GLOBAL_NAME','SYS')
--------------------------------------------------------------------------------

  CREATE OR REPLACE FORCE VIEW "SYS"."GLOBAL_NAME" ("GLOBAL_NAME") AS
  select value$ from sys.props$ where name = 'GLOBAL_DB_NAME'

SQL> select * from props$ where value$='XIFENFEI';

NAME                           VALUE$
------------------------------ ----------------------------------------
COMMENT$
--------------------------------------------------------------------------
GLOBAL_DB_NAME                 XIFENFEI
Global database name

SQL> select dump('GLOBAL_DB_NAME','16') from dual;

DUMP('GLOBAL_DB_NAME','16')
--------------------------------------------------------
Typ=96 Len=14: 47,4c,4f,42,41,4c,5f,44,42,5f,4e,41,4d,45
--得出GLOBAL_DB_NAME存储字16进制字符串为:0e474c4f42414c5f44425f4e414d45

bbed操作

--通过第三方工具定位props$表中的GLOBAL_DB_NAME列在数据块0x00400321的31行上

[oracle@xifenfei ~]$ bbed listfile=listfile mode=edit password=blockedit

BBED: Release 2.0.0.0.0 - Limited Production on Thu Aug 9 00:26:12 2012

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

************* !!! For Oracle Internal Use only !!! ***************

BBED> info
 File#  Name                                                        Size(blks)
 -----  ----                                                        ----------
     1  /u01/oracle/oradata/ora11g/system01.dbf                              0
     2  /u01/oracle/oradata/ora11g/sysaux01.dbf                              0
     3  /u01/oracle/oradata/ora11g/undotbs01.dbf                             0
     4  /u01/oracle/oradata/ora11g/users01.dbf                               0

BBED> set block 801
        BLOCK#          801

BBED> map
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801                                   Dba:0x00400321
------------------------------------------------------------
 KTB Data Block (Table/Cluster)

 struct kcbh, 20 bytes                      @0       

 struct ktbbh, 72 bytes                     @20      

 struct kdbh, 14 bytes                      @92      

 struct kdbt[1], 4 bytes                    @106     

 sb2 kdbr[37]                               @110     

 ub1 freespace[5771]                        @184     

 ub1 rowdata[2233]                          @5955    

 ub4 tailchk                                @8188    

BBED>  p *kdbr[31]
rowdata[0]
----------
ub1 rowdata[0]                              @5955     0x2c

BBED> x /rccc
rowdata[0]                                  @5955    
----------
flag@5955: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@5956: 0x02
cols@5957:    3

col   0[14] @5958: GLOBAL_DB_NAME
col    1[0] @5973: *NULL*
col   2[20] @5974: Global database name

BBED> set count 64
        COUNT           64

BBED> f /x 0e474c4f
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets: 5958 to 6021           Dba:0x00400321
------------------------------------------------------------------------
 0e474c4f 42414c5f 44425f4e 414d45ff 14476c6f 62616c20 64617461 62617365 
 206e616d 652c0003 0e474c4f 42414c5f 44425f4e 414d4506 4f524131 31471447 

 <32 bytes per line>

BBED> f
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets: 5998 to 6061           Dba:0x00400321
------------------------------------------------------------------------
 0e474c4f 42414c5f 44425f4e 414d4506 4f524131 31471447 6c6f6261 6c206461 
 74616261 7365206e 616d652c 00030a44 4254494d 455a4f4e 45053030 3a30300c 

 <32 bytes per line>

BBED> f
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets: 6460 to 6523           Dba:0x00400321
------------------------------------------------------------------------
 0e474c4f 42414c5f 44425f4e 414d4508 53454544 44415441 14476c6f 62616c20 
 64617461 62617365 206e616d 652c0003 114e4c53 5f524442 4d535f56 45525349 

 <32 bytes per line>

BBED> f
BBED-00212: search string not found

BBED> d /v offset 5958
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801     Offsets: 5958 to 6021  Dba:0x00400321
-------------------------------------------------------
 0e474c4f 42414c5f 44425f4e 414d45ff l .GLOBAL_DB_NAME.
 14476c6f 62616c20 64617461 62617365 l .Global database
 206e616d 652c0003 0e474c4f 42414c5f l  name,...GLOBAL_
 44425f4e 414d4506 4f524131 31471447 l DB_NAME.ORA11G.G

 <16 bytes per line>

BBED> d /v offset 5998
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801     Offsets: 5998 to 6061  Dba:0x00400321
-------------------------------------------------------
 0e474c4f 42414c5f 44425f4e 414d4506 l .GLOBAL_DB_NAME.
 4f524131 31471447 6c6f6261 6c206461 l ORA11G.Global da
 74616261 7365206e 616d652c 00030a44 l tabase name,...D
 4254494d 455a4f4e 45053030 3a30300c l BTIMEZONE.00:00.

 <16 bytes per line>

BBED> d /v offset 6460
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801     Offsets: 6460 to 6523  Dba:0x00400321
-------------------------------------------------------
 0e474c4f 42414c5f 44425f4e 414d4508 l .GLOBAL_DB_NAME.
 53454544 44415441 14476c6f 62616c20 l SEEDDATA.Global 
 64617461 62617365 206e616d 652c0003 l database name,..
 114e4c53 5f524442 4d535f56 45525349 l .NLS_RDBMS_VERSI

 <16 bytes per line>
--通过时上面的查找可以知道数据库对GLOBAL_DB_NAME有进行两次update操作
--GLOBAL_DB_NAME初始化值SEEDDATA,第一次更新为ORA11G,第二次更新为null

BBED> p  kdbr[31]
sb2 kdbr[31]                                @172      5863
--这里可以发现我们看到offset 5995开始有值,但是row directory却指向了5863

BBED> d offset 5863 count 128
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets: 5863 to 5990           Dba:0x00400321
------------------------------------------------------------------------
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 2c02030e 
 474c4f42 414c5f44 425f4e41 4d45ff14 476c6f62 616c2064 61746162 61736520 

 <32 bytes per line>
--通过dump看到row directory指向的值和实际的数据相差(5995-5863),都是0

BBED> d /v offset 5900
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801     Offsets: 5900 to 6027  Dba:0x00400321
-------------------------------------------------------
 00000000 00000000 00000000 00000000 l ................
 00000000 00000000 00000000 00000000 l ................
 00000000 00000000 00000000 00000000 l ................
 00000000 0000002c 02030e47 4c4f4241 l .......,...GLOBA
 4c5f4442 5f4e414d 45ff1447 6c6f6261 l L_DB_NAME..Globa
 6c206461 74616261 7365206e 616d652c l l database name,
 00030e47 4c4f4241 4c5f4442 5f4e414d l ...GLOBAL_DB_NAM
 45064f52 41313147 14476c6f 62616c20 l E.ORA11G.Global 

 <16 bytes per line>
--需要指定的新值前面也存在同样的0,所以模仿的处理方法,让row directory同样向前偏移92

BBED> m /x 0f17 offset 172
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets:  172 to  299           Dba:0x00400321
------------------------------------------------------------------------
 0f176018 16189a17 5e173d17 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> p kdbr[31]
sb2 kdbr[31]                                @172      5903

BBED> p *kdbr[31]
rowdata[40]
-----------
ub1 rowdata[40]                             @5995     0x2c

BBED> x /rccc
rowdata[40]                                 @5995    
-----------
flag@5995: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@5996: 0x00
cols@5997:    3

col   0[14] @5998: GLOBAL_DB_NAME
col    1[6] @6013: ORA11G
col   2[20] @6020: Global database name

修改lock信息
BBED> m /x 02 offset 5996
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets: 5996 to 6123           Dba:0x00400321
------------------------------------------------------------------------
 02030e47 4c4f4241 4c5f4442 5f4e414d 45064f52 41313147 14476c6f 62616c20 
 64617461 62617365 206e616d 652c0003 0a444254 494d455a 4f4e4505 30303a30 
 300c4442 2074696d 65207a6f 6e652c00 02174e4f 5f555345 5249445f 56455249 
 46494552 5f53414c 54203633 39364335 38414231 37414530 30374539 41373238 

 <32 bytes per line>

BBED> d offset 5955
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets: 5955 to 6082           Dba:0x00400321
------------------------------------------------------------------------
 2c02030e 474c4f42 414c5f44 425f4e41 4d45ff14 476c6f62 616c2064 61746162 
 61736520 6e616d65 2c02030e 474c4f42 414c5f44 425f4e41 4d45064f 52413131 
 4714476c 6f62616c 20646174 61626173 65206e61 6d652c00 030a4442 54494d45 
 5a4f4e45 0530303a 30300c44 42207469 6d65207a 6f6e652c 0002174e 4f5f5553 

 <32 bytes per line>

BBED> m /x 2c00  
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets: 5955 to 6082           Dba:0x00400321
------------------------------------------------------------------------
 2c00030e 474c4f42 414c5f44 425f4e41 4d45ff14 476c6f62 616c2064 61746162 
 61736520 6e616d65 2c02030e 474c4f42 414c5f44 425f4e41 4d45064f 52413131 
 4714476c 6f62616c 20646174 61626173 65206e61 6d652c00 030a4442 54494d45 
 5a4f4e45 0530303a 30300c44 42207469 6d65207a 6f6e652c 0002174e 4f5f5553 

 <32 bytes per line>

--验证块
BBED> sum apply
Check value for File 1, Block 801:
current = 0xe836, required = 0xe836

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/oracle/oradata/ora11g/system01.dbf
BLOCK = 801

Block Checking: DBA = 4195105, Block Type = KTB-managed data block
data header at 0xb677b25c
kdbchk: the amount of space used is not equal to block size
        used=2056 fsc=6 avsp=6040 dtl=8096
Block 801 failed with check code 6110

DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0

--修改_ktbitfsc信息,让其通过块验证
BBED> p ktbbhitl
struct ktbbhitl[0], 24 bytes                @44      
   struct ktbitxid, 8 bytes                 @44      
      ub2 kxidusn                           @44       0x0004
      ub2 kxidslt                           @46       0x000e
      ub4 kxidsqn                           @48       0x000001d4
   struct ktbituba, 8 bytes                 @52      
      ub4 kubadba                           @52       0x00c00a93
      ub2 kubaseq                           @56       0x0083
      ub1 kubarec                           @58       0x33
   ub2 ktbitflg                             @60       0x8000 (KTBFCOM)
   union _ktbitun, 2 bytes                  @62      
      b2 _ktbitfsc                          @62       0
      ub2 _ktbitwrp                         @62       0x0000
   ub4 ktbitbas                             @64       0x000c78fe
struct ktbbhitl[1], 24 bytes                @68      
   struct ktbitxid, 8 bytes                 @68      
      ub2 kxidusn                           @68       0x000a
      ub2 kxidslt                           @70       0x0016
      ub4 kxidsqn                           @72       0x000001eb
   struct ktbituba, 8 bytes                 @76      
      ub4 kubadba                           @76       0x00c0015b
      ub2 kubaseq                           @80       0x008f
      ub1 kubarec                           @82       0x0d
   ub2 ktbitflg                             @84       0x0001 (NONE)
   union _ktbitun, 2 bytes                  @86      
      b2 _ktbitfsc                          @86       6
      ub2 _ktbitwrp                         @86       0x0006
   ub4 ktbitbas                             @88       0x00000000

BBED> m /x 00 offset 86
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets:   86 to  213           Dba:0x00400321
------------------------------------------------------------------------
 00000000 00000001 25000600 5c00e716 98179e17 00002500 701f571e 92189c1e 
 101efb1d ffff981d 551d0f1d d91cb71c 941c731c 4a1c1a1c ef1bc51b 9c1b701b 
 471b191b d61a931a 3c1a101a ee19bc19 86194e19 0d190f17 60181618 9a175e17 
 3d170000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> sum apply
Check value for File 1, Block 801:
current = 0xe830, required = 0xe830

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/oracle/oradata/ora11g/system01.dbf
BLOCK = 801

Block Checking: DBA = 4195105, Block Type = KTB-managed data block
data header at 0xb677b25c
kdbchk: space available on commit is incorrect
        tosp=6046 fsc=0 stb=0 avsp=6040
Block 801 failed with check code 6111

DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 1
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0

--修改kdbhtosp信息
BBED> p kdbh
struct kdbh, 14 bytes                       @92      
   ub1 kdbhflag                             @92       0x00 (NONE)
   b1 kdbhntab                              @93       1
   b2 kdbhnrow                              @94       37
   sb2 kdbhfrre                             @96       6
   sb2 kdbhfsbo                             @98       92
   sb2 kdbhfseo                             @100      5863
   b2 kdbhavsp                              @102      6040
   b2 kdbhtosp                              @104      6046

BBED> d offset 102
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets:  102 to  229           Dba:0x00400321
------------------------------------------------------------------------
 98179e17 00002500 701f571e 92189c1e 101efb1d ffff981d 551d0f1d d91cb71c 
 941c731c 4a1c1a1c ef1bc51b 9c1b701b 471b191b d61a931a 3c1a101a ee19bc19 
 86194e19 0d190f17 60181618 9a175e17 3d170000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> d offset 104
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets:  104 to  231           Dba:0x00400321
------------------------------------------------------------------------
 9e170000 2500701f 571e9218 9c1e101e fb1dffff 981d551d 0f1dd91c b71c941c 
 731c4a1c 1a1cef1b c51b9c1b 701b471b 191bd61a 931a3c1a 101aee19 bc198619 
 4e190d19 0f176018 16189a17 5e173d17 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> m /x 98
 File: /u01/oracle/oradata/ora11g/system01.dbf (1)
 Block: 801              Offsets:  104 to  231           Dba:0x00400321
------------------------------------------------------------------------
 98170000 2500701f 571e9218 9c1e101e fb1dffff 981d551d 0f1dd91c b71c941c 
 731c4a1c 1a1cef1b c51b9c1b 701b471b 191bd61a 931a3c1a 101aee19 bc198619 
 4e190d19 0f176018 16189a17 5e173d17 00000000 00000000 00000000 00000000 
 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

--至此修改row directory指针完成
BBED> sum apply
Check value for File 1, Block 801:
current = 0xe836, required = 0xe836

BBED> verify
DBVERIFY - Verification starting
FILE = /u01/oracle/oradata/ora11g/system01.dbf
BLOCK = 801


DBVERIFY - Verification complete

Total Blocks Examined         : 1
Total Blocks Processed (Data) : 1
Total Blocks Failing   (Data) : 0
Total Blocks Processed (Index): 0
Total Blocks Failing   (Index): 0
Total Blocks Empty            : 0
Total Blocks Marked Corrupt   : 0
Total Blocks Influx           : 0

重新启动数据库

SQL> startup 
ORACLE instance started.

Total System Global Area  313860096 bytes
Fixed Size                  1344652 bytes
Variable Size             260049780 bytes
Database Buffers           46137344 bytes
Redo Buffers                6328320 bytes
Database mounted.
Database opened.
SQL> select * from global_name ;

GLOBAL_NAME
--------------------------------------------------------
ORA11G

SQL> 
此条目发表在 非常规恢复 分类目录,贴了 , 标签。将固定链接加入收藏夹。

bbed 恢复 GLOBAL_NAME 为空故障》有 5 条评论

  1. 惜分飞 说:

    是指针,关于数据块的校验部分,自己慢慢去折磨,一句话讲不清楚

    惜分飞,

    谢谢大神,受益非浅!还有点疑问:p kdbr中值最小的就是数据位置的指针么? 还有数据校验这一块还有点不清楚。能给指点指点么 ?感谢感谢!

  2. san 说:

    惜分飞,

    谢谢大神,受益非浅!还有点疑问:p kdbr中值最小的就是数据位置的指针么? 还有数据校验这一块还有点不清楚。能给指点指点么 ? 感谢感谢!

  3. 惜分飞 说:

    对数据库启动过程进行跟踪,可以定位到sql,也可以定位到block,然后dump block分析

    请问大神,你怎么知道修改lock信息 时应该改为哪些值?

    还有如果不知道global_name被改为空,怎么定位这个问题?

  4. san 说:

    请问大神,你怎么知道修改lock信息 时应该改为哪些值?

    还有如果不知道global_name被改为空,怎么定位这个问题?

  5. 惜分飞 说:

    Ora-00600 [18062] During Database Startup

    Applies to:
    Oracle Server - Enterprise Edition - Version: 11.2.0.2.0 and later   [Release: 11.2 and later ]
    Information in this document applies to any platform.
    
    Symptoms
    After updating the GLOBAL_NAME and setting it to NULL:
    UPDATE GLOBAL_NAME SET GLOBAL_NAME = '';
    COMMIT;
    
    
    The database is not starting up:
    SQL> shutdown immediate
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL> startup
    ORACLE instance started.
    
    Total System Global Area 422670336 bytes
    Fixed Size 1344616 bytes
    Variable Size 318770072 bytes
    Database Buffers 96468992 bytes
    Redo Buffers 6086656 bytes
    Database mounted.
    ORA-01092: ORACLE instance terminated. Disconnection forced
    ORA-00600: internal error code, arguments: [18062], [], [], [], [], [], [], [],
    [], [], [], []
    Process ID: 3552
    Session ID: 1 Serial number: 5
    
    Cause
    The steps that can cause the error:
    SQL> select * from global_name;
    
    GLOBAL_NAME
    --------------------------------------------------------------------------------
    ORCL1.LOCALDOMAIN
    
    SQL> UPDATE GLOBAL_NAME SET GLOBAL_NAME = '';
    
    1 row updated.
    
    SQL> commit;
    
    Commit complete.
    
    SQL> shutdown immediate
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL> startup
    ORACLE instance started.
    
    Total System Global Area 422670336 bytes
    Fixed Size 1344616 bytes
    Variable Size 318770072 bytes
    Database Buffers 96468992 bytes
    Redo Buffers 6086656 bytes
    Database mounted.
    ORA-01092: ORACLE instance terminated. Disconnection forced
    ORA-00600: internal error code, arguments: [18062], [], [], [], [], [], [], [],
    [], [], [], []
    Process ID: 3552
    Session ID: 1 Serial number: 5
    
    Solution
    Steps to resolve the error:
    
    1) Mount the DB
    SQL> conn / as sysdba
    Connected to an idle instance.
    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area 422670336 bytes 
    Fixed Size 1344616 bytes 
    Variable Size 318770072 bytes 
    Database Buffers 96468992 bytes 
    Redo Buffers 6086656 bytes 
    Database mounted.
    
    2) Find the PID of the Oracle process
    
    oracle@localhost:[oracle@localhost ~]$ ps -ef | grep LOCAL | grep -v grep
    oracle 4030 3417 0 11:51 ? 00:00:00 oracleorcl1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
    
    
    3) Here the PID is 4030 which we will use in the GDB
    oracle@localhost:[oracle@localhost ~]$ gdb $ORACLE_HOME/bin/oracle 4030
    
    
    GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2)
    Copyright (C) 2009 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later 
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law. Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "i386-redhat-linux-gnu".
    For bug reporting instructions, please see:
    ...
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/bin/oracle...(no debugging symbols found)...done.
    Attaching to program: /home/oracle/app/oracle/product/11.2.0/dbhome_2/bin/oracle, process 4030
    warning: .dynamic section for "/lib/librt.so.1" is not at the expected address
    warning: difference appears to be caused by prelink, adjusting expectations
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libodm11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libodm11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libcell11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libcell11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libskgxp11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libskgxp11.so
    Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
    Loaded symbols for /lib/librt.so.1
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libnnz11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libnnz11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libclsra11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libclsra11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libdbcfg11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libdbcfg11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libhasgen11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libhasgen11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libskgxn2.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libskgxn2.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libocr11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libocr11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libocrb11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libocrb11.so
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libocrutl11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libocrutl11.so
    Reading symbols from /usr/lib/libaio.so.1...(no debugging symbols found)...done.
    Loaded symbols for /usr/lib/libaio.so.1
    Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
    Loaded symbols for /lib/libdl.so.2
    Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
    Loaded symbols for /lib/libm.so.6
    Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
    [Thread debugging using libthread_db enabled]
    Loaded symbols for /lib/libpthread.so.0
    Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
    Loaded symbols for /lib/libnsl.so.1
    Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
    Loaded symbols for /lib/libc.so.6
    Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
    Loaded symbols for /lib/ld-linux.so.2
    Reading symbols from /usr/lib/libnuma.so.1...(no debugging symbols found)...done.
    Loaded symbols for /usr/lib/libnuma.so.1
    Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
    Loaded symbols for /lib/libnss_files.so.2
    Reading symbols from /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libnque11.so...(no debugging symbols found)...done.
    Loaded symbols for /home/oracle/app/oracle/product/11.2.0/dbhome_2/lib/libnque11.so
    0x00d24402 in __kernel_vsyscall ()
    
    
    4) Insert a breakpoint at kokiasg
    (gdb) break kokiasg
    Breakpoint 1 at 0x99f4d04
    
    
    5) Continue the debugging
    (Gdb) continue
    Continuing.
    
    
    6) Open the DB
    SQL> alter database open
    
    The hang will happen at the following point in the alert log due to the above break point
    Database Characterset is WE8MSWIN1252
    
    
    The gdb output will show:
    Breakpoint 1 , 0x0a3d404d in kokiasg ()
    
    
    7) Open a new SQLPLUS session as the first one is currently hanging due to the 
         break-point and restore the GLOBAL_NAME to its original value:
    
    SQLPLUS / as sysdba
    
    SQL> select status from v$instance;
    
    STATUS
    -----------
    OPEN
    
    SQL > update global_name set global_name = 'ORCL1.LOCALDOMAIN';
    
    1 row updated.
    
    SQL > commit;
    
    
    8) In another session connect to SQLPLUS and check the GLOBAL_NAME:
    SQL> select * from global_name;
    
    GLOBAL_NAME
    ----------------------
    ORCL1.LOCALDOMAIN
    
    9) Kill the GDB session which is currently suspending the database OPEN:
    (Gdb) kill
    Kill the program being debugged? (y or n) y
    (Gdb) quit
    
    At this moment, sqlplus will prompt:
    alter database open
    *
    ERROR at line 1 :
    ORA - 03113 : end - of - file on communication channel
    
    10) Restart the DB.