Hdr: 10321285 11.1.0.7.0 RDBMS 11.1.0.7.0 DIAG ADR PRODID-5 PORTID-212 ORA-600
Abstract: ORA-600 [DBGRMBLCP_CORRUPT_PAGE]
 
*** 11/21/10 07:41 am *** (CHG: RDBMS Ver.-> NULL -> 11.1.0.7.0)
*** 11/21/10 07:41 am ***
----
 
PROBLEM:
--------
1. Clear description of the problem encountered
 
On Sep. 5th, the following internal errors were written to the alert log:
 
ORA-600 [dbgrmblcp_corrupt_page], 
[/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
[]
ORA-600 [dbgrmblgp_get_page_1], [16], [675499854], [42], []
 
can also see:
ORA-51108: unable to access diagnostic repository - retry command
ORA-48223: Interrupt Requested - Fetch Aborted - Return Code [51014]
ORA-6512: at "SYS.DBMS_IR", line 522
 
This was a weekend - should have been little to no activity
These errors were raised several times on one day, did not reappear since 
then.
 
2. Pertinent configuration information (MTS/OPS/distributed/etc)
 
3. Indication of the frequency and predictability of the problem
So far, this happened only on Sep. 5th at Customer-side.
Not reproducible
 
4. Sequence of events leading to the problem
undetermined
 
5. Technical impact on the customer. Include persistent after effects.
Customer is concerned about their ADR repository.
Is this the visible sign of something being corrupt in their ADR Repository?
 
DIAGNOSTIC ANALYSIS:
--------------------
Alert log shows:
 
Sun Sep 05 11:36:41 2010
Errors in file 
/pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j001_3485708.trc  
(incident=326889):
ORA-600: internal error code, arguments: [dbgrmblcp_corrupt_page], 
[/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
[], [], [], [], [], [], [], [], []
Incident details in: 
/pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326889/pextapps_j
001_3485708_i326889.trc
Errors in file 
/pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j001_3485708.trc  
(incident=326890):
ORA-600: internal error code, arguments: [dbgrmblgp_get_page_1], [16], 
[675499854], [42], [], [], [], [], [], [], [], []
Incident details in: 
/pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326890/pextapps_j
001_3485708_i326890.trc
Sun Sep 05 11:36:45 2010
Trace dumping is performing id=[cdmp_20100905113645]
Sun Sep 05 11:36:45 2010
Sweep Incident[326889]: completed
Errors in file 
/pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j001_3485708.trc:
ORA-12012: error on auto execute of job 11375
ORA-51108: unable to access diagnostic repository - retry command
ORA-48223: Interrupt Requested - Fetch Aborted - Return Code [51014]
ORA-6512: at "SYS.DBMS_IR", line 522
Sun Sep 05 11:37:41 2010
Errors in file 
/pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
(incident=326831):
ORA-600: internal error code, arguments: [dbgrmblcp_corrupt_page], 
[/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
[], [], [], [], [], [], [], [], []
Incident details in: 
/pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326831/pextapps_j
000_1769628_i326831.trc
Sun Sep 05 11:37:45 2010
Sweep Incident[326890]: completed
Sun Sep 05 11:38:00 2010
Errors in file 
/pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
(incident=326832):
ORA-600: internal error code, arguments: [dbgrmblgp_get_page_1], [16], 
[675499854], [42], [], [], [], [], [], [], [], []
Incident details in: 
/pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326832/pextapps_j
000_1769628_i326832.trc
Sun Sep 05 11:38:00 2010
Trace dumping is performing id=[cdmp_20100905113800]
Trace dumping is performing id=[cdmp_20100905113803]
Sun Sep 05 11:38:43 2010
Errors in file 
/pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
(incident=326833):
ORA-600: internal error code, arguments: [dbgrmblcp_corrupt_page], 
[/pextapps/admin/diag/rdbms/pextapps/pextapps/metadata/HM_FINDING.ams], [16], 
[], [], [], [], [], [], [], [], []
Incident details in: 
/pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326833/pextapps_j
000_1769628_i326833.trc
Sun Sep 05 11:38:45 2010
Sweep Incident[326832]: completed
Sweep Incident[326831]: completed
Errors in file 
/pextapps/admin/diag/rdbms/pextapps/pextapps/trace/pextapps_j000_1769628.trc  
(incident=326834):
ORA-600: internal error code, arguments: [dbgrmblgp_get_page_1], [16], 
[675499854], [42], [], [], [], [], [], [], [], []
Incident details in: 
/pextapps/admin/diag/rdbms/pextapps/pextapps/incident/incdir_326834/pextapps_j
000_1769628_i326834.trc
Sun Sep 05 11:38:47 2010
Trace dumping is performing id=[cdmp_20100905113847]
Trace dumping is performing id=[cdmp_20100905113850]
Sun Sep 05 11:39:45 2010
Sweep Incident[326834]: completed
Sweep Incident[326833]: completed
 
 
Asked CT to run the following commands:
 adrci> set home rdbms
 adrci> show catalog
 adrci> describe hm_finding
 adrci> query (RUN_ID, NAME) HM_FINDING
which all worked fine
 
Obtained the content of their /diag/rdbms/.../.../metadata/' folder
 
WORKAROUND:
-----------
 
RELATED BUGS:
-------------
Bug 10303857 - QCN: ORA-700[DBGRMBLCP_CORRUPT_PAGE] / 
ORA-600[DBGRMBLGP_GET_PAGE_1] ON NT 
Bug 6595098 - ORA-48132 AND ORA-600 [DBGRMBLCP_CORRUPT_PAGE] ETC DURING OPEN 
DATABASE
 
REPRODUCIBILITY:
----------------
For the time being, these errors did not reappear.
 
TEST CASE:
----------
 
STACK TRACE:
------------
skdstdst ksedst1 ksedst dbkedDefDump ksedmp ksfdmp dbgexPhaseII
dbgexProcessError dbgeExecuteForError dbgePostErrorKGE dbkePostKGE_kgsf
kgeadse kgerinv_internal kgesoftnmierr
dbgrmblcp_corrupt_page dbgrmblgp_get_page dbgrmblrp_read_page 
dbgrmqmgp_get_page dbgrmqmif_internal_fetch dbgrmqmf_fetch_real 
dbgrmqmf_fetch
dbgrip_fetch_record dbgrip_relation_iterator dbghmo_close_findin 
dbkh_close_finding  kcidr_file_header_check dbkird_df_check 
dbkh_run_check_internal dbkh_run_revalidate check dbkifRevalidateFinding 
dbkifReevaluateOpenFailures dbkiicd_reeval pevm_icd_call_commo pfrinstr_ICAL 
pfrrun_no_tool pfrrun plsql_run peidxr_run peidxexe kkxdexe kkxmpexe kgmexwi 
kgmexec evapls evaopn2 kkxmexcs opiexe kpoal8 opiodr kpoodr upirtrc kpurcsc 
kpuexec OCIStmtExecute jslvec_execcb1 jslvswu jslve_execute0 jslve_execute 
rpiswu2 kkjex1e kkjsexe kkjrdp opirip opidrv sou2o opimai_real main start