分类目录归档：GoldenGate

经常和ogg打交道的同仁们，WARNING OGG-01223 TCP/IP error 111这个错误肯定不陌生，今天发现MOS上关于WARNING OGG-01223 TCP/IP error 111错误处理思路及其方案的文档写的非常好，看了之后深受启发，贡献出来给大家
WARNING OGG-01223 TCP/IP error 111
//files.xifenfei.com/wp-content/uploads/2011/11/WARNING_OGG-01223_TCPIP_error_111.pdf

Cause

This is generally due due to basic issues like:
* The remote manager process is not running or is running with incorrect port number
* The RMTHOST parameter in the pump extract is not configured correctly.
In other cases, the issue could be due to firewalls that forbid the connection, 
blocking certain ports or processes. 
This is generally seen when there is a firewall between the source and target machine and 
either the ports are not open or just the manager port is open.

Solution

The Extract, Replicat and Ggsci processes use ports normally starting at port 7840 and ascend sequentially. 
The ggsci command 'send manager getportinfo detail ' will retrieve the current list of ports that have been
allocated by Manager and their corresponding process IDs.
If you have some port restrictions, then you could use the DYNAMICPORTLIST with some range so that the
collector process will allocate the ports from that range. 
In general to overcome this issue, we could do something like below 

1. Change the target manager parameter file to use something like the following 

port 7809 
dynamicportlist 7810-7820 

2. Stop and start the manager 
3. Open ports 7809 through 7820 in the firewall 
4. Re-start the source pump 
The port range used in dynamicportlist(7810-7820) and the manager port 7809 is just an example. 
You can define your own ports there and have them open. 

OGG Extract Pump abended with ERROR OGG-01224 TCP/IP error 110 (Connection timed out); retries exceeded

1、异常现象

[oracle@localhost ~]$ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.1.1.0.0 Build 078
Linux, x64, 64bit (optimized), Oracle 10 on Jul 28 2010 13:21:11

Copyright (C) 1995, 2010, Oracle and/or its affiliates. All rights reserved.



GGSCI (localhost.localdomain) 1> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     EXT-ECP     00:00:00      unknown     
EXTRACT     RUNNING     EXT-EDS     00:00:00      unknown     
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      unknown     

2、尝试关闭异常进程重启

GGSCI (localhost.localdomain) 2> stop *

Sending STOP request to EXTRACT EXT-ECP ...

ERROR: sending message to EXTRACT EXT-ECP (Timeout waiting for message).

Sending STOP request to EXTRACT EXT-EDS ...

ERROR: sending message to EXTRACT EXT-EDS (Timeout waiting for message).

Sending STOP request to EXTRACT EXT-XZ ...

ERROR: sending message to EXTRACT EXT-XZ (Timeout waiting for message).

Sending STOP request to EXTRACT P-EDS ...

ERROR: sending message to EXTRACT P-EDS (Timeout waiting for message).

Sending STOP request to EXTRACT P-XZ ...

ERROR: sending message to EXTRACT P-XZ (Timeout waiting for message).

Sending STOP request to REPLICAT REP-BOS ...

ERROR: sending message to REPLICAT REP-BOS (Timeout waiting for message).

GGSCI (localhost.localdomain) 3> stop mgr!

Sending STOP request to MANAGER ...
Request processed.
Manager stopped.

GGSCI (localhost.localdomain) 4> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     STOPPED                                           
EXTRACT     RUNNING     EXT-ECP     00:00:00      unknown     
EXTRACT     RUNNING     EXT-EDS     00:00:00      unknown     
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      unknown     

GGSCI (localhost.localdomain) 5> kill EXT-ECP 

ERROR: Manager not currently running.

GGSCI (localhost.localdomain) 6> kill EXT-EDS 

ERROR: Manager not currently running.


GGSCI (localhost.localdomain) 7> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     STOPPED                                           
EXTRACT     RUNNING     EXT-ECP     00:00:00      unknown     
EXTRACT     RUNNING     EXT-EDS     00:00:00      unknown     
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      unknown   

GGSCI (localhost.localdomain) 8> exit  
--使用stop 进程，stop mgr，kill 进程都不能正常关闭这些进程

3、系统系统级别kill相关ogg进程

[oracle@localhost OGG]$ ps -ef|grep /opt/OGG
oracle    7479     1  0 Nov10 ?        00:03:31 /opt/OGG/extract PARAMFILE /opt/OGG/dirprm/ext-ecp.prm REPORTFILE /opt/OGG/dirrpt/EXT-ECP.rpt PROCESSID EXT-ECP USESUBDIRS
oracle    7480     1  0 Nov10 ?        00:02:30 /opt/OGG/extract PARAMFILE /opt/OGG/dirprm/ext-eds.prm REPORTFILE /opt/OGG/dirrpt/EXT-EDS.rpt PROCESSID EXT-EDS USESUBDIRS
oracle    7482     1  0 Nov10 ?        00:03:07 /opt/OGG/extract PARAMFILE /opt/OGG/dirprm/ext-xz.prm REPORTFILE /opt/OGG/dirrpt/EXT-XZ.rpt PROCESSID EXT-XZ USESUBDIRS
oracle    7483     1  0 Nov10 ?        00:00:01 /opt/OGG/extract PARAMFILE /opt/OGG/dirprm/p-eds.prm REPORTFILE /opt/OGG/dirrpt/P-EDS.rpt PROCESSID P-EDS USESUBDIRS
oracle    7485     1  0 Nov10 ?        00:00:03 /opt/OGG/replicat PARAMFILE /opt/OGG/dirprm/rep-bos.prm REPORTFILE /opt/OGG/dirrpt/REP-BOS.rpt PROCESSID REP-BOS USESUBDIRS
oracle    7518     1  0 Nov10 ?        00:00:01 ./server -p 7847 -k -l /opt/OGG/ggserr.log
oracle    7677     1  0 Nov10 ?        00:00:15 /opt/OGG/extract PARAMFILE /opt/OGG/dirprm/p-xz.prm REPORTFILE /opt/OGG/dirrpt/P-XZ.rpt PROCESSID P-XZ USESUBDIRS
oracle   25261 25112  0 12:48 pts/1    00:00:00 grep /opt/OGG
[oracle@localhost OGG]$ kill -9 7479 7480 7482 7483 7485  7518 7677
[oracle@localhost OGG]$ ps -ef|grep /opt/OGG
oracle   25264 25112  0 12:48 pts/1    00:00:00 grep /opt/OGG

4、重启所有ogg进程

[oracle@localhost OGG]$ ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.1.1.0.0 Build 078
Linux, x64, 64bit (optimized), Oracle 10 on Jul 28 2010 13:21:11

Copyright (C) 1995, 2010, Oracle and/or its affiliates. All rights reserved.



GGSCI (localhost.localdomain) 1> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     STOPPED                                           
EXTRACT     ABENDED     EXT-ECP     00:00:00      unknown     
EXTRACT     ABENDED     EXT-EDS     00:00:00      unknown     
EXTRACT     ABENDED     EXT-XZ      00:00:00      unknown     
EXTRACT     ABENDED     P-EDS       00:00:00      unknown     
EXTRACT     ABENDED     P-XZ        00:00:00      unknown     
REPLICAT    ABENDED     REP-BOS     00:00:00      unknown     
--进程状态还是异常

GGSCI (localhost.localdomain) 2> start mgr

Manager started.


GGSCI (localhost.localdomain) 3> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     EXT-ECP     00:00:00      unknown     
EXTRACT     RUNNING     EXT-EDS     00:00:00      unknown     
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      unknown     
--进程起来了，但是Time Since Chkpt还是不正确

GGSCI (localhost.localdomain) 4> stop ext-ecp

Sending STOP request to EXTRACT EXT-ECP ...
Request processed.


GGSCI (localhost.localdomain) 5> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     STOPPED     EXT-ECP     unknown       00:00:02    
EXTRACT     RUNNING     EXT-EDS     00:00:00      unknown     
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      unknown     
--关闭EXT-ECP测试，状态正常

GGSCI (localhost.localdomain) 6> start ext-ecp

Sending START request to MANAGER ...
EXTRACT EXT-ECP starting


GGSCI (localhost.localdomain) 7> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     EXT-ECP     unknown       00:00:14    
EXTRACT     RUNNING     EXT-EDS     00:00:00      unknown     
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      unknown     
--Lag异常，等待恢复

GGSCI (localhost.localdomain) 8> stop ext-eds

Sending STOP request to EXTRACT EXT-EDS ...

Recovery is not complete.  This normal stop will wait and checkpoint recovery's 
work when recovery has finished. To force Extract to stop now, 
use the SEND EXTRACT EXT-EDS, FORCESTOP command.
--因为恢复没有完成导致该提示，可以忽略，等待

GGSCI (localhost.localdomain) 9> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     EXT-ECP     unknown       00:00:02    
EXTRACT     STOPPED     EXT-EDS     01:51:12      00:00:01    
EXTRACT     RUNNING     EXT-IM      00:00:00      1059:44:26  
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      unknown     


GGSCI (localhost.localdomain) 10> start ext-eds

Sending START request to MANAGER ...
EXTRACT EXT-EDS starting


GGSCI (localhost.localdomain) 11> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     EXT-ECP     99:53:02      00:00:01    
EXTRACT     RUNNING     EXT-EDS     01:51:12      00:00:10    
EXTRACT     RUNNING     EXT-XZ      00:00:00      unknown     
EXTRACT     RUNNING     P-EDS       00:00:00      unknown     
EXTRACT     RUNNING     P-XZ        00:00:00      unknown     
REPLICAT    RUNNING     REP-BOS     00:00:00      00:00:00    


GGSCI (localhost.localdomain) 12> stop ext-xz

Sending STOP request to EXTRACT EXT-XZ ...
Request processed.


GGSCI (localhost.localdomain) 13> start ext-xz

Sending START request to MANAGER ...
EXTRACT EXT-XZ starting

GGSCI (localhost.localdomain) 15> stop p-eds

Sending STOP request to EXTRACT P-EDS ...
Request processed.


GGSCI (localhost.localdomain) 16> start p-eds

Sending START request to MANAGER ...
EXTRACT P-EDS starting


GGSCI (localhost.localdomain) 17> stop p-xz

Sending STOP request to EXTRACT P-XZ ...
Request processed.


GGSCI (localhost.localdomain) 18> start p-xz

Sending START request to MANAGER ...
EXTRACT P-XZ starting


GGSCI (localhost.localdomain) 19> info all

Program     Status      Group       Lag           Time Since Chkpt

MANAGER     RUNNING                                           
EXTRACT     RUNNING     EXT-ECP     00:00:00      00:00:01    
EXTRACT     RUNNING     EXT-EDS     00:00:00      00:00:10    
EXTRACT     RUNNING     EXT-IM      00:00:00      1059:45:28  
EXTRACT     RUNNING     EXT-XZ      00:00:00      00:00:07    
EXTRACT     RUNNING     P-EDS       00:00:00      00:00:04    
EXTRACT     RUNNING     P-XZ        00:00:00      00:00:05    
REPLICAT    RUNNING     REP-BOS     00:00:00      00:00:05    
--重启所有异常进程，ogg工作正常

GGSCI (localhost.localdomain) 20> 

5、总结处理步骤
强制关闭mgr,系统级别kill相关ogg进程,开启ogg主进程,重启相关进程