drdb脑裂分析解决

1、脑裂原因
1.1)drdb两边的磁盘数据不一致,并且不知道自动恢复,举例说明产生该现象原因:
a是从节点、b是主节点
1.1.1)a节点磁盘不能正常写入数据(磁盘、主机、网络等原因)
1.1.2)a、b两个节点角色发生改变(a为主节点,b磁盘不可用,两边的数据未一致),a中对磁盘进行了操作
1.1.3)此时b磁盘恢复可用,但是因为a、b磁盘不一致,出现脑裂
1.2)官方描述
DRBD detects split brain at the time connectivity becomes available again and the peer nodes exchange the initial DRBD protocol handshake. If DRBD detects that both nodes are (or were at some point, while disconnected) in the primary role, it immediately tears down the replication connection. The tell-tale sign of this is a message like the following appearing in the system log:
Split-Brain detected, dropping connection!

2、出现脑裂的现象
2.1)开机界面

2.2)进入系统查看drdb状态
2.2.1)备节点
[root@node1 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:13:40
0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r—-
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:620
2.2.2)主节点
[root@node1 ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:13:40
m:res cs ro ds p mounted fstype
0:r0 StandAlone Secondary/Unknown UpToDate/DUnknown r—-

3、处理脑裂
需要选择一个节点为主节点(本实验选择node2为主节点)
3.1)从节点上
[root@node1 ~]# drbdadm secondary r0
[root@node1 ~]# drbdadm — –discard-my-data connect r0
[root@node1 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:13:40
0: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r—-
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:620
3.2)主节点,通过cat /proc/drbd查看状态,如果不是WFConnection状态,需要再手动连接
[root@node2 ~]# drbdadm connect r0

4、再次查看drdb状态
4.1)查看从节点
[root@node1 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:13:40
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r—-
ns:0 nr:1156 dw:1156 dr:0 al:0 bm:16 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
4.2)查看主节点
[root@node2 ~]# service drbd status
drbd driver loaded OK; device status:
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:13:40
m:res cs ro ds p mounted fstype
0:r0 Connected Primary/Secondary UpToDate/UpToDate C /opt/mysql ext3
通过查看drbd状态发现,脑裂问题解决,node1中的数据和node2中的相同

发表在 Linux高可用 | 评论关闭

heartbeat+drbd+mysql高可用配置

一、heartbeat
1、安装heartbeat
Heartbeat安装及简单配置

2、配置参数
2.1)/etc/ha.d/ha.cf
logfile /var/log/ha-log
keepalive 2
deadtime 30
warntime 10
initdead 120
udpport 694
ucast eth1 10.10.10.2
auto_failback off
watchdog /dev/watchdog
node node1
node node2
ping 192.168.1.1
respawn hacluster /usr/lib/heartbeat/ipfail

2.2)/etc/ha.d/haresources
node1 IPaddr::192.168.1.100/24/eth0/ drbddisk Filesystem::/dev/drbd0::/opt/mysql::ext3 mysql

2.3)/etc/ha.d/authkeys
auth 1
1 crc

二、DRBD
1、DRBD安装
DRBD安装配置说明

2、/etc/drbd.conf

global {
usage-count yes;
}
common {
  syncer { rate 10M; }
}
resource r0 {
protocol C;
disk {
    on-io-error   detach;
  }
net {
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict   disconnect;
}
on node1 {
    device     /dev/drbd0;
    disk       /dev/sdb1;
    address    10.10.10.1:7788;
    meta-disk  internal;
  }
on node2 {
    device    /dev/drbd0;
    disk      /dev/sdb1;
    address   10.10.10.2:7788;
    meta-disk internal;
  }
}

三、mysql
1、mysql安装
mysql 5.5二进制文件安装

四、服务配置
1、配置mysql服务器(不要设置开机启动,由heartbeat控制)
修改mysql.server文件
basedir=/opt/mysql/product/5.5
datadir=/opt/mysql/mysqldata
mv /opt/mysql/product/5.5/support-files/mysql.server /etc/init.d/mysql

2、配置heartbeat
chkconfig –add heartbeat
chkconfig heartbeat on

发表在 Linux高可用 | 一条评论

mysql 5.5二进制文件安装

1、创建相关目录用户
su – root
groupadd mysql
useradd -g mysql -p xifenfei -s /bin/bash -m mysql -d /opt/mysql
MYSQL_BASE=/opt/mysql
mkdir -p $MYSQL_BASE/product/5.5
mkdir -p $MYSQL_BASE/mysqldata
mkdir -p $MYSQL_BASE/mysqllog
mkdir -p /var/run/mysqld/
chmod -R 777 $MYSQL_BASE
chown -R mysql:mysql $MYSQL_BASE
chown -R mysql:mysql /var/run/mysqld

3、环境变量配置
export MYSQL_BASE=/opt/mysql
export basedir=$MYSQL_BASE/product/5.5
export datadir=$MYSQL_BASE/mysqldata
export LD_LIBRARY_PATH=$basedir/lib:/lib:/usr/lib:/usr/local/lib
export TMPDIR=/tmp
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=$basedir:$basedir/bin:${PATH}:$LD_LIBRARY_PATH

4、调整mysql 用户系统限制(可选)
编辑文件:/etc/security/limits.conf 加入以下语句:
mysql soft nproc 2047
mysql hard nproc 16384
mysql soft nofile 1024
mysql hard nofile 65536
确认如下语句是否存在于/etc/pam.d/login,如果不存在请增加:
session required pam_limits.so
如果用户SHELL 用的是Bourne, Bash, 或者Korn shell 修改/etc/profile,并增加如
下内容。(在SuSe 操作系统上,则需要修改/etc/profile.local):
if [ $USER = "mysql" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

5、安装数据库
5.1)解压数据库至/opt/mysql/product/5.5中
5.2)安装数据
[mysql@localhost ~]$ cd product/5.5/scripts/
[mysql@localhost scripts]$ ./mysql_install_db –basedir=$basedir –datadir=$datadir –user=mysql

5.3)设置my.cnf参数
vi /etc/my.cnf

[mysqladmin]
socket =/var/run/mysqld/mysqld.sock
[client]
port = 3306
socket =/var/run/mysqld/mysqld.sock
[mysqld]
port = 3306
socket = /var/run/mysqld/mysqld.sock
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
basedir = /opt/mysql/product/5.5
datadir = /opt/mysql/mysqldata
skip-external-locking = 1
interactive_timeout = 1200
wait_timeout = 1200
character-set-server = utf8
back_log = 500
default_time_zone = '+08:00'
max_connections = 1000
max_connect_errors = 1000
connect_timeout = 300
table_open_cache = 100
max_allowed_packet = 16M
binlog_cache_size = 8M
max_heap_table_size = 64M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 32
thread_concurrency = 16
query_cache_size = 0M
default-storage-engine = INNODB
thread_stack = 192K
transaction_isolation = READ-COMMITTED
tmp_table_size = 64M
log-bin =/opt/mysql/mysqllog/mysqlbin
log-error=/opt/mysql/mysqllog/mysqld.err
expire_logs_days = 7
binlog_format = ROW
max_binlog_size = 300M
slow_query_log = 1
slow_query_log_file =/opt/mysql/mysqllog/mysqld-slow
long_query_time = 10
tmpdir = /tmp
server-id = 1
key_buffer_size = 8M
read_buffer_size = 1M
read_rnd_buffer_size = 1M
bulk_insert_buffer_size = 1M
myisam_sort_buffer_size = 128K
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 2G
innodb_data_file_path = ibdata1:1024M:autoextend
innodb_autoextend_increment = 64
innodb_file_per_table = 1
innodb_data_home_dir = /opt/mysql/mysqldata
innodb_file_io_threads = 4
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 8M
innodb_log_file_size = 512M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 40
innodb_lock_wait_timeout = 120
innodb_locks_unsafe_for_binlog = 1
innodb_autoinc_lock_mode = 2
skip-name-resolve
lower_case_table_names=1
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
auto-rehash
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192

5.4)异常处理
5.4.1)直接通过mysqld_safe不能正常启动数据库
ln -s /opt/mysql/product/5.5 /usr/local/mysql
或者
ln -s /opt/mysql/product/5.5/bin/mysqld /usr/local/mysql/bin/mysqld

5.4.2)mysql_secure_installation不能正常执行
1)
执行:/opt/mysql/product/5.5/bin/mysql_secure_installation出现以下错误,解决方法(ln -s /var/run/mysqld/mysqld.sock /tmp/mysql.sock)
Enter current password for root (enter for none):
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)

2)
修改mysql_secure_installation 脚本的do_query 函数(推荐处理方法)
do_query() {
echo “$1″ >$command
#sed ‘s,^,> ,’ < $command # Debugging # mysql --defaults-file=$config <$command ----此行修改如下: mysql --defaults-extra-file=$config <$command return $? } 6、添加开机启动服务
修改mysql.server文件
basedir=/opt/mysql/product/5.5
datadir=/opt/mysql/mysqldata
[root@localhost ~]# cp /opt/mysql/product/5.5/support-files/mysql.server /etc/init.d/mysqld
[root@localhost ~]# chkconfig –add mysqld
[root@localhost ~]# chkconfig mysqld on
[root@localhost ~]# chkconfig –list mysqld
mysqld 0:off 1:off 2:on 3:on 4:on 5:on 6:off

发表在 MySQL安装配置 | 评论关闭