DRBD(Distributed Replicated Block Device),分布式复制块设备,是一种通过TCP/IP网络实现块设备数据实时镜像的方案。利用这种方案,单一主节点模式(single primary mode)双机系统能够实时地将业务数据保存在主备节点的磁盘中,正常情况下两个节点的数据是一模一样的。

HeartBeat在此是用来做热备切换的,两节点间通过心跳线连接,如果主节点死机的话,可以通过HA将DRBD快速切换到另外一个节点,同时抢占虚拟IP,并启动MySQL,对MySQL客户端来说,只有很短时间的中断。此模式能有效提高MySQL的可用性。

此文档参考了DRBD(http://www.drbd.org), HA(http://linux-ha.org), MySQL的相关官方文档。安装笔记由鱼漂(ipaddr)原创,最新版本可以在http://www.eit.name/找到,转载请注明。

--------------------------------
一.环境说明
--------------------------------
OS: CentOS 5.5 32位
MySQL: 系统自带
DRBD: 8.3.8.1编译安装 (http://www.drbd.org/)
HeartBeat: 3.0.3编译安装 (http://linux-ha.org/)

====Node1====
Hostname: db_node1.systemadmin.cn
Ip: 192.168.0.201
====Node2====
Hostname: db_node2.systemadmin.cn
Ip: 192.168.0.202

HB的虚IP: 192.168.0.200

用来做DRBD的分区:/dev/sdb1
挂载点:/data/dbdata

--------------------------------
二. 编译安装DRBD
--------------------------------
www.drbd.org下载drbd-8.3.8.1.tar.gz,分别在两个节点上编译安装:
注意,DRBD需要内核模块支持,2.6.33后的Linux内核已内置DRBD的支持,在此版本前,安装时需要kernel-devel包,同时编译drbd时要带上--with-km。

#./configure --prefix=/ --with-km
#make
#make install
#chkconfig --add drbd
#chkconfig drbd on

检查安装是否成功:
#drbdadm
看是否以执行成功?
#cat /proc/drbd
是否可以看到版本信息

重启后再次检查cat /proc/drbd

--------------------------------
三. 配置DRBD
--------------------------------
注意,新版本的DRBD的配置文件已拆分成多个,全部放在/etc/drbd.d/目录,具体可参考/etc/drbd.conf中的配置.
如果开了iptables防火墙,需要打开对应的7789端口。

在主节点配置/etc/drbd.d/dbdata.res:
=========dbdata.res begin===========
resource dbdata {
on db_node1.systemadmin.cn {
device /dev/drbd1;
disk /dev/sdb1;
address 192.168.0.201:7789;
meta-disk internal;
}
on db_node2.systemadmin.cn {
device /dev/drbd1;
disk /dev/sdb1;
address 192.168.0.202:7789;
meta-disk internal;
}
}
=========dbdata.res end===========

创建drbd设备
#drbdadm create-md all
#/etc/init.d/drbd restart

查看是否有相应的块设备:
#ls -l /dev/drbd1
#cat /proc/drbd

设置当前节点为主节点并进行格式化:
#drbdadm -- --overwrite-data-of-peer primary all
#mkfs.ext3 /dev/drbd1
#mount /dev/drbd1 /data/dbdata
#mkdir /data/dbdata/etc
#mkdir /data/dbdata/data

在从节点做如下配置:
复制dbdata.res到从节点的/etc/drbd.d/
#drbdadm create-md all
#/etc/init.d/drbd restart

通过cat /prod/drbd查看是否开始复制,以及两节点的状态。

此时开始同步两个节点的磁盘,需要一定时间,在同步完成前,请不要重启,否则会重新同步。

同步完成后,如果两个节点都是Secondary/Secondary,并且已经UpToDate了,可以挑一个节点设置为主节点:
#drbdadm primary all

DRDB的一些常规操作:
切换主从节点前,要注意umount
主从切换: drbdadm primary all, drbdadm secondary all
节点间连接:drbdadm connect|disconnect all

DRBD脑裂后的处理:
脑裂后,两个节点间数据不同步,主从关系失效,需要按下面的步骤修复:
a.在从节点如下操作:
#drbdadm secondary dbdata
#drbdadm -- --discard-my-data connect dbdata
b.在主节点上,通过cat /proc/drbd查看状态,如果不是WFConnection状态,需要再手动连接:
#drbdadm connect dbdadta

--------------------------------
四. 迁移MySQL到DRBD磁盘
--------------------------------
确保drbd当前工作正常,将/data/dbdata加载,再在主节点按如下步骤操作。
a. 关闭MySQL: /etc/init.d/mysqld stop
b. #mv /etc/my.cnf /data/dbdata/etc/
c. #ln -s /data/dbdata/etc/my.cnf /etc/my.cnf
d. 修改my.cnf,将数文件文件指向 /data/dbdata/data
e. 将原来的数据文件移动到 /data/dbdata/data
f. 再启动MySQL

从节点只需要删除/etc/my.cnf,同时,建立链接到/data/dbdata/etc/my.cnf即可。

现在已完成MySQL+DRBD的配置,但切换时仍然是手动进行,假设主节点从node1转到node2,手动操作如下:
a. 在node1上停止MySQL
b. 在node1上unmout /data/dbdata
c. 在node1上执地drbdadm secondary all,两个节点都是secondary状态后,才可以将node2设置为primary
d. 在node2上执行drbdadm primary all,将node2变成主节点
e. 观察两个节点的/proc/drbd是否正常
f. 在node2上mount /dev/drbd1 /data/dbdata
g. 在node2上面启动MySQL

鱼漂建议:据说MyISAM在DRBD上容易造成数据丢失,或表文件损坏,所以此模式下最好用InnoDB,打开Binlog,同时,innodb_flush_log_at_trx_commit=1可以保证数据完整,但会严重影响性能。设置为2的话性能约有10倍提升,不过故障时可能会丢失1秒的数据。

--------------------------------
五. 安装配置HA
--------------------------------
需分别下载和安装:
* Cluster Glue 1.0.6: glue-1.0.6.tar.bz2
* Resource Agents 1.0.3: agents-1.0.3.tar.bz2
* Heartbeat 3.0.3: Heartbeat-3-0-STABLE-3.0.3.tar.bz2

鱼漂特别提醒,编译安装过程中,还会连接网络下载文件和校验xml,请在安装前确保网络连接和DNS解析正常。

A. 先安装glue
1. 从linux-ha.org下载 glue-1.0.6.tar.bz2
2. 解压
3. 编译安装:
./autogen.sh
./configure

打开lib/stonith/main.c,将version相关的注释掉,否则编译不过,有点诡异。
1、找到其64行,将其注释掉。
2、找到其76到81行全部注释掉。
3、找到其390行,将其注释

groupadd haclient
useradd -g haclient hacluster
make
make install

B. 安装Cluster agent
#./autogen.sh
#./configure
#make
#make install


C. 安装HA3.0.3
下载HA3.0.3解压并进入到源代码目录:
#./bootstrap
#./ConfigureMe configure
#make
#make install

D. 配置/etc/ha.d/ha.cf
=========ha.cf begin==============
logfacility local0
keepalive 500ms
deadtime 10
warntime 5
initdead 60
mcast eth0 225.0.0.1 694 1 0
auto_failback off
node db_node1.systemadmin.cn
node db_node2.systemadmin.cn
=========ha.cf end==============
注意,防火墙需要打开端口为694,目标Ip为225.0.0.1的udp通信

E. 配置/etc/ha.d/haresources
========haresource begin===========
db_node1.systemadmin.cn drbddisk Filesystem::/dev/drbd1::/data/dbdata::ext3 IPaddr::192.168.0.200/24/eth0 mysql
========haresource end===========

F.生成/etc/ha.d/authkeys
#( echo -ne "auth 1\n1 sha1 "; \
dd if=/dev/urandom bs=512 count=1 | openssl md5 ) \
> /etc/ha.d/authkeys
#chmod 0600 /etc/ha.d/authkeys


G. 编写MySQL启动脚本/etc/ha.d/resource.d/mysql
=========mysql being============
#!/bin/bash
#
# Author: ipaddr ( admin.net [#] 163.com )
#
. /etc/ha.d/shellfuncs
case "$1" in
start)
res=`/etc/init.d/mysqld start`
ret=$?
ha_log $res
exit $ret
;;
stop)
res=`/etc/init.d/mysqld stop`
ret=$?
ha_log $res
exit $ret
;;
status)
if [[ `ps -ef | grep '[m]ysqld'` > 1 ]] ; then
echo "running"
else
echo "stopped"
fi
;;
*)
echo "Usage: mysql {start|stop|status}"
exit 1
;;
esac
exit 0
=========mysql end============


F. 拷贝一些常用脚本
1. 从Heartbeat-3-0源码目录的heartbeat/lib/复制hb_standby, hb_takeover到/usr/lib/heartbeat 并设置为可执行.
2. 将Cluster agent的一些文件拷到/etc/ha.d/:
cp /usr/etc/ha.d/shellfuncs /etc/ha.d/
cp -rf /etc/ha.d/* /usr/etc/ha.d/
(这个操作很奇怪,Cluster Agent与HA使用不同的目录,但又要共用文件,不知在编译agent时是否可以使用--prefix=/来解决,由于编译时间实在太长,所有没有测试)

G. 使用:
1. 设置为自启动:
chkconfig mysqld off
chkconfig --add heartbeat
chkconfig heartbeat on

2. 手动切换两个节点:
/usr/lib/heartbeat/hb_takeover 将当前节点设为主节点
/usr/lib/heartbeat/hb_standby 将当前节点设为从节点
3. 查看HA状态
查看当前节点heartbeat状态:
#/usr/lib/heartbeat/heartbeat -s

查看当前节点heartbeat状态:
#/usr/bin/cl_status hbstatus

列出所有节点:
#/usr/bin/cl_status listnodes
(如果是64位系统,HA的安装目录通常为/usr/lib64/)

一些题外话:
个人觉得HA3.0还不够成熟,尤其是在编译安装时,非常麻烦,主要有几个小问题:
a. glue必须把version相关的注释掉才可以编译通过,很诡异;
b. 个人认为HA是一个整体,把它拆成三个软件包编译安装实在是很难理解;
c. 安装完后,还需要手动复制一些脚本文件,另外,agent和ha默认使用不同的/etc/目录也很郁闷;
d. 几个包都要从网络下载xml文件,编译过程实在太久,花了一个下午才把agent编译完成,为什么不把相关xml打包在源码中一起下载呢?

当然,瑕不掩玉,HA还是非常了不起的一个开源作品。

附 MySQL的DRBD文档:
[file]attachment/201008/1283093603_0.rar[/file]

I needed to autoload some modules on some CentOS boxen and had a really hard time finding the most modern way to load them. After having them in /etc/rc.d/rc.local i found that i could also put them into /etc/rc.modules which then helped me find the following lines in /etc/rc.sysinit:

# Load other user-defined modules
for file in /etc/sysconfig/modules/*.modules ; do
[ -x $file ] && $file
done
# Load modules (for backward compatibility with VARs)
if [ -f /etc/rc.modules ]; then
/etc/rc.modules
fi

The cleanest way therefore seems to be like so:

echo "modprobe some_module" > /etc/sysconfig/modules/something.modules
chmod +x /etc/sysconfig/modules/something.modules

This kind of files could even be installed from a RPM without needing any special pre-install/uninstall handling.
This probably also applies to various Flavor of Redhat and Fedora. Personally i prefer gentoos nice modules.autoload file.

Recently I put a second, faster network card in a server. On booting OpenSuse 10 assigned the new card the name eth2 and the existing, built-in Ethernet device eth0. A number of applications, for example Samba and dnsmasq, typically bind to an Ethernet name rather than a specific IP or MAC address. It is possible to change the individual configuration files for each of these services but this is a little ugly considering my goal was to install the new hardware and disable the existing device, leaving everything else untouched.

A tidier solution is to assign eth0 to the new card and eth1 to the older (unused) device. Figuring out how to do this is a little confusing, there is no Yast option to configure network names and manually editing /etc/sysconfig/network/ifcfg-eth(mac address) provides no help either. Instead you must edit the file /etc/udev/rules.d/30-net_persistent_names.rules and change the device name associated to the relevant network MAC address. In a two card setup the file will look a little like this (each network device entry is on a single line):

SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="MAC", IMPORT="/sbin/rename_netiface %k eth0"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="MAC", IMPORT="/sbin/rename_netiface %k eth1"

Where SYSFS{address} is set to the MAC addresses of each ethernet device. Change this file so that the correct names are associated to the appropriate MAC devices and reboot. Afterwards you will find your network cards are named in the order that you desire and all the applications that refer to eth0 work as if nothing has changed. It is a simple change and something that really should figure as part of the advanced network device options in Yast.

分页: 1/1 第一页 1 最后页 [ 显示模式: 摘要 | 列表 ]