memcached 双机热备一、需求背景发现公司好多项目都用到memched。

并且都是单点，memched服务器挂了就会影响业务，于是百度之，发现memched不可以集群，也没有看到有什么高可用方案，但是发现了repcached（memched主从复制），结合LVS的NAT模式不同端口转发实现memched双机HA，本文章记录本人的部署过程，从属实战，没有文字性的说明，欢迎各位指点。

二、环境：1、系统环境：1.CentOS release 6.4 (Final)2、网络环境1.调度机（master）：2.vip：172.28.26.1003.vip1:172.28.16.100（lvs转发网关）4.eth1:172.28.26.101 （内网）5.eth2:172.28.16.101（lvs转发网段）6.调度机（backup）：7.eth1:172.28.26.99 （内网）8.eth2:172.28.16.99（lvs转发网段）9.memched备节点：10.eth1:172.28.26.102 （内网）11.eth2:172.28.16.102（lvs转发网段）12.memched主节点：13.eth1:172.28.26.103 （内网）14.eth2:172.28.16.103（lvs转发网段）3、路由策略1.调度机（master）：2.echo '201 eth1'>> /etc/iproute2/rt_tables3.echo 'default table eth1 via 172.28.26.1 dev eth1'> /etc/sysconfig/network-scripts/route-eth14.echo 'from 172.28.26.101/255.255.255.255 table eth1'> /etc/sysconfig/network-scripts/rule-eth15.echo 'from 172.28.26.100/255.255.255.255 table eth1'>> /etc/sysconfig/network-scripts/rule-eth16.echo 'from 172.28.16.101/32 table ZW_LVS_LAN'> /etc/sysconfig/network-scripts/rule-eth27.memched节点1：8.echo 'default table ZW_LVS_LAN via 172.28.16.100'> /etc/sysconfig/network-scripts/route-eth29.echo 'from 172.28.16.0/24 table ZW_LVS_LAN'> /etc/sysconfig/network-scripts/rule-eth210.echo '202 ZW_LVS_LAN'>> /etc/iproute2/rt_tables11.memched节点2：12.echo 'default table ZW_LVS_LAN via 172.28.16.100'> /etc/sysconfig/network-scripts/route-eth213.echo 'from 172.28.16.0/24 table ZW_LVS_LAN'> /etc/sysconfig/network-scripts/rule-eth214.echo '202 ZW_LVS_LAN'>> /etc/iproute2/rt_tables4、内核参数1.fs.file-max= 10000002.kernel.core_uses_pid = 13.kernel.msgmax = 10485604.kernel.msgmnb = 10737418245.kernel.shmall = 42949672966.kernel.shmmax = 687194767367.kernel.sysrq = 0dev_max_backlog = 1048576.core.rmem_default = 2097152.core.rmem_max = 16777216.core.somaxconn = 1048576.core.wmem_default = 2097152.core.wmem_max = 16777216.ipv4.conf.default.accept_source_route = 0.ipv4.conf.default.rp_filter = 1.ipv4.ip_forward = 1.ipv4.ip_local_port_range = 1024 65000.ipv4.neigh.default.gc_thresh1 = 10240.ipv4.neigh.default.gc_thresh2 = 40960.ipv4.neigh.default.gc_thresh3 = 81920.ipv4.tcp_fin_timeout = 1.ipv4.tcp_keepalive_intvl = 15.ipv4.tcp_keepalive_probes = 5.ipv4.tcp_keepalive_time = 30.ipv4.tcp_max_orphans = 3276800.ipv4.tcp_max_syn_backlog = 1048576.ipv4.tcp_max_tw_buckets = 50000.ipv4.tcp_mem = 94500000 915000000 927000000.ipv4.tcp_orphan_retries = 3.ipv4.tcp_reordering = 5.ipv4.tcp_retrans_collapse = 0.ipv4.tcp_retries2 = 5.ipv4.tcp_rmem = 4096 87380 4194304.ipv4.tcp_sack = 1.ipv4.tcp_synack_retries = 1.ipv4.tcp_syncookies = 0.ipv4.tcp_syn_retries = 1.ipv4.tcp_timestamps = 1.ipv4.tcp_tw_recycle = 1.ipv4.tcp_tw_reuse = 1.ipv4.tcp_window_scaling = 1.ipv4.tcp_wmem = 4096 16384 4194304.ipv6.conf.all.disable_ipv6 = 1.ipv6.conf.default.disable_ipv6 = 1.ipv4.conf.eth0.rp_filter = 1 #memched节点不需要.ipv4.conf.eth1.rp_filter = 1 #memched节点不需要.ipv4.conf.eth2.rp_filter = 1 #memched节点不需要.ipv4.conf.all.rp_filter = 0 #memched节点不需要三、服务安装1、调度机（master、backup）1.yum -y install ipvsadm keepalived sendmail2、memched节点1、memched节点21.wget /files/memcached-1.4.15.tar.gz2.wget /~provos/libevent-1.4.13-stable.tar.gz3.wget /repcached/memcached-1.2.8-repcached-2.2.tar.gz1.tar -zxvf libevent-1.4.13-stable.tar.gz2.cd libevent-1.4.13-stable3../configure --prefix=/usr/local4.make5.make install7.tar -zxvf memcached-1.4.15.tar.gz8.cd memcached-1.4.159../configure --with-libevent=/usr/local10.make11.make install12.13.tar -zxvf memcached-1.2.8-repcached-2.2.tar.gz14.cd memcached-1.2.8-repcached-2.215../configure --enable-replication16.make17.make install四、keepalived配置（master、backup）[root@LVS1 keepalived]# cat /etc/keepalived/gobal_module1.! global configure file2.3.global_defs {4.notification_email {5.navyaijm@6.}7.notification_email_from navyaijm@8.smtp_server 127.0.0.19.smtp_connect_timeout 3010.router_id ZH_DG_LVS111.}12.13.[root@LVS1 keepalived]# cat /etc/keepalived/keepalived.conf14.! Keepalived main configure file15.include /etc/keepalived/gobal_module16.include /etc/keepalived/vrrpd_module17.include /etc/keepalived/lvs_module18.[root@LVS1 keepalived]# cat /etc/keepalived/vrrpd_module1.vrrp_sync_group ZH_DG_Memcached1 {2.group{3.ZH_DG_WAN14.ZH_DG_LAN15.}6.smtp_alter7.}9.vrrp_instance ZH_DG_WAN1 {10.state MASTER （从上为：backup）11.interface eth112.virtual_router_id 10 （从为：9）13.priority 10014.advert_int 115.authentication {16.auth_type PASS17.auth_pass 08756CD018.}19.virtual_ipaddress {20.172.28.26.100 # vip21.}22.}23.24.25.vrrp_instance ZH_DG_LAN2 {26.state MASTER （从上为：backup）27.interface eth228.virtual_router_id 10 （从为：9）29.priority 10030.advert_int 131.authentication {32.auth_type PASS33.auth_pass 6F8DBC2E34.}35.virtual_ipaddress {36.172.28.16.100 # Memcached 虚拟网关37.}38.}[root@LVS1 keepalived]# cat /etc/keepalived/lvs_module1.virtual_server 172.28.26.100 11211 {2.delay_loop 63.lb_algo wrr4.lb_kind NAT5.persistence_timeout 606.protocol TCP7.8.include /etc/keepalived/realserver/172.28.26.102_11234.conf9.include /etc/keepalived/realserver/172.28.26.103_11233.conf10.}11.12.13.[root@LVS1 keepalived]# cat /etc/keepalived/realserver/172.28.26.102_11234.conf14.real_server 172.28.16.102 11234 {15.weight 1 #权重16.inhibit_on_failure #故障会修改权重为零17.TCP_CHECK {18.connect_timeout 10 #10秒无响应超时19.nb_get_retry 3 #重连次数20.delay_before_retry 3 #重连间隔，单位为秒21.connect_port 11234 #检测端口22.}23.}24.25.26.[root@LVS1 keepalived]# cat /etc/keepalived/realserver/172.28.26.103_11233.conf27.real_server 172.28.16.103 11233 {28.weight 1 #权重29.inhibit_on_failure #故障会修改权重为零30.TCP_CHECK {31.connect_timeout 10 #10秒无响应超时32.nb_get_retry 3 #重连次数33.delay_before_retry 3 #重连间隔，单位为秒34.connect_port 11233 #检测端口35.}36.}五、启动服务1、keepalived启动（master和backup）1./etc/init.d/keepalived startmemcached的启动主节点：1./usr/local/bin/memcached -d -v -l 0.0.0.0 -p 11233 -X 11244 -uroot （11244是数据同步端口）备节点：1./usr/local/bin/memcached -d -v -l 0.0.0.0 -p 11234 -uroot -x 172.28.16.103 -X 11244（11234 是服务监听端口，11244是监听数据同步端口；-x 指定masterIP，-X指定master数据同步监听端口）1.PS：2.1、如果启动的时候报错如下：3.[root@yw_memcached_slave ~]#/usr/local/bin/memcached -d -v -l 0.0.0.0 -p 11233 -X 11244 -uroot4./usr/local/bin/memcached: error while loading shared libraries: libevent-1.4.so.2: cannot open shared object file: No such file or directory5.请yum安装libeven6.[root@yw_memcached_slave ~]# yum -y install libeven7.2、 repcached是日本人开发的实现memcached复制功能，它是一个单master单slave的方案，但它的master/slave都是可读写的，而且可以相互同步，如果master坏掉，slave侦测到连接断了，它会自动listen而成为master；而如果slave 坏掉，master也会侦测到连接断，它就会重新listen等待新的slave加入；master没有抢占功能，如果master挂掉再起来只能是从了，并且永远只能是从。