LVS/NAT + keepalived负载均衡实现-白红宇

LVS/NAT + keepalived负载均衡实现

阅读量：7234 次

发布时间：2019-06-29

本文共 11699 字，大约阅读时间需要 38 分钟。

一、keepalived简介

keepalived是分布式部署解决系统高可用的软件，结合lvs(LinuxVirtual Server)使用，解决单机宕机的问题。

keepalived是一个基于VRRP协议来实现IPVS的高可用的解决方案。对于LVS负载均衡来说，如果前端的调度器direct发生故障，则后端的realserver是无法接受请求并响应的。因此，保证前端direct的高可用性是非常关键的，否则后端的服务器是无法进行服务的。而我们的keepalived就可以用来解决单点故障（如LVS的前端direct故障）问题。keepalived的主要工作原理是：运行keepalived的两台服务器，其中一台为MASTER，另一台为BACKUP，正常情况下，所有的数据转换功能和ARP请求响应都是由MASTER完成的，一旦MASTER发生故障，则BACKUP会马上接管MASTER的工作，这种切换时非常迅速的。

二、测试环境

下面拿4台虚拟机进行环境测试,实验环境为centos6.6 x86_64,在lvs NAT模式环境下只有前端两台keepalived调度机有公网ip，后端真实机只有内网ip，具体用途和ip如下

服务器类型	公网ip	内网ip
LVS VIP	192.168.214.70	192.168.211.254
Keepalived Master	192.168.214.76	192.168.211.76
Keepalived Backup	192.168.214.77	192.168.211.77
Realserver A		192.168.211.79
Realserver B		192.168.211.83

三、软件安装

1、安装lvs所需包ipvsadm

yum install -y ipvsadm

ln -s /usr/src/kernels/`uname -r` /usr/src/linux

lsmod |grep ip_vs

#注意Centos 6.X安装lvs，使用1.26版本。并且需要先安装yuminstall libnl* popt* -y

执行ipvsadm(modprobe ip_vs)把ip_vs模块加载到内核

[root@test85 ~]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

#IP Virtual Server version 1.2.1 ---- ip_vs内核模块版本

2、安装keepalived

yum install -y keepalived

chkconfig keepalived on

注：在centos7系列系统中开机自动启动使用systemctl enable keepalived

四、开启路由转发

在lvs/NAT模式下前端两台调度机192.168.211.76和192.168.211.77上需要开启路由转发功能

vi /etc/sysctl.conf

修改net.ipv4.ip_forward = 0为net.ipv4.ip_forward= 1

保存退出后，使用systcl –p命令让其生效

五、keepalived配置

先看下211.76 keepalivedmaster主机上的配置

具体详细的参数说明见上篇LVS/DR + keepalived负载均衡实现

[root@localhost ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalivedglobal_defs {   notification_email {     charles@test.com    }   notification_email_from ReportLog@test.com    #smtp_server 127.0.0.1    #smtp_connect_timeout 30    router_id LVS_1   #lvs_id LVS_EXAMPLE_01 }vrrp_sync_group VG1 {   group {      VI_1      VI_GATEWAY   }} vrrp_instance VI_1 {    state MASTER    interface eth0    #公网ip网口    virtual_router_id 51    priority 150    advert_int 1    lvs_sync_daemon_inteface eth0    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        192.168.214.70    #公网vip地址    }}vrrp_instance VI_GATEWAY {    state MASTER    interface eth1         #内网ip网口    lvs_sync_daemon_inteface eth1    virtual_router_id 52    priority 150    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        192.168.211.254  #内网vip地址    }}virtual_server 192.168.214.70 80 {    delay_loop 6    lb_algo rr    lb_kind NAT         #使用lvs的NAT工作模式    #nat_mask 255.255.255.0    persistence_timeout 10     protocol TCP    real_server 192.168.211.79 80 {        TCP_CHECK {            connect_timeout 3            connect_port 80            nb_get_retry 3            delay_before_retry 3        }    }   real_server 192.168.211.83 80 {        TCP_CHECK {            connect_timeout 3            connect_port 80            nb_get_retry 3            delay_before_retry 3        }    }}

再看下211.77 keepalived BACKUP主机上的配置

[root@localhost ~]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalivedglobal_defs {   notification_email {     charles@test.com    }   notification_email_from ReportLog@test.com    #smtp_server 127.0.0.1    #smtp_connect_timeout 30   router_id LVS_1   #lvs_id LVS_EXAMPLE_02 }vrrp_sync_group VG1 {   group {      VI_1      VI_GATEWAY   }} vrrp_instance VI_1 {    state BACKUP    interface eth0    lvs_sync_daemon_inteface eth0    virtual_router_id 51    priority 100    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        192.168.214.70     }}vrrp_instance VI_GATEWAY {    state BACKUP    interface eth1    lvs_sync_daemon_inteface eth1    virtual_router_id 52    priority 100    advert_int 1    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        192.168.211.254     }}virtual_server 192.168.214.70 80 {    delay_loop 6    lb_algo rr    lb_kind NAT    #nat_mask 255.255.255.0    persistence_timeout 10    protocol TCP real_server 192.168.211.79 80 {        TCP_CHECK {            connect_timeout 3            connect_port 80            nb_get_retry 3            delay_before_retry 3        }    }real_server 192.168.211.83 80 {        TCP_CHECK {            connect_timeout 3            connect_port 80            nb_get_retry 3            delay_before_retry 3        }    }}

六、后端真实机配置

后端两台web服务真实机192.168.211.79和192.168.211.83需要配置默认网关为vip内网ip地址192.168.211.254。

注意，线上环境如果更改默认网关为vip内网地址后，可能会造成连接不上服务器，需要提前添加相关静态路由。

七、启动keepalived服务及查看相关信息

在211.76和211.77上分别启动keepalived服务

在211.76keepalived Master主机上查看信息

通过ip addr可以看到vip 地址已经绑定在eth0和eth1网口

[root@localhost ~]# ip addr1: lo: 
     
       mtu65536 qdisc noqueue state UNKNOWN    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00   inet 127.0.0.1/8 scope host lo   inet6 ::1/128 scope host       valid_lft forever preferred_lft forever2: eth0:
      
        mtu 1500 qdisc pfifo_fast state UP qlen1000   link/ether 52:54:00:55:b2:d4 brd ff:ff:ff:ff:ff:ff   inet 192.168.214.76/24 brd 192.168.214.255 scope global eth0   inet 192.168.214.70/32 scope global eth0   inet6 fe80::5054:ff:fe55:b2d4/64 scope link       valid_lft forever preferred_lft forever3: eth1:
       
         mtu 1500 qdisc pfifo_fast state UP qlen1000   link/ether 52:54:00:85:11:95 brd ff:ff:ff:ff:ff:ff   inet 192.168.211.76/24 brd 192.168.211.255 scope global eth1   inet 192.168.211.254/32 scope global eth1   inet6 fe80::5054:ff:fe85:1195/64 scope link       valid_lft forever preferred_lft forever在211.76上查看日志信息，看到已成功进入keepalived主机模式[root@localhost ~]# tail -f/var/log/messagesMay 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_GATEWAY) setting protocol VIPs.May 10 10:25:47 localhost Keepalived_healthcheckers[6209]:Netlink reflector reports IP 192.168.211.254 addedMay 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_GATEWAY) Sending gratuitous ARPs oneth1 for 192.168.211.254May 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Group(VG1) Syncing instances to MASTER stateMay 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) EnteringMASTER STATEMay 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) setting protocol VIPs.May 10 10:25:47 localhost Keepalived_healthcheckers[6209]:Netlink reflector reports IP 192.168.214.70 addedMay 10 10:25:47 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) Sendinggratuitous ARPs on eth0 for 192.168.214.70May 10 10:25:52 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_GATEWAY) Sendinggratuitous ARPs on eth1 for 192.168.211.254May 10 10:25:52 localhostKeepalived_vrrp[6210]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for192.168.214.70

在211.77上查看日志信息，看到已成功进入keepalived备机模式

[root@localhost ~]# tail -f/var/log/messagesMay 10 10:27:54 localhostKeepalived_vrrp[6249]: Opening file '/etc/keepalived/keepalived.conf'.May 10 10:27:54 localhostKeepalived_vrrp[6249]: Configuration is using : 69554 BytesMay 10 10:27:54 localhostKeepalived_vrrp[6249]: Using LinkWatch kernel netlink reflector...May 10 10:27:54 localhostKeepalived_vrrp[6249]: VRRP_Instance(VI_1) Entering BACKUP STATEMay 10 10:27:54 localhostKeepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Entering BACKUP STATEMay 10 10:27:54 localhost Keepalived_vrrp[6249]:VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]May 10 10:27:54 localhostKeepalived_vrrp[6249]: VRRP sockpool: [ifindex(3), proto(112), unicast(0),fd(12,13)]May 10 10:27:54 localhostKeepalived_healthcheckers[6248]: Using LinkWatch kernel netlink reflector...May 10 10:27:54 localhostKeepalived_healthcheckers[6248]: Activating healthchecker for service[192.168.211.79]:80May 10 10:27:54 localhostKeepalived_healthcheckers[6248]: Activating healthchecker for service[192.168.211.83]:80

通过ipvsadm -L –n 查看相应lvs连接信息

[root@localhost ~]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.214.70:80 rr persistent 10

-> 192.168.211.79:80 Masq 1 0 0

-> 192.168.211.83:80 Masq 1 3 0

八、keepalived测试

使用vip地址192.168.214.70访问后端192.168.211.79和192.168.211.83的页面

正常访问没问题后，我们来模拟lvs集群故障，让主备进行自动切换下

首先把keepalived master主机211.76宕机，看备机能否接管过来，vip地址是否会漂移过来

在211.77上查看日志，发现备机成功切换成了主机状态，vip地址成功漂移了过来

[root@localhost ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen1000

link/ether 52:54:00:1b:a2:11 brd ff:ff:ff:ff:ff:ff

inet 192.168.214.77/24 brd 192.168.214.255 scope global eth0

inet 192.168.214.70/32 scope global eth0

inet6 fe80::5054:ff:fe1b:a211/64 scope link

valid_lft forever preferred_lft forever

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP>mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 52:54:00:64:47:7d brd ff:ff:ff:ff:ff:ff

inet 192.168.211.77/24 brd 192.168.211.255 scope global eth1

inet 192.168.211.254/32 scope global eth1

inet6 fe80::5054:ff:fe64:477d/64 scope link

valid_lft forever preferred_lft forever

[root@localhost ~]# tail -f /var/log/messagesMay 10 10:27:54 localhost Keepalived_vrrp[6249]: Opening file '/etc/keepalived/keepalived.conf'.May 10 10:27:54 localhost Keepalived_vrrp[6249]: Configuration is using : 69554 BytesMay 10 10:27:54 localhost Keepalived_vrrp[6249]: Using LinkWatch kernel netlink reflector...May 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Entering BACKUP STATEMay 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Entering BACKUP STATEMay 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]May 10 10:27:54 localhost Keepalived_vrrp[6249]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(12,13)]May 10 10:27:54 localhost Keepalived_healthcheckers[6248]: Using LinkWatch kernel netlink reflector...May 10 10:27:54 localhost Keepalived_healthcheckers[6248]: Activating healthchecker for service [192.168.211.79]:80May 10 10:27:54 localhost Keepalived_healthcheckers[6248]: Activating healthchecker for service [192.168.211.83]:80May 10 11:09:26 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Transition to MASTER STATEMay 10 11:09:26 localhost Keepalived_vrrp[6249]: VRRP_Group(VG1) Syncing instances to MASTER stateMay 10 11:09:26 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Transition to MASTER STATEMay 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Entering MASTER STATEMay 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) setting protocol VIPs.May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Sending gratuitous ARPs on eth1 for 192.168.211.254May 10 11:09:27 localhost Keepalived_healthcheckers[6248]: Netlink reflector reports IP 192.168.211.254 addedMay 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Entering MASTER STATEMay 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) setting protocol VIPs.May 10 11:09:27 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.70May 10 11:09:27 localhost Keepalived_healthcheckers[6248]: Netlink reflector reports IP 192.168.214.70 addedMay 10 11:09:32 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_GATEWAY) Sending gratuitous ARPs on eth1 for 192.168.211.254May 10 11:09:32 localhost Keepalived_vrrp[6249]: VRRP_Instance(VI_1) Sending

如果想了解更多，请关注我们的公众号

公众号ID:opdevos

扫码关注