在AWS ec2上實現Redis哨兵+複製+VIP漂移方案

目前Redis的HA架構中使用Redis sentinel哨兵模式的居多,使用該架構後,應用只需要連接到哨兵集群,再由哨兵提供主節點的IP位址,應用拿到主節點的地址後再次發起連接請求,該種方案成熟穩定,但是可以看到這種方式也有一定的弊端,即應用需要發起兩次連接,而且還要在代碼依賴的第三方庫支持哨兵部署,目前公司緩存方案採用的是AWS的elasticache服務,只提供了一個endpoint給應用,應用連接該endpoint後就會被轉發到對應的主節點,即使發生主從切換,應用不需要做出任何改變,通過endpoint還是可以連接到切換後的新的主節點,所以主從切換對應用來說也是透明的,如果我們想在哨兵模式中實現該endpoint功能,就需要在哨兵中引入VIP的方案,即應用連接的是VIP,默認VIP在主節點,當主從切換後,VIP漂移到了新的主節點,這樣就實現了elasticache的endpoint功能,目前我們準備採用該方案來實現elasticache到ec2 redis的數據遷移,這樣應用幾乎不要改動太多代碼,只需要在遷移完成後,將連接的endpoint修改為ec2的VIP即可.

集群規劃方案:

集群架構

在上述集群方案中

• VIP默認在主節點上,當哨兵檢測到主節點故障後,從節點failover變成主節點,同時VIP漂移到新的主節點上
• 從節點的虛擬網卡ens5:0默認處於關閉狀態,進行主從切換後由腳本啟動該網卡.

主節點和從節點的redis配置文件

bind 0.0.0.0protected-mode notcp-backlog 2048timeout 86400tcp-keepalive 0daemonize yessupervised systemdpidfile &34;loglevel noticelogfile &34;databases 64save 900 1save 300 10save 60 10000stop-writes-on-bgsave-error yesrdbcompression yesrdbchecksum yesdbfilename &34;dir &34;replica-serve-stale-data yesreplica-read-only yesrepl-diskless-sync norepl-diskless-sync-delay 5repl-ping-replica-period 5repl-timeout 30repl-disable-tcp-nodelay norepl-backlog-size 10mbrepl-backlog-ttl 7200replica-priority 100maxclients 65000maxmemory 14gbmaxmemory-policy allkeys-lrumaxmemory-samples 5replica-ignore-maxmemory yeslazyfree-lazy-eviction nolazyfree-lazy-expire nolazyfree-lazy-server-del noreplica-lazy-flush noappendonly noappendfilename &34;appendfsync everysecno-appendfsync-on-rewrite noauto-aof-rewrite-percentage 100auto-aof-rewrite-min-size 64mbaof-load-truncated yesslowlog-log-slower-than 10000slowlog-max-len 256aof-use-rdb-preamble yes

主節點和從節點的sentinel配置文件

protected-mode noport 26379daemonize yessupervised systemdpidfile &34;logfile &34;dir &34;sentinel myid eb94724e56ac043f0d38f002e40ff77793fa1434sentinel deny-scripts-reconfig yessentinel monitor userredis-b 10.191.102.157 6379 2sentinel down-after-milliseconds userredis-b 5000sentinel failover-timeout userredis-b 30000sentinel notification-script userredis-b /usr/local/redis/dba/shell/notify.shsentinel client-reconfig-script userredis-b /usr/local/redis/dba/shell/reconfig.sh

其中notification-script和client-reconfig-script參數只要在主節點和從節點上配置即可.

這裡注意最後面一行的參數,該行參數定義發生主從切換後sentinel執行的腳本,sentinel會向該攪拌傳遞8個參數,每個參數如下所示.

<master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>

其中第六個參數<to-ip>就是集群failover後新的主節點信息,我們在準備腳本的時候接收sentinel傳遞的第六個參數,相當於腳本知道了新的主節點信息,那麼我們就可以在腳本中定義在新的主節點和老的主節點上做的操作,例如在老的主節點將VIP解綁,並將VIP綁定到新的主節點上.

我們準備的切換VIP的腳本,AWS ec2的vip解綁和重新綁定到新的ec2的機器上需要調用aws專有的命令,如果是普通的伺服器的話,直接用ip address或者ifconfig將VIP配到新的機器即可.

The following arguments are passed to the script: <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port> <state> is currently always &34;34;leader&34;observer& the old address of the master and the new address of the elected replica define global envionment variablesec1_userredis_b_01_ids=&34;ec1_userredis_b_01_ip=&34;ec1_userredis_b_01_hostname=&34;ec1_userredis_b_02_ids=&34;ec1_userredis_b_02_ip=&34;ec1_userredis_b_02_hostname=&34;vip=&34;new_master_ip=$6email_list=&34;my_ip=`netstat -rn | grep UG | awk &39; | /usr/bin/xargs -i /sbin/ifconfig {} | grep &34; | awk &39;`my_if=`netstat -rn | grep UG| awk &39;`my_vif=&34;current_time=`date &34;` find out new master and old master instance nework interface IDnew_master_instance_interface_id=`/usr/local/bin/aws ec2 describe-instances --region us-east-1 --instance-ids ${new_master_instance_id} --query &39; --output text`old_master_instance_interface_id=`/usr/local/bin/aws ec2 describe-instances --region us-east-1 --instance-ids ${old_master_instance_id} --query &39; --output text` When Slave: Stop old master vitrual ip netork card if [ ${new_master_ip} = ${my_ip} ]; then retry_unassign_times=0 /usr/local/bin/aws ec2 unassign-private-ip-addresses --region us-east-1 --network-interface-id ${old_master_instance_interface_id} --private-ip-addresses ${vip} get_unassign_status=$? while [ ${get_unassign_status} -ne 0 ] do let retry_unassign_times++ if [ ${retry_unassign_times} == 3 ];then /bin/mail -s &34; ${email_list} < /dev/null exit 2 fi sleep 1 /usr/local/bin/aws ec2 unassign-private-ip-addresses --region us-east-1 --network-interface-id ${old_master_instance_interface_id} --private-ip-addresses ${vip} get_unassign_status=$? done /bin/mail -s &34; ${email_list} < /dev/null retry_assign_times=0 /usr/local/bin/aws ec2 assign-private-ip-addresses --region us-east-1 --allow-reassignment --network-interface-id ${new_master_instance_interface_id} --private-ip-addresses ${vip} get_assign_status=$? while [ ${get_assign_status} -ne 0 ] do let retry_assign_times++ if [ ${retry_assign_times} == 3 ];then /bin/mail -s &34; ${email_list} < /dev/null exit 2 fi sleep 1 /usr/local/bin/aws ec2 assign-private-ip-addresses --region us-east-1 --allow-reassignment --network-interface-id ${new_master_instance_interface_id} --private-ip-addresses ${vip} get_assign_status=$? done /bin/mail -s &34; ${email_list} < /dev/null retry_up_times=0 /sbin/ifup ${my_vif} get_up_status=$? while [ ${get_up_status} -ne 0 ] do let retry_up_times++ if [ ${retry_up_times} == 3 ];then /bin/mail -s &34; ${email_list} < /dev/null exit 2 fi sleep 1 /sbin/ifup ${my_vif} get_up_status=$? done /bin/mail -s &34; ${email_list} < /dev/null /sbin/arping -q -c 3 -A ${vip} -I ${my_if}else /sbin/ifdown ${my_vif} if [ $? -ne 0 ]; then /bin/mail -s &34; ${email_list} < /dev/null exit 2 else /bin/mail -s &34; ${email_list} < /dev/null fifi

這裡需要特別留意一下:

• 如果定義的腳本有語法錯誤等問題,sentinel不會告知你的腳本是否有錯.
• sentinel的進程以超級用戶root來運行,否則ifup,ifdown等命令sentinel無法正常調用.即使加了sudo並且給運行sentinel的用戶sudo的權限.
• 在初次運行時VIP需要手動綁定到主節點上.