使用kubeadm的方式搭建K8S高可用集群

2020-10-23 極客運維

PS: 最近經常有朋友問我有沒有用kubeadm搭建高可用集群的文檔，說實在的我確實沒有，我自己測試的話就用kubeadm單master版，公司用的話就用二進位搭建的。所以就找了個下班時間搭建測試了一番。希望對大家有幫助！如果覺得有用的話就幫忙點個關注或轉發吧，哈哈~

節點規劃信息

基礎環境配置

環境信息

環境初始化

環境初始化（1）、配置主機名，以k8s-master01為例

hostnamectl set-hostname k8s-master01

（1）、配置主機hosts映射

10.1.10.100 k8s-master0110.1.10.101 k8s-master0210.1.10.102 k8s-master0310.1.10.103 k8s-node0110.1.10.200 k8s-lb

配置完後可以通過如下命令測試

for host in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-lb;do ping -c 1 $host;done

這裡ping k8s-node01不通，是因為我們還沒配置VIP

（2）、禁用防火牆

systemctl stop firewalldsystemctl disable firewalld

（3）、關閉selinux

setenforce 0sed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/sysconfig/selinuxsed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config

（4）、關閉swap分區

swapoff -a && sysctl -w vm.swappiness=0

（5）、時間同步

yum install chrony -ysystemctl enable chronydsystemctl start chronydchronyc sources

（6）、配置ulimt

ulimit -SHn 65535

（7）、配置內核參數

cat >> /etc/sysctl.d/k8s.conf << EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1vm.swappiness=0EOF

使之生效

sysctl -p

（8）、master之間添加互信（按需）

ssh-keygenssh-copy-id 10.1.10.101ssh-copy-id 10.1.10.102

內核升級

由於centos7.6的系統默認內核版本是3.10，3.10的內核有很多BUG，最常見的一個就是group memory leak。
（1）、下載所需要的內核版本，我這裡採用rpm安裝，所以直接下載的rpm包

wget https://cbs.centos.org/kojifiles/packages/kernel/4.9.220/37.el7/x86_64/kernel-4.9.220-37.el7.x86_64.rpm

（2）、執行rpm升級即可

rpm -ivh kernel-4.9.220-37.el7.x86_64.rpm

（3）、升級完reboot，然後查看內核是否成功升級

rebootuname -r

組件安裝

安裝ipvs

（1）、安裝ipvs需要的軟體
由於我準備使用ipvs作為kube-proxy的代理模式，所以需要安裝相應的軟體包。

yum install ipvsadm ipset sysstat conntrack libseccomp -y

（2）、加載模塊

cat > /etc/sysconfig/modules/ipvs.modules <<EOF#!/bin/bashmodprobe -- ip_vsmodprobe -- ip_vs_rrmodprobe -- ip_vs_wrrmodprobe -- ip_vs_shmodprobe -- nf_conntrackmodprobe -- ip_tablesmodprobe -- ip_setmodprobe -- xt_setmodprobe -- ipt_setmodprobe -- ipt_rpfiltermodprobe -- ipt_REJECTmodprobe -- ipipEOF

注意：在內核4.19版本nf_conntrack_ipv4已經改為nf_conntrack

配置重啟自動加載

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

安裝docker-ce

# 安裝需要的軟體yum install -y yum-utils device-mapper-persistent-data lvm2# 添加yum源yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

查看是否有docker-ce包

# yum list | grep docker-cecontainerd.io.x86_64 1.2.13-3.1.el7 docker-ce-stabledocker-ce.x86_64 3:19.03.8-3.el7 docker-ce-stabledocker-ce-cli.x86_64 1:19.03.8-3.el7 docker-ce-stabledocker-ce-selinux.noarch 17.03.3.ce-1.el7 docker-ce-stable

安裝docker-ce

yum install docker-ce-19.03.8-3.el7 -ysystemctl start dockersystemctl enable docker

配置鏡像加速

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.iosystemctl restart docker

安裝kubernetes組件

添加yum源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=0repo_gpgcheck=0gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpgEOF

安裝軟體

yum install -y kubelet-1.18.2-0 kubeadm-1.18.2-0 kubectl-1.18.2-0 --disableexcludes=kubernetes

將kubelet設置為開機自啟動

systemctl enable kubelet.service

以上操作在所有節點執行

集群初始化

配置VIP

高可用採用的是HAProxy+Keepalived，HAProxy和KeepAlived以守護進程的方式在所有Master節點部署。

安裝軟體

yum install keepalived haproxy -y

配置haproxy

所有master節點的配置相同，如下：

#---------------------------------------------------------------------# Global settings#---------------------------------------------------------------------global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats#---------------------------------------------------------------------# common defaults that all the 'listen' and 'backend' sections will# use if not designated in their block#---------------------------------------------------------------------defaults mode http log global option httplog option dontlognull option http-server-close option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000#---------------------------------------------------------------------# kubernetes apiserver frontend which proxys to the backends#---------------------------------------------------------------------frontend kubernetes mode tcp bind *:16443 option tcplog default_backend kubernetes-apiserver#---------------------------------------------------------------------# round robin balancing between the various backends#---------------------------------------------------------------------backend kubernetes-apiserver mode tcp balance roundrobin server k8s-master01 10.1.10.100:6443 check server k8s-master02 10.1.10.101:6443 check server k8s-master03 10.1.10.102:6443 check#---------------------------------------------------------------------# collection haproxy statistics message#---------------------------------------------------------------------listen stats bind *:9999 stats auth admin:P@ssW0rd stats refresh 5s stats realm HAProxy\ Statistics stats uri /admin?stats

配置keepalived

k8s-master01

! Configuration File for keepalivedglobal_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr vrrp_garp_interval 0 vrrp_gna_interval 0}# 定義腳本vrrp_script check_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 2 weight -5 fall 3 rise 2 }vrrp_instance VI_1 { state MASTER interface eth33 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.1.10.200 } # 調用腳本 track_script { check_apiserver }}

k8s-master02

! Configuration File for keepalivedglobal_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr vrrp_garp_interval 0 vrrp_gna_interval 0}# 定義腳本vrrp_script check_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 2 weight -5 fall 3 rise 2 }vrrp_instance VI_1 { state MASTER interface eth33 virtual_router_id 51 priority 99 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.1.10.200 } # 調用腳本 track_script { check_apiserver }}

k8s-master03

! Configuration File for keepalivedglobal_defs { notification_email { acassen@firewall.loc failover@firewall.loc sysadmin@firewall.loc } notification_email_from Alexandre.Cassen@firewall.loc smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr vrrp_garp_interval 0 vrrp_gna_interval 0}# 定義腳本vrrp_script check_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 2 weight -5 fall 3 rise 2 }vrrp_instance VI_1 { state MASTER interface ens33 virtual_router_id 51 priority 98 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.1.10.200 } # 調用腳本 #track_script { # check_apiserver #}}

先把健康檢查關閉，等部署好了過後再打開

編寫健康檢測腳本check-apiserver.sh

#!/bin/bashfunction check_apiserver(){ for ((i=0;i<5;i++)) do apiserver_job_id=${pgrep kube-apiserver} if [[ ! -z ${apiserver_job_id} ]];then return else sleep 2 fi done apiserver_job_id=0}# 1->running 0->stoppedcheck_apiserverif [[ $apiserver_job_id -eq 0 ]];then /usr/bin/systemctl stop keepalived exit 1else exit 0fi

啟動haproxy和keepalived

systemctl enable --now keepalivedsystemctl enable --now haproxy

部署master

（1）、在k8s-master01上，編寫kubeadm.yaml配置文件，如下：

cat >> kubeadm.yaml <<EOFapiVersion: kubeadm.k8s.io/v1beta2kind: ClusterConfigurationkubernetesVersion: v1.18.2imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containerscontrolPlaneEndpoint: "k8s-lb:16443"networking: dnsDomain: cluster.local podSubnet: 192.168.0.0/16 serviceSubnet: 10.96.0.0/12---apiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationfeatureGates: SupportIPVSProxyMode: truemode: ipvsEOF

提前下載鏡像

kubeadm config images pull --config kubeadm.yaml

進行初始化，看到initialized successfully標識初始化成功。

kubeadm init --config kubeadm.yaml --upload-certs

.......Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of the control-plane node running the following command on each as root: kubeadm join k8s-lb:16443 --token 3k4vr0.x3y2nc3ksfnei4y1 \ --discovery-token-ca-cert-hash sha256:a5f761f332bd45a199d0676875e7f58c323226df6fb9b4f0b977b6f63b252791 \ --control-plane --certificate-key f25e738324e4f027703f24b55d47d28f692b4edc21c2876171ff87877dc8f2efPlease note that the certificate-key gives access to cluster sensitive data, keep it secret!As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:kubeadm join k8s-lb:16443 --token 3k4vr0.x3y2nc3ksfnei4y1 \ --discovery-token-ca-cert-hash sha256:a5f761f332bd45a199d0676875e7f58c323226df6fb9b4f0b977b6f63b252791

配置環境變量

cat >> /root/.bashrc <<EOFexport KUBECONFIG=/etc/kubernetes/admin.confEOFsource /root/.bashrc

查看節點狀態

# kubectl get nodesNAME STATUS ROLES AGE VERSIONk8s-master01 NotReady master 3m1s v1.18.2

安裝網絡插件

wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml

如果有節點是多網卡，所以需要在資源清單文件中指定內網網卡
vi calico.yaml

......spec: containers: - env: - name: DATASTORE_TYPE value: kubernetes - name: IP_AUTODETECTION_METHOD # DaemonSet中添加該環境變量 value: interface=ens33 # 指定內網網卡 - name: WAIT_FOR_DATASTORE value: "true"......

kubectl apply -f calico.yaml # 安裝calico網絡插件
當網絡插件安裝完成後，查看node節點信息如下：

# kubectl get nodesNAME STATUS ROLES AGE VERSIONk8s-master01 Ready master 10m v1.18.2

可以看到狀態已經從NotReady變為ready了。
（2）、將master02加入集群
提前下載鏡像

kubeadm config images pull --config kubeadm.yaml

加入集群

kubeadm join k8s-lb:16443 --token 3k4vr0.x3y2nc3ksfnei4y1 \ --discovery-token-ca-cert-hash sha256:a5f761f332bd45a199d0676875e7f58c323226df6fb9b4f0b977b6f63b252791 \ --control-plane --certificate-key f25e738324e4f027703f24b55d47d28f692b4edc21c2876171ff87877dc8f2ef

輸出如下：

...This node has joined the cluster and a new control plane instance was created:* Certificate signing request was sent to apiserver and approval was received.* The Kubelet was informed of the new secure connection details.* Control plane (master) label and taint were applied to the new node.* The Kubernetes control plane instances scaled up.* A new etcd member was added to the local/stacked etcd cluster.To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/configRun 'kubectl get nodes' to see this node join the cluster....

配置環境變量

cat >> /root/.bashrc <<EOFexport KUBECONFIG=/etc/kubernetes/admin.confEOFsource /root/.bashrc

另一臺的操作一樣。
查看集群狀態

# kubectl get nodes NAME STATUS ROLES AGE VERSIONk8s-master01 Ready master 41m v1.18.2k8s-master02 Ready master 29m v1.18.2k8s-master03 Ready master 27m v1.18.2

查看集群組件狀態

# kubectl get pod -n kube-system -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScalico-kube-controllers-77c5fc8d7f-stl57 1/1 Running 0 26m 192.168.32.130 k8s-master01 <none> <none>calico-node-ppsph 1/1 Running 0 26m 10.1.10.100 k8s-master01 <none> <none>calico-node-tl6sq 0/1 Init:2/3 0 26m 10.1.10.101 k8s-master02 <none> <none>calico-node-w92qh 1/1 Running 0 26m 10.1.10.102 k8s-master03 <none> <none>coredns-546565776c-vtlhr 1/1 Running 0 42m 192.168.32.129 k8s-master01 <none> <none>coredns-546565776c-wz9bk 1/1 Running 0 42m 192.168.32.131 k8s-master01 <none> <none>etcd-k8s-master01 1/1 Running 0 42m 10.1.10.100 k8s-master01 <none> <none>etcd-k8s-master02 1/1 Running 0 30m 10.1.10.101 k8s-master02 <none> <none>etcd-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>kube-apiserver-k8s-master01 1/1 Running 0 42m 10.1.10.100 k8s-master01 <none> <none>kube-apiserver-k8s-master02 1/1 Running 0 30m 10.1.10.101 k8s-master02 <none> <none>kube-apiserver-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>kube-controller-manager-k8s-master01 1/1 Running 1 42m 10.1.10.100 k8s-master01 <none> <none>kube-controller-manager-k8s-master02 1/1 Running 1 30m 10.1.10.101 k8s-master02 <none> <none>kube-controller-manager-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>kube-proxy-6sbpp 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>kube-proxy-dpppr 1/1 Running 0 42m 10.1.10.100 k8s-master01 <none> <none>kube-proxy-ln7l7 1/1 Running 0 30m 10.1.10.101 k8s-master02 <none> <none>kube-scheduler-k8s-master01 1/1 Running 1 42m 10.1.10.100 k8s-master01 <none> <none>kube-scheduler-k8s-master02 1/1 Running 1 30m 10.1.10.101 k8s-master02 <none> <none>kube-scheduler-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>

查看CSR

kubectl get csrNAME AGE SIGNERNAME REQUESTOR CONDITIONcsr-cfl2w 42m kubernetes.io/kube-apiserver-client-kubelet system:node:k8s-master01 Approved,Issuedcsr-mm7g7 28m kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:3k4vr0 Approved,Issuedcsr-qzn6r 30m kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:3k4vr0 Approved,Issued

部署node

node節點只需加入集群即可

kubeadm join k8s-lb:16443 --token 3k4vr0.x3y2nc3ksfnei4y1 \ --discovery-token-ca-cert-hash sha256:a5f761f332bd45a199d0676875e7f58c323226df6fb9b4f0b977b6f63b252791

輸出日誌如下：

W0509 23:24:12.159733 10635 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.[preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/[preflight] Reading configuration from the cluster...[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"[kubelet-start] Starting the kubelet[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster:* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

然後查看集群節點信息

# kubectl get nodes NAME STATUS ROLES AGE VERSIONk8s-master01 Ready master 47m v1.18.2k8s-master02 Ready master 35m v1.18.2k8s-master03 Ready master 32m v1.18.2k8s-node01 Ready node01 55s v1.18.2

測試切換

關閉一臺master主機，看集群是否可用。
關閉master01主機，然後查看整個集群。

# 模擬關掉keepalivedsystemctl stop keepalived# 然後查看集群是否可用[root@k8s-master03 ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONk8s-master01 Ready master 64m v1.18.2k8s-master02 Ready master 52m v1.18.2k8s-master03 Ready master 50m v1.18.2k8s-node01 Ready <none> 18m v1.18.2[root@k8s-master03 ~]# kubectl get pod -n kube-systemNAME READY STATUS RESTARTS AGEcalico-kube-controllers-77c5fc8d7f-stl57 1/1 Running 0 49mcalico-node-8t5ft 1/1 Running 0 19mcalico-node-ppsph 1/1 Running 0 49mcalico-node-tl6sq 1/1 Running 0 49mcalico-node-w92qh 1/1 Running 0 49mcoredns-546565776c-vtlhr 1/1 Running 0 65mcoredns-546565776c-wz9bk 1/1 Running 0 65metcd-k8s-master01 1/1 Running 0 65metcd-k8s-master02 1/1 Running 0 53metcd-k8s-master03 1/1 Running 0 51mkube-apiserver-k8s-master01 1/1 Running 0 65mkube-apiserver-k8s-master02 1/1 Running 0 53mkube-apiserver-k8s-master03 1/1 Running 0 51mkube-controller-manager-k8s-master01 1/1 Running 2 65mkube-controller-manager-k8s-master02 1/1 Running 1 53mkube-controller-manager-k8s-master03 1/1 Running 0 51mkube-proxy-6sbpp 1/1 Running 0 51mkube-proxy-dpppr 1/1 Running 0 65mkube-proxy-ln7l7 1/1 Running 0 53mkube-proxy-r5ltk 1/1 Running 0 19mkube-scheduler-k8s-master01 1/1 Running 2 65mkube-scheduler-k8s-master02 1/1 Running 1 53mkube-scheduler-k8s-master03 1/1 Running 0 51m

到此集群搭建完了，然後可以開啟keepalived的檢查腳本了。另外一些組件就自己自行安裝。

安裝自動補全命令

yum install -y bash-completionsource /usr/share/bash-completion/bash_completionsource <(kubectl completion bash)echo "source <(kubectl completion bash)" >> ~/.bashrc

使用kubeadm的方式搭建K8S高可用集群

節點規劃信息

基礎環境配置

環境信息

環境初始化

內核升級

組件安裝

安裝ipvs

安裝docker-ce

安裝kubernetes組件

集群初始化

配置VIP

安裝軟體

配置haproxy

配置keepalived

部署master

部署node

測試切換

安裝自動補全命令

相關焦點

kubeadm搭建高可用集群

k8s集群搭建

詳解kubeadm安裝k8s集群常見問題

使用Kubeadm設置Kubernetes集群的高可用性

kubeadm創建高可用集群v1.19.1

k8s版本平滑升級

利用kubeadm快速構建k8s-1.18.5環境

kubeadm創建k8s集群一直失敗，排查1天，最後竟然是

k8s集群CI/CD集成介紹二：rancher搭建k8s集群環境

搭建最新的k8s集群

半小時搭建一套帶Dashboard的kubernetes（K8S）集群

Centos7搭建k8s環境教程,一次性成功,收藏了!

kubeadm安裝kubernetes/k8s的詳細筆記(包括各種坑和注意事項)

五分鐘極速搭建kubernetes集群

k8s集群構建：官方推薦之KubeAdm安裝

我司基於K8s高可用集群架構

kubeadm部署單master節點

Centos7搭建k8s環境教程，一次性成功，收藏了

教你一次性成功安裝K8S集群(基於一主兩從模式)

用rancher2分分鐘搭建k8s集群