Kubernetes 1.14 二進位集群安裝

Kubernetes 1.14 二進位集群安裝

Kubernetes

本系列文檔將介紹如何使用二進位部署Kubernetes v1.14集群的所有部署，而不是使用自動化部署(kubeadm)集群。在部署過程中，將詳細列出各個組件啟動參數，以及相關配置說明。在學習完本文檔後，將理解k8s各個組件的交互原理，並且可以快速解決實際問題。

本文檔適用於Centos7.4及以上版本，隨著各個組件的更新，本文檔提供了相關鏡像的包，及時版本更新也不會影響文檔的使用。 如果有文檔相關問題可以直接在網站下面註冊回復，或者點擊右下角加群，我將在12小時內回復您。 並且建議您使用的環境及配置和我相同！

組件版本

Kubernetes 1.14.2Docker 18.09 (docker使用官方的腳本安裝，後期可能升級為新的版本，但是不影響)Etcd 3.3.13Flanneld 0.11.0組件說明

kube-apiserver

使用節點本地Nginx 4層透明代理實現高可用 (也可以使用haproxy，只是起到代理apiserver的作用)關閉非安全埠8080和匿名訪問使用安全埠6443接受https請求嚴格的認知和授權策略 (x509、token、rbac)開啟bootstrap token認證，支持kubelet TLS bootstrapping；使用https訪問kubelet、etcdkube-controller-manager

3節點高可用 (在k8s中，有些組件需要選舉，所以使用奇數為集群高可用方案)關閉非安全埠，使用10252接受https請求使用kubeconfig訪問apiserver的安全扣使用approve kubelet證書籤名請求(CSR)，證書過期後自動輪轉各controller使用自己的ServiceAccount訪問apiserverkube-scheduler

3節點高可用；使用kubeconfig訪問apiserver安全埠kubelet

使用kubeadm動態創建bootstrap token使用TLS bootstrap機制自動生成client和server證書，過期後自動輪轉在kubeletConfiguration類型的JSON文件配置主要參數關閉只讀埠，在安全埠10250接受https請求，對請求進行認真和授權，拒絕匿名訪問和非授權訪問使用kubeconfig訪問apiserver的安全埠kube-proxy

使用kubeconfig訪問apiserver的安全埠在KubeProxyConfiguration類型JSON文件配置為主要參數使用ipvs代理模式集群插件

DNS 使用功能、性能更好的coredns網絡 使用Flanneld 作為集群網絡插件一、初始化環境

集群機器

192.168.0.50 k8s-01192.168.0.51 k8s-02192.168.0.52 k8s-03#node節點192.168.0.53 k8s-04#node節點只運行node，但是設置證書的時候要添加這個ip本文檔的所有etcd集群、master集群、worker節點均使用以上三臺機器，並且初始化步驟需要在所有機器上執行命令。如果沒有特殊命令，所有操作均在192.168.0.50上進行操作

node節點後面會有操作，但是在初始化這步，是所有集群機器。包括node節點，我上面沒有列出node節點

修改主機名所有機器設置永久主機名

hostnamectl set-hostname abcdocker-k8s01 #所有機器按照要求修改bash #刷新主機名接下來我們需要在所有機器上添加hosts解析

cat >>/etc/hosts <<EOF192.168.0.50 k8s-01192.168.0.51 k8s-02192.168.0.52 k8s-03192.168.0.53 k8s-04EOF設置免密我們只在k8s-01上設置免密即可

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repocurl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repoyum install -y expect#分發公鑰ssh-keygen -t rsa -P ""-f /root/.ssh/id_rsafor i in k8s-01 k8s-02 k8s-03 k8s-04;doexpect -c "spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i expect { \"*yes/no*\" {send \"yes\r\"; exp_continue} \"*password*\" {send \"123456\r\"; exp_continue} \"*Password*\" {send \"123456\r\";} } "done#我這裡密碼是123456 大家按照自己主機的密碼進行修改就可以更新PATH變量本次的k8s軟體包的目錄全部存放在/opt下

[root@abcdocker-k8s01 ~]# echo 'PATH=/opt/k8s/bin:$PATH'>>/etc/profile[root@abcdocker-k8s01 ~]# source /etc/profile[root@abcdocker-k8s01 ~]# env|grep PATHPATH=/opt/k8s/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin安裝依賴包在每臺伺服器上安裝依賴包

yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget關閉防火牆 Linux 以及swap分區

systemctl stop firewalldsystemctl disable firewalldiptables -F && iptables -X && iptables -F -t nat && iptables -X -t natiptables -P FORWARD ACCEPTswapoff -ased -i '/ swap / s/^\(.*\)$/#\1/g'/etc/fstabsetenforce 0sed -i 's/^SELINUX=.*/SELINUX=disabled/'/etc/selinux/config#如果開啟了swap分區，kubelet會啟動失敗(可以通過設置參數——-fail-swap-on設置為false)升級內核Docker overlay2需要使用kernel 4.x版本，所以我們需要升級內核 我這裡的內核使用4.18.9

CentOS 7.x 系統自帶的 3.10.x 內核存在一些 Bugs，導致運行的 Docker、Kubernetes 不穩定，例如：

高版本的 docker(1.13 以後) 啟用了 3.10 kernel 實驗支持的 kernel memory account 功能(無法關閉)，當節點壓力大如頻繁啟動和停止容器時會導致 cgroup memory leak；網絡設備引用計數洩漏，會導致類似於報錯："kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1";解決方案如下：

升級內核到 4.4.X 以上；或者，手動編譯內核，disable CONFIG_MEMCG_KMEM 特性；或者，安裝修復了該問題的 Docker 18.09.1 及以上的版本。但由於 kubelet 也會設置 kmem（它 vendor 了 runc），所以需要重新編譯 kubelet 並指定 GOFLAGS="-tags=nokmem"；exportKernel_Version=4.18.9-1wget http://mirror.rc.usf.edu/compute_lock/elrepo/kernel/el7/x86_64/RPMS/kernel-ml{,-devel}-${Kernel_Version}.el7.elrepo.x86_64.rpmyum localinstall -y kernel-ml*#如果是手動下載內核rpm包，直接執行後面yum install -y kernel-ml*即可修改內核啟動順序,默認啟動的順序應該為1,升級以後內核是往前面插入,為0（如果每次啟動時需要手動選擇哪個內核,該步驟可以省略）

grub2-set-default0&& grub2-mkconfig -o /etc/grub2.cfg使用下面命令看看確認下是否啟動默認內核指向上面安裝的內核

grubby --default-kernel#這裡的輸出結果應該為我們升級後的內核信息重啟加載新內核 （升級完內核順便update一下）

reboot加載內核模塊首先我們要檢查是否存在所需的內核模塊

find /lib/modules/`uname -r`/-name "ip_vs_rr*"find /lib/modules/`uname -r`/-name "br_netfilter*"1.加載內核，加入開機啟動 (2選1即可)

cat >/etc/rc.local<< EOFmodprobe ip_vs_rrmodprobe br_netfilterEOF2.使用systemd-modules-load加載內核模塊

cat >/etc/modules-load.d/ipvs.conf << EOF ip_vs_rr br_netfilterEOFsystemctl enable --now systemd-modules-load.service驗證模塊是否加載成功

lsmod |egrep " ip_vs_rr|br_netfilter"為什麼要使用IPVS,從k8s的1.8版本開始，kube-proxy引入了IPVS模式，IPVS模式與iptables同樣基於Netfilter，但是採用的hash表，因此當service數量達到一定規模時，hash查表的速度優勢就會顯現出來，從而提高service的服務性能。ipvs依賴於nf_conntrack_ipv4內核模塊,4.19包括之後內核裡改名為nf_conntrack,1.13.1之前的kube-proxy的代碼裡沒有加判斷一直用的nf_conntrack_ipv4,好像是1.13.1後的kube-proxy代碼裡增加了判斷,我測試了是會去load nf_conntrack使用ipvs正常優化內核參數

cat > kubernetes.conf <<EOFnet.bridge.bridge-nf-call-iptables=1net.bridge.bridge-nf-call-ip6tables=1net.ipv4.ip_forward=1net.ipv4.tcp_tw_recycle=0vm.swappiness=0# 禁止使用 swap 空間，只有當系統 OOM 時才允許使用它vm.overcommit_memory=1# 不檢查物理內存是否夠用vm.panic_on_oom=0# 開啟 OOMfs.inotify.max_user_instances=8192fs.inotify.max_user_watches=1048576fs.file-max=52706963fs.nr_open=52706963net.ipv6.conf.all.disable_ipv6=1net.netfilter.nf_conntrack_max=2310720EOFcp kubernetes.conf /etc/sysctl.d/kubernetes.confsysctl -p /etc/sysctl.d/kubernetes.conf需要關閉tcp_tw_recycle，否則和NAT衝突，會導致服務不通 關閉IPV6，防止觸發Docker BUG

設置系統時區

timedatectl set-timezone Asia/Shanghai#將當前的 UTC 時間寫入硬體時鐘timedatectl set-local-rtc 0#重啟依賴於系統時間的服務systemctl restart rsyslog systemctl restart crond創建相關目錄

mkdir -p /opt/k8s/{bin,work}/etc/{kubernetes,etcd}/cert設置分髮腳本參數後續所有的使用環境變量都定義在environment.sh中，需要根據個人機器及網絡環境修改。並且需要拷貝到所有節點的/opt/k8s/bin目錄下

#!/usr/bin/bash# 生成 EncryptionConfig 所需的加密 keyexport ENCRYPTION_KEY=$(head -c 32/dev/urandom | base64)# 集群各機器 IP 數組export NODE_IPS=(192.168.0.50192.168.0.51192.168.0.52192.168.0.53)# 集群各 IP 對應的主機名數組export NODE_NAMES=(k8s-01 k8s-02 k8s-03 k8s-04)# 集群MASTER機器 IP 數組export MASTER_IPS=(192.168.0.50192.168.0.51192.168.0.52)# 集群所有的master Ip對應的主機export MASTER_NAMES=(k8s-01 k8s-02 k8s-03)# etcd 集群服務地址列表export ETCD_ENDPOINTS="https://192.168.0.50:2379,https://192.168.0.51:2379,https://192.168.0.52:2379"# etcd 集群間通信的 IP 和埠export ETCD_NODES="k8s-01=https://192.168.0.50:2380,k8s-02=https://192.168.0.51:2380,k8s-03=https://192.168.0.52:2380"# etcd 集群所有node ipexport ETCD_IPS=(192.168.0.50192.168.0.51192.168.0.52192.168.0.53)# kube-apiserver 的反向代理(kube-nginx)地址埠export KUBE_APISERVER="https://192.168.0.54:8443"# 節點間網際網路接口名稱export IFACE="eth0"# etcd 數據目錄export ETCD_DATA_DIR="/data/k8s/etcd/data"# etcd WAL 目錄，建議是 SSD 磁碟分區，或者和 ETCD_DATA_DIR 不同的磁碟分區export ETCD_WAL_DIR="/data/k8s/etcd/wal"# k8s 各組件數據目錄export K8S_DIR="/data/k8s/k8s"# docker 數據目錄#export DOCKER_DIR="/data/k8s/docker"## 以下參數一般不需要修改# TLS Bootstrapping 使用的 Token，可以使用命令 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成#BOOTSTRAP_TOKEN="41f7e4ba8b7be874fcff18bf5cf41a7c"# 最好使用 當前未用的網段 來定義服務網段和 Pod 網段# 服務網段，部署前路由不可達，部署後集群內路由可達(kube-proxy 保證)SERVICE_CIDR="10.254.0.0/16"# Pod 網段，建議 /16 段地址，部署前路由不可達，部署後集群內路由可達(flanneld 保證)CLUSTER_CIDR="172.30.0.0/16"# 服務埠範圍 (NodePort Range)export NODE_PORT_RANGE="1024-32767"# flanneld 網絡配置前綴export FLANNEL_ETCD_PREFIX="/kubernetes/network"# kubernetes 服務 IP (一般是 SERVICE_CIDR 中第一個IP)export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"# 集群 DNS 服務 IP (從 SERVICE_CIDR 中預分配)export CLUSTER_DNS_SVC_IP="10.254.0.2"# 集群 DNS 域名（末尾不帶點號）export CLUSTER_DNS_DOMAIN="cluster.local"# 將二進位目錄 /opt/k8s/bin 加到 PATH 中export PATH=/opt/k8s/bin:$PATH請根據IP進行修改

分發環境變量腳本

source environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp environment.sh root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip}"chmod +x /opt/k8s/bin/* "done二、k8s集群部署

創建CA證書和秘鑰為確保安全，kubernetes各個組件需要使用x509證書對通信進行加密和認證 CA(Certificate Authority)是自籤名的根證書，用來籤名後續創建的其他證書。本文章使用CloudFlare的PKI工具cfssl創建所有證書。

注意:如果沒有特殊指明，本文檔的所有操作均在k8s-01節點執行，遠程分發到其他節點

安裝cfssl工具集

mkdir -p /opt/k8s/cert && cd /opt/k8swget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64mv cfssl_linux-amd64 /opt/k8s/bin/cfsslwget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64mv cfssljson_linux-amd64 /opt/k8s/bin/cfssljsonwget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64mv cfssl-certinfo_linux-amd64 /opt/k8s/bin/cfssl-certinfochmod +x /opt/k8s/bin/*export PATH=/opt/k8s/bin:$PATH創建根證書 (CA)

CA證書是集群所有節點共享的，只需要創建一個CA證書，後續創建的所有證書都是由它籤名

創建配置文件

CA配置文件用於配置根證書的使用場景(profile)和具體參數 (usage、過期時間、服務端認證、客戶端認證、加密等)

cd /opt/k8s/workcat > ca-config.json <<EOF{"signing":{"default":{"expiry":"87600h"},"profiles":{"kubernetes":{"usages":["signing","key encipherment","server auth","client auth"],"expiry":"87600h"}}}}EOF######################signing 表示該證書可用於籤名其它證書，生成的ca.pem證書找中CA=TRUEserver auth 表示client可以用該證書對server提供的證書進行驗證client auth 表示server可以用該證書對client提供的證書進行驗證創建證書籤名請求文件

cd /opt/k8s/workcat > ca-csr.json <<EOF{"CN":"kubernetes","key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"k8s","OU":"4Paradigm"}],"ca":{"expiry":"876000h"}}EOF#######################CN CommonName,kube-apiserver從證書中提取該欄位作為請求的用戶名(UserName)，瀏覽器使用該欄位驗證網站是否合法O Organization,kube-apiserver 從證書中提取該欄位作為請求用戶和所屬組(Group)kube-apiserver將提取的User、Group作為RBAC授權的用戶和標識生成CA證書和私鑰

cd /opt/k8s/workcfssl gencert -initca ca-csr.json | cfssljson -bare cals ca*分發證書

#將生成的CA證書、秘鑰文件、配置文件拷貝到所有節點的/etc/kubernetes/cert目錄下cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p /etc/kubernetes/cert" scp ca*.pem ca-config.json root@${node_ip}:/etc/kubernetes/certdone部署kubectl命令行工具kubectl默認從~/.kube/config讀取kube-apiserver地址和認證信息。kube/config只需要部署一次，生成的kubeconfig文件是通用的

下載和解壓kubectl

cd /opt/k8s/workwget http://down.i4t.com/k8s1.14/kubernetes-client-linux-amd64.tar.gztar -xzvf kubernetes-client-linux-amd64.tar.gz分發所有使用kubectl節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp kubernetes/client/bin/kubectl root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip}"chmod +x /opt/k8s/bin/*"done創建admin證書和私鑰kubectl與apiserver https通信，apiserver對提供的證書進行認證和授權。kubectl作為集群的管理工具，需要被授予最高權限，這裡創建具有最高權限的admin證書

創建證書籤名請求

cd /opt/k8s/workcat > admin-csr.json <<EOF{"CN":"admin","hosts":[],"key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"system:masters","OU":"4Paradigm"}]}EOF###################● O 為system:masters，kube-apiserver收到該證書後將請求的Group設置為system:masters●預定的ClusterRoleBinding cluster-admin將Group system:masters與Role cluster-admin綁定，該Role授予API的權限●該證書只有被kubectl當做client證書使用，所以hosts欄位為空生成證書和私鑰

cd /opt/k8s/workcfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes admin-csr.json | cfssljson -bare adminls admin*創建kubeconfig文件kubeconfig為kubectl的配置文件，包含訪問apiserver的所有信息，如apiserver地址、CA證書和自身使用的證書

cd /opt/k8s/worksource /opt/k8s/bin/environment.sh# 設置集群參數kubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kubectl.kubeconfig#設置客戶端認證參數kubectl config set-credentials admin \--client-certificate=/opt/k8s/work/admin.pem \--client-key=/opt/k8s/work/admin-key.pem \--embed-certs=true \--kubeconfig=kubectl.kubeconfig# 設置上下文參數kubectl config set-context kubernetes \--cluster=kubernetes \--user=admin \--kubeconfig=kubectl.kubeconfig# 設置默認上下文kubectl config use-context kubernetes --kubeconfig=kubectl.kubeconfig################--certificate-authority 驗證kube-apiserver證書的根證書--client-certificate、--client-key 剛生成的admin證書和私鑰，連接kube-apiserver時使用--embed-certs=true將ca.pem和admin.pem證書嵌入到生成的kubectl.kubeconfig文件中(如果不加入，寫入的是證書文件路徑，後續拷貝kubeconfig到其它機器時，還需要單獨拷貝證書)分發kubeconfig文件 分發到所有使用kubectl命令的節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p ~/.kube" scp kubectl.kubeconfig root@${node_ip}:~/.kube/configdone#保存文件名為~/.kube/config部署ETCD集群這裡使用的ETCD為三節點高可用集群，步驟如下

下載和分發etcd二進位文件創建etcd集群各節點的x509證書，用於加密客戶端(如kubectl)與etcd集群、etcd集群之間的數據流創建etcd的system unit文件，配置服務參數檢查集群工作狀態 etcd集群各節點的名稱和IP如下 k8s-01 192.168.0.50 k8s-02 192.168.0.51 k8s-03 192.168.0.52注意: 沒有特殊說明都在k8s-01節點操作下載和分發etcd二進位文件

cd /opt/k8s/workwget http://down.i4t.com/k8s1.14/etcd-v3.3.13-linux-amd64.tar.gztar -xvf etcd-v3.3.13-linux-amd64.tar.gz分發二進位文件到集群節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${ETCD_IPS[@]}do echo ">>> ${node_ip}" scp etcd-v3.3.13-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin ssh root@${node_ip}"chmod +x /opt/k8s/bin/*"done創建etcd證書和私鑰

cd /opt/k8s/workcat > etcd-csr.json <<EOF{"CN":"etcd","hosts":["127.0.0.1","192.168.0.50","192.168.0.51","192.168.0.52"],"key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"k8s","OU":"4Paradigm"}]}EOF#host欄位指定授權使用該證書的etcd節點IP或域名列表，需要將etcd集群的3個節點都添加其中生成證書和私鑰

cd /opt/k8s/workcfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes etcd-csr.json | cfssljson -bare etcdls etcd*pem分發證書和私鑰到etcd各個節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${ETCD_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p /etc/etcd/cert" scp etcd*.pem root@${node_ip}:/etc/etcd/cert/done創建etcd的啟動文件 (這裡將配置文件也存放在啟動文件裡)

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > etcd.service.template<<EOF[Unit]Description=EtcdServerAfter=network.targetAfter=network-online.targetWants=network-online.targetDocumentation=https://github.com/coreos[Service]Type=notifyWorkingDirectory=${ETCD_DATA_DIR}ExecStart=/opt/k8s/bin/etcd \\--data-dir=${ETCD_DATA_DIR} \\--wal-dir=${ETCD_WAL_DIR} \\--name=##NODE_NAME## \\--cert-file=/etc/etcd/cert/etcd.pem \\--key-file=/etc/etcd/cert/etcd-key.pem \\--trusted-ca-file=/etc/kubernetes/cert/ca.pem \\--peer-cert-file=/etc/etcd/cert/etcd.pem \\--peer-key-file=/etc/etcd/cert/etcd-key.pem \\--peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\--peer-client-cert-auth \\--client-cert-auth \\--listen-peer-urls=https://##NODE_IP##:2380 \\--initial-advertise-peer-urls=https://##NODE_IP##:2380 \\--listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\--advertise-client-urls=https://##NODE_IP##:2379 \\--initial-cluster-token=etcd-cluster-0 \\--initial-cluster=${ETCD_NODES} \\--initial-cluster-state=new \\--auto-compaction-mode=periodic \\--auto-compaction-retention=1 \\--max-request-bytes=33554432 \\--quota-backend-bytes=6442450944 \\--heartbeat-interval=250 \\--election-timeout=2000Restart=on-failureRestartSec=5LimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF配置說明 (此處不需要修改任何配置)

WorkDirectory、–data-dir 指定etcd工作目錄和數據存儲為${ETCD_DATA_DIR},需要在啟動前創建這個目錄 (後面跟著我操作就可以，會有創建步驟)–wal-dir 指定wal目錄，為了提高性能，一般使用SSD和–data-dir不同的盤–name 指定節點名稱，當–initial-cluster-state值為new時，–name的參數值必須位於–initial-cluster列表中–cert-file、–key-file ETCD server與client通信時使用的證書和私鑰–trusted-ca-file 籤名client證書的CA證書，用於驗證client證書–peer-cert-file、–peer-key-file ETCD與peer通信使用的證書和私鑰–peer-trusted-ca-file 籤名peer證書的CA證書，用於驗證peer證書為各個節點分發啟動文件

#分發會將配置文件中的#替換成ipcd /opt/k8s/worksource /opt/k8s/bin/environment.shfor(( i=0; i <3; i++))do sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/"-e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template> etcd-${ETCD_IPS[i]}.service donels *.service#NODE_NAMES 和 NODE_IPS 為相同長度的 bash 數組，分別為節點名稱和對應的 IP；分發生成的etcd啟動文件到對應的伺服器

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.servicedone重命名etcd啟動文件並啟動etcd服務 etcd首次進程啟動會等待其他節點加入etcd集群，執行啟動命令會卡頓一會，為正常現象

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd "&done#這裡我們創建了etcd的工作目錄檢查啟動結果

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status etcd|grep Active"done正常狀態

如果etcd集群狀態不是active (running)，請使用下面命令查看etcd日誌

journalctl -fu etcd驗證ETCD集群狀態 不是完etcd集群後，在任一etcd節點執行下命令

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ETCDCTL_API=3/opt/k8s/bin/etcdctl \--endpoints=https://${node_ip}:2379 \--cacert=/etc/kubernetes/cert/ca.pem \--cert=/etc/etcd/cert/etcd.pem \--key=/etc/etcd/cert/etcd-key.pem endpoint healthdone正常狀態如下

我們還可以通過下面命令查看當前etcd集群leader

source /opt/k8s/bin/environment.shETCDCTL_API=3/opt/k8s/bin/etcdctl \-w table --cacert=/etc/kubernetes/cert/ca.pem \--cert=/etc/etcd/cert/etcd.pem \--key=/etc/etcd/cert/etcd-key.pem \--endpoints=${ETCD_ENDPOINTS} endpoint status正常狀態如下

部署Flannel網絡Kubernetes要求集群內各個節點(包括master)能通過Pod網段互聯互通，Flannel使用vxlan技術為各個節點創建一個互通的Pod網絡，使用的埠為8472.第一次啟動時，從etcd獲取配置的Pod網絡，為本節點分配一個未使用的地址段，然後創建flannel.1網絡接口(也可能是其它名稱)flannel將分配給自己的Pod網段信息寫入/run/flannel/docker文件，docker後續使用這個文件中的環境變量設置Docker0網橋，從而從這個地址段為本節點的所有Pod容器分配IP

下載分發flanneld二進位文件(本次flanneld不使用Pod運行)

cd /opt/k8s/workmkdir flannelwget http://down.i4t.com/k8s1.14/flannel-v0.11.0-linux-amd64.tar.gztar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannel分發二進位文件到所有集群的節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp flannel/{flanneld,mk-docker-opts.sh} root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip}"chmod +x /opt/k8s/bin/*"done創建Flannel證書和私鑰flanneld從etcd集群存取網段分配信息，而etcd集群開啟了雙向x509證書認證，所以需要為flannel生成證書和私鑰

創建證書籤名請求

cd /opt/k8s/workcat > flanneld-csr.json <<EOF{"CN":"flanneld","hosts":[],"key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"k8s","OU":"4Paradigm"}]}EOF生成證書和私鑰

cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneldls flanneld*pem將生成的證書和私鑰分發到所有節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p /etc/flanneld/cert" scp flanneld*.pem root@${node_ip}:/etc/flanneld/certdone向etcd寫入Pod網段信息

cd /opt/k8s/worksource /opt/k8s/bin/environment.shetcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/opt/k8s/work/ca.pem \--cert-file=/opt/k8s/work/flanneld.pem \--key-file=/opt/k8s/work/flanneld-key.pem \ mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'注意: flanneld當前版本v0.11.0不支持etcd v3，故使用etcd v2 API寫入配置Key和網段數據； 寫入的Pod網段${CLUSTER_CIDR}地址段(如/16)必須小於SubnetLen，必須與kube-controller-manager的–cluster-cidr參數一致

創建flanneld的啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > flanneld.service << EOF[Unit]Description=Flanneld overlay address etcd agentAfter=network.targetAfter=network-online.targetWants=network-online.targetAfter=etcd.serviceBefore=docker.service[Service]Type=notifyExecStart=/opt/k8s/bin/flanneld \\-etcd-cafile=/etc/kubernetes/cert/ca.pem \\-etcd-certfile=/etc/flanneld/cert/flanneld.pem \\-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\-etcd-endpoints=${ETCD_ENDPOINTS} \\-etcd-prefix=${FLANNEL_ETCD_PREFIX} \\-iface=${IFACE} \\-ip-masqExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/dockerRestart=alwaysRestartSec=5StartLimitInterval=0[Install]WantedBy=multi-user.targetRequiredBy=docker.serviceEOFmk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網段信息寫入 /run/flannel/docker 文件，後續 docker 啟動時使用這個文件中的環境變量配置 docker0 網橋；flanneld 使用系統預設路由所在的接口與其它節點通信，對於有多個網絡接口（如內網和公網）的節點，可以用 -iface 參數指定通信接口;flanneld 運行時需要 root 權限；-ip-masq: flanneld 為訪問 Pod 網絡外的流量設置 SNAT 規則，同時將傳遞給 Docker 的變量 –ip-masq（/run/flannel/docker 文件中）設置為 false，這樣 Docker 將不再創建 SNAT 規則； Docker 的 –ip-masq 為 true 時，創建的 SNAT 規則比較「暴力」：將所有本節點 Pod 發起的、訪問非 docker0 接口的請求做 SNAT，這樣訪問其他節點 Pod 的請求來源 IP 會被設置為 flannel.1 接口的 IP，導致目的 Pod 看不到真實的來源 Pod IP。 flanneld 創建的 SNAT 規則比較溫和，只對訪問非 Pod 網段的請求做 SNAT。分發啟動文件到所有節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp flanneld.service root@${node_ip}:/etc/systemd/system/done啟動flanneld服務

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"done檢查啟動結果

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status flanneld|grep Active"done檢查分配給flanneld的Pod網段信息

source /opt/k8s/bin/environment.shetcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/etc/kubernetes/cert/ca.pem \--cert-file=/etc/flanneld/cert/flanneld.pem \--key-file=/etc/flanneld/cert/flanneld-key.pem \get ${FLANNEL_ETCD_PREFIX}/config查看已分配的Pod子網網段列表

source /opt/k8s/bin/environment.shetcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/etc/kubernetes/cert/ca.pem \--cert-file=/etc/flanneld/cert/flanneld.pem \--key-file=/etc/flanneld/cert/flanneld-key.pem \ ls ${FLANNEL_ETCD_PREFIX}/subnets查看某Pod網段對應節點IP和flannel接口地址

source /opt/k8s/bin/environment.shetcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/etc/kubernetes/cert/ca.pem \--cert-file=/etc/flanneld/cert/flanneld.pem \--key-file=/etc/flanneld/cert/flanneld-key.pem \get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.16.0-21#後面節點IP需要根據我們查出來的地址進行修改查看節點flannel網絡信息

ip addr showflannel.1網卡的地址為分配的pod自網段的第一個個IP (.0)，且是/32的地址

ip addr show|grep flannel.1到其它節點 Pod 網段請求都被轉發到 flannel.1 網卡； flanneld 根據 etcd 中子網段的信息，如 ${FLANNEL_ETCD_PREFIX}/subnets/172.30.80.0-21，來決定進請求發送給哪個節點的互聯 IP； 驗證各節點能通過 Pod 網段互通 在各節點上部署 flannel 後，檢查是否創建了 flannel 接口(名稱可能為 flannel0、flannel.0、flannel.1 等)：

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh ${node_ip}"/usr/sbin/ip addr show flannel.1|grep -w inet"donekube-apiserver 高可用

使用Nginx 4層透明代理功能實現k8s節點(master節點和worker節點)高可用訪問kube-apiserver的步驟控制節點的kube-controller-manager、kube-scheduler是多實例部署，所以只要一個實例正常，就可以保證集群高可用集群內的Pod使用k8s服務域名kubernetes訪問kube-apiserver，kube-dns會自動解析多個kube-apiserver節點的IP，所以也是高可用的在每個Nginx進程，後端對接多個apiserver實例，Nginx對他們做健康檢查和負載均衡kubelet、kube-proxy、controller-manager、schedule通過本地nginx (監聽我們vip 192.158.0.54)訪問kube-apiserver，從而實現kube-apiserver高可用下載編譯nginx (k8s-01安裝就可以，後面有拷貝步驟)

cd /opt/k8s/workwget http://down.i4t.com/k8s1.14/nginx-1.15.3.tar.gztar -xzvf nginx-1.15.3.tar.gz#編譯cd /opt/k8s/work/nginx-1.15.3mkdir nginx-prefix./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module make && make install#############--without-http_scgi_module --without-http_fastcgi_module--with-stream：開啟4層透明轉發(TCP Proxy)功能；--without-xxx：關閉所有其他功能，這樣生成的動態連結二進位程序依賴最小；查看 nginx 動態連結的庫：

ldd ./nginx-prefix/sbin/nginx由於只開啟了 4 層透明轉發功能，所以除了依賴 libc 等作業系統核心 lib 庫外，沒有對其它 lib 的依賴(如 libz、libssl 等)，這樣可以方便部署到各版本作業系統中

創建目錄結構

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}done拷貝二進位程序到其他主機 (有報錯執行2遍就可以)

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx ssh root@${node_ip}"chmod a+x /opt/k8s/kube-nginx/sbin/*" ssh root@${node_ip}"mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}" sleep 3done配置Nginx文件，開啟4層透明轉發

cd /opt/k8s/workcat > kube-nginx.conf <<EOFworker_processes 1;events { worker_connections 1024;}stream { upstream backend { hash $remote_addr consistent; server 192.168.0.50:6443 max_fails=3 fail_timeout=30s; server 192.168.0.51:6443 max_fails=3 fail_timeout=30s; server 192.168.0.52:6443 max_fails=3 fail_timeout=30s;} server { listen *:8443; proxy_connect_timeout 1s; proxy_pass backend;}}EOF#這裡需要將server替換我們自己的地址分發配置文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-nginx.conf root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.confdone配置Nginx啟動文件

cd /opt/k8s/workcat > kube-nginx.service <<EOF[Unit]Description=kube-apiserver nginx proxyAfter=network.targetAfter=network-online.targetWants=network-online.target[Service]Type=forkingExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -tExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginxExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reloadPrivateTmp=trueRestart=alwaysRestartSec=5StartLimitInterval=0LimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF分發nginx啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-nginx.service root@${node_ip}:/etc/systemd/system/done啟動 kube-nginx 服務

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable kube-nginx && systemctl start kube-nginx"done檢查 kube-nginx 服務運行狀態

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status kube-nginx |grep 'Active:'"doneKeepLived 部署前面我們也說了，高可用方案需要一個VIP，供集群內部訪問

在所有master節點安裝keeplived

yum install -y keepalived接下來我們要配置keeplive服務

192.168.0.50配置

cat >/etc/keepalived/keepalived.conf <<EOF!ConfigurationFilefor keepalivedglobal_defs { router_id 192.168.0.50}vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 8443" interval 2 weight -20}vrrp_instance VI_1 { state MASTERinterface eth0 virtual_router_id 251 priority 100 advert_int 1 mcast_src_ip 192.168.0.50 nopreempt authentication { auth_type PASS auth_pass 11111111} track_script { chk_nginx} virtual_ipaddress {192.168.0.54}}EOF## 192.168.0.50 為節點IP，192.168.0.54位VIP將配置拷貝到其他節點，並替換相關IP

for node_ip in192.168.0.50192.168.0.51192.168.0.52do echo ">>> ${node_ip}" scp /etc/keepalived/keepalived.conf $node_ip:/etc/keepalived/keepalived.confdone#替換IPssh root@192.168.0.51 sed -i 's#192.168.0.50#192.168.0.51#g'/etc/keepalived/keepalived.confssh root@192.168.0.52 sed -i 's#192.168.0.50#192.168.0.52#g'/etc/keepalived/keepalived.conf#192.168.0.50不替換是因為已經修改好了創建健康檢查腳本

vim /opt/check_port.sh CHK_PORT=$1if[-n "$CHK_PORT"];then PORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l`if[ $PORT_PROCESS -eq 0];then echo "Port $CHK_PORT Is Not Used,End."exit1fielse echo "Check Port Cant Be Empty!"fi啟動keeplived

for NODE in k8s-01 k8s-02 k8s-03;do echo "--- $NODE ---" scp -r /opt/check_port.sh $NODE:/etc/keepalived/ ssh $NODE 'systemctl enable --now keepalived'done啟動完畢後ping 192.168.0.54 (VIP)

[root@abcdocker-k8s03 ~]# ping 192.168.0.54PING 192.168.0.54(192.168.0.54)56(84) bytes of data.64 bytes from192.168.0.54: icmp_seq=1 ttl=64 time=0.055 ms^C---192.168.0.54 ping statistics ---1 packets transmitted,1 received,0% packet loss, time 0msrtt min/avg/max/mdev =0.055/0.055/0.055/0.000 ms#如果沒有啟動，請檢查原因。 ps -ef|grep keep 檢查是否啟動成功#沒有啟動成功，請執行下面命令，從新啟動。啟動成功vip肯定就通了systemctl start keepalived部署master節點kubernetes master節點運行組件如下:kube-apiserver、kube-scheduler、kube-controller-manager、kube-nginx

kube-apiserver、kube-scheduler、kube-controller-manager均以多實例模式運行kube-scheduler和kube-controller-manager會自動選舉一個leader實例，其他實例處於阻塞模式，當leader掛了後，重新選舉產生的leader，從而保證服務可用性kube-apiserver是無狀態的，需要通過kube-nginx進行代理訪問，從而保證服務可用性以下操作都在K8s-01操作

下載kubernetes二進位包，並分發到所有master節點

cd /opt/k8s/workwget http://down.i4t.com/k8s1.14/kubernetes-server-linux-amd64.tar.gztar -xzvf kubernetes-server-linux-amd64.tar.gzcd kubernetestar -xzvf kubernetes-src.tar.gz將壓縮包的文件拷貝到所有master節點上

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp kubernetes/server/bin/{apiextensions-apiserver,cloud-controller-manager,kube-apiserver,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${node_ip}:/opt/k8s/bin/ ssh root@${node_ip}"chmod +x /opt/k8s/bin/*"done創建Kubernetes 證書和私鑰創建籤證籤名請求

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > kubernetes-csr.json <<EOF{"CN":"kubernetes","hosts":["127.0.0.1","192.168.0.50","192.168.0.51","192.168.0.52","192.168.0.54","10.254.0.1","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local."],"key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"k8s","OU":"4Paradigm"}]}EOF#需要將集群的所有IP及VIP添加進去hosts 欄位指定授權使用該證書的IP和域名列表，這裡列出了master節點IP、kubernetes服務的IP和域名 kubernetes serviceIP是apiserver自動創建的，一般是–service-cluster-ip-range參數指定的網段的第一個IP

$ kubectl get svc kubernetesNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEkubernetes ClusterIP10.254.0.1<none>443/TCP 31d生成證書和私鑰

cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes ls kubernetes*pem將生成的證書和私鑰文件拷貝到所有master節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p /etc/kubernetes/cert" scp kubernetes*.pem root@${node_ip}:/etc/kubernetes/cert/done創建加密配置文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > encryption-config.yaml <<EOFkind:EncryptionConfigapiVersion: v1resources:- resources:- secrets providers:- aescbc: keys:- name: key1 secret: ${ENCRYPTION_KEY}- identity:{}EOF將加密配置文件拷貝到master節點的/etc/kubernetes目錄下

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp encryption-config.yaml root@${node_ip}:/etc/kubernetes/done創建審計策略文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > audit-policy.yaml <<EOFapiVersion: audit.k8s.io/v1beta1kind:Policyrules:# The following requests were manually identified as high-volume and low-risk, so drop them.- level:None resources:-group:"" resources:- endpoints- services- services/status users:-'system:kube-proxy' verbs:- watch- level:None resources:-group:"" resources:- nodes- nodes/status userGroups:-'system:nodes' verbs:-get- level:None namespaces:- kube-system resources:-group:"" resources:- endpoints users:-'system:kube-controller-manager'-'system:kube-scheduler'-'system:serviceaccount:kube-system:endpoint-controller' verbs:-get- update- level:None resources:-group:"" resources:- namespaces- namespaces/status- namespaces/finalize users:-'system:apiserver' verbs:-get# Don't log HPA fetching metrics.- level:None resources:-group: metrics.k8s.io users:-'system:kube-controller-manager' verbs:-get- list# Don't log these read-only URLs.- level:None nonResourceURLs:-'/healthz*'-/version-'/swagger*'# Don't log events requests.- level:None resources:-group:"" resources:- events# node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes- level:Request omitStages:-RequestReceived resources:-group:"" resources:- nodes/status- pods/status users:- kubelet-'system:node-problem-detector'-'system:serviceaccount:kube-system:node-problem-detector' verbs:- update- patch- level:Request omitStages:-RequestReceived resources:-group:"" resources:- nodes/status- pods/status userGroups:-'system:nodes' verbs:- update- patch# deletecollection calls can be large, don't log responses for expected namespace deletions- level:Request omitStages:-RequestReceived users:-'system:serviceaccount:kube-system:namespace-controller' verbs:- deletecollection# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,# so only log at the Metadata level.- level:Metadata omitStages:-RequestReceived resources:-group:"" resources:- secrets- configmaps-group: authentication.k8s.io resources:- tokenreviews# Get repsonses can be large; skip them.- level:Request omitStages:-RequestReceived resources:-group:""-group: admissionregistration.k8s.io-group: apiextensions.k8s.io-group: apiregistration.k8s.io-group: apps-group: authentication.k8s.io-group: authorization.k8s.io-group: autoscaling-group: batch-group: certificates.k8s.io-group: extensions-group: metrics.k8s.io-group: networking.k8s.io-group: policy-group: rbac.authorization.k8s.io-group: scheduling.k8s.io-group: settings.k8s.io-group: storage.k8s.io verbs:-get- list- watch# Default level for known APIs- level:RequestResponse omitStages:-RequestReceived resources:-group:""-group: admissionregistration.k8s.io-group: apiextensions.k8s.io-group: apiregistration.k8s.io-group: apps-group: authentication.k8s.io-group: authorization.k8s.io-group: autoscaling-group: batch-group: certificates.k8s.io-group: extensions-group: metrics.k8s.io-group: networking.k8s.io-group: policy-group: rbac.authorization.k8s.io-group: scheduling.k8s.io-group: settings.k8s.io-group: storage.k8s.io# Default level for all other requests.- level:Metadata omitStages:-RequestReceivedEOF分發審計策略文件：

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp audit-policy.yaml root@${node_ip}:/etc/kubernetes/audit-policy.yamldone創建證書籤名請求

cat > proxy-client-csr.json <<EOF{"CN":"aggregator","hosts":[],"key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"k8s","OU":"4Paradigm"}]}EOFCN名稱需要位於kube-apiserver的–requestherader-allowed-names參數中，否則後續訪問metrics時會提示權限不足生成證書和私鑰

cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \-ca-key=/etc/kubernetes/cert/ca-key.pem \-config=/etc/kubernetes/cert/ca-config.json \-profile=kubernetes proxy-client-csr.json | cfssljson -bare proxy-clientls proxy-client*.pem將生成的證書和私鑰文件拷貝到master節點

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp proxy-client*.pem root@${node_ip}:/etc/kubernetes/cert/done創建kube-apiserver啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > kube-apiserver.service.template<<EOF[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target[Service]WorkingDirectory=${K8S_DIR}/kube-apiserverExecStart=/opt/k8s/bin/kube-apiserver \\--advertise-address=##NODE_IP## \\--default-not-ready-toleration-seconds=360 \\--default-unreachable-toleration-seconds=360 \\--feature-gates=DynamicAuditing=true \\--max-mutating-requests-inflight=2000 \\--max-requests-inflight=4000 \\--default-watch-cache-size=200 \\--delete-collection-workers=2 \\--encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\--etcd-cafile=/etc/kubernetes/cert/ca.pem \\--etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\--etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\--etcd-servers=${ETCD_ENDPOINTS} \\--bind-address=##NODE_IP## \\--secure-port=6443 \\--tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\--tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\--insecure-port=0 \\--audit-dynamic-configuration \\--audit-log-maxage=15 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-truncate-enabled \\--audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\--audit-policy-file=/etc/kubernetes/audit-policy.yaml \\--profiling \\--anonymous-auth=false \\--client-ca-file=/etc/kubernetes/cert/ca.pem \\--enable-bootstrap-token-auth \\--requestheader-allowed-names="aggregator" \\--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-extra-headers-prefix="X-Remote-Extra-" \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User \\--service-account-key-file=/etc/kubernetes/cert/ca.pem \\--authorization-mode=Node,RBAC \\--runtime-config=api/all=true \\--enable-admission-plugins=NodeRestriction \\--allow-privileged=true \\--apiserver-count=3 \\--event-ttl=168h \\--kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\--kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\--kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\--kubelet-https=true \\--kubelet-timeout=10s \\--proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\--proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\--service-cluster-ip-range=${SERVICE_CIDR} \\--service-node-port-range=${NODE_PORT_RANGE} \\--logtostderr=true \\--v=2Restart=on-failureRestartSec=10Type=notifyLimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF參數配置說明

--advertise-address：apiserver 對外通告的 IP（kubernetes 服務後端節點 IP）；--default-*-toleration-seconds：設置節點異常相關的閾值；--max-*-requests-inflight：請求相關的最大閾值；--etcd-*：訪問 etcd 的證書和 etcd 伺服器地址；--experimental-encryption-provider-config：指定用於加密 etcd 中 secret 的配置；--bind-address： https 監聽的 IP，不能為127.0.0.1，否則外界不能訪問它的安全埠6443；--secret-port：https 監聽埠；--insecure-port=0：關閉監聽 http 非安全埠(8080)；--tls-*-file：指定 apiserver 使用的證書、私鑰和 CA 文件；--audit-*：配置審計策略和審計日誌文件相關的參數；--client-ca-file：驗證 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)請求所帶的證書；--enable-bootstrap-token-auth：啟用 kubelet bootstrap 的 token 認證；--requestheader-*：kube-apiserver 的 aggregator layer 相關的配置參數，proxy-client & HPA 需要使用；--requestheader-client-ca-file：用於籤名--proxy-client-cert-file 和--proxy-client-key-file 指定的證書；在啟用了 metric aggregator 時使用；--requestheader-allowed-names：不能為空，值為逗號分割的--proxy-client-cert-file 證書的 CN 名稱，這裡設置為"aggregator"；--service-account-key-file：籤名ServiceAccountToken的公鑰文件，kube-controller-manager 的--service-account-private-key-file 定私鑰文件，兩者配對使用；--runtime-config=api/all=true：啟用所有版本的APIs，如 autoscaling/v2alpha1；--authorization-mode=Node,RBAC、--anonymous-auth=false：開啟Node和 RBAC 授權模式，拒絕未授權的請求；--enable-admission-plugins：啟用一些默認關閉的 plugins；--allow-privileged：運行執行 privileged 權限的容器；--apiserver-count=3：指定 apiserver 實例的數量；--event-ttl：指定 events 的保存時間；--kubelet-：如果指定，則使用 https 訪問 kubelet APIs；需要為證書對應的用戶(上面 kubernetes.pem 證書的用戶為 kubernetes)用戶定義 RBAC 規則，否則訪問 kubelet API 時提示未授權；--proxy-client-*：apiserver 訪問 metrics-server 使用的證書；--service-cluster-ip-range：指定ServiceCluster IP 地址段；--service-node-port-range：指定NodePort的埠範圍；如果 kube-apiserver 機器沒有運行 kube-proxy，則還需要添加--enable-aggregator-routing=true參數；關於--requestheader-XXX 相關參數，參考：https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.mdhttps://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/注意: requestheader-client-ca-file指定的CA證書，必須具有client auth and server auth 如果–requestheader-allowed-names為空，或者–proxy-client-cert-file證書的CN名稱不在allowed-names中，則後續查看node或者Pods的metrics失敗

為各個節點創建和分發kube-apiserver啟動文件替換模板文件的變量，為各個節點生成啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor(( i=0; i <3; i++))#這裡是三個節點所以為3,請根據實際情況修改,後邊不在提示,同理do sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/"-e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template> kube-apiserver-${MASTER_IPS[i]}.service donels kube-apiserver*.service分發apiserver啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.servicedone啟動apiserver

source /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p ${K8S_DIR}/kube-apiserver" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"done檢查服務是否正常

source /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status kube-apiserver |grep 'Active:'"done確保狀態為active (running)，否則查看日誌，確認原因

journalctl -u kube-apiserver列印kube-apiserver寫入etcd數據

source /opt/k8s/bin/environment.shETCDCTL_API=3 etcdctl \--endpoints=${ETCD_ENDPOINTS} \--cacert=/opt/k8s/work/ca.pem \--cert=/opt/k8s/work/etcd.pem \--key=/opt/k8s/work/etcd-key.pem \get/registry/--prefix --keys-only檢查kube-apiserver監聽的埠

netstat -lntup|grep kubetcp 00192.168.0.50:64430.0.0.0:* LISTEN 11739/kube-apiserve檢查集群信息

$ kubectl cluster-infoKubernetes master is running at https://192.168.0.54:8443To further debug and diagnose cluster problems,use'kubectl cluster-info dump'.$ kubectl get all --all-namespacesNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdefault service/kubernetes ClusterIP10.254.0.1<none>443/TCP 3m5s$ kubectl get componentstatusesNAME STATUS MESSAGE ERRORscheduler UnhealthyGet http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refusedcontroller-manager UnhealthyGet http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refusedetcd-2Healthy{"health":"true"}etcd-0Healthy{"health":"true"}etcd-1Healthy{"health":"true"}如果提示有報錯，請檢查~/.kube/config以及配置證書是否有問題

授權kube-apiserver訪問kubelet API的權限在執行kubectl命令時，apiserver會將請求轉發到kubelet的https埠。這裡定義的RBAC規則，授權apiserver使用的證書(kubernetes.pem)用戶名(CN:kubernetes)訪問kubelet API的權限

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes部署高可用kube-controller-manager集群該集群包含三個節點，啟動後通過競爭選舉機制產生一個leader節點，其他節點為阻塞狀態。當leader節點不可用時，阻塞節點將會在此選舉產生新的leader，從而保證服務的高可用。為保證通信安全，這裡採用x509證書和私鑰，kube-controller-manager在與apiserver的安全埠（http 10252）通信使用；

創建kube-controller-manager證書和私鑰 創建證書籤名請求

cd /opt/k8s/workcat > kube-controller-manager-csr.json <<EOF{"CN":"system:kube-controller-manager","key":{"algo":"rsa","size":2048},"hosts":["127.0.0.1","192.168.0.50","192.168.0.51","192.168.0.52"],"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"system:kube-controller-manager","OU":"4Paradigm"}]}EOFhost列表包含所有的kube-controller-manager節點IP(VIP不需要輸入)CN和O均為system:kube-controller-manager，kubernetes內置的ClusterRoleBindings system:kube-controller-manager賦予kube-controller-manager工作所需權限生成證書和私鑰

cd /opt/k8s/workcfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-managerls kube-controller-manager*pem將生成的證書和私鑰分發到所有master節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-controller-manager*.pem root@${node_ip}:/etc/kubernetes/cert/done創建和分發kubeconfig文件

#kube-controller-manager使用kubeconfig文件訪問apiserver#該文件提供了apiserver地址、嵌入的CA證書和kube-controller-manager證書cd /opt/k8s/worksource /opt/k8s/bin/environment.shkubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-controller-manager.kubeconfigkubectl config set-credentials system:kube-controller-manager \--client-certificate=kube-controller-manager.pem \--client-key=kube-controller-manager-key.pem \--embed-certs=true \--kubeconfig=kube-controller-manager.kubeconfigkubectl config set-context system:kube-controller-manager \--cluster=kubernetes \--user=system:kube-controller-manager \--kubeconfig=kube-controller-manager.kubeconfigkubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig分發kubeconfig到所有master節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-controller-manager.kubeconfig root@${node_ip}:/etc/kubernetes/done創建kube-controller-manager啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > kube-controller-manager.service.template<<EOF[Unit]Description=KubernetesControllerManagerDocumentation=https://github.com/GoogleCloudPlatform/kubernetes[Service]WorkingDirectory=${K8S_DIR}/kube-controller-managerExecStart=/opt/k8s/bin/kube-controller-manager \\--profiling \\--cluster-name=kubernetes \\--controllers=*,bootstrapsigner,tokencleaner \\--kube-api-qps=1000 \\--kube-api-burst=2000 \\--leader-elect \\--use-service-account-credentials\\--concurrent-service-syncs=2 \\--bind-address=0.0.0.0 \\#--secure-port=10252 \\--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\#--port=0 \\--authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\--client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-allowed-names="" \\--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-extra-headers-prefix="X-Remote-Extra-" \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User \\--authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\--experimental-cluster-signing-duration=876000h \\--horizontal-pod-autoscaler-sync-period=10s \\--concurrent-deployment-syncs=10 \\--concurrent-gc-syncs=30 \\--node-cidr-mask-size=24 \\--service-cluster-ip-range=${SERVICE_CIDR} \\--pod-eviction-timeout=6m \\--terminated-pod-gc-threshold=10000 \\--root-ca-file=/etc/kubernetes/cert/ca.pem \\--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\--logtostderr=true \\--v=2Restart=on-failureRestartSec=5[Install]WantedBy=multi-user.targetEOF參數解釋

–port=0：關閉監聽非安全埠（http），同時 –address 參數無效，–bind-address 參數有效；–secure-port=10252、–bind-address=0.0.0.0: 在所有網絡接口監聽 10252 埠的 https /metrics 請求；–kubeconfig：指定 kubeconfig 文件路徑，kube-controller-manager 使用它連接和驗證 kube-apiserver；–authentication-kubeconfig 和 –authorization-kubeconfig：kube-controller-manager 使用它連接 apiserver，對 client 的請求進行認證和授權。kube-controller-manager 不再使用 –tls-ca-file 對請求 https metrics 的 Client 證書進行校驗。如果沒有配置這兩個 kubeconfig 參數，則 client 連接 kube-controller-manager https 埠的請求會被拒絕(提示權限不足)。–cluster-signing-*-file：籤名 TLS Bootstrap 創建的證書；–experimental-cluster-signing-duration：指定 TLS Bootstrap 證書的有效期；–root-ca-file：放置到容器 ServiceAccount 中的 CA 證書，用來對 kube-apiserver 的證書進行校驗；–service-account-private-key-file：籤名 ServiceAccount 中 Token 的私鑰文件，必須和 kube-apiserver 的 –service-account-key-file 指定的公鑰文件配對使用；–service-cluster-ip-range ：指定 Service Cluster IP 網段，必須和 kube-apiserver 中的同名參數一致；–leader-elect=true：集群運行模式，啟用選舉功能；被選為 leader 的節點負責處理工作，其它節點為阻塞狀態；–controllers=*,bootstrapsigner,tokencleaner：啟用的控制器列表，tokencleaner 用於自動清理過期的 Bootstrap token；–horizontal-pod-autoscaler-*：custom metrics 相關參數，支持 autoscaling/v2alpha1；–tls-cert-file、–tls-private-key-file：使用 https 輸出 metrics 時使用的 Server 證書和秘鑰；–use-service-account-credentials=true: kube-controller-manager 中各 controller 使用 serviceaccount 訪問 kube-apiserver； 替換啟動文件，並分髮腳本

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor(( i=0; i <3; i++))do sed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/"-e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-controller-manager.service.template> kube-controller-manager-${MASTER_IPS[i]}.service donels kube-controller-manager*.service分發到所有master節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-controller-manager-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-controller-manager.servicedone啟動服務

source /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p ${K8S_DIR}/kube-controller-manager" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"done檢查運行狀態

source /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status kube-controller-manager|grep Active"done檢查服務狀態

netstat -lnpt | grep kube-conttcp6 00:::10252:::* LISTEN 13279/kube-controlltcp6 00:::10257:::* LISTEN 13279/kube-controllkube-controller-manager 創建權限ClusteRole system:kube-controller-manager的權限太小，只能創建secret、serviceaccount等資源,將controller的權限分散到ClusterRole system:controller:xxx中

$ kubectl describe clusterrole system:kube-controller-managerName: system:kube-controller-managerLabels: kubernetes.io/bootstrapping=rbac-defaultsAnnotations: rbac.authorization.kubernetes.io/autoupdate:truePolicyRule:ResourcesNon-ResourceURLsResourceNamesVerbs secrets [][][create deleteget update] endpoints [][][create get update] serviceaccounts [][][create get update] events [][][create patch update] tokenreviews.authentication.k8s.io [][][create] subjectaccessreviews.authorization.k8s.io [][][create] configmaps [][][get] namespaces [][][get]*.*[][][list watch]需要在 kube-controller-manager 的啟動參數中添加 –use-service-account-credentials=true 參數，這樣 main controller 會為各 controller 創建對應的 ServiceAccount XXX-controller。內置的 ClusterRoleBinding system:controller:XXX 將賦予各 XXX-controller ServiceAccount 對應的 ClusterRole system:controller:XXX 權限。

$ kubectl get clusterrole|grep controllersystem:controller:attachdetach-controller 22msystem:controller:certificate-controller 22msystem:controller:clusterrole-aggregation-controller 22msystem:controller:cronjob-controller 22msystem:controller:daemon-set-controller 22msystem:controller:deployment-controller 22msystem:controller:disruption-controller 22msystem:controller:endpoint-controller 22msystem:controller:expand-controller 22msystem:controller:generic-garbage-collector 22msystem:controller:horizontal-pod-autoscaler 22msystem:controller:job-controller 22msystem:controller:namespace-controller 22msystem:controller:node-controller 22msystem:controller:persistent-volume-binder 22msystem:controller:pod-garbage-collector 22msystem:controller:pv-protection-controller 22msystem:controller:pvc-protection-controller 22msystem:controller:replicaset-controller 22msystem:controller:replication-controller 22msystem:controller:resourcequota-controller 22msystem:controller:route-controller 22msystem:controller:service-account-controller 22msystem:controller:service-controller 22msystem:controller:statefulset-controller 22msystem:controller:ttl-controller 22msystem:kube-controller-manager 22m以 deployment controller 為例：

$ kubectl describe clusterrole system:controller:deployment-controllerName: system:controller:deployment-controllerLabels: kubernetes.io/bootstrapping=rbac-defaultsAnnotations: rbac.authorization.kubernetes.io/autoupdate:truePolicyRule:ResourcesNon-ResourceURLsResourceNamesVerbs replicasets.apps [][][create deleteget list patch update watch] replicasets.extensions [][][create deleteget list patch update watch] events [][][create patch update] pods [][][get list update watch] deployments.apps [][][get list update watch] deployments.extensions [][][get list update watch] deployments.apps/finalizers [][][update] deployments.apps/status [][][update] deployments.extensions/finalizers [][][update] deployments.extensions/status [][][update]查看當前的 leader

$ kubectl get endpoints kube-controller-manager --namespace=kube-system -o yamlapiVersion: v1kind:Endpointsmetadata: annotations: control-plane.alpha.kubernetes.io/leader:'{"holderIdentity":"abcdocker-k8s01_56e187ed-bc5b-11e9-b4a3-000c291b8bf5","leaseDurationSeconds":15,"acquireTime":"2019-08-11T17:13:29Z","renewTime":"2019-08-11T17:19:06Z","leaderTransitions":0}' creationTimestamp:"2019-08-11T17:13:29Z" name: kube-controller-managernamespace: kube-system resourceVersion:"848" selfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid:56e64ea1-bc5b-11e9-b77e-000c291b8bf5部署高可用kube-scheduler創建 kube-scheduler 證書和私鑰 創建證書籤名請求：

cd /opt/k8s/workcat > kube-scheduler-csr.json <<EOF{"CN":"system:kube-scheduler","hosts":["127.0.0.1","192.168.0.50","192.168.0.51","192.168.0.52"],"key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"system:kube-scheduler","OU":"4Paradigm"}]}EOFhosts 列表包含所有 kube-scheduler 節點 IP；CN 和 O 均為 system:kube-scheduler，kubernetes 內置的 ClusterRoleBindings system:kube-scheduler 將賦予 kube-scheduler 工作所需的權限；生成證書和私鑰：

cd /opt/k8s/workcfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-schedulerls kube-scheduler*pem將生成的證書和私鑰分發到所有 master 節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp kube-scheduler*.pem root@${node_ip}:/etc/kubernetes/cert/done創建和分發 kubeconfig 文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shkubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-scheduler.kubeconfigkubectl config set-credentials system:kube-scheduler \--client-certificate=kube-scheduler.pem \--client-key=kube-scheduler-key.pem \--embed-certs=true \--kubeconfig=kube-scheduler.kubeconfigkubectl config set-context system:kube-scheduler \--cluster=kubernetes \--user=system:kube-scheduler \--kubeconfig=kube-scheduler.kubeconfigkubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig分發 kubeconfig 到所有 master 節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" scp kube-scheduler.kubeconfig root@${node_ip}:/etc/kubernetes/done創建 kube-scheduler 配置文件

cd /opt/k8s/workcat >kube-scheduler.yaml.template<<EOFapiVersion: kubescheduler.config.k8s.io/v1alpha1kind:KubeSchedulerConfigurationbindTimeoutSeconds:600clientConnection: burst:200 kubeconfig:"/etc/kubernetes/kube-scheduler.kubeconfig" qps:100enableContentionProfiling:falseenableProfiling:truehardPodAffinitySymmetricWeight:1healthzBindAddress:127.0.0.1:10251leaderElection: leaderElect:truemetricsBindAddress:##NODE_IP##:10251EOF–kubeconfig：指定 kubeconfig 文件路徑，kube-scheduler 使用它連接和驗證 kube-apiserver；–leader-elect=true：集群運行模式，啟用選舉功能；被選為 leader 的節點負責處理工作，其它節點為阻塞狀態；替換模板文件中的變量：

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor(( i=0; i <3; i++))do sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/"-e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-scheduler.yaml.template> kube-scheduler-${NODE_IPS[i]}.yamldonels kube-scheduler*.yaml分發 kube-scheduler 配置文件到所有 master 節點：

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-scheduler-${node_ip}.yaml root@${node_ip}:/etc/kubernetes/kube-scheduler.yamldone創建kube-scheduler啟動文件

cd /opt/k8s/workcat > kube-scheduler.service.template<<EOF[Unit]Description=KubernetesSchedulerDocumentation=https://github.com/GoogleCloudPlatform/kubernetes[Service]WorkingDirectory=${K8S_DIR}/kube-schedulerExecStart=/opt/k8s/bin/kube-scheduler \\--config=/etc/kubernetes/kube-scheduler.yaml \\--bind-address=##NODE_IP## \\--secure-port=10259 \\--port=0 \\--tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\--tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \\--authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\--client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-allowed-names="" \\--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-extra-headers-prefix="X-Remote-Extra-" \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User \\--authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\--logtostderr=true \\--v=2Restart=alwaysRestartSec=5StartLimitInterval=0[Install]WantedBy=multi-user.targetEOF分發配置文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor(( i=0; i <3; i++))do sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/"-e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-scheduler.service.template> kube-scheduler-${NODE_IPS[i]}.service donels kube-scheduler*.servicecd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" scp kube-scheduler-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-scheduler.servicedone啟動kube-scheduler

source /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p ${K8S_DIR}/kube-scheduler" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler"done檢查服務運行狀態

source /opt/k8s/bin/environment.shfor node_ip in ${MASTER_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status kube-scheduler|grep Active"done查看輸出的 metrics 注意：以下命令在 kube-scheduler 節點上執行。 kube-scheduler 監聽 10251 和 10251 埠： 10251：接收 http 請求，非安全埠，不需要認證授權； 10259：接收 https 請求，安全埠，需要認證授權； 兩個接口都對外提供 /metrics 和 /healthz 的訪問。

curl -s http://192.168.0.50:10251/metrics|head# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.# TYPE apiserver_audit_event_total counterapiserver_audit_event_total 0# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.# TYPE apiserver_audit_requests_rejected_total counterapiserver_audit_requests_rejected_total 0# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.# TYPE apiserver_client_certificate_expiration_seconds histogramapiserver_client_certificate_expiration_seconds_bucket{le="0"}0apiserver_client_certificate_expiration_seconds_bucket{le="1800"}0查看當前leader

$ kubectl get endpoints kube-scheduler --namespace=kube-system -o yamlapiVersion: v1kind:Endpointsmetadata: annotations: control-plane.alpha.kubernetes.io/leader:'{"holderIdentity":"abcdocker-k8s01_72210df0-bc5d-11e9-9ca8-000c291b8bf5","leaseDurationSeconds":15,"acquireTime":"2019-08-11T17:28:35Z","renewTime":"2019-08-11T17:31:06Z","leaderTransitions":0}' creationTimestamp:"2019-08-11T17:28:35Z" name: kube-schedulernamespace: kube-system resourceVersion:"1500" selfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler uid:72bcd72f-bc5d-11e9-b77e-000c291b8bf5work節點安裝kubernetes work節點運行如下組件: >docker、kubelet、kube-proxy、flanneld、kube-nginx

前面已經安裝flanneld這就不在安裝了

安裝依賴包

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"yum install -y epel-release" ssh root@${node_ip}"yum install -y conntrack ipvsadm ntp ntpdate ipset jq iptables curl sysstat libseccomp && modprobe ip_vs "done部署Docker組件

我們在所有節點安裝docker，這裡使用阿里雲的yum安裝

Docker步驟需要在所有節點安裝

yum install -y yum-utils device-mapper-persistent-data lvm2yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repoyum makecache fastyum -y install docker-ce創建配置文件

mkdir -p /etc/docker/cat >/etc/docker/daemon.json <<EOF{"exec-opts":["native.cgroupdriver=systemd"],"registry-mirrors":["https://hjvrgh7a.mirror.aliyuncs.com"],"log-driver":"json-file","log-opts":{"max-size":"100m"},"storage-driver":"overlay2"}EOF#這裡配置當時鏡像加速器,可以不進行配置，但是建議配置要添加我們harbor倉庫需要在添加下面一行"insecure-registries":["harbor.i4t.com"],默認docker hub需要https協議，使用上面配置不需要配置https修改Docker啟動參數

這裡需要在所有的節點上修改docker配置！！

EnvironmentFile=-/run/flannel/dockerExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock完整配置如下

$ cat /usr/lib/systemd/system/docker.service[Unit]Description=DockerApplicationContainerEngineDocumentation=https://docs.docker.comBindsTo=containerd.serviceAfter=network-online.target firewalld.service containerd.serviceWants=network-online.targetRequires=docker.socket[Service]Type=notify# the default is not to use systemd for cgroups because the delegate issues still# exists and systemd currently does not support the cgroup feature set required# for containers run by dockerExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sockEnvironmentFile=-/run/flannel/dockerExecReload=/bin/kill -s HUP $MAINPIDTimeoutSec=0RestartSec=2Restart=always# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.# Both the old, and new location are accepted by systemd 229 and up, so using the old location# to make them work for either version of systemd.StartLimitBurst=3# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make# this option work for either version of systemd.StartLimitInterval=60s# Having non-zero Limit*s causes performance problems due to accounting overhead# in the kernel. We recommend using cgroups to do container-local accounting.LimitNOFILE=infinityLimitNPROC=infinityLimitCORE=infinity# Comment TasksMax if your systemd version does not support it.# Only systemd 226 and above support this option.TasksMax=infinity# set delegate yes so that systemd does not reset the cgroups of docker containersDelegate=yes# kill only the docker process, not all processes in the cgroupKillMode=process[Install]WantedBy=multi-user.target啟動 docker 服務

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable docker && systemctl restart docker"done檢查服務運行狀態

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status docker|grep Active"done檢查 docker0 網橋

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"/usr/sbin/ip addr show flannel.1 && /usr/sbin/ip addr show docker0"done查看 docker 的狀態信息

docker info#查看docker版本以及存儲引擎是否是overlay2以上Docker步驟，有很多需要進入每臺伺服器進行修改配置文件！！

部署kubelet組件kubelet運行在每個worker節點上，接收kube-apiserver發送的請求，管理Pod容器，執行交互命令 kubelet啟動時自動向kube-apiserver註冊節點信息，內置的cAdivsor統計和監控節點的資源使用資源情況。為確保安全，部署時關閉了kubelet的非安全http埠，對請求進行認證和授權，拒絕未授權的訪問

創建kubelet bootstrap kubeconfig文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_name in ${NODE_NAMES[@]}do echo ">>> ${node_name}"# 創建 tokenexport BOOTSTRAP_TOKEN=$(kubeadm token create \--description kubelet-bootstrap-token \--groups system:bootstrappers:${node_name} \--kubeconfig ~/.kube/config)# 設置集群參數 kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/cert/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig# 設置客戶端認證參數 kubectl config set-credentials kubelet-bootstrap \--token=${BOOTSTRAP_TOKEN} \--kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig# 設置上下文參數 kubectl config set-context default \--cluster=kubernetes \--user=kubelet-bootstrap \--kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig# 設置默認上下文 kubectl config use-context default--kubeconfig=kubelet-bootstrap-${node_name}.kubeconfigdone向kubeconfig寫入的是token，bootstrap結束後kube-controller-manager為kubelet創建client和server證書查看kubeadm為各個節點創建的token

$ kubeadm token list --kubeconfig ~/.kube/configTOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPSds9td8.wazmxhtaznrweknk 23h2019-08-13T01:54:57+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-01hy5ssz.4zi4e079ovxba52x23h2019-08-13T01:54:58+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-03pkkcl0.l7syoup3jedt7c3l 23h2019-08-13T01:54:57+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-02tubfqq.mja239hszl4rmron 23h2019-08-13T01:54:58+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-04token有效期為1天，超期後將不能被用來bootstrap kubelet，且會被kube-controller-manager的token cleaner清理kube-apiserver接收kubelet的bootstrap token後，將請求的user設置為system:bootstrap; group設置為system:bootstrappers，後續將為這個group設置ClusterRoleBinding查看各token關聯的Secret

$ kubectl get secrets -n kube-system|grep bootstrap-tokenbootstrap-token-ds9td8 bootstrap.kubernetes.io/token 73m15sbootstrap-token-hy5ssz bootstrap.kubernetes.io/token 73m14sbootstrap-token-pkkcl0 bootstrap.kubernetes.io/token 73m15sbootstrap-token-tubfqq bootstrap.kubernetes.io/token 73m14s分發 bootstrap kubeconfig 文件到所有 worker 節點

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_name in ${NODE_NAMES[@]}do echo ">>> ${node_name}" scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfigdone創建和分發kubelet參數配置

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > kubelet-config.yaml.template<<EOFkind:KubeletConfigurationapiVersion: kubelet.config.k8s.io/v1beta1address:"##NODE_IP##"staticPodPath:""syncFrequency:1mfileCheckFrequency:20shttpCheckFrequency:20sstaticPodURL:""port:10250readOnlyPort:0rotateCertificates:trueserverTLSBootstrap:trueauthentication: anonymous: enabled:false webhook: enabled:true x509: clientCAFile:"/etc/kubernetes/cert/ca.pem"authorization: mode:WebhookregistryPullQPS:0registryBurst:20eventRecordQPS:0eventBurst:20enableDebuggingHandlers:trueenableContentionProfiling:truehealthzPort:10248healthzBindAddress:"##NODE_IP##"clusterDomain:"${CLUSTER_DNS_DOMAIN}"clusterDNS:-"${CLUSTER_DNS_SVC_IP}"nodeStatusUpdateFrequency:10snodeStatusReportFrequency:1mimageMinimumGCAge:2mimageGCHighThresholdPercent:85imageGCLowThresholdPercent:80volumeStatsAggPeriod:1mkubeletCgroups:""systemCgroups:""cgroupRoot:""cgroupsPerQOS:truecgroupDriver: systemdruntimeRequestTimeout:10mhairpinMode: promiscuous-bridgemaxPods:220podCIDR:"${CLUSTER_CIDR}"podPidsLimit:-1resolvConf:/etc/resolv.confmaxOpenFiles:1000000kubeAPIQPS:1000kubeAPIBurst:2000serializeImagePulls:falseevictionHard: memory.available:"100Mi"nodefs.available:"10%"nodefs.inodesFree:"5%"imagefs.available:"15%"evictionSoft:{}enableControllerAttachDetach:truefailSwapOn:truecontainerLogMaxSize:20MicontainerLogMaxFiles:10systemReserved:{}kubeReserved:{}systemReservedCgroup:""kubeReservedCgroup:""enforceNodeAllocatable:["pods"]EOFaddress：kubelet 安全埠（https，10250）監聽的地址，不能為 127.0.0.1，否則 kube-apiserver、heapster 等不能調用 kubelet 的 API；readOnlyPort=0：關閉只讀埠(默認 10255)，等效為未指定；authentication.anonymous.enabled：設置為 false，不允許匿名訪問 10250 埠；authentication.x509.clientCAFile：指定籤名客戶端證書的 CA 證書，開啟 HTTP 證書認證；authentication.webhook.enabled=true：開啟 HTTPs bearer token 認證；對於未通過 x509 證書和 webhook 認證的請求(kube-apiserver 或其他客戶端)，將被拒絕，提示 Unauthorized；authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查詢 kube-apiserver 某 user、group 是否具有操作資源的權限(RBAC)；featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自動 rotate 證書，證書的有效期取決於 kube-controller-manager 的 –experimental-cluster-signing-duration 參數；需要 root 帳戶運行；為各個節點創建和分發kubelet配置文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" sed -e "s/##NODE_IP##/${node_ip}/" kubelet-config.yaml.template> kubelet-config-${node_ip}.yaml.template scp kubelet-config-${node_ip}.yaml.template root@${node_ip}:/etc/kubernetes/kubelet-config.yamldone創建和分發kubelet啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > kubelet.service.template<<EOF[Unit]Description=KubernetesKubeletDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=docker.serviceRequires=docker.service[Service]WorkingDirectory=${K8S_DIR}/kubeletExecStart=/opt/k8s/bin/kubelet \\--allow-privileged=true \\--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\--cert-dir=/etc/kubernetes/cert \\--cni-conf-dir=/etc/cni/net.d \\--container-runtime=docker \\--container-runtime-endpoint=unix:///var/run/dockershim.sock \\--root-dir=${K8S_DIR}/kubelet \\--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\--config=/etc/kubernetes/kubelet-config.yaml \\--hostname-override=##NODE_NAME## \\--pod-infra-container-image=registry.cn-beijing.aliyuncs.com/k8s_images/pause-amd64:3.1 \\--image-pull-progress-deadline=15m \\--volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\--logtostderr=true \\--v=2Restart=alwaysRestartSec=5StartLimitInterval=0[Install]WantedBy=multi-user.targetEOF如果設置了 –hostname-override 選項，則 kube-proxy 也需要設置該選項，否則會出現找不到 Node 的情況；–bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用該文件中的用戶名和 token 向 kube-apiserver 發送 TLS Bootstrapping 請求；K8S approve kubelet 的 csr 請求後，在 –cert-dir 目錄創建證書和私鑰文件，然後寫入 –kubeconfig 文件；–pod-infra-container-image 不使用 redhat 的 pod-infrastructure:latest 鏡像，它不能回收容器的殭屍；分發啟動文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_name in ${NODE_NAMES[@]}do echo ">>> ${node_name}" sed -e "s/##NODE_NAME##/${node_name}/" kubelet.service.template> kubelet-${node_name}.service scp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.servicedoneBootstrapTokenAuth和授予權限 kubelet 啟動時查找--kubeletconfig參數對應的文件是否存在，如果不存在則使用--bootstrap-kubeconfig 指定的 kubeconfig 文件向kube-apiserver 發送證書籤名請求(CSR)。 kube-apiserver 收到 CSR 請求後，對其中的Token進行認證，認證通過後將請求的 user 設置為 system:bootstrap:，group設置為system:bootstrappers，這一過程稱為BootstrapTokenAuth。創建user和group的CSR權限，不創建kubelet會啟動失敗

$ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers啟動 kubelet 服務

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/" ssh root@${node_ip}"/usr/sbin/swapoff -a" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"done關閉 swap 分區，否則 kubelet 會啟動失敗；

kubelet 啟動後使用 –bootstrap-kubeconfig 向 kube-apiserver 發送 CSR 請求，當這個 CSR 被 approve 後，kube-controller-manager 為 kubelet 創建 TLS 客戶端證書、私鑰和 –kubeletconfig 文件。 注意：kube-controller-manager 需要配置 –cluster-signing-cert-file 和 –cluster-signing-key-file 參數，才會為 TLS Bootstrap 創建證書和私鑰。

$ kubectl get csrNAME AGE REQUESTOR CONDITIONcsr-22kt238s system:bootstrap:pkkcl0 Pendingcsr-f9trc 37s system:bootstrap:tubfqq Pendingcsr-v7jt2 38s system:bootstrap:ds9td8 Pendingcsr-zrww2 37s system:bootstrap:hy5ssz Pending這裡4個節點均處於pending(等待)狀態

自動approve CSR請求

創建三個ClusterRoleBinding，分別用於自動approve client、renew client、renew server證書

cd /opt/k8s/workcat > csr-crb.yaml <<EOF# Approve all CSRs for the group "system:bootstrappers" kind:ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name:auto-approve-csrs-for-group subjects:- kind:Group name: system:bootstrappers apiGroup: rbac.authorization.k8s.io roleRef: kind:ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient apiGroup: rbac.authorization.k8s.io---# To let a node of the group "system:nodes" renew its own credentials kind:ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: node-client-cert-renewal subjects:- kind:Group name: system:nodes apiGroup: rbac.authorization.k8s.io roleRef: kind:ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient apiGroup: rbac.authorization.k8s.io---# A ClusterRole which instructs the CSR approver to approve a node requesting a# serving cert matching its client cert.kind:ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: approve-node-server-renewal-csrrules:- apiGroups:["certificates.k8s.io"] resources:["certificatesigningrequests/selfnodeserver"] verbs:["create"]---# To let a node of the group "system:nodes" renew its own server credentials kind:ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: node-server-cert-renewal subjects:- kind:Group name: system:nodes apiGroup: rbac.authorization.k8s.io roleRef: kind:ClusterRole name: approve-node-server-renewal-csr apiGroup: rbac.authorization.k8s.ioEOFkubectl apply -f csr-crb.yamlauto-approve-csrs-for-group 自動approve node的第一次CSR，注意第一次CSR時，請求的Group為system:bootstrappersnode-client-cert-renewal 自動approve node後續過期的client證書，自動生成的證書Group為system:nodesnode-server-cert-renewal 自動approve node後續過期的server證書，自動生成的證書Group查看kubelet 等待1-10分鐘，3個節點的CSR都會自動approved

$ kubectl get csrNAME AGE REQUESTOR CONDITIONcsr-22kt24m48s system:bootstrap:pkkcl0 Approved,Issuedcsr-d8tvc 77s system:node:k8s-01Pendingcsr-f9trc 4m47s system:bootstrap:tubfqq Approved,Issuedcsr-kcdvx 76s system:node:k8s-02Pendingcsr-m8k8t 75s system:node:k8s-04Pendingcsr-v7jt2 4m48s system:bootstrap:ds9td8 Approved,Issuedcsr-wwvwd 76s system:node:k8s-03Pendingcsr-zrww2 4m47s system:bootstrap:hy5ssz Approved,IssuedPending的CSR用於創建kubelet serve證書，需要手動approve (後面步驟)

目前所有節點均為ready狀態

[root@abcdocker-k8s01 work]# kubectl get nodeNAME STATUS ROLES AGE VERSIONk8s-01Ready<none>2m29s v1.14.2k8s-02Ready<none>2m28s v1.14.2k8s-03Ready<none>2m28s v1.14.2k8s-04Ready<none>2m27s v1.14.2kube-controller-manager為各node生成了kubeconfig文件和公鑰

$ ls -l /etc/kubernetes/kubelet.kubeconfig-rw--1 root root 2313Aug1202:04/etc/kubernetes/kubelet.kubeconfig$ ls -l /etc/kubernetes/cert/|grep kubelet-rw--1 root root 1273Aug1202:07 kubelet-client-2019-08-12-02-07-59.pemlrwxrwxrwx 1 root root 59Aug1202:07 kubelet-client-current.pem ->/etc/kubernetes/cert/kubelet-client-2019-08-12-02-07-59.pem手動approve server cert csr

基於安全考慮，CSR approving controllers不會自動approve kubelet server證書籤名請求，需要手動approve

kubectl get csr | grep Pending| awk '{print $1}'| xargs kubectl certificate approve

kubelet API接口

kubelet啟動後監聽多個埠，用於接受kube-apiserver或其他客戶端發送的請求

netstat -lntup|grep kubelettcp 00192.168.0.50:102480.0.0.0:* LISTEN 49491/kubelettcp 00127.0.0.1:457370.0.0.0:* LISTEN 49491/kubelettcp 00192.168.0.50:102500.0.0.0:* LISTEN 49491/kubelet10248: healthz http 服務；10250: https 服務，訪問該埠時需要認證和授權（即使訪問 /healthz 也需要）； 未開啟只讀埠 10255；從 K8S v1.10 開始，去除了 –cadvisor-port 參數（默認 4194 埠），不支持訪問 cAdvisor UI & APIbear token認證和授權創建一個ServiceAccount，將它和ClusterRole system:kubelet-api-admin綁定，從而具有調用kubelet API的權限

kubectl create sa kubelet-api-testkubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-testSECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')TOKEN=$(kubectl describe secret ${SECRET}| grep -E '^token'| awk '{print $2}')echo ${TOKEN}部署kube-proxy組件kube-proxy運行在所有worker節點上，它監聽apiserver中service和endpoint的變化情況，創建路由規則提供服務IP和負載均衡功能。這裡使用ipvs模式的kube-proxy進行部署

在各個節點需要安裝ipvsadm和ipset命令，加載ip_vs內核模塊

創建kube-proxy證書籤名請求

cd /opt/k8s/workcat > kube-proxy-csr.json <<EOF{"CN":"system:kube-proxy","key":{"algo":"rsa","size":2048},"names":[{"C":"CN","ST":"BeiJing","L":"BeiJing","O":"k8s","OU":"4Paradigm"}]}EOFCN：指定該證書的 User 為 system:kube-proxy；預定義的 RoleBinding system:node-proxier 將User system:kube-proxy 與 Role system:node-proxier 綁定，該 Role 授予了調用 kube-apiserver Proxy 相關 API 的權限；該證書只會被 kube-proxy 當做 client 證書使用，所以 hosts 欄位為空；生成證書和私鑰：

cd /opt/k8s/workcfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxyls kube-proxy*創建和分發 kubeconfig 文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shkubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-proxy.kubeconfigkubectl config set-credentials kube-proxy \--client-certificate=kube-proxy.pem \--client-key=kube-proxy-key.pem \--embed-certs=true \--kubeconfig=kube-proxy.kubeconfigkubectl config set-context default \--cluster=kubernetes \--user=kube-proxy \--kubeconfig=kube-proxy.kubeconfigkubectl config use-context default--kubeconfig=kube-proxy.kubeconfig–embed-certs=true：將 ca.pem 和 admin.pem 證書內容嵌入到生成的kubectl-proxy.kubeconfig文件中(不加時，寫入的是證書文件路徑)； 分發 kubeconfig 文件：cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_name in ${NODE_NAMES[@]}do echo ">>> ${node_name}" scp kube-proxy.kubeconfig root@${node_name}:/etc/kubernetes/done創建kube-proxy配置文件從v1.10開始，kube-proxy部分參數可以配置在文件中，可以使用–write-config-to選項生成該配置文件

cd /opt/k8s/workcat > kube-proxy-config.yaml.template<<EOFkind:KubeProxyConfigurationapiVersion: kubeproxy.config.k8s.io/v1alpha1clientConnection: burst:200 kubeconfig:"/etc/kubernetes/kube-proxy.kubeconfig" qps:100bindAddress:##NODE_IP##healthzBindAddress:##NODE_IP##:10256metricsBindAddress:##NODE_IP##:10249enableProfiling:trueclusterCIDR: ${CLUSTER_CIDR}hostnameOverride:##NODE_NAME##mode:"ipvs"portRange:""kubeProxyIPTablesConfiguration: masqueradeAll:falsekubeProxyIPVSConfiguration: scheduler: rr excludeCIDRs:[]EOFbindAddress: 監聽地址；clientConnection.kubeconfig: 連接 apiserver 的 kubeconfig 文件；-clusterCIDR: kube-proxy 根據 –cluster-cidr判斷集群內部和外部流量，指定 –cluster-cidr 或 –masquerade-all 選項後 kube-proxy 才會對訪問 Service IP 的請求做 SNAT；hostnameOverride: 參數值必須與 kubelet 的值一致，否則 kube-proxy 啟動後會找不到該 Node，從而不會創建任何 ipvs 規則；mode: 使用 ipvs 模式；分發和創建kube-proxy配置文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor(( i=0; i <4; i++))do echo ">>> ${NODE_NAMES[i]}" sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/"-e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-proxy-config.yaml.template> kube-proxy-config-${NODE_NAMES[i]}.yaml.template scp kube-proxy-config-${NODE_NAMES[i]}.yaml.template root@${NODE_NAMES[i]}:/etc/kubernetes/kube-proxy-config.yamldone#我這裡一共有4個節點要運行，所以這裡寫4，這是整個集群的node節點的數量！ 這裡一定要注意修改！！創建和分發 kube-proxy systemd unit 文件

cd /opt/k8s/worksource /opt/k8s/bin/environment.shcat > kube-proxy.service <<EOF[Unit]Description=KubernetesKube-ProxyServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target[Service]WorkingDirectory=${K8S_DIR}/kube-proxyExecStart=/opt/k8s/bin/kube-proxy \\--config=/etc/kubernetes/kube-proxy-config.yaml \\--logtostderr=true \\--v=2Restart=on-failureRestartSec=5LimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF分發 kube-proxy systemd unit 文件：

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_name in ${NODE_NAMES[@]}do echo ">>> ${node_name}" scp kube-proxy.service root@${node_name}:/etc/systemd/system/done啟動 kube-proxy 服務

cd /opt/k8s/worksource /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"mkdir -p ${K8S_DIR}/kube-proxy" ssh root@${node_ip}"modprobe ip_vs_rr" ssh root@${node_ip}"systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"done檢查啟動結果

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"systemctl status kube-proxy|grep Active"done檢查監聽埠

[root@abcdocker-k8s01 work]# netstat -lnpt|grep kube-proxtcp 00192.168.0.50:102490.0.0.0:* LISTEN 55015/kube-proxytcp 00192.168.0.50:102560.0.0.0:* LISTEN 55015/kube-proxy查看ipvs路由規則

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh root@${node_ip}"/usr/sbin/ipvsadm -ln"done正常輸出如下

可見所有通過 https 訪問 K8S SVC kubernetes 的請求都轉發到 kube-apiserver 節點的 6443 埠；

驗證集群功能現在使用daemonset驗證master和worker節點是否正常

[root@abcdocker-k8s01 work]# kubectl get nodeNAME STATUS ROLES AGE VERSIONk8s-01Ready<none>20m v1.14.2k8s-02Ready<none>20m v1.14.2k8s-03Ready<none>20m v1.14.2k8s-04Ready<none>20m v1.14.2創建測試yaml文件

cd /opt/k8s/workcat > nginx-ds.yml <<EOFapiVersion: v1kind:Servicemetadata: name: nginx-ds labels: app: nginx-dsspec: type:NodePort selector: app: nginx-ds ports:- name: http port:80 targetPort:80---apiVersion: extensions/v1beta1kind:DaemonSetmetadata: name: nginx-ds labels: addonmanager.kubernetes.io/mode:Reconcilespec:template: metadata: labels: app: nginx-ds spec: containers:- name:my-nginx image: daocloud.io/library/nginx:1.13.0-alpine ports:- containerPort:80EOF執行測試

kubectl create -f nginx-ds.yml這裡pod已經啟動成功

[root@abcdocker-k8s01 work]# kubectl get pod -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESnginx-ds-29n8p1/1Running0116s172.17.0.2 k8s-02<none><none>nginx-ds-7zhbb1/1Running0116s172.30.96.2 k8s-01<none><none>nginx-ds-kvr7q 1/1Running0116s172.17.0.2 k8s-04<none><none>nginx-ds-lk9dv 1/1Running0116s172.17.0.2 k8s-03<none><none>檢查各節點的Pod IP 連通性 這裡看到pod的IP，我們將ip複製一下

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh ${node_ip}"ping -c 1 172.17.0.2" ssh ${node_ip}"ping -c 1 172.30.96.2" ssh ${node_ip}"ping -c 1 172.17.0.2"done檢查服務IP和埠可達性

[root@abcdocker-k8s01 work]# kubectl get svc |grep nginx-dsnginx-ds NodePort10.254.248.73<none>80:15402/TCP 4m11s我們在任意節點訪問server IP

source /opt/k8s/bin/environment.shfor node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}" ssh ${node_ip}"curl -s 10.254.248.73"done#這裡請根據上面查看的svc IP進行修改此時我們已經可以使用任意節點IP+15402埠訪問nginx (這個埠是通過kubectl get svc獲取到的，每個人的埠可能不一樣。請按照實際情況進行修改！)

CoreDNS安裝上面我們驗證的集群內部網絡，已經沒有問題。接下來進行安裝DNS

這裡的所有操作在k8s01上執行即可

for NODE in192.168.0.50192.168.0.51192.168.0.52192.168.0.53;do echo "--- $NODE ---" ssh $NODE "wget -P /opt/ http://down.i4t.com/coredns_v1.4.tar" ssh $NODE "docker load -i /opt/coredns_v1.4.tar"done#下載鏡像並分發鏡像下載coredns yaml文件

wget -P /opt/ http://down.i4t.com/k8s1.14/coredns.yaml創建coredns

kubectl create -f /opt/coredns.yaml#這裡已經鏡像讓你們手動下載了，沒有下載請看docker步驟,最後一步執行完畢後，pod啟動成功 (Running狀態為正常)

kubectl get pod -n kube-system -l k8s-app=kube-dnsNAME READY STATUS RESTARTS AGEcoredns-d7964c8db-vgl5l 1/1Running021scoredns-d7964c8db-wvz5k 1/1Running021scoredns啟動之後，我們需要測一下dns功能是否正常

溫馨提示：busybox高版本有nslookup Bug，不建議使用高版本，請按照我的版本進行操作即可！

創建一個yaml文件測試是否正常

cat<<EOF | kubectl apply -f -apiVersion: v1kind:Podmetadata: name: busyboxnamespace:defaultspec: containers:- name: busybox image: busybox:1.28.3 command:- sleep-"3600" imagePullPolicy:IfNotPresent restartPolicy:AlwaysEOF創建後Pod我們進行檢查

kubectl get podNAME READY STATUS RESTARTS AGEbusybox 1/1Running04s使用nslookup查看是否能返回地址

kubectl exec-ti busybox -- nslookup kubernetesServer:10.254.0.2Address1:10.254.0.2 kube-dns.kube-system.svc.cluster.localName: kubernetesAddress1:10.254.0.1 kubernetes.default.svc.cluster.local默認kubectl沒有table補全命令，如果需要補全請參考下面文章

這裡我們也沒有安裝dashboard，需要dashboard請參考下面

需要prometheus監控請參考下面文章

如果有遇到安裝問題請及時在下方留言，或者加入QQ群找管理員協助解決！