在《當Docker遇到systemd》一文中，我提到過這兩天兒一直在做的一個task：使用kubeadm在Ubuntu 16.04上安裝部署Kubernetes的最新發布版本-k8s 1.5.1。
年中，Docker宣布在Docker engine中集成swarmkit工具包，這一announcement在輕量級容器界引發軒然大波。畢竟開發者是懶惰的^0^，有了docker swarmkit，驅動developer去安裝其他容器編排工具的動力在哪裡呢？即便docker engine還不是當年那個被人們高頻使用的IE瀏覽器。作為針對Docker公司這一市場行為的回應，容器集群管理和服務編排領先者Kubernetes在三個月後發布了Kubernetes1.4.0版本。在這個版本中K8s新增了kubeadm工具。kubeadm的使用方式有點像集成在docker engine中的swarm kit工具，旨在改善開發者在安裝、調試和使用k8s時的體驗，降低安裝和使用門檻。理論上通過兩個命令：init和join即可搭建出一套完整的Kubernetes cluster。
不過，和初入docker引擎的swarmkit一樣，kubeadm目前也在active development中，也不是那麼stable，因此即便在當前最新的k8s 1.5.1版本中，它仍然處於Alpha狀態，官方不建議在Production環境下使用。每次執行kubeadm init時，它都會列印如下提醒日誌：
[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.
不過由於之前部署的k8s 1.3.7集群運行良好，這給了我們在k8s這條路上繼續走下去並走好的信心。但k8s在部署和管理方面的體驗的確是太繁瑣了，於是我們準備試驗一下kubeadm是否能帶給我們超出預期的體驗。之前在aliyun ubuntu 14.04上安裝kubernetes 1.3.7的經驗和教訓，讓我略微有那麼一丟丟底氣，但實際安裝過程依舊是一波三折。這既與kubeadm的unstable有關，同樣也與cni、第三方網絡add-ons的質量有關。無論哪一方出現問題都會讓你的install過程異常坎坷曲折。
一、環境與約束在kubeadm支持的Ubuntu 16.04+, CentOS 7 or HypriotOS v1.0.1+三種作業系統中，我們選擇了Ubuntu 16.04。由於阿里雲尚無官方16.04 Image可用，我們新開了兩個Ubuntu 14.04ECS實例，並通過apt-get命令手工將其升級到Ubuntu 16.04.1，詳細版本是：Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-58-generic x86_64)。
Ubuntu 16.04使用了systemd作為init system，在安裝和配置Docker時，可以參考我的這篇《當Docker遇到system》。Docker版本我選擇了目前可以得到的lastest stable release: 1.12.5。
# docker versionClient: Version: 1.12.5 API version: 1.24 Go version: go1.6.4 Git commit: 7392c3b Built: Fri Dec 16 02:42:17 2016 OS/Arch: linux/amd64Server: Version: 1.12.5 API version: 1.24 Go version: go1.6.4 Git commit: 7392c3b Built: Fri Dec 16 02:42:17 2016 OS/Arch: linux/amd64
至於Kubernetes版本，前面已經提到過了，我們就使用最新發布的Kubernetes 1.5.1版本。1.5.1是1.5.0的一個緊急fix版本，主要」to address default flag values which in isolation were not problematic, but in concert could result in an insecure cluster」。官方建議skip 1.5.0，直接用1.5.1。
這裡再重申一下：Kubernetes的安裝、配置和調通是很難的，在阿里雲上調通就更難了，有時還需要些運氣。Kubernetes、Docker、cni以及各種網絡Add-ons都在active development中，也許今天還好用的step、tip和trick，明天就out-dated，因此在借鑑本文的操作步驟時，請謹記這些^0^。
二、安裝包準備我們這次新開了兩個ECS實例，一個作為master node，一個作為minion node。Kubeadm默認安裝時，master node將不會參與Pod調度，不會承載work load，即不會有非核心組件的Pod在Master node上被創建出來。當然通過kubectl taint命令可以解除這一限制，不過這是後話了。
集群拓撲：
master node：10.47.217.91，主機名：iZ25beglnhtZminion node：10.28.61.30，主機名：iZ2ze39jeyizepdxhwqci6Z
本次安裝的主參考文檔就是Kubernetes官方的那篇《Installing Kubernetes on Linux with kubeadm》。
本小節，我們將進行安裝包準備，即將kubeadm以及此次安裝所需要的k8s核心組件統統下載到上述兩個Node上。注意：如果你有加速器，那麼本節下面的安裝過程將尤為順利，反之，… 。以下命令，在兩個Node上均要執行。
1、添加apt-key# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -OK
2、添加Kubernetes源並更新包信息添加Kubernetes源到sources.list.d目錄下：
# cat <<EOF > /etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF# cat /etc/apt/sources.list.d/kubernetes.listdeb http://apt.kubernetes.io/ kubernetes-xenial main
更新包信息：
# apt-get update... ...Hit:2 http://mirrors.aliyun.com/ubuntu xenial InReleaseHit:3 https://apt.dockerproject.org/repo ubuntu-xenial InReleaseGet:4 http://mirrors.aliyun.com/ubuntu xenial-security InRelease [102 kB]Get:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [6,299 B]Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages [1,739 B]Get:6 http://mirrors.aliyun.com/ubuntu xenial-updates InRelease [102 kB]Get:7 http://mirrors.aliyun.com/ubuntu xenial-proposed InRelease [253 kB]Get:8 http://mirrors.aliyun.com/ubuntu xenial-backports InRelease [102 kB]Fetched 568 kB in 19s (28.4 kB/s)Reading package lists... Done
3、下載Kubernetes核心組件在此次安裝中，我們通過apt-get就可以下載Kubernetes的核心組件，包括kubelet、kubeadm、kubectl和kubernetes-cni等。
# apt-get install -y kubelet kubeadm kubectl kubernetes-cniReading package lists... DoneBuilding dependency treeReading state information... DoneThe following package was automatically installed and is no longer required: libtimedate-perlUse 'apt autoremove' to remove it.The following additional packages will be installed: ebtables ethtool socatThe following NEW packages will be installed: ebtables ethtool kubeadm kubectl kubelet kubernetes-cni socat0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.Need to get 37.6 MB of archives.After this operation, 261 MB of additional disk space will be used.Get:2 http://mirrors.aliyun.com/ubuntu xenial/main amd64 ebtables amd64 2.0.10.4-3.4ubuntu1 [79.6 kB]Get:6 http://mirrors.aliyun.com/ubuntu xenial/main amd64 ethtool amd64 1:4.5-1 [97.5 kB]Get:7 http://mirrors.aliyun.com/ubuntu xenial/universe amd64 socat amd64 1.7.3.1-1 [321 kB]Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.3.0.1-07a8a2-00 [6,877 kB]Get:3 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.5.1-00 [15.1 MB]Get:4 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.5.1-00 [7,954 kB]Get:5 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.6.0-alpha.0-2074-a092d8e0f95f52-00 [7,120 kB]Fetched 37.6 MB in 36s (1,026 kB/s)... ...Unpacking kubeadm (1.6.0-alpha.0-2074-a092d8e0f95f52-00) ...Processing triggers for systemd (229-4ubuntu13) ...Processing triggers for ureadahead (0.100.0-19) ...Processing triggers for man-db (2.7.5-1) ...Setting up ebtables (2.0.10.4-3.4ubuntu1) ...update-rc.d: warning: start and stop actions are no longer supported; falling back to defaultsSetting up ethtool (1:4.5-1) ...Setting up kubernetes-cni (0.3.0.1-07a8a2-00) ...Setting up socat (1.7.3.1-1) ...Setting up kubelet (1.5.1-00) ...Setting up kubectl (1.5.1-00) ...Setting up kubeadm (1.6.0-alpha.0-2074-a092d8e0f95f52-00) ...Processing triggers for systemd (229-4ubuntu13) ...Processing triggers for ureadahead (0.100.0-19) ...... ...
下載後的kube組件並未自動運行起來。在 /lib/systemd/system下面我們能看到kubelet.service：
# ls /lib/systemd/system|grep kubekubelet.service//kubelet.service[Unit]Description=kubelet: The Kubernetes Node AgentDocumentation=http://kubernetes.io/docs/[Service]ExecStart=/usr/bin/kubeletRestart=alwaysStartLimitInterval=0RestartSec=10[Install]WantedBy=multi-user.target
kubelet的版本：
# kubelet --versionKubernetes v1.5.1
k8s的核心組件都有了，接下來我們就要boostrap kubernetes cluster了。同時，問題也就隨之而來了，而這些問題以及問題的解決才是本篇要說明的重點。
三、初始化集群前面說過，理論上通過kubeadm使用init和join命令即可建立一個集群，這init就是在master節點對集群進行初始化。和k8s 1.4之前的部署方式不同的是，kubeadm安裝的k8s核心組件都是以容器的形式運行於master node上的。因此在kubeadm init之前，最好給master node上的docker engine掛上加速器代理，因為kubeadm要從gcr.io/google_containers repository中pull許多核心組件的images，大約有如下一些：
gcr.io/google_containers/kube-controller-manager-amd64 v1.5.1 cd5684031720 2 weeks ago 102.4 MBgcr.io/google_containers/kube-apiserver-amd64 v1.5.1 8c12509df629 2 weeks ago 124.1 MBgcr.io/google_containers/kube-proxy-amd64 v1.5.1 71d2b27b03f6 2 weeks ago 175.6 MBgcr.io/google_containers/kube-scheduler-amd64 v1.5.1 6506e7b74dac 2 weeks ago 53.97 MBgcr.io/google_containers/etcd-amd64 3.0.14-kubeadm 856e39ac7be3 5 weeks ago 174.9 MBgcr.io/google_containers/kubedns-amd64 1.9 26cf1ed9b144 5 weeks ago 47 MBgcr.io/google_containers/dnsmasq-metrics-amd64 1.0 5271aabced07 7 weeks ago 14 MBgcr.io/google_containers/kube-dnsmasq-amd64 1.4 3ec65756a89b 3 months ago 5.13 MBgcr.io/google_containers/kube-discovery-amd64 1.0 c5e0c9a457fc 3 months ago 134.2 MBgcr.io/google_containers/exechealthz-amd64 1.2 93a43bfb39bf 3 months ago 8.375 MBgcr.io/google_containers/pause-amd64 3.0 99e59f495ffa 7 months ago 746.9 kB
在Kubeadm的文檔中，Pod Network的安裝是作為一個單獨的步驟的。kubeadm init並沒有為你選擇一個默認的Pod network進行安裝。我們將首選Flannel 作為我們的Pod network，這不僅是因為我們的上一個集群用的就是flannel，而且表現穩定。更是由於Flannel就是coreos為k8s打造的專屬overlay network add-ons。甚至於flannel repository的readme.md都這樣寫著：「flannel is a network fabric for containers, designed for Kubernetes」。如果我們要使用Flannel，那麼在執行init時，按照kubeadm文檔要求，我們必須給init命令帶上option：–pod-network-cidr=10.244.0.0/16。
1、執行kubeadm init執行kubeadm init命令：
# kubeadm init --pod-network-cidr=10.244.0.0/16[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.[preflight] Running pre-flight checks[preflight] Starting the kubelet service[init] Using Kubernetes version: v1.5.1[tokens] Generated token: "2e7da9.7fc5668ff26430c7"[certificates] Generated Certificate Authority key and certificate.[certificates] Generated API Server key and certificate[certificates] Generated Service Account signing keys[certificates] Created keys and certificates in "/etc/kubernetes/pki"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"[apiclient] Created API client, waiting for the control plane to become ready //如果沒有掛加速器，可能會在這裡hang住。[apiclient] All control plane components are healthy after 54.789750 seconds[apiclient] Waiting for at least one node to register and become ready[apiclient] First node is ready after 1.003053 seconds[apiclient] Creating a test deployment[apiclient] Test deployment succeeded[token-discovery] Created the kube-discovery deployment, waiting for it to become ready[token-discovery] kube-discovery is ready after 62.503441 seconds[addons] Created essential addon: kube-proxy[addons] Created essential addon: kube-dnsYour Kubernetes master has initialized successfully!You should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:http://kubernetes.io/docs/admin/addons/You can now join any number of machines by running the following on each node:kubeadm join --token=2e7da9.7fc5668ff26430c7 123.56.200.187
init成功後的master node有啥變化？k8s的核心組件均正常啟動：
# ps -ef|grep kuberoot 2477 2461 1 16:36 ? 00:00:04 kube-proxy --kubeconfig=/run/kubeconfigroot 30860 1 12 16:33 ? 00:01:09 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.localroot 30952 30933 0 16:33 ? 00:00:01 kube-scheduler --address=127.0.0.1 --leader-elect --master=127.0.0.1:8080root 31128 31103 2 16:33 ? 00:00:11 kube-controller-manager --address=127.0.0.1 --leader-elect --master=127.0.0.1:8080 --cluster-name=kubernetes --root-ca-file=/etc/kubernetes/pki/ca.pem --service-account-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem --insecure-experimental-approve-all-kubelet-csrs-for-group=system:kubelet-bootstrap --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16root 31223 31207 2 16:34 ? 00:00:10 kube-apiserver --insecure-bind-address=127.0.0.1 --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota --service-cluster-ip-range=10.96.0.0/12 --service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem --client-ca-file=/etc/kubernetes/pki/ca.pem --tls-cert-file=/etc/kubernetes/pki/apiserver.pem --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem --token-auth-file=/etc/kubernetes/pki/tokens.csv --secure-port=6443 --allow-privileged --advertise-address=123.56.200.187 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --anonymous-auth=false --etcd-servers=http://127.0.0.1:2379root 31491 31475 0 16:35 ? 00:00:00 /usr/local/bin/kube-discovery
而且是多以container的形式啟動：
# docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESc16c442b7eca gcr.io/google_containers/kube-proxy-amd64:v1.5.1 "kube-proxy --kubecon" 6 minutes ago Up 6 minutes k8s_kube-proxy.36dab4e8_kube-proxy-sb4sm_kube-system_43fb1a2c-cb46-11e6-ad8f-00163e1001d7_2ba1648e9f73998e01d7 gcr.io/google_containers/kube-discovery-amd64:1.0 "/usr/local/bin/kube-" 8 minutes ago Up 8 minutes k8s_kube-discovery.7130cb0a_kube-discovery-1769846148-6z5pw_kube-system_1eb97044-cb46-11e6-ad8f-00163e1001d7_fd49c2e3dd5412e5e15c gcr.io/google_containers/kube-apiserver-amd64:v1.5.1 "kube-apiserver --ins" 9 minutes ago Up 9 minutes k8s_kube-apiserver.1c5a91d9_kube-apiserver-iz25beglnhtz_kube-system_eea8df1717e9fea18d266103f9edfac3_8cae848560017f8819b2 gcr.io/google_containers/etcd-amd64:3.0.14-kubeadm "etcd --listen-client" 9 minutes ago Up 9 minutes k8s_etcd.c323986f_etcd-iz25beglnhtz_kube-system_3a26566bb004c61cd05382212e3f978f_06d517eb03c2463aba9c gcr.io/google_containers/kube-controller-manager-amd64:v1.5.1 "kube-controller-mana" 9 minutes ago Up 9 minutes k8s_kube-controller-manager.d30350e1_kube-controller-manager-iz25beglnhtz_kube-system_9a40791dd1642ea35c8d95c9e610e6c1_3b05cb8afb9a724540a7 gcr.io/google_containers/kube-scheduler-amd64:v1.5.1 "kube-scheduler --add" 9 minutes ago Up 9 minutes k8s_kube-scheduler.ef325714_kube-scheduler-iz25beglnhtz_kube-system_dc58861a0991f940b0834f8a110815cb_9b3ccda2.... ...
不過這些核心組件並不是跑在pod network中的（沒錯，此時的pod network還沒有創建），而是採用了host network。以kube-apiserver的pod信息為例：
kube-system kube-apiserver-iz25beglnhtz 1/1 Running 0 1h 10.47.217.91 iz25beglnhtz
kube-apiserver的IP是host ip，從而推斷容器使用的是host網絡，這從其對應的pause容器的network屬性就可以看出：
# docker ps |grep apiservera5a76bc59e38 gcr.io/google_containers/kube-apiserver-amd64:v1.5.1 "kube-apiserver --ins" About an hour ago Up About an hour k8s_kube-apiserver.2529402_kube-apiserver-iz25beglnhtz_kube-system_25d646be9a0092138dc6088fae6f1656_ec0079fcef4d3bf057a6 gcr.io/google_containers/pause-amd64:3.0 "/pause" About an hour ago Up About an hour k8s_POD.d8dbe16c_kube-apiserver-iz25beglnhtz_kube-system_25d646be9a0092138dc6088fae6f1656_bbfd8a31
inspect pause容器，可以看到pause container的NetworkMode的值：
"NetworkMode": "host",
如果kubeadm init執行過程中途出現了什麼問題，比如前期忘記掛加速器導致init hang住，你可能會ctrl+c退出init執行。重新配置後，再執行kubeadm init，這時你可能會遇到下面kubeadm的輸出：
# kubeadm init --pod-network-cidr=10.244.0.0/16[kubeadm] WARNING: kubeadm is in alpha, please do not use it for production clusters.[preflight] Running pre-flight checks[preflight] Some fatal errors occurred: Port 10250 is in use /etc/kubernetes/manifests is not empty /etc/kubernetes/pki is not empty /var/lib/kubelet is not empty /etc/kubernetes/admin.conf already exists /etc/kubernetes/kubelet.conf already exists[preflight] If you know what you are doing, you can skip pre-flight checks with `--skip-preflight-checks`
kubeadm會自動檢查當前環境是否有上次命令執行的「殘留」。如果有，必須清理後再行執行init。我們可以通過」kubeadm reset」來清理環境，以備重來。
# kubeadm reset[preflight] Running pre-flight checks[reset] Draining node: "iz25beglnhtz"[reset] Removing node: "iz25beglnhtz"[reset] Stopping the kubelet service[reset] Unmounting mounted directories in "/var/lib/kubelet"[reset] Removing kubernetes-managed containers[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/etcd][reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki][reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf]
鑑於文章長度超過公眾號文章限制，請點擊「查看全文」閱讀文章全部內容。