和我一起學k8s（二）二進位部署一套k8s集群

作者：DevOps旭

來自：DevOps探路者

一、k8s環境準備

1、軟體版本

kubernetes v1.18.8docker v19.03.9-ceEtcd v3.4.9calico

2、伺服器

master 192.168.1.31node1 192.168.1.32node2 192.168.1.33

3、配置主機名

cat >> /etc/hosts <<EOF 192.168.1.31 k8s01 192.168.1.32 k8s02 192.168.1.33 k8s03EOF

4、免密鑰

在k8s-master節點上操作

ssh-keygen -f /root/.ssh/id_rsa -P &39; -qssh-copy-id k8s01ssh-copy-id k8s02ssh-copy-id k8s03

5、安裝依賴

在所有機器上執行

yum install -y epel-release conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget unzip net-tools

6、關閉防火牆、selinux、swap分區

在所有機器上執行

systemctl stop firewalldsystemctl disable firewalldiptables -F && iptables -X && iptables -F -t nat && iptables -X -t natiptables -P FORWARD ACCEPTswapoff -ased -i &\1/g&39;s/^SELINUX=.*/SELINUX=disabled/& 禁止使用 swap 空間，只有當系統 OOM 時才允許使用它vm.overcommit_memory=1 開啟 OOMfs.inotify.max_user_watches=1048576fs.inotify.max_user_instances=8192   fs.file-max=52706963fs.nr_open=52706963net.ipv6.conf.all.disable_ipv6=1net.netfilter.nf_conntrack_max=2310720vm.dirty_bytes=15728640EOFcp kubernetes.conf /etc/sysctl.d/kubernetes.conf 34;kernel:unregister_netdevice: waiting for eth0 to become free. Usage count = 1&34;signing&34;default&34;expiry&34;87600h&34;profiles&34;etcd&34;expiry&34;87600h&34;usages&34;signing&34;key encipherment&34;server auth&34;client auth&34;CN&34;etcd CA&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;L&34;Beijing&34;ST&34;Beijing&34;CN&34;etcd&34;hosts&34;192.168.1.31&34;192.168.1.32&34;192.168.1.33&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;L&34;BeiJing&34;ST&34;BeiJing& 在k8s01上cat > /opt/etcd/cfg/etcd.conf << EOF34;etcd-1&34;/var/lib/etcd/default.etcd&34;https://192.168.1.31:2380&34;https://192.168.1.31:2379&[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS=&34;ETCD_ADVERTISE_CLIENT_URLS=&34;ETCD_INITIAL_CLUSTER=&34;ETCD_INITIAL_CLUSTER_TOKEN=&34;ETCD_INITIAL_CLUSTER_STATE=&34;EOF[Member]ETCD_NAME=&34;ETCD_DATA_DIR=&34;ETCD_LISTEN_PEER_URLS=&34;ETCD_LISTEN_CLIENT_URLS=&34;34;https://192.168.1.32:2380&34;https://192.168.1.32:2379&34;etcd-1=https://192.168.1.31:2380,etcd-2=https://192.168.1.32:2380,etcd-3=https://192.168.1.33:2380&34;etcd-cluster&34;new& 在k8s03上cat > /opt/etcd/cfg/etcd.conf << EOF34;etcd-3&34;/var/lib/etcd/default.etcd&34;https://192.168.1.33:2380&34;https://192.168.1.33:2379&[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS=&34;ETCD_ADVERTISE_CLIENT_URLS=&34;ETCD_INITIAL_CLUSTER=&34;ETCD_INITIAL_CLUSTER_TOKEN=&34;ETCD_INITIAL_CLUSTER_STATE=&34;EOF​

創建etcd的systemd

cat > /usr/lib/systemd/system/etcd.service << EOF[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.target[Service]Type=notifyEnvironmentFile=/opt/etcd/cfg/etcd.confExecStart=/opt/etcd/bin/etcd \--cert-file=/opt/etcd/ssl/server.pem \--key-file=/opt/etcd/ssl/server-key.pem \--peer-cert-file=/opt/etcd/ssl/server.pem \--peer-key-file=/opt/etcd/ssl/server-key.pem \--trusted-ca-file=/opt/etcd/ssl/ca.pem \--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \--logger=zapRestart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF​

啟動etcd

systemctl daemon-reloadsystemctl start etcdsystemctl enable etcd​

查看集群狀態

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=&34; endpoint health

查看當前leader

[root@k8s01 ~]34;https://192.168.1.31:2379,https://192.168.1.32:2379,https://192.168.1.33:2379& githubwget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

2、部署docker

tar zxvf docker-19.03.9.tgzmv docker/* /usr/bin

3、創建docker的systemd

cat > /usr/lib/systemd/system/docker.service << EOF[Unit]Description=Docker Application Container EngineDocumentation=https://docs.docker.comAfter=network-online.target firewalld.serviceWants=network-online.target[Service]Type=notifyExecStart=/usr/bin/dockerdExecReload=/bin/kill -s HUP $MAINPIDLimitNOFILE=infinityLimitNPROC=infinityLimitCORE=infinityTimeoutStartSec=0Delegate=yesKillMode=processRestart=on-failureStartLimitBurst=3StartLimitInterval=60s[Install]WantedBy=multi-user.targetEOF​

4、創建配置文件

mkdir /etc/dockermkdir /mnt/docker-libcat > /etc/docker/daemon.json << EOF{    &34;: [&34;],    &34;: &34;}EOF​

5、啟動docker

systemctl daemon-reloadsystemctl start dockersystemctl enable docker

雲計算

四、部署kubernetes

1、下載kubernetes最新的二進位文件

wget https://dl.k8s.io/v1.19.0/kubernetes-server-linux-amd64.tar.gztar zvxf kubernetes-server-linux-amd64.tar.gzcd kubernetes/server/binmv kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/binmv kubectl /usr/bin/

2、部署apiserver

34;signing&34;default&34;expiry&34;87600h&34;profiles&34;kubernetes&34;expiry&34;87600h&34;usages&34;signing&34;key encipherment&34;server auth&34;client auth&34;CN&34;kubernetes&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;L&34;Beijing&34;ST&34;Beijing&34;O&34;k8s&34;OU&34;System& 生成證書cfssl gencert -initca ca-csr.json | cfssljson -bare ca -​34;CN&34;kubernetes&34;hosts&34;10.0.0.1&34;127.0.0.1&34;192.168.1.31&34;192.168.1.32&34;192.168.1.33&34;kubernetes&34;kubernetes.default&34;kubernetes.default.svc&34;kubernetes.default.svc.cluster&34;kubernetes.default.svc.cluster.local&34;key&34;algo&34;rsa&34;size&34;names&34;C&34;CN&34;L&34;BeiJing&34;ST&34;BeiJing&34;O&34;k8s&34;OU&34;System& 生成證書cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server​34;--logtostderr=false \\--v=2 \\--log-dir=/opt/kubernetes/logs \\--etcd-servers=https://192.168.31.71:2379,https://192.168.31.72:2379,https://192.168.31.73:2379 \\--bind-address=192.168.31.71 \\--secure-port=6443 \\--advertise-address=192.168.31.71 \\--allow-privileged=true \\--service-cluster-ip-range=10.0.0.0/24 \\--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\--authorization-mode=RBAC,Node \\--enable-bootstrap-token-auth=true \\--token-auth-file=/opt/kubernetes/cfg/token.csv \\--service-node-port-range=30000-32767 \\--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\--tls-cert-file=/opt/kubernetes/ssl/server.pem \\--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\--client-ca-file=/opt/kubernetes/ssl/ca.pem \\--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\--etcd-cafile=/opt/etcd/ssl/ca.pem \\--etcd-certfile=/opt/etcd/ssl/server.pem \\--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\--audit-log-maxage=30 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-path=/opt/kubernetes/logs/k8s-audit.log&34;system:node-bootstrapper&34;--logtostderr=false \\--v=2 \\--log-dir=/opt/kubernetes/log \\--leader-elect=true \\--master=127.0.0.1:8080 \\--bind-address=127.0.0.1 \\--allocate-node-cidrs=true \\--cluster-cidr=10.244.0.0/16 \\--service-cluster-ip-range=10.0.0.0/24 \\--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\--root-ca-file=/opt/kubernetes/ssl/ca.pem \\--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\--experimental-cluster-signing-duration=87600h0m0s&34;--logtostderr=false \--v=2 \--log-dir=/opt/kubernetes/log \--leader-elect \--master=127.0.0.1:8080 \--bind-address=127.0.0.1& k8s01cat > /opt/kubernetes/cfg/kubelet.conf << EOFKUBELET_OPTS=& k8s02cat > /opt/kubernetes/cfg/kubelet.conf << EOFKUBELET_OPTS=& k8s03cat > /opt/kubernetes/cfg/kubelet.conf << EOFKUBELET_OPTS=&34;kubelet-bootstrap&34;kubelet-bootstrap& 查看kubelet證書請求kubectl get csr34;--logtostderr=false \\--v=2 \\--log-dir=/opt/kubernetes/log \\--config=/opt/kubernetes/cfg/kube-proxy-config.yml& k8s01cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOFkind: KubeProxyConfigurationapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 0.0.0.0metricsBindAddress: 0.0.0.0:10249clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfighostnameOverride: k8s01clusterCIDR: 10.0.0.0/24EOF​ k8s03cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOFkind: KubeProxyConfigurationapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 0.0.0.0metricsBindAddress: 0.0.0.0:10249clientConnection: kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfighostnameOverride: k8s03clusterCIDR: 10.0.0.0/24EOF

生成kube-proxy.kubeconfig文件

cd ~/ssl/kubernetescat > kube-proxy-csr.json << EOF{  &34;: &34;,  &34;: [],  &34;: {    &34;: &34;,    &34;: 2048 },  &34;: [   {      &34;: &34;,      &34;: &34;,      &34;: &34;,      &34;: &34;,      &34;: &34;   } ]}EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

生成kubeconfig文件

kubectl config set-cluster kubernetes \  --certificate-authority=/opt/kubernetes/ssl/ca.pem \  --embed-certs=true \  --server=http://192.168.1.31:6443 \  --kubeconfig=kube-proxy.kubeconfigkubectl config set-credentials kube-proxy \  --client-certificate=./kube-proxy.pem \  --client-key=./kube-proxy-key.pem \  --embed-certs=true \  --kubeconfig=kube-proxy.kubeconfigkubectl config set-context default \  --cluster=kubernetes \  --user=kube-proxy \  --kubeconfig=kube-proxy.kubeconfigkubectl config use-context default --kubeconfig=kube-proxy.kubeconfig​cp kube-proxy.kubeconfig /opt/kubernetes/cfg/scp kube-proxy.kubeconfig k8s02:/opt/kubernetes/cfg/scp kube-proxy.kubeconfig k8s03:/opt/kubernetes/cfg/

生成systemd管理kube-proxy

cat > /usr/lib/systemd/system/kube-proxy.service << EOF[Unit]Description=Kubernetes ProxyAfter=network.target[Service]EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.confExecStart=/opt/kubernetes/cfg/kube-proxy \$KUBE_PROXY_OPTSRestart=on-failureLimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF​

啟動並設置開機啟動

systemctl daemon-reloadsystemctl start kube-proxysystemctl enable kube-proxy

7、cni網絡

下載CNI二進位文件

wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgzmkdir -p /opt/cni/bintar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin​

部署cni網絡

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.ymlkubectl apply -f kube-flannel.yml​

8、授權apiserver訪問kubelet

cat > apiserver-to-kubelet-rbac.yaml << EOFapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: annotations:   rbac.authorization.kubernetes.io/autoupdate: &34; labels:   kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubeletrules:  - apiGroups:      - &34;   resources:      - nodes/proxy      - nodes/stats      - nodes/log      - nodes/spec      - nodes/metrics      - pods/log   verbs:      - &34;---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: system:kube-apiserver namespace: &34;roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubeletsubjects:  - apiGroup: rbac.authorization.k8s.io   kind: User   name: kubernetesEOF​kubectl apply -f apiserver-to-kubelet-rbac.yaml​

至此單master的k8s集群部署完畢，下面我去討論多master。