k8s監控系統prometheus-operator

簡介

prometheus-operator
Prometheus：一個非常優秀的監控工具或者說是監控方案。它提供了數據搜集、存儲、處理、可視化和告警一套完整的解決方案。作為kubernetes官方推薦的監控系統，用Prometheus來監控kubernetes集群的狀況和運行在集群上的應用運行狀況。

Prometheus架構圖

那Prometheus Operator是做什麼的呢？
Operator是由CoreOS公司開發的，用來擴展 Kubernetes API，特定的應用程式控制器，它用來創建、配置和管理複雜的有狀態應用，如資料庫、緩存和監控系統。
可以理解為，Prometheus Operator就是用於管理部署Prometheus到kubernetes的工具，其目的是簡化和自動化對Prometheus組件的維護。

Prometheus Operator架構

部署前準備

1、克隆kube-prometheus項目

[root@k8s-master001 opt] ls -altotal 20drwxr-xr-x. 10 root root  140 Sep 14 21:25 .drwxr-xr-x. 12 root root 4096 Sep 14 21:11 ..drwxr-xr-x.  2 root root 4096 Sep 14 21:23 adapterdrwxr-xr-x.  2 root root  189 Sep 14 21:22 alertmanagerdrwxr-xr-x.  2 root root  241 Sep 14 21:22 exporterdrwxr-xr-x.  2 root root  254 Sep 14 21:23 grafanadrwxr-xr-x.  2 root root  272 Sep 14 21:22 metricsdrwxr-xr-x.  2 root root 4096 Sep 14 21:25 prometheusdrwxr-xr-x.  2 root root 4096 Sep 14 21:23 serviceMonitordrwxr-xr-x.  2 root root 4096 Sep 14 21:11 setup

3、修改yaml文件中的nodeSelector
首先查看下現在Node節點的標籤

[root@k8s-master001 manifests] grep -A1 nodeSelector  prometheus/*prometheus/prometheus-prometheus.yaml:  nodeSelector:prometheus/prometheus-prometheus.yaml:  nodeSelector:prometheus/prometheus-prometheus.yaml-    kubernetes.io/os: linux

部署kube-prometheus

1、安裝operator

[root@k8s-master001 manifests] kubectl  get po -n monitoringNAME                                   READY   STATUS    RESTARTS   AGEprometheus-operator-74d54b5cfc-xgqg7   2/2     Running   0          2m40s

2、安裝adapter

[root@k8s-master001 manifests] kubectl  get po -n monitoringNAME                                   READY   STATUS    RESTARTS   AGEprometheus-adapter-557648f58c-9x446    1/1     Running   0          41sprometheus-operator-74d54b5cfc-xgqg7   2/2     Running   0          4m33s

3、安裝alertmanager

[root@k8s-master001 manifests] kubectl  get po -n monitoringNAME                                   READY   STATUS    RESTARTS   AGEalertmanager-main-0                    2/2     Running   0          53malertmanager-main-1                    2/2     Running   0          3m3salertmanager-main-2                    2/2     Running   0          53m

4、安裝exporter

[root@k8s-master001 manifests] kubectl  get po -n monitoring NAME                                   READY   STATUS    RESTARTS   AGEnode-exporter-2rvtt                    2/2     Running   0          108snode-exporter-9kwb6                    2/2     Running   0          108snode-exporter-9zlbb                    2/2     Running   0          108s

5、安裝metrics

[root@k8s-master001 manifests] kubectl  get po -n monitoringNAME                                   READY   STATUS    RESTARTS   AGEkube-state-metrics-85cb9cfd7c-v9c4f    3/3     Running   0          2m8s

6、安裝prometheus

[root@k8s-master001 manifests] kubectl  get po -n monitoringNAME                                   READY   STATUS    RESTARTS   AGEprometheus-k8s-0                       3/3     Running   1          94sprometheus-k8s-1                       3/3     Running   1          94s

7、安裝grafana

root@k8s-master001 manifests] kubectl  get po -n monitoringNAME                                   READY   STATUS    RESTARTS   AGEgrafana-b558fb99f-87spq                1/1     Running   0          3m14s

8、安裝serviceMonitor

[root@k8s-master001 manifests] kubectl  get po -n monitoring NAME                                   READY   STATUS    RESTARTS   AGEalertmanager-main-0                    2/2     Running   0          90malertmanager-main-1                    2/2     Running   0          40malertmanager-main-2                    2/2     Running   0          90mgrafana-b558fb99f-87spq                1/1     Running   0          4m56skube-state-metrics-85cb9cfd7c-v9c4f    3/3     Running   0          10mnode-exporter-2rvtt                    2/2     Running   0          35mnode-exporter-9kwb6                    2/2     Running   0          35mnode-exporter-9zlbb                    2/2     Running   0          35mprometheus-adapter-557648f58c-9x446    1/1     Running   0          91mprometheus-k8s-0                       3/3     Running   1          7m49sprometheus-k8s-1                       3/3     Running   1          7m49sprometheus-operator-74d54b5cfc-xgqg7   2/2     Running   0          95mNAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGEservice/alertmanager-main       ClusterIP   10.98.96.94     <none>        9093/TCP                     91mservice/alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   91mservice/grafana                 ClusterIP   10.108.204.33   <none>        3000/TCP                     6m30sservice/kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP            12mservice/node-exporter           ClusterIP   None            <none>        9100/TCP                     36mservice/prometheus-adapter      ClusterIP   10.98.16.117    <none>        443/TCP                      93mservice/prometheus-k8s          ClusterIP   10.109.119.37   <none>        9090/TCP                     9m22sservice/prometheus-operated     ClusterIP   None            <none>        9090/TCP                     9m24sservice/prometheus-operator     ClusterIP   None            <none>        8443/TCP                     97m

10、使用nodeport暴露grafana和prometheus服務，訪問UI界面

---apiVersion: v1kind: Servicemetadata:  name: grafana-svc  namespace: monitoringspec:  type: NodePort  ports:  - port: 3000    targetPort: 3000  selector:    app: grafana---apiVersion: v1kind: Servicemetadata:  name: prometheus-svc  namespace: monitoringspec:  type: NodePort  ports:  - port: 9090    targetPort: 9090  selector:    prometheus: k8s

查看結果

[root@k8s-master001 manifests]FFFFFF; --tt-darkmode-color: EF7060; --tt-darkmode-bgcolor:  grep -A2 -B2  selector serviceMonitor/prometheus-serviceMonitorKube*serviceMonitor/prometheus-serviceMonitorKubeControllerManager.yaml-    matchNames:serviceMonitor/prometheus-serviceMonitorKubeControllerManager.yaml-    - kube-systemserviceMonitor/prometheus-serviceMonitorKubeControllerManager.yaml:  selector:serviceMonitor/prometheus-serviceMonitorKubeControllerManager.yaml-    matchLabels:serviceMonitor/prometheus-serviceMonitorKubeControllerManager.yaml-      k8s-app: kube-controller-manager--serviceMonitor/prometheus-serviceMonitorKubelet.yaml-    matchNames:serviceMonitor/prometheus-serviceMonitorKubelet.yaml-    - kube-systemserviceMonitor/prometheus-serviceMonitorKubelet.yaml:  selector:serviceMonitor/prometheus-serviceMonitorKubelet.yaml-    matchLabels:serviceMonitor/prometheus-serviceMonitorKubelet.yaml-      k8s-app: kubelet--serviceMonitor/prometheus-serviceMonitorKubeScheduler.yaml-    matchNames:serviceMonitor/prometheus-serviceMonitorKubeScheduler.yaml-    - kube-systemserviceMonitor/prometheus-serviceMonitorKubeScheduler.yaml:  selector:serviceMonitor/prometheus-serviceMonitorKubeScheduler.yaml-    matchLabels:serviceMonitor/prometheus-serviceMonitorKubeScheduler.yaml-      k8s-app: kube-scheduler

2、創建kube-controller-manager和kube-scheduler service
k8s v1.19默認使用https，kube-controller-manager埠10257 kube-scheduler埠10259
kube-controller-manager-scheduler.yml

apiVersion: v1kind: Servicemetadata:  namespace: kube-system  name: kube-controller-manager  labels:    k8s-app: kube-controller-managerspec:  selector:    component: kube-controller-manager  type: ClusterIP  clusterIP: None  ports:  - name: https-metrics    port: 10257    targetPort: 10257    protocol: TCP---apiVersion: v1kind: Servicemetadata:  namespace: kube-system  name: kube-scheduler  labels:    k8s-app: kube-schedulerspec:  selector:    component: kube-scheduler  type: ClusterIP  clusterIP: None  ports:  - name: https-metrics    port: 10259    targetPort: 10259    protocol: TCP

執行命令

[root@k8s-master001 manifests] kubectl  get svc -n kube-system NAME                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                        AGEkube-controller-manager   ClusterIP   None         <none>        10257/TCP                      37mkube-scheduler            ClusterIP   None         <none>        10259/TCP                      37m

3、創建kube-controller-manager和kube-scheduler endpoint
注意：addresses改成集群實際的IP
kube-ep.yml

apiVersion: v1kind: Endpointsmetadata:  labels:    k8s-app: kube-controller-manager  name: kube-controller-manager  namespace: kube-systemsubsets:- addresses:  - ip: 10.26.25.20  - ip: 10.26.25.21  - ip: 10.26.25.22  ports:  - name: https-metrics    port: 10257    protocol: TCP---apiVersion: v1kind: Endpointsmetadata:  labels:    k8s-app: kube-scheduler  name: kube-scheduler  namespace: kube-systemsubsets:- addresses:  - ip: 10.26.25.20  - ip: 10.26.25.21  - ip: 10.26.25.22  ports:  - name: https-metrics    port: 10259    protocol: TCP

[root@k8s-master001 manifests] kubectl  get ep -n kube-systemNAME                      ENDPOINTS                                                        AGEkube-controller-manager   10.26.25.20:10257,10.26.25.21:10257,10.26.25.22:10257            16mkube-scheduler            10.26.25.20:10259,10.26.25.21:10259,10.26.25.22:10259            16m

現在看下頁面上prometheus target，已經能看到kube-controller-manager和kube-scheduler被監控了

坑位二

1、默認清理下，kube-controller-manager和kube-scheduler綁定IP為127.0.0.1，如果需要監控這兩個服務，需要修改kube-controller-manager和kube-scheduler配置，讓其綁定到0.0.0.0
2、配置文件所在目錄/etc/kubernetes/manifests
修改kube-controller-manager.yaml中--bind-address=0.0.0.0
修改kube-scheduler.yaml中--bind-address=0.0.0.0
3、重啟kubelet:systemctl restart kubelet
4、查看是否生效，返回200即為成功

[root@k8s-master002 manifests] curl -I -k https://10.26.25.20:10259/healthzHTTP/1.1 200 OKCache-Control: no-cache, privateContent-Type: text/plain; charset=utf-8X-Content-Type-Options: nosniffDate: Tue, 15 Sep 2020 06:19:36 GMTContent-Length: 2

最後

kube-prometheus配置很多，這裡只是做了最基礎的設置。更多需求請自行查看官方文檔

註：文中圖片來源於網絡，如有侵權，請聯繫我及時刪除。