「Kubernetes 系列」 Prometheus Operator 定製指南

前置要求

• Kubernetes版本：1.16
• Kube-Prometheus版本：release-0.4
• 編譯工具：jb、jsontoyaml、jsonnet

Kube-Prometheus簡介

Git地址：https://github.com/coreos/kube-prometheus.git

Prometheus 在K8S 中部署主要通過Coreos 所開發的Prometheus Operator 方式部署的，Operator也是Coreos開發的用於擴展K8S API的，具體會單獨一章細講。這裡我們針對Prometheus監控感受下它。

包含組件：

• The Prometheus Operator
• 高可用Prometheus
• 高可用 Alertmanager
• Prometheus node-exporter
• Prometheus Adapter for Kubernetes Metrics APIs
• kube-state-metrics
• Grafana

格式：倉庫代碼格式都是jsonnet格式的文件，方便我們高效定製開發，但是相應的也需要一些學習門檻。

工具安裝

go get github.com/brancz/gojsontoyaml

go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb (更新jb)

jb update (更新kube-prometheus)

部署

git clone jsonnet

定製編譯入口

Prometheus 和 Grafana 都不提供靜態的Configmap 配置文件方式。

默認會有一個基礎版的example.jsonnet，這裡面定義了我們所需組件的一些變量和配置等。同樣在examples目錄中有一些特殊定製的案例：

這裡我們會修改這個example.jsonnet文件加入一些生產環境所需配置：

• Prometheus持久化存儲（prometheus-data：10G）
• Grafana 增加配置（GF_AUTH_GENERIC_OAUTH_ENABLED&39;true&39;ksonnet/ksonnet.beta.4/k.libsonnet&34;persistentVolumeClaim:: {&persistentvolumeclaim-v1-core (defines variable named &39; of type &39;)

  local env = k.apps.v1.deployment.mixin.spec.template.spec.containersType.envType;

  local kp =

  (import &39;) +

  (import &39;) +

  (import &39;) +

  // Uncomment the following imports to enable its patches

  // (import &39;) +

  // (import &39;) +

  // (import &39;) +

  // (import &39;) +

  // (import &39;) +

  {

  _config+:: {

  namespace: &39;,

  versions+:: {

  grafana: &39;,

  },

  grafana+:: {

  plugins+: [&39;],

  env+: [

  env.new(&39;, &39;),

  ],

  },

  },

  prometheus+:: {

  prometheus+: {

  spec+: { // https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md39;t specified for &39;, then by default the &39; arg will be passed to prometheus by prometheus-operator.

  // The possible values for a prometheus <duration> are:

  // * https://github.com/prometheus/common/blob/c7de230/model/time.go34;^([0-9]+)(y|w|d|h|m|s|ms)$&39;30d&39;storage.volumeClaimTemplate&39;t created), prometheus will be created with an EmptyDir for the &39; volume (for the prom tsdb).

  // This &39; causes the following to be automatically created (via dynamic provisioning) for each prometheus pod:

  // * PersistentVolumeClaim (and a corresponding PersistentVolume)

  // * the actual volume (per the StorageClassName specified below)

  storage: { // https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md39;pvc&core.v1.persistentVolumeClaim.new

  pvc.mixin.spec.withAccessModes(&39;) +

  // https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/39;requests&storage-resource-quota (defines &39;)

  pvc.mixin.spec.resources.withRequests({ storage: &39; }) +

  // A StorageClass of the following name (which can be seen via `kubectl get storageclass` from a node in the given K8s cluster) must exist prior to kube-prometheus being deployed.

  pvc.mixin.spec.withStorageClassName(&39;),

  // The following &39; is only needed if you&manual-storage-provisioning).

  // And note that this is not supported/allowed by AWS - uncommenting the following &39; line (when deploying kube-prometheus to a K8s cluster in AWS) will cause the pvc to be stuck in the Pending status and have the following error:

  // * &34;ssd&39;

  //pvc.mixin.spec.selector.withMatchLabels({}),

  }, // storage

  }, // spec

  }, // prometheus

  }, // prometheus

  };

  { [&39; + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +

  {

  [&39; + name]: kp.prometheusOperator[name]

  for name in std.filter((function(name) name != &39;), std.objectFields(kp.prometheusOperator))

  } +

  // serviceMonitor is separated so that it can be created after the CRDs are ready

  { &39;: kp.prometheusOperator.serviceMonitor } +

  { [&39; + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +

  { [&39; + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +

  { [&39; + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +

  { [&39; + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +

  { [&39; + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +

  { [&39; + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

  編譯生成K8S manifest文件

  cd kube-prometheus

  jb install (生成vendor目錄。關鍵步驟，否則下面編譯無法通過)

  ./build.sh example.jsonnet

  (which uses vendor/ to render all manifests in a json structure of {filename: manifest-content})

  (import &39;) +

  (import &39;) +

  查看生成的manifest文件：

  

  apiVersion: monitoring.coreos.com/v1

  kind: Prometheus

  metadata:

  labels:

  prometheus: k8s

  name: k8s

  namespace: monitoring

  spec:

  alerting:

  alertmanagers:

  - name: alertmanager-main

  namespace: monitoring

  port: web

  image: quay.io/prometheus/prometheus:v2.19.2

  nodeSelector:

  kubernetes.io/os: linux

  podMonitorNamespaceSelector: {}

  podMonitorSelector: {}

  replicas: 2

  resources:

  requests:

  memory: 400Mi

  retention: 30d

  ruleSelector:

  matchLabels:

  prometheus: k8s

  role: alert-rules

  securityContext:

  fsGroup: 2000

  runAsNonRoot: true

  runAsUser: 1000

  serviceAccountName: prometheus-k8s

  serviceMonitorNamespaceSelector: {}

  serviceMonitorSelector: {}

  storage:

  volumeClaimTemplate:

  apiVersion: v1

  kind: PersistentVolumeClaim

  spec:

  accessModes:

  - ReadWriteOnce

  resources:

  requests:

  storage: 10Gi

  storageClassName: prometheus-data

  version: v2.19.2

  部署

  $ kubectl apply -f manifests/setup

  $ kubectl apply -f manifests/

  

  每天三分鐘，知識效率高。輕文章，更高效。

  歡迎關注同名微信公眾帳號極客24h。