kube-prometheus-stack 自定义 alertmanager 配置推送webhook

创建AlertmanagerConfig资源

在没有使用 prometheus-operator 的情况下，需要手动配置 alertmanager.yaml 来路由&发送从 prometheus 接收的警报。

使用 prometheus-operator 之后，事情变得简单一些。只需要创建 AlertmanagerConfig 资源，prometheus-operator 会自动 merge 所有的 AlertmanagerConfig 资源生成/更新 alertmanager.yaml，并通知 alertmanager 重载配置。

默认情况下，prometheus-operator会关注所有namespace下的所有AlertmanagerConfig：

kubectl get -n kube-prom alertmanagerskubectl get -n kube-prom alertmanagers/kube-promethues-stack-kube-alertmanager -o yaml# spec.alertmanagerConfigNamespaceSelector: {}，表示不作筛选
# spec.alertmanagerConfigSelector: {}，表示不作筛选

创建一个简单警报路由规则

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:name: testwebhooknamespace: kube-prom
spec:route:receiver: webhookgroupBy: ["instance", "job"]groupWait: "10s"groupInterval: "20s"repeatInterval: "30s"receivers:- name: webhookwebhookConfigs:- url: "http://10.0.2.11:8080/webhook/send"sendResolved: trueinhibitRules:- sourceMatch:- name: severityvalue: 'critical'targetMatch:- name: severityvalue: 'warning'equal: ['instance']

参考：

https://github.com/prometheus-community/helm-charts/issues/2224
https://kkgithub.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#alertmanagerconfig

kubectl apply -f alertmanager-config.yaml

kubectl edit svc kube-promethues-stack-kube-alertmanager -n kube-prom
kubectl get svc kube-promethues-stack-kube-alertmanager -n kube-prom

创建资源后，打开alertmanager管理后台 http://10.0.2.12:32466/#/status 页面，确认 Config 已经包含相关的配置信息（可能需要稍等一会）。

AlertmanagerConfig 资源详情参考：https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#alertmanagerconfig

创建 PrometheusRule 资源

类似 AlertmanagerConfig，可以通过创建 PrometheusRule 资源来创建警报规则（rule），prometheus-operator 会自动把所有 rule 配置 merge 到 prometheus.yml。

默认情况下，prometheus-operator 会关注所有 namespace 下匹配 label release=kube-prometheus-stack 的 PrometheusRule ：

kubectl get -n kube-prom prometheuses
kubectl get -n kube-prom prometheuses/kube-promethues-stack-kube-prometheus -o yaml
# spec.ruleNamespaceSelector: {}，表示不作筛选
# spec.ruleSelector:
#   matchLabels:
#     release: kube-prometheus-stack

创建一个能立即触发报警的规则:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:labels:prometheus: k8sole: alert-rulesname: kube-prom-kube-prom-stack-kube-prome-prometheus.rulesnamespace: kube-prom
spec:groups:- name: diskrules:- alert: diskFreeannotations:value: "{{$value}}"summary: "{{ $labels.job }}  项目实例 {{ $labels.instance }} 磁盘使用率大于 80%"description: "{{ $labels.instance }}  {{ $labels.mountpoint }}  磁盘使用率大于80%  (当前的值: {{ $value }}%),请及时处理"expr: |(1-(node_filesystem_free_bytes{fstype=~"ext4|xfs",mountpoint!="/boot"} / node_filesystem_size_bytes{fstype=~"ext4|xfs",mountpoint!="/boot"}) )*100 > 80for: 1mlabels:severity: warning

kubectl apply -f prometheus-rule.yaml

注意：labels 的severity: warning和前面创建 AlertmanagerConfig 的 inhibitRules 配置匹配，为什么需要namespace: kube-prom？prometheus-operator 会在 AlertmanagerConfig 的 matchers 强制加上这个标签，issue 讨论：https://github.com/prometheus-operator/prometheus-operator/issues/3737

kubectl edit svc kube-promethues-stack-kube-prometheus -n kube-prom 
kubectl get svc kube-promethues-stack-kube-prometheus -n kube-prom

创建资源后，打开prometheus管理后台 http://10.0.2.12:30133/rules页面，搜索diskFree确认能找到新添加的规则（可能需要稍等一会)。

PrometheusRule 资源详情参考：https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrule

编写 /webhook/send 接口

创建springboot项目，添加如下依赖

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.olive</groupId><artifactId>test-promethues</artifactId><version>0.0.1-SNAPSHOT</version><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId><version>3.2.0</version></dependency><dependency><groupId>com.alibaba.fastjson2</groupId><artifactId>fastjson2</artifactId><version>2.0.49</version></dependency></dependencies>
</project>

创建 controller

package com.olive;import java.time.LocalDateTime;
import java.util.HashMap;
import java.util.Map;import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RestController;import com.alibaba.fastjson2.JSON;@RestController
public class RevcController {@PostMapping("/webhook/send")public Map<String, String> create(@RequestBody Map<String, Object> entity) {System.out.println(LocalDateTime.now());System.out.println(JSON.toJSONString(entity));Map<String, String> result = new HashMap<String, String>();result.put("code", "success");return result;}}

创建springboot引导类

package com.olive;import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;@SpringBootApplication
public class App {public static void main(String[] args) {SpringApplication.run(App.class, args);}
}

参考：

https://www.cnblogs.com/roy2220/p/14867024.html