Prometheus监控案例-tomcat、mysql、redis、haproxy、nginx

监控tomcat

tomcat自身并不能提供监控指标数据，需要借助第三方exporter实现：https://github.com/nlighten/tomcat_exporter

构建镜像

基于tomcat官方镜像，重新制作一个镜像，将tomcat-exporter和tomcat整合到一起。Ddockerfile如下：

FROM tomcat:8.5.73
LABEL maintainer="[e-mail]"ADD server.xml /usr/local/tomcat/conf/RUN mkdir -p /data/tomcat/webappsADD myapp /data/tomcat/webapps/myapp#添加tomcat-expoter相关文件
ADD metrics.war  /data/tomcat/webapps
ADD simpleclient-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
ADD tomcat_exporter_client-0.0.17.jar /usr/local/tomcat/lib/EXPOSE 8080 8443ADD run_tomcat.sh /usr/bin/CMD ["/usr/bin/run_tomcat.sh"]

myapp/目录下是一个测试页面index.jsp

root@master-01:~/dockerfile/tomcat# cat myapp/index.jsp
tomcat myapp

tomcat配置文件server.xml中需要指定app路径，并且打开war包的自动解压和部署功能

root@master-01:~/dockerfile/tomcat# cat server.xml |grep /data/tomcat<Host name="localhost"  appBase="/data/tomcat/webapps"  unpackWARs="true" autoDeploy="true">

run_tomcat.sh是启动tomcat脚本

root@master-01:~/dockerfile/tomcat# cat run_tomcat.sh
#!/bin/bash
/usr/local/tomcat/bin/catalina.sh starttail -f /etc/hosts

执行构建，上传到私有仓库

docker build -t harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73 .
docker push harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73

运行一个测试容器，验证metrics接口是否可以访问

docker run -d --name tomcat -p 8080:8080 harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73

部署tomcat

以Deployment方式运行tomcat

apiVersion: apps/v1
kind: Deployment
metadata:name: tomcat-deploymentnamespace: default
spec:selector:matchLabels:app: tomcatreplicas: 1template:metadata:labels:app: tomcatannotations:prometheus.io/scrape: 'true'prometheus.io/port: "8080"spec:containers:- name: tomcatimage: harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73imagePullPolicy: Alwaysports:- containerPort: 8080securityContext:privileged: true
---
kind: Service
apiVersion: v1
metadata:annotations:prometheus.io/scrape: 'true'prometheus.io/port: '8080'name: tomcat-service
spec:selector:app: tomcatports:- nodePort: 31080port: 80protocol: TCPtargetPort: 8080type: NodePort

将此yaml文件创建到集群上，然后等待Pod就绪

配置Prometheus采集数据

如果是部署在集群内部的Prometheus，建议使用角色为pod或endpoints的基于API Server的服务发现来发现target并监控。具体job配置可以参考之前的文章《Prometheus服务发现》

如果是集群外的Prometheus，添加一个job，内容如下：

- job_name: tomcat-exporterstatic_configs:- targets: ["192.168.122.12:31080"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

在Grafana上导入模板，查看监控数据。模板地址：https://github.com/nlighten/tomcat_exporter/blob/master/dashboard/example.json

监控mysql

Prometheus官方提供了mysql-expoter用于采集mysql指标数据，项目地址：https://github.com/prometheus/mysqld_exporter

部署mysql

直接以apt方式安装mysql

apt -y install mariadb-server
mysql

mysql授权

创建一个账户并授权，之后mysql-exporter会使用此账户来访问mysql查询相关信息

> CREATE USER 'mysql-exporter'@'localhost' IDENTIFIED BY 'Admin@123' WITH MAX_USER_CONNECTIONS 3;
> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysql-exporter'@'localhost';mysql -u mysql-exporter -p -h localhost;	#验证权限

部署mysql-exporter

下载安装包

wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz
tar xf mysqld_exporter-0.14.0.linux-amd64.tar.gz
mv mysqld_exporter-0.14.0.linux-amd64/mysqld_exporter /usr/bin/mysqld_exporter -h #查看帮助信息

准备mysql-exporter配置文件

root@prometheus-node-02:~# mkdir /etc/mysqld-exporter
root@prometheus-node-02:~# cat /etc/mysqld-exporter/.my.cnf
[client]
#指定连接mysql的账户和密码
user = mysql-exporter
password = Admin@123

准备service文件

root@prometheus-node-02:~# cat /lib/systemd/system/mysql-exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target[Service]
ExecStart=/usr/bin/mysqld_exporter --config.my-cnf=/etc/mysql-exporter/.my.cnf --web.listen-address=192.168.122.23:9104[Install]
WantedBy=multi-user.target

启动服务

systemctl daemon-reload
systemctl start mysql-exporter.service
systemctl status mysql-exporter.service
systemctl enable mysql-exporter.service

访问mysql-exporter指标接口测试

配置Prometheus采集数据

编辑Prometheus配置，添加一个job，内容如下：

  - job_name: mysql-exporterstatic_configs:- targets: ["192.168.122.23:9104"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板地址：https://github.com/prometheus/mysqld_exporter/blob/main/mysqld-mixin/dashboards/mysql-overview.json

监控redis

redis-exporter项目地址：https://github.com/oliver006/redis_exporter

部署redis&redis-exporter

将redis和redis-exporter运行在同一个Pod中，然后以Deployment方式部署Pod

apiVersion: apps/v1
kind: Deployment
metadata:name: redis-deploy
spec:replicas: 1selector:matchLabels:app: redistemplate:metadata:labels:app: redisannotations:prometheus.io/scrape: "true"pometheus.io/port: "9121"spec:containers:- name: redisimage: redisports:- containerPort: 6379resources:requests:cpu: 100mmemory: 100Mi- name: redis-exporterimage: oliver006/redis_exporterports:- containerPort: 9121resources:requests:cpu: 100mmemory: 100Mi

kind: Service
apiVersion: v1
metadata:annotations:prometheus.io/scrape: 'true'prometheus.io/port: "9121"name: redis-exporter-service
spec:selector:app: redisports:- nodePort: 31082name: promport: 9121protocol: TCPtargetPort: 9121type: NodePort---
kind: Service
apiVersion: v1
metadata:name: redis-redis-service
spec:selector:app: redisports:- nodePort: 31081name: redisport: 6379protocol: TCPtargetPort: 6379type: NodePort

将yaml文件创建到集群上，等待Pod就绪

通过 任意节点:31082/metrics 访问redis-exporter指标接口测试

配置Prometheus采集数据

编辑Prometheus配置，添加job，内容如下：

  - job_name: redis-exporterstatic_configs:- targets: ["192.168.122.13:31082"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板ID：14615

监控Haproxy

Prometheus官方提供了haproxy-exporter用于采集haproxy指标数据，项目地址：https://github.com/prometheus/haproxy_exporter

部署haproxy-exporter

下载安装包

wget https://github.com/prometheus/haproxy_exporter/releases/download/v0.15.0/haproxy_exporter-0.15.0.linux-amd64.tar.gz
tar xf haproxy_exporter-0.15.0.linux-amd64.tar.gz
mv haproxy_exporter-0.15.0.linux-amd64/haproxy_exporter /usr/bin/
haproxy_exporter -h		#查看帮助信息

haproxy-exporter可以通过两种方式连接haproxy来获取数据：

1. 通过socket文件连接haproxy获取数据，这种方式只能获取本机的haproxy数据
2. 通过haproxy状态页连接haproxy获取数据，这种方式可以获取本机或远程主机的数据，但需要在haproxy配置文件中开启状态页面配置

启动hapeoxy-exporter

#如果通过haproxy状态页获取数据，运行下面命令，其中haadmin:123456表示访问状态页的用户名密码
haproxy_exporter --haproxy.scrape-uri="http://haadmin:123456@127.0.0.1:9999/haproxy-status;csv" &#如果通过socket文件获取数据，运行下面命令
haproxy_exporter --haproxy.scrape-uri="unix:/run/haproxy/admin.sock" &

访问haproxy-exporter指标接口验证，默认端口9101

配置Prometheus采集数据

编辑Prometheus配置添加job，内容如下：

  - job_name: haproxy-exporterstatic_configs:- targets: ["192.168.122.14:9101", "192.168.122.15:9101"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板ID: 2428

监控nginx

nginx-exporter项目地址：https://github.com/hnlq715/nginx-vts-exporter

此nginx-exporter依赖一个nginx模块nginx-module-vts（https://github.com/vozlt/nginx-module-vts），所以需要编译安装nginx，并加载此模块。

nginx-module-vts模块会收集nginx指标数据，然后提供一个状态页面，nginx-exporter通过此状态页获取指标数据，再将数据转换为Prometheus需要的数据格式提供给Prometheus。

部署nginx

下载nginx源码包和nginx-module-vts模块

wget http://nginx.org/download/nginx-1.22.1.tar.gz
wget https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.1.tar.gz
tar xf nginx-module-vts-0.2.1.tar.gz -C /usr/local/src/
tar xvf nginx-1.22.1.tar.gz
cd nginx-1.22.1/./configure --prefix=/usr/local/nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-file-aio \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module \
--add-module=/usr/local/src/nginx-module-vts-0.2.1/

修改nginx配置文件

http {vhost_traffic_status_zone;	#启用状态页...server {...location /status {	#状态页面location配置vhost_traffic_status_display;vhost_traffic_status_display_format html;}}
}

启动nginx

/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx

访问状态页测试

还可以用json格式输出数据

部署nginx-exporter

wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
tar xf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter /usr/bin/
nginx-vts-exporter  -nginx.scrape_uri http://192.168.122.23/status/format/json &	#启动nginx-exporter

访问nginx-exporter指标接口测试

配置Prometheus采集数据

编辑Prometheus配置，添加Job，内容如下：

  - job_name: nginx-exporterstatic_configs:- targets: ["192.168.122.23:9913"]

配置修改完成后，重启Prometheus。然后通过Prometheus界面查看target状态

Grafana导入模板，查看数据。模板地址：https://github.com/hnlq715/nginx-vts-exporter/blob/master/dashboard/nginx-vts-exporter.json

监控ingress-nginx-controller

ingress-nginx-controller程序自身可以提供指标数据，默认通过端口10254/metrics提供指标数据，在部署ingress-nginx-controller时候需要在容器上定义10254端口，并在ingress-nginx-controller对应的Service资源上也定义10254端口

        ports:- containerPort: 80name: httpprotocol: TCP- containerPort: 443name: httpsprotocol: TCP- containerPort: 8443name: webhookprotocol: TCP- name: metricscontainerPort: 10254protocol: TCP

apiVersion: v1
kind: Service
metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/version: 1.3.1name: ingress-nginx-controllernamespace: ingress-nginx
spec:ipFamilies:- IPv4ipFamilyPolicy: SingleStackports:- appProtocol: httpname: httpport: 80protocol: TCPtargetPort: http- appProtocol: httpsname: httpsport: 443protocol: TCPtargetPort: https- name: metricsport: 10254targetPort: 10254protocol: TCPselector:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/name: ingress-nginxtype: NodePort

之后就可以通过nodePort访问ingress-nginx-controller的指标接口

然后在Prometheus中配置中添加Job，内容如下

  - job_name: nginx-ingress-controllerstatic_configs:- targets: ["192.168.122.11:38144"]

配置修改完成后，重启Prometheus。通过Prometheus界面查看target状态

Grafana导入模板，查看监控数据。模板ID：9614