【Kubernetes】日志平台EFK+Logstash+Kafka【实战】

一，环境准备

（1）下载镜像包（共3个）：

elasticsearch-7-12-1.tar.gz
fluentd-containerd.tar.gz
kibana-7-12-1.tar.gz

（2）在node节点导入镜像：

ctr -n=k8s.io images import elasticsearch-7-12-1.tar.gz 
ctr -n=k8s.io images import kibana-7-12-1.tar.gz 
ctr -n=k8s.io images import fluentd-containerd.tar.gz

（3）在master节点导入镜像：

ctr -n=k8s.io images import fluentd-containerd.tar.gz

二，实战操作详细步骤

1，创建命名空间

kubectl create ns kube-logging

---

2，创建elasticsearch的 service

# vim elasticsearch_svc.yaml kind: Service                  # service服务
apiVersion: v1
metadata:name: elasticsearchnamespace: kube-logging      # 指定命名空间labels:app: elasticsearch
spec:selector:app: elasticsearchclusterIP: None              # lesshead类型ports:- port: 9200               # 指定端口9200name: rest- port: 9300                # 指定端口9300name: inter-node

---

3，安装elasticsearch：类型Statefulset

（1）创建NFS

在所有节点执行如下命令，通过NFS 再创建存储类，实现存储类动态供给

1-1 安装NFS

#yum安装nfs
yum install nfs-utils -y#启动nfs服务
systemctl start nfs#设置nfs开机自启动
systemctl enable nfs.service

1-2 配置NSF

仅在master节点配置nfs文件，将master作为服务端。

# master上创建一个nfs共享目录
mkdir /data/v1 -p#编辑/etc/exports文件
vim /etc/exports
/data/v1 *(rw,no_root_squash)#加载配置，使配置生效
exportfs -arv# 重新启动nfs
systemctl restart nfs

（2）创建存储供应商（基于NFS）

2-1 创建sa账号

# vim serviceaccount.yaml apiVersion: v1
kind: ServiceAccount
metadata:name: nfs-provisioner

说明：为了使Pod里面的进程调用Kubernetes API或其他外部服务。从而指定serviceaccount之后，把pod创建出来，使用这个pod时，就有了我们指定的账户的权限。

2-2 对sa账号授权

kubectl create clusterrolebinding nfs-provisioner-clusterrolebinding --clusterrole=cluster-admin --serviceaccount=default:nfs-provisioner

2-3 创建nfs-provisioner供应商：类型Deployment

在node节点上，导入镜像：nfs-subdir-external-provisioner.tar.gz
创建供应商：

ctr -n=k8s.io images import nfs-client-provisioner.tar.gz

# vim deployment.yamlkind: Deployment
apiVersion: apps/v1
metadata:name: nfs-provisioner
spec:selector:matchLabels:app: nfs-provisionerreplicas: 1strategy:type: Recreatetemplate:metadata:labels:app: nfs-provisionerspec:serviceAccount: nfs-provisionercontainers:- name: nfs-provisionerimage: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0imagePullPolicy: IfNotPresentvolumeMounts:- name: nfs-client-rootmountPath: /persistentvolumesenv:- name: PROVISIONER_NAMEvalue: example.com/nfs- name: NFS_SERVERvalue: 192.168.40.180 #这个需要写nfs服务端所在的ip地址，大家需要写自己安装了nfs服务的机器ip- name: NFS_PATHvalue: /data/v1  #这个是nfs服务端共享的目录volumes:- name: nfs-client-rootnfs:server: 192.168.40.180   # 和上面保持一直path: /data/v1           # 和上面保持一直

#验证nfs是否创建成功
kubectl get pods | grep nfs

（3）创建Storageclass

# vim class.yamlapiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: do-block-storage
provisioner: example.com/nfs

注意：
provisioner: example.com/nfs
该值需要和 nfs-provisioner 配置的 PROVISIONER_NAME 处的value值保持一致。

（4）安装elasticsearch

在node节点，导入镜像：elasticsearch-7-12-1.tar.gz

ctr -n=k8s.io images import elasticsearch-7-12-1.tar.gz

# vim elasticsearch-statefulset.yamlapiVersion: apps/v1
kind: StatefulSet
metadata:name: es-clusternamespace: kube-logging
spec:serviceName: elasticsearchreplicas: 3selector:matchLabels:app: elasticsearchtemplate:metadata:labels:app: elasticsearchspec:containers:- name: elasticsearchimage: elasticsearch:7.12.1imagePullPolicy: IfNotPresentresources:limits:cpu: 1000mrequests:cpu: 100mports:- containerPort: 9200name: restprotocol: TCP- containerPort: 9300name: inter-nodeprotocol: TCPvolumeMounts:- name: datamountPath: /usr/share/elasticsearch/dataenv:- name: cluster.namevalue: k8s-logs- name: node.namevalueFrom:fieldRef:fieldPath: metadata.name- name: discovery.seed_hostsvalue: "es-cluster-0.elasticsearch.kube-logging.svc.cluster.local,es-cluster-1.elasticsearch.kube-logging.svc.cluster.local,es-cluster-2.elasticsearch.kube-logging.svc.cluster.local"   # 创建3个pod的完整域名，可简写- name: cluster.initial_master_nodesvalue: "es-cluster-0,es-cluster-1,es-cluster-2"- name: ES_JAVA_OPTSvalue: "-Xms512m -Xmx512m"initContainers:- name: fix-permissionsimage: busyboximagePullPolicy: IfNotPresentcommand: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]securityContext:privileged: truevolumeMounts:- name: datamountPath: /usr/share/elasticsearch/data- name: increase-vm-max-mapimage: busyboximagePullPolicy: IfNotPresentcommand: ["sysctl", "-w", "vm.max_map_count=262144"]securityContext:privileged: true- name: increase-fd-ulimitimage: busyboximagePullPolicy: IfNotPresentcommand: ["sh", "-c", "ulimit -n 65536"]securityContext:privileged: truevolumeClaimTemplates:- metadata:name: datalabels:app: elasticsearchspec:accessModes: [ "ReadWriteOnce" ]storageClassName: do-block-storageresources:requests:storage: 10Gi

注意：chmod -R 1000:1000 /usr/：中，这是用户ID（UID）和组ID（GID）的组合，分别用冒号分隔。
在大多数Linux发行版中，UID和GID为1000通常分配给第一个非root用户（即，安装系统后创建的第一个用户账户）。
这意味着该命令将文件或目录的所有者和组更改为UID和GID都为1000的用户和组。

---

4，安装kibana

创建前端service，及代理的后端pod1个

# vim kibana.yamlapiVersion: v1
kind: Service
metadata:name: kibananamespace: kube-logginglabels:app: kibana
spec:type: NodePortports:- port: 5601selector:app: kibana
---
apiVersion: apps/v1
kind: Deployment
metadata:name: kibananamespace: kube-logginglabels:app: kibana
spec:replicas: 1selector:matchLabels:app: kibanatemplate:metadata:labels:app: kibanaspec:containers:- name: kibanaimage:  kibana:7.12.1imagePullPolicy: IfNotPresentresources:limits:cpu: 1000mrequests:cpu: 100menv:- name: ELASTICSEARCH_URLvalue: http://elasticsearch:9200ports:- containerPort: 5601

kubectl get pods -n kube-loggingkubectl get svc -n kube-logging

浏览器访问地址：
http://<任意节点ip>：<前端service映射端口(32059)> :

---

5，安装fluentd：类型Daemonset

daemonset控制器可以保证集群中的每个节点都可以运行同样fluentd的pod副本。
从而，可以收集k8s集群中每个节点的日志，将应用应用程序容器的输入输出日志，重定向到node节点里的json文件中即可。

Fluentd不但可以把容器日志转换成指定的格式发送到elasticsearch集群中，还可以采集kubelet、kube-proxy、docker的日志。

创建fluentd服务的sa账号，并分配角色授权：

# vim fluentd.yaml apiVersion: v1
kind: ServiceAccount
metadata:name: fluentdnamespace: kube-logginglabels:app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: fluentdlabels:app: fluentd
rules:
- apiGroups:- ""resources:- pods- namespacesverbs:- get- list- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: fluentd
roleRef:kind: ClusterRolename: fluentdapiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccountname: fluentdnamespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: fluentdnamespace: kube-logginglabels:app: fluentd
spec:selector:matchLabels:app: fluentdtemplate:metadata:labels:app: fluentdspec:serviceAccount: fluentdserviceAccountName: fluentdtolerations:- key: node-role.kubernetes.io/control-planeeffect: NoSchedulecontainers:- name: fluentdimage: docker.io/fluent/fluentd-kubernetes-daemonset:v1.16-debian-elasticsearch7-1imagePullPolicy: IfNotPresentenv:- name:  FLUENT_ELASTICSEARCH_HOSTvalue: "elasticsearch.kube-logging.svc.cluster.local"- name:  FLUENT_ELASTICSEARCH_PORTvalue: "9200"- name: FLUENT_ELASTICSEARCH_SCHEMEvalue: "http"- name: FLUENTD_SYSTEMD_CONFvalue: disable            - name: FLUENT_CONTAINER_TAIL_PARSER_TYPEvalue: "cri"- name: FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMATvalue: "%Y-%m-%dT%H:%M:%S.%L%z"            resources:limits:memory: 512Mirequests:cpu: 100mmemory: 200MivolumeMounts:- name: varlogmountPath: /var/log- name: containersmountPath: /var/log/containersreadOnly: trueterminationGracePeriodSeconds: 30volumes:- name: varloghostPath:path: /var/log- name: containershostPath:path: /var/log/containers

【注意】日志格式化。容器运行时为containerd时，才加入。为docker时，不用加。
- name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
value: “cri”
- name: FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT
value: “%Y-%m-%dT%H:%M:%S.%L%z”

6，配置连接

https://www.elastic.co/guide/en/kibana/7.12/kuery-query.html

https://www.elastic.co/guide/en/kibana/7.12/kuery-query.html