飞天使-k8sv1.14二进制安装
文章目录
- 安装前准备
- 安装前设置分发脚本
- 开始安装k8s集群
- cfssl 安装
- 部署kubectl命令行工具
- 创建admin证书和私钥
- 创建kubeconfig文件
- 部署ETCD集群
- 部署Flannel网络
- kube-apiserver 高可用
- KeepLived 部署
- 部署master节点
- 部署高可用kube-controller-manager集群
- kube-controller-manager 创建权限
- 部署高可用kube-scheduler
- work节点安装
- 部署Docker组件
- 部署kubelet组件
- 部署kube-proxy组件
- 验证集群功能
- CoreDNS安装
- 参考文章
安装前准备
192.168.100.26 k8s-01
192.168.100.27 k8s-02
192.168.100.28 k8s-03#node节点
192.168.100.29 k8s-04 #node节点只运行node，但是设置证书的时候要添加这个ip
本文档的所有etcd集群、master集群、worker节点均使用以上三台机器，并且初始化步骤需要在所有机器上执行命令。如果没有特殊命令，所有操作均在192.168.100.26上进行操作
hostnamectl set-hostname k8s-01 #所有机器按照要求修改
hostnamectl set-hostname k8s-02
hostnamectl set-hostname k8s-03
hostnamectl set-hostname k8s-04bash #刷新主机名
# 设置/etc/hosts
cat >> /etc/hosts <<EOF
192.168.100.26 k8s-01
192.168.100.27 k8s-02
192.168.100.28 k8s-03
192.168.100.29 k8s-04
EOF# 只要在k8s-01 设置免密，主机密码 123456,各位自己改成自己的
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y expectsed -i "s@PermitRootLogin no@PermitRootLogin yes@g" /etc/ssh/sshd_config
sed -i "s@PasswordAuthentication no@PasswordAuthentication yes@g" /etc/ssh/sshd_config
systemctl restart sshd
#分发公钥
ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa
for i in k8s-01 k8s-02 k8s-03 k8s-04;do
expect -c "
spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$iexpect {\"*yes/no*\" {send \"yes\r\"; exp_continue}\"*password*\" {send \"123456\r\"; exp_continue}\"*Password*\" {send \"123456\r\";}} "
done # 更新path ，本次路径全部在/opt 目录下
echo 'PATH=/opt/k8s/bin:$PATH' >>/etc/profile
source /etc/profile
env|grep PATH# 内核以及基础配置更新,内核升级步骤可以自选
yum -y install gcc gcc-c++ make autoconf libtool-ltdl-devel gd-devel freetype-devel libxml2-devel libjpeg-devel libpng-devel openssh-clients openssl-devel curl-devel bison patch libmcrypt-devel libmhash-devel ncurses-devel binutils compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel glibc glibc-common glibc-devel libgcj libtiff pam-devel libicu libicu-devel gettext-devel libaio-devel libaio libgcc libstdc++ libstdc++-devel unixODBC unixODBC-devel numactl-devel glibc-headers sudo bzip2 mlocate flex lrzsz sysstat lsof setuptool system-config-network-tui system-config-firewall-tui ntsysv ntp pv lz4 dos2unix unix2dos rsync dstat iotop innotop mytop telnet iftop expect cmake nc gnuplot screen xorg-x11-utils xorg-x11-xinit rdate bc expat-devel compat-expat1 tcpdump sysstat man nmap curl lrzsz elinks finger bind-utils traceroute mtr ntpdate zip unzip vim wget net-tools
systemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
iptables -P FORWARD ACCEPT
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config# reboot 后yum update -y 执行下# 加载内核模块调优
find /lib/modules/`uname -r`/ -name "ip_vs_rr*"
find /lib/modules/`uname -r`/ -name "br_netfilter*"
cat > /etc/rc.local << EOF
modprobe ip_vs_rr
modprobe br_netfilter
EOF
cat > /etc/modules-load.d/ipvs.conf << EOFip_vs_rrbr_netfilter
EOF
systemctl enable --now systemd-modules-load.service
#验证是否成功
lsmod |egrep " ip_vs_rr|br_netfilter"cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.overcommit_memory=1 # 不检查物理内存是否够用
vm.panic_on_oom=0 # 开启 OOM
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
cp kubernetes.conf /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf
需要关闭tcp_tw_recycle，否则和NAT冲突，会导致服务不通
关闭IPV6，防止触发Docker BUGtimedatectl set-timezone Asia/Shanghai#将当前的 UTC 时间写入硬件时钟
timedatectl set-local-rtc 0#重启依赖于系统时间的服务
systemctl restart rsyslog
systemctl restart crond
mkdir -p /opt/k8s/{bin,work} /etc/{kubernetes,etcd}/cert
#在所有节点上执行，因为flanneld是在所有节点运行的
安装前设置分发脚本
后续所有的使用环境变量都定义在environment.sh中，需要根据个人机器及网络环境修改。并且需要拷贝到所有节点的/opt/k8s/bin目录下
cat environment.sh
#!/usr/bin/bash
# 生成 EncryptionConfig 所需的加密 key
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
# 集群各机器 IP 数组
export NODE_IPS=( 192.168.100.26 192.168.100.27 192.168.100.28 192.168.100.29 )
# 集群各 IP 对应的主机名数组
export NODE_NAMES=(k8s-01 k8s-02 k8s-03 k8s-04)
# 集群MASTER机器 IP 数组
export MASTER_IPS=(192.168.100.26 192.168.100.27 192.168.100.28 )
# 集群所有的master Ip对应的主机
export MASTER_NAMES=(k8s-01 k8s-02 k8s-03)
# etcd 集群服务地址列表
export ETCD_ENDPOINTS="https://192.168.100.26:2379,https://192.168.100.27:2379,https://192.168.100.28:2379"
# etcd 集群间通信的 IP 和端口
export ETCD_NODES="k8s-01=https://192.168.100.26:2380,k8s-02=https://192.168.100.27:2380,k8s-03=https://192.168.100.28:2380"
# etcd 集群所有node ip
export ETCD_IPS=(192.168.100.26 192.168.100.27 192.168.100.28 192.168.100.29 )
# kube-apiserver 的反向代理(kube-nginx)地址端口
export KUBE_APISERVER="https://192.168.100.199:8443"
# 节点间互联网络接口名称
export IFACE="eth0"
# etcd 数据目录
export ETCD_DATA_DIR="/data/k8s/etcd/data"
# etcd WAL 目录，建议是 SSD 磁盘分区，或者和 ETCD_DATA_DIR 不同的磁盘分区
export ETCD_WAL_DIR="/data/k8s/etcd/wal"
# k8s 各组件数据目录
export K8S_DIR="/data/k8s/k8s"
# docker 数据目录
#export DOCKER_DIR="/data/k8s/docker"
## 以下参数一般不需要修改
# TLS Bootstrapping 使用的 Token，可以使用命令 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成
#BOOTSTRAP_TOKEN="41f7e4ba8b7be874fcff18bf5cf41a7c"
# 最好使用 当前未用的网段 来定义服务网段和 Pod 网段
# 服务网段，部署前路由不可达，部署后集群内路由可达(kube-proxy 保证)
SERVICE_CIDR="10.254.0.0/16"
# Pod 网段，建议 /16 段地址，部署前路由不可达，部署后集群内路由可达(flanneld 保证)
CLUSTER_CIDR="172.30.0.0/16"
# 服务端口范围 (NodePort Range)
export NODE_PORT_RANGE="1024-32767"
# flanneld 网络配置前缀
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
# kubernetes 服务 IP (一般是 SERVICE_CIDR 中第一个IP)
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"
# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
export CLUSTER_DNS_SVC_IP="10.254.0.2"
# 集群 DNS 域名（末尾不带点号）
export CLUSTER_DNS_DOMAIN="cluster.local"
# 将二进制目录 /opt/k8s/bin 加到 PATH 中
export PATH=/opt/k8s/bin:$PATH#分发环境变量
source environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"scp environment.sh root@${node_ip}:/opt/k8s/bin/ssh root@${node_ip} "chmod +x /opt/k8s/bin/* "
done
开始安装k8s集群
cfssl 安装
为确保安全，kubernetes各个组件需要使用x509证书对通信进行加密和认证
CA(Certificate Authority)是自签名的根证书，用来签名后续创建的其他证书。本文章使用CloudFlare的PKI工具cfssl创建所有证书。注意: 如果没有特殊指明，本文档的所有操作均在k8s-01节点执行，远程分发到其他节点
#安装cfssl
mkdir -p /opt/k8s/cert && cd /opt/k8s
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
mv cfssl_linux-amd64 /opt/k8s/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
mv cfssljson_linux-amd64 /opt/k8s/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /opt/k8s/bin/cfssl-certinfo
chmod +x /opt/k8s/bin/*
export PATH=/opt/k8s/bin:$PATHcd /opt/k8s/work
cat > ca-config.json <<EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"kubernetes": {"usages": ["signing","key encipherment","server auth","client auth"],"expiry": "87600h"}}}
}
EOF######################
signing 表示该证书可用于签名其它证书，生成的ca.pem证书找中CA=TRUE
server auth 表示client可以用该证书对server提供的证书进行验证
client auth 表示server可以用该证书对client提供的证书进行验证cd /opt/k8s/work
cat > ca-csr.json <<EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "4Paradigm"}],"ca": {"expiry": "876000h"}
}
EOF#######################
CN CommonName,kube-apiserver从证书中提取该字段作为请求的用户名(User Name)，浏览器使用该字段验证网站是否合法
O Organization,kube-apiserver 从证书中提取该字段作为请求用户和所属组(Group)
kube-apiserver将提取的User、Group作为RBAC授权的用户和标识cd /opt/k8s/work
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls ca*#分发证书
#将生成的CA证书、秘钥文件、配置文件拷贝到所有节点的/etc/kubernetes/cert目录下cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p /etc/kubernetes/cert"scp ca*.pem ca-config.json root@${node_ip}:/etc/kubernetes/certdone部署kubectl命令行工具
cd /opt/k8s/work
wget http://down.i4t.com/k8s1.14/kubernetes-client-linux-amd64.tar.gz
tar -xzvf kubernetes-client-linux-amd64.tar.gz
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"scp kubernetes/client/bin/kubectl root@${node_ip}:/opt/k8s/bin/ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"done创建admin证书和私钥
kubectl与apiserver https通信，apiserver对提供的证书进行认证和授权。kubectl作为集群的管理工具，需要被授予最高权限，这里创建具有最高权限的admin证书创建证书签名请求
cd /opt/k8s/work
cat > admin-csr.json <<EOF
{"CN": "admin","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "system:masters","OU": "4Paradigm"}]
}
EOF###################
● O 为system:masters，kube-apiserver收到该证书后将请求的Group设置为system:masters
● 预定的ClusterRoleBinding cluster-admin将Group system:masters与Role cluster-admin绑定，该Role授予API的权限
● 该证书只有被kubectl当做client证书使用，所以hosts字段为空
cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes admin-csr.json | cfssljson -bare admin
ls admin*
创建kubeconfig文件
kubeconfig为kubectl的配置文件，包含访问apiserver的所有信息，如apiserver地址、CA证书和自身使用的证书
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
# 设置集群参数
kubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kubectl.kubeconfig
#设置客户端认证参数
kubectl config set-credentials admin \--client-certificate=/opt/k8s/work/admin.pem \--client-key=/opt/k8s/work/admin-key.pem \--embed-certs=true \--kubeconfig=kubectl.kubeconfig
# 设置上下文参数
kubectl config set-context kubernetes \--cluster=kubernetes \--user=admin \--kubeconfig=kubectl.kubeconfig
# 设置默认上下文
kubectl config use-context kubernetes --kubeconfig=kubectl.kubeconfig################
--certificate-authority 验证kube-apiserver证书的根证书
--client-certificate、--client-key 刚生成的admin证书和私钥，连接kube-apiserver时使用
--embed-certs=true 将ca.pem和admin.pem证书嵌入到生成的kubectl.kubeconfig文件中 (如果不加入，写入的是证书文件路径，后续拷贝kubeconfig到其它机器时，还需要单独拷贝证书)cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p ~/.kube"scp kubectl.kubeconfig root@${node_ip}:~/.kube/configdone部署ETCD集群
cd /opt/k8s/work
wget http://down.i4t.com/k8s1.14/etcd-v3.3.13-linux-amd64.tar.gz
tar -xvf etcd-v3.3.13-linux-amd64.tar.gz
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}doecho ">>> ${node_ip}"scp etcd-v3.3.13-linux-amd64/etcd* root@${node_ip}:/opt/k8s/binssh root@${node_ip} "chmod +x /opt/k8s/bin/*"done# 创建etcd 证书密钥cd /opt/k8s/work
cat > etcd-csr.json <<EOF
{"CN": "etcd","hosts": ["127.0.0.1","192.168.100.26","192.168.100.27","192.168.100.28"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "4Paradigm"}]
}
EOF#host字段指定授权使用该证书的etcd节点IP或域名列表，需要将etcd集群的3个节点都添加其中cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*pemcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${ETCD_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p /etc/etcd/cert"scp etcd*.pem root@${node_ip}:/etc/etcd/cert/done# 创建启动文件cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > etcd.service.template <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos
[Service]
Type=notify
WorkingDirectory=${ETCD_DATA_DIR}
ExecStart=/opt/k8s/bin/etcd \\--data-dir=${ETCD_DATA_DIR} \\--wal-dir=${ETCD_WAL_DIR} \\--name=##NODE_NAME## \\--cert-file=/etc/etcd/cert/etcd.pem \\--key-file=/etc/etcd/cert/etcd-key.pem \\--trusted-ca-file=/etc/kubernetes/cert/ca.pem \\--peer-cert-file=/etc/etcd/cert/etcd.pem \\--peer-key-file=/etc/etcd/cert/etcd-key.pem \\--peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \\--peer-client-cert-auth \\--client-cert-auth \\--listen-peer-urls=https://##NODE_IP##:2380 \\--initial-advertise-peer-urls=https://##NODE_IP##:2380 \\--listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \\--advertise-client-urls=https://##NODE_IP##:2379 \\--initial-cluster-token=etcd-cluster-0 \\--initial-cluster=${ETCD_NODES} \\--initial-cluster-state=new \\--auto-compaction-mode=periodic \\--auto-compaction-retention=1 \\--max-request-bytes=33554432 \\--quota-backend-bytes=6442450944 \\--heartbeat-interval=250 \\--election-timeout=2000
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF配置说明 (此处不需要修改任何配置)WorkDirectory、--data-dir 指定etcd工作目录和数据存储为${ETCD_DATA_DIR},需要在启动前创建这个目录 (后面跟着我操作就可以，会有创建步骤)
--wal-dir 指定wal目录，为了提高性能，一般使用SSD和--data-dir不同的盘
--name 指定节点名称，当--initial-cluster-state值为new时，--name的参数值必须位于--initial-cluster列表中
--cert-file、--key-file ETCD server与client通信时使用的证书和私钥
--trusted-ca-file 签名client证书的CA证书，用于验证client证书
--peer-cert-file、--peer-key-file ETCD与peer通信使用的证书和私钥
--peer-trusted-ca-file 签名peer证书的CA证书，用于验证peer证书#分发会将配置文件中的#替换成ip
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))dosed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${ETCD_IPS[i]}/" etcd.service.template > etcd-${ETCD_IPS[i]}.service done
ls *.service#NODE_NAMES 和 NODE_IPS 为相同长度的 bash 数组，分别为节点名称和对应的 IP；cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.servicedone重命名etcd启动文件并启动etcd服务etcd首次进程启动会等待其他节点加入etcd集群，执行启动命令会卡顿一会，为正常现象cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p ${ETCD_DATA_DIR} ${ETCD_WAL_DIR}"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl restart etcd " &done#这里我们创建了etcd的工作目录#检查启动结果
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status etcd|grep Active"done正常状态下为这种情况
# for node_ip in ${MASTER_IPS[@]}
> do
> echo ">>> ${node_ip}"
> ssh root@${node_ip} "systemctl status etcd|grep Active"
> done
>>> 192.168.100.26Active: active (running) since Mon 2023-08-21 16:10:15 CST; 52s ago
>>> 192.168.100.27Active: active (running) since Mon 2023-08-21 16:10:15 CST; 52s ago
>>> 192.168.100.28Active: active (running) since Mon 2023-08-21 16:10:15 CST; 52s ago可以用这个命令查看journalctl -fu etcd验证etcd是否正常
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ETCDCTL_API=3 /opt/k8s/bin/etcdctl \--endpoints=https://${node_ip}:2379 \--cacert=/etc/kubernetes/cert/ca.pem \--cert=/etc/etcd/cert/etcd.pem \--key=/etc/etcd/cert/etcd-key.pem endpoint healthdone正常为如下操作
# for node_ip in ${MASTER_IPS[@]}
> do
> echo ">>> ${node_ip}"
> ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
> --endpoints=https://${node_ip}:2379 \
> --cacert=/etc/kubernetes/cert/ca.pem \
> --cert=/etc/etcd/cert/etcd.pem \
> --key=/etc/etcd/cert/etcd-key.pem endpoint health
> done
>>> 192.168.100.26
https://192.168.100.26:2379 is healthy: successfully committed proposal: took = 2.247021ms
>>> 192.168.100.27
https://192.168.100.27:2379 is healthy: successfully committed proposal: took = 2.080702ms
>>> 192.168.100.28
https://192.168.100.28:2379 is healthy: successfully committed proposal: took = 2.262168ms通过etcd 查看集群的leader
source /opt/k8s/bin/environment.sh
ETCDCTL_API=3 /opt/k8s/bin/etcdctl \-w table --cacert=/etc/kubernetes/cert/ca.pem \--cert=/etc/etcd/cert/etcd.pem \--key=/etc/etcd/cert/etcd-key.pem \--endpoints=${ETCD_ENDPOINTS} endpoint status
# ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
> -w table --cacert=/etc/kubernetes/cert/ca.pem \
> --cert=/etc/etcd/cert/etcd.pem \
> --key=/etc/etcd/cert/etcd-key.pem \
> --endpoints=${ETCD_ENDPOINTS} endpoint status
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://192.168.100.26:2379 | 4fb7383fdc28cdcb | 3.3.13 | 20 kB | true | 2 | 8 |
| https://192.168.100.27:2379 | e69d641fa8a08905 | 3.3.13 | 20 kB | false | 2 | 8 |
| https://192.168.100.28:2379 | 1f54d9cf6d187788 | 3.3.13 | 20 kB | false | 2 | 8 |
+-----------------------------+------------------+---------+---------+-----------+-----------+------------+部署Flannel网络
Kubernetes要求集群内各个节点(包括master)能通过Pod网段互联互通，Flannel使用vxlan技术为各个节点创建一个互通的Pod网络，使用的端口为8472.第一次启动时，从etcd获取配置的Pod网络，为本节点分配一个未使用的地址段，然后创建flannel.1网络接口(也可能是其它名称)flannel将分配给自己的Pod网段信息写入/run/flannel/docker文件，docker后续使用这个文件中的环境变量设置Docker0网桥，从而从这个地址段为本节点的所有Pod容器分配IP下载分发flanneld二进制文件 (本次flanneld不使用Pod运行)
cd /opt/k8s/work
mkdir flannel
wget http://down.i4t.com/k8s1.14/flannel-v0.11.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.11.0-linux-amd64.tar.gz -C flannelflanneld从etcd集群存取网段分配信息，而etcd集群开启了双向x509证书认证，所以需要为flannel生成证书和私钥创建证书签名请求
cd /opt/k8s/work
cat > flanneld-csr.json <<EOF
{"CN": "flanneld","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "4Paradigm"}]
}
EOFcfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
ls flanneld*pemcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p /etc/flanneld/cert"scp flanneld*.pem root@${node_ip}:/etc/flanneld/certdone
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
etcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/opt/k8s/work/ca.pem \--cert-file=/opt/k8s/work/flanneld.pem \--key-file=/opt/k8s/work/flanneld-key.pem \mk ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 21, "Backend": {"Type": "vxlan"}}'
注意:flanneld当前版本v0.11.0不支持etcd v3，故使用etcd v2 API写入配置Key和网段数据；写入的Pod网段${CLUSTER_CIDR}地址段(如/16)必须小于SubnetLen，必须与kube-controller-manager的--cluster-cidr参数一致#创建flanneld的启动文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\-etcd-cafile=/etc/kubernetes/cert/ca.pem \\-etcd-certfile=/etc/flanneld/cert/flanneld.pem \\-etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\-etcd-endpoints=${ETCD_ENDPOINTS} \\-etcd-prefix=${FLANNEL_ETCD_PREFIX} \\-iface=${IFACE} \\-ip-masq
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网段信息写入 /run/flannel/docker 文件，后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥；
flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 -iface 参数指定通信接口;
flanneld 运行时需要 root 权限；
-ip-masq: flanneld 为访问 Pod 网络外的流量设置 SNAT 规则，同时将传递给 Docker 的变量 --ip-masq（/run/flannel/docker 文件中）设置为 false，这样 Docker 将不再创建 SNAT 规则； Docker 的 --ip-masq 为 true 时，创建的 SNAT 规则比较“暴力”：将所有本节点 Pod 发起的、访问非 docker0 接口的请求做 SNAT，这样访问其他节点 Pod 的请求来源 IP 会被设置为 flannel.1 接口的 IP，导致目的 Pod 看不到真实的来源 Pod IP。 flanneld 创建的 SNAT 规则比较温和，只对访问非 Pod 网段的请求做 SNAT。
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"scp flanneld.service root@${node_ip}:/etc/systemd/system/done
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl restart flanneld"done
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status flanneld|grep Active"done# 检查分配给flanneld的Pod网段信息
source /opt/k8s/bin/environment.sh
etcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/etc/kubernetes/cert/ca.pem \--cert-file=/etc/flanneld/cert/flanneld.pem \--key-file=/etc/flanneld/cert/flanneld-key.pem \get ${FLANNEL_ETCD_PREFIX}/config#查看已分配的Pod子网网段列表source /opt/k8s/bin/environment.sh
etcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/etc/kubernetes/cert/ca.pem \--cert-file=/etc/flanneld/cert/flanneld.pem \--key-file=/etc/flanneld/cert/flanneld-key.pem \ls ${FLANNEL_ETCD_PREFIX}/subnets
#查看某Pod网段对应节点IP和flannel接口地址
source /opt/k8s/bin/environment.sh
etcdctl \--endpoints=${ETCD_ENDPOINTS} \--ca-file=/etc/kubernetes/cert/ca.pem \--cert-file=/etc/flanneld/cert/flanneld.pem \--key-file=/etc/flanneld/cert/flanneld-key.pem \get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.104.0-21#后面节点IP需要根据我们查出来的地址进行修改
# etcdctl \
> --endpoints=${ETCD_ENDPOINTS} \
> --ca-file=/etc/kubernetes/cert/ca.pem \
> --cert-file=/etc/flanneld/cert/flanneld.pem \
> --key-file=/etc/flanneld/cert/flanneld-key.pem \
> ls ${FLANNEL_ETCD_PREFIX}/subnets
/kubernetes/network/subnets/172.30.104.0-21
/kubernetes/network/subnets/172.30.224.0-21
/kubernetes/network/subnets/172.30.24.0-21
/kubernetes/network/subnets/172.30.64.0-21
[root@k8s-01 work]# source /opt/k8s/bin/environment.sh
[root@k8s-01 work]# source /opt/k8s/bin/environment.sh
[root@k8s-01 work]# etcdctl \
> --endpoints=${ETCD_ENDPOINTS} \
> --ca-file=/etc/kubernetes/cert/ca.pem \
> --cert-file=/etc/flanneld/cert/flanneld.pem \
> --key-file=/etc/flanneld/cert/flanneld-key.pem \
> get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.104.0-21
{"PublicIP":"192.168.100.28","BackendType":"vxlan","BackendData":{"VtepMAC":"9a:aa:f4:f0:2c:e8"}}# 查看节点flannel网络信息
ip addr show
flannel.1网卡的地址为分配的pod自网段的第一个个IP (.0)，且是/32的地址
ip addr show|grep flannel.1
到其它节点 Pod 网段请求都被转发到 flannel.1 网卡；flanneld 根据 etcd 中子网段的信息，如 ${FLANNEL_ETCD_PREFIX}/subnets/172.30.80.0-21，来决定进请求发送给哪个节点的互联 IP；验证各节点能通过 Pod 网段互通在各节点上部署 flannel 后，检查是否创建了 flannel 接口(名称可能为 flannel0、flannel.0、flannel.1 等)：
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"done
kube-apiserver 高可用
使用Nginx 4层透明代理功能实现k8s节点(master节点和worker节点)高可用访问kube-apiserver的步骤
控制节点的kube-controller-manager、kube-scheduler是多实例部署，所以只要一个实例正常，就可以保证集群高可用
集群内的Pod使用k8s服务域名kubernetes访问kube-apiserver，kube-dns会自动解析多个kube-apiserver节点的IP，所以也是高可用的
在每个Nginx进程，后端对接多个apiserver实例，Nginx对他们做健康检查和负载均衡
kubelet、kube-proxy、controller-manager、schedule通过本地nginx (监听我们vip 192.158.0.54)访问kube-apiserver，从而实现kube-apiserver高可用
下载编译nginx (k8s-01安装就可以，后面有拷贝步骤)
cd /opt/k8s/work
wget http://down.i4t.com/k8s1.14/nginx-1.15.3.tar.gz
tar -xzvf nginx-1.15.3.tar.gz#编译
cd /opt/k8s/work/nginx-1.15.3
mkdir nginx-prefix
./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module
make && make install#############
--without-http_scgi_module --without-http_fastcgi_module
--with-stream：开启 4 层透明转发(TCP Proxy)功能；
--without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小；查看安装后的动态库
ldd ./nginx-prefix/sbin/nginx
由于只开启了 4 层透明转发功能，所以除了依赖 libc 等操作系统核心 lib 库外，没有对其它 lib 的依赖(如 libz、libssl 等)，这样可以方便部署到各版本操作系统中cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}done#拷贝二进制到其他文件，一次执行不成功，就多执行一次
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx root@${node_ip}:/opt/k8s/kube-nginx/sbin/kube-nginxssh root@${node_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*"ssh root@${node_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}"sleep 3done# 配置Nginx文件，开启4层透明转发cd /opt/k8s/work
cat > kube-nginx.conf <<EOF
worker_processes 1;
events {worker_connections 1024;
}
stream {upstream backend {hash $remote_addr consistent;server 192.168.100.26:6443 max_fails=3 fail_timeout=30s;server 192.168.100.27:6443 max_fails=3 fail_timeout=30s;server 192.168.100.28:6443 max_fails=3 fail_timeout=30s;}server {listen *:8443;proxy_connect_timeout 1s;proxy_pass backend;}
}
EOF#这里需要将server替换我们自己的地址cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-nginx.conf root@${node_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.confdone#配置启动文件
cd /opt/k8s/work
cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t
ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx
ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOFcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-nginx.service root@${node_ip}:/etc/systemd/system/donecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl start kube-nginx"donecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status kube-nginx |grep 'Active:'"done
KeepLived 部署
需要一个vip
在所有master节点安装keeplived
yum install -y keepalived在192.168.100.26 上操作
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {router_id 192.168.100.26
}
vrrp_script chk_nginx {script "/etc/keepalived/check_port.sh 8443"interval 2weight -20
}
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 251priority 100advert_int 1mcast_src_ip 192.168.100.26nopreemptauthentication {auth_type PASSauth_pass 11111111}track_script {chk_nginx}virtual_ipaddress {192.168.100.199}
}
EOF## 192.168.100.26 为节点IP，192.168.100.199位VIP#将配置拷贝到其他节点，并替换相关IP
for node_ip in 192.168.100.26 192.168.100.27 192.168.100.28doecho ">>> ${node_ip}"scp /etc/keepalived/keepalived.conf $node_ip:/etc/keepalived/keepalived.confdone#替换IP
ssh root@192.168.100.27 sed -i 's#192.168.100.26#192.168.100.27#g' /etc/keepalived/keepalived.conf
ssh root@192.168.100.28 sed -i 's#192.168.100.26#192.168.100.28#g' /etc/keepalived/keepalived.conf#192.168.100.26不替换是因为已经修改好了# 创建健康检查脚本
vim /opt/check_port.sh
CHK_PORT=$1if [ -n "$CHK_PORT" ];thenPORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l`if [ $PORT_PROCESS -eq 0 ];thenecho "Port $CHK_PORT Is Not Used,End."exit 1fielseecho "Check Port Cant Be Empty!"fifor NODE in k8s-01 k8s-02 k8s-03; doecho "--- $NODE ---"scp -r /opt/check_port.sh $NODE:/etc/keepalived/ssh $NODE 'systemctl enable --now keepalived'
done#没有启动成功，请执行下面命令，从新启动。启动成功vip肯定就通了
systemctl start keepalived部署master节点
kubernetes master节点运行组件如下:kube-apiserver、kube-scheduler、kube-controller-manager、kube-nginxkube-apiserver、kube-scheduler、kube-controller-manager均以多实例模式运行
kube-scheduler和kube-controller-manager会自动选举一个leader实例，其他实例处于阻塞模式，当leader挂了后，重新选举产生的leader，从而保证服务可用性
kube-apiserver是无状态的，需要通过kube-nginx进行代理访问，从而保证服务可用性
以下操作都在K8s-01操作下载kubernetes二进制包，并分发到所有master节点cd /opt/k8s/work
wget http://down.i4t.com/k8s1.14/kubernetes-server-linux-amd64.tar.gz
tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes
tar -xzvf kubernetes-src.tar.gzcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kubernetes/server/bin/kube-apiserver root@${node_ip}:/opt/k8s/bin/scp kubernetes/server/bin/{apiextensions-apiserver,cloud-controller-manager,kube-controller-manager,kube-proxy,kube-scheduler,kubeadm,kubectl,kubelet,mounter} root@${node_ip}:/opt/k8s/bin/ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"done#同时将kubelet kube-proxy拷贝到所有节点cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"scp kubernetes/server/bin/{kubelet,kube-proxy} root@${node_ip}:/opt/k8s/bin/ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"done#添加密钥
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kubernetes-csr.json <<EOF
{"CN": "kubernetes","hosts": ["127.0.0.1","192.168.100.26","192.168.100.27","192.168.100.28","192.168.100.199","10.254.0.1","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local."],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "4Paradigm"}]
}
EOF#需要将集群的所有IP及VIP添加进去
#如果要添加注意最后的逗号，不要忘记添加，否则下一步报错
hosts 字段指定授权使用该证书的IP和域名列表，这里列出了master节点IP、kubernetes服务的IP和域名kubernetes serviceIP是apiserver自动创建的，一般是--service-cluster-ip-range参数指定的网段的第一个IP
$ kubectl get svc kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.254.0.1 443/TCP 31d#目前我们是看不到cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetesls kubernetes*pemcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p /etc/kubernetes/cert"scp kubernetes*.pem root@${node_ip}:/etc/kubernetes/cert/donecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > encryption-config.yaml <<EOF
kind: EncryptionConfig
apiVersion: v1
resources:- resources:- secretsproviders:- aescbc:keys:- name: key1secret: ${ENCRYPTION_KEY}- identity: {}
EOFcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp encryption-config.yaml root@${node_ip}:/etc/kubernetes/done#创建审核文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > audit-policy.yaml <<EOF
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:# The following requests were manually identified as high-volume and low-risk, so drop them.- level: Noneresources:- group: ""resources:- endpoints- services- services/statususers:- 'system:kube-proxy'verbs:- watch- level: Noneresources:- group: ""resources:- nodes- nodes/statususerGroups:- 'system:nodes'verbs:- get- level: Nonenamespaces:- kube-systemresources:- group: ""resources:- endpointsusers:- 'system:kube-controller-manager'- 'system:kube-scheduler'- 'system:serviceaccount:kube-system:endpoint-controller'verbs:- get- update- level: Noneresources:- group: ""resources:- namespaces- namespaces/status- namespaces/finalizeusers:- 'system:apiserver'verbs:- get# Don't log HPA fetching metrics.- level: Noneresources:- group: metrics.k8s.iousers:- 'system:kube-controller-manager'verbs:- get- list# Don't log these read-only URLs.- level: NonenonResourceURLs:- '/healthz*'- /version- '/swagger*'# Don't log events requests.- level: Noneresources:- group: ""resources:- events# node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes- level: RequestomitStages:- RequestReceivedresources:- group: ""resources:- nodes/status- pods/statususers:- kubelet- 'system:node-problem-detector'- 'system:serviceaccount:kube-system:node-problem-detector'verbs:- update- patch- level: RequestomitStages:- RequestReceivedresources:- group: ""resources:- nodes/status- pods/statususerGroups:- 'system:nodes'verbs:- update- patch# deletecollection calls can be large, don't log responses for expected namespace deletions- level: RequestomitStages:- RequestReceivedusers:- 'system:serviceaccount:kube-system:namespace-controller'verbs:- deletecollection# Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,# so only log at the Metadata level.- level: MetadataomitStages:- RequestReceivedresources:- group: ""resources:- secrets- configmaps- group: authentication.k8s.ioresources:- tokenreviews# Get repsonses can be large; skip them.- level: RequestomitStages:- RequestReceivedresources:- group: ""- group: admissionregistration.k8s.io- group: apiextensions.k8s.io- group: apiregistration.k8s.io- group: apps- group: authentication.k8s.io- group: authorization.k8s.io- group: autoscaling- group: batch- group: certificates.k8s.io- group: extensions- group: metrics.k8s.io- group: networking.k8s.io- group: policy- group: rbac.authorization.k8s.io- group: scheduling.k8s.io- group: settings.k8s.io- group: storage.k8s.ioverbs:- get- list- watch# Default level for known APIs- level: RequestResponseomitStages:- RequestReceivedresources:- group: ""- group: admissionregistration.k8s.io- group: apiextensions.k8s.io- group: apiregistration.k8s.io- group: apps- group: authentication.k8s.io- group: authorization.k8s.io- group: autoscaling- group: batch- group: certificates.k8s.io- group: extensions- group: metrics.k8s.io- group: networking.k8s.io- group: policy- group: rbac.authorization.k8s.io- group: scheduling.k8s.io- group: settings.k8s.io- group: storage.k8s.io# Default level for all other requests.- level: MetadataomitStages:- RequestReceived
EOFcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp audit-policy.yaml root@${node_ip}:/etc/kubernetes/audit-policy.yamldone
cat > proxy-client-csr.json <<EOF
{"CN": "aggregator","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "4Paradigm"}]
}
EOFCN名称需要位于kube-apiserver的--requestherader-allowed-names参数中，否则后续访问metrics时会提示权限不足
生成证书和私钥source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp proxy-client*.pem root@${node_ip}:/etc/kubernetes/cert/donecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-apiserver.service.template <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=${K8S_DIR}/kube-apiserver
ExecStart=/opt/k8s/bin/kube-apiserver \\--advertise-address=##NODE_IP## \\--default-not-ready-toleration-seconds=360 \\--default-unreachable-toleration-seconds=360 \\--feature-gates=DynamicAuditing=true \\--max-mutating-requests-inflight=2000 \\--max-requests-inflight=4000 \\--default-watch-cache-size=200 \\--delete-collection-workers=2 \\--encryption-provider-config=/etc/kubernetes/encryption-config.yaml \\--etcd-cafile=/etc/kubernetes/cert/ca.pem \\--etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \\--etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \\--etcd-servers=${ETCD_ENDPOINTS} \\--bind-address=##NODE_IP## \\--secure-port=6443 \\--tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \\--tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \\--insecure-port=0 \\--audit-dynamic-configuration \\--audit-log-maxage=15 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-truncate-enabled \\--audit-log-path=${K8S_DIR}/kube-apiserver/audit.log \\--audit-policy-file=/etc/kubernetes/audit-policy.yaml \\--profiling \\--anonymous-auth=false \\--client-ca-file=/etc/kubernetes/cert/ca.pem \\--enable-bootstrap-token-auth \\--requestheader-allowed-names="aggregator" \\--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-extra-headers-prefix="X-Remote-Extra-" \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User \\--service-account-key-file=/etc/kubernetes/cert/ca.pem \\--authorization-mode=Node,RBAC \\--runtime-config=api/all=true \\--enable-admission-plugins=NodeRestriction \\--allow-privileged=true \\--apiserver-count=3 \\--event-ttl=168h \\--kubelet-certificate-authority=/etc/kubernetes/cert/ca.pem \\--kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \\--kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \\--kubelet-https=true \\--kubelet-timeout=10s \\--proxy-client-cert-file=/etc/kubernetes/cert/proxy-client.pem \\--proxy-client-key-file=/etc/kubernetes/cert/proxy-client-key.pem \\--service-cluster-ip-range=${SERVICE_CIDR} \\--service-node-port-range=${NODE_PORT_RANGE} \\--logtostderr=true \\--v=2
Restart=on-failure
RestartSec=10
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF--advertise-address：apiserver 对外通告的 IP（kubernetes 服务后端节点 IP）；
--default-*-toleration-seconds：设置节点异常相关的阈值；
--max-*-requests-inflight：请求相关的最大阈值；
--etcd-*：访问 etcd 的证书和 etcd 服务器地址；
--experimental-encryption-provider-config：指定用于加密 etcd 中 secret 的配置；
--bind-address： https 监听的 IP，不能为 127.0.0.1，否则外界不能访问它的安全端口 6443；
--secret-port：https 监听端口；
--insecure-port=0：关闭监听 http 非安全端口(8080)；
--tls-*-file：指定 apiserver 使用的证书、私钥和 CA 文件；
--audit-*：配置审计策略和审计日志文件相关的参数；
--client-ca-file：验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书；
--enable-bootstrap-token-auth：启用 kubelet bootstrap 的 token 认证；
--requestheader-*：kube-apiserver 的 aggregator layer 相关的配置参数，proxy-client & HPA 需要使用；
--requestheader-client-ca-file：用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书；在启用了 metric aggregator 时使用；
--requestheader-allowed-names：不能为空，值为逗号分割的 --proxy-client-cert-file 证书的 CN 名称，这里设置为 "aggregator"；
--service-account-key-file：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 --service-account-private-key-file 定私钥文件，两者配对使用；
--runtime-config=api/all=true： 启用所有版本的 APIs，如 autoscaling/v2alpha1；
--authorization-mode=Node,RBAC、--anonymous-auth=false： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求；
--enable-admission-plugins：启用一些默认关闭的 plugins；
--allow-privileged：运行执行 privileged 权限的容器；
--apiserver-count=3：指定 apiserver 实例的数量；
--event-ttl：指定 events 的保存时间；
--kubelet-：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权；
--proxy-client-*：apiserver 访问 metrics-server 使用的证书；
--service-cluster-ip-range： 指定 Service Cluster IP 地址段；
--service-node-port-range： 指定 NodePort 的端口范围；
如果 kube-apiserver 机器没有运行 kube-proxy，则还需要添加 --enable-aggregator-routing=true 参数；关于 --requestheader-XXX 相关参数，参考：
https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md
https://docs.bitnami.com/kubernetes/how-to/configure-autoscaling-custom-metrics/
注意: requestheader-client-ca-file指定的CA证书，必须具有client auth and server auth如果--requestheader-allowed-names为空，或者--proxy-client-cert-file证书的CN名称不在allowed-names中，则后续查看node或者Pods的metrics失败为各个节点创建和分发kube-apiserver启动文件替换模板文件的变量，为各个节点生成启动文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ )) #这里是三个节点所以为3,请根据实际情况修改,后边不在提示,同理dosed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-apiserver.service.template > kube-apiserver-${MASTER_IPS[i]}.service done
ls kube-apiserver*.servicecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.servicedonesource /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-apiserver"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"done
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status kube-apiserver |grep 'Active:'"done
#如果开启不了，查看日志 journalctl -u kube-apiserver#打印kube-apiserver写入etcd数据
source /opt/k8s/bin/environment.sh
ETCDCTL_API=3 etcdctl \--endpoints=${ETCD_ENDPOINTS} \--cacert=/opt/k8s/work/ca.pem \--cert=/opt/k8s/work/etcd.pem \--key=/opt/k8s/work/etcd-key.pem \get /registry/ --prefix --keys-only#检查集群信息
netstat -lntup|grep kube[root@k8s-01 work]# kubectl cluster-info
Kubernetes master is running at https://192.168.100.199:8443To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@k8s-01 work]# kubectl get all --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 2m26s
[root@k8s-01 work]# kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"} 如果提示有报错，请检查~/.kube/config以及配置证书是否有问题授权kube-apiserver访问kubelet API的权限在执行kubectl命令时，apiserver会将请求转发到kubelet的https端口。这里定义的RBAC规则，授权apiserver使用的证书(kubernetes.pem)用户名(CN:kubernetes)访问kubelet API的权限kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes部署高可用kube-controller-manager集群
该集群包含三个节点，启动后通过竞争选举机制产生一个leader节点，其他节点为阻塞状态。当leader节点不可用时，阻塞节点将会在此选举产生新的leader，从而保证服务的高可用。为保证通信安全，这里采用x509证书和私钥，kube-controller-manager在与apiserver的安全端口（http 10252）通信使用；创建kube-controller-manager证书和私钥创建证书签名请求
cd /opt/k8s/work
cat > kube-controller-manager-csr.json <<EOF
{"CN": "system:kube-controller-manager","key": {"algo": "rsa","size": 2048},"hosts": ["127.0.0.1","192.168.100.26","192.168.100.27","192.168.100.28"],"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "system:kube-controller-manager","OU": "4Paradigm"}]
}
EOF#这里的IP地址为master iphost列表包含所有的kube-controller-manager节点IP(VIP不需要输入)
CN和O均为system:kube-controller-manager，kubernetes内置的ClusterRoleBindings system:kube-controller-manager赋予kube-controller-manager工作所需权限
生成证书和私钥cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
ls kube-controller-manager*pem将生成的证书和私钥分发到所有master节点
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-controller-manager*.pem root@${node_ip}:/etc/kubernetes/cert/done# 创建和分发kubeconfig文件
#kube-controller-manager使用kubeconfig文件访问apiserver
#该文件提供了apiserver地址、嵌入的CA证书和kube-controller-manager证书cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \--client-certificate=kube-controller-manager.pem \--client-key=kube-controller-manager-key.pem \--embed-certs=true \--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager \--cluster=kubernetes \--user=system:kube-controller-manager \--kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfigcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-controller-manager.kubeconfig root@${node_ip}:/etc/kubernetes/done创建kube-controller-manager启动文件cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-controller-manager.service.template <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=${K8S_DIR}/kube-controller-manager
ExecStart=/opt/k8s/bin/kube-controller-manager \\--profiling \\--cluster-name=kubernetes \\--controllers=*,bootstrapsigner,tokencleaner \\--kube-api-qps=1000 \\--kube-api-burst=2000 \\--leader-elect \\--use-service-account-credentials\\--concurrent-service-syncs=2 \\--bind-address=0.0.0.0 \\#--secure-port=10252 \\--tls-cert-file=/etc/kubernetes/cert/kube-controller-manager.pem \\--tls-private-key-file=/etc/kubernetes/cert/kube-controller-manager-key.pem \\#--port=0 \\--authentication-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\--client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-allowed-names="" \\--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-extra-headers-prefix="X-Remote-Extra-" \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User \\--authorization-kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\--cluster-signing-cert-file=/etc/kubernetes/cert/ca.pem \\--cluster-signing-key-file=/etc/kubernetes/cert/ca-key.pem \\--experimental-cluster-signing-duration=876000h \\--horizontal-pod-autoscaler-sync-period=10s \\--concurrent-deployment-syncs=10 \\--concurrent-gc-syncs=30 \\--node-cidr-mask-size=24 \\--service-cluster-ip-range=${SERVICE_CIDR} \\--pod-eviction-timeout=6m \\--terminated-pod-gc-threshold=10000 \\--root-ca-file=/etc/kubernetes/cert/ca.pem \\--service-account-private-key-file=/etc/kubernetes/cert/ca-key.pem \\--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\--logtostderr=true \\--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF参数解释--port=0：关闭监听非安全端口（http），同时 --address 参数无效，--bind-address 参数有效；
--secure-port=10252、--bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求；
--kubeconfig：指定 kubeconfig 文件路径，kube-controller-manager 使用它连接和验证 kube-apiserver；
--authentication-kubeconfig 和 --authorization-kubeconfig：kube-controller-manager 使用它连接 apiserver，对 client 的请求进行认证和授权。kube-controller-manager 不再使用 --tls-ca-file 对请求 https metrics 的 Client 证书进行校验。如果没有配置这两个 kubeconfig 参数，则 client 连接 kube-controller-manager https 端口的请求会被拒绝(提示权限不足)。
--cluster-signing-*-file：签名 TLS Bootstrap 创建的证书；
--experimental-cluster-signing-duration：指定 TLS Bootstrap 证书的有效期；
--root-ca-file：放置到容器 ServiceAccount 中的 CA 证书，用来对 kube-apiserver 的证书进行校验；
--service-account-private-key-file：签名 ServiceAccount 中 Token 的私钥文件，必须和 kube-apiserver 的 --service-account-key-file 指定的公钥文件配对使用；
--service-cluster-ip-range ：指定 Service Cluster IP 网段，必须和 kube-apiserver 中的同名参数一致；
--leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；
--controllers=*,bootstrapsigner,tokencleaner：启用的控制器列表，tokencleaner 用于自动清理过期的 Bootstrap token；
--horizontal-pod-autoscaler-*：custom metrics 相关参数，支持 autoscaling/v2alpha1；
--tls-cert-file、--tls-private-key-file：使用 https 输出 metrics 时使用的 Server 证书和秘钥；
--use-service-account-credentials=true: kube-controller-manager 中各 controller 使用 serviceaccount 访问 kube-apiserver；cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))dosed -e "s/##NODE_NAME##/${MASTER_NAMES[i]}/" -e "s/##NODE_IP##/${MASTER_IPS[i]}/" kube-controller-manager.service.template > kube-controller-manager-${MASTER_IPS[i]}.service done
ls kube-controller-manager*.servicecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-controller-manager-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-controller-manager.servicedonesource /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-controller-manager"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager"done
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status kube-controller-manager|grep Active"done#检查服务状态
[root@k8s-01 work]# netstat -lnpt | grep kube-cont
tcp6 0 0 :::10257 :::* LISTEN 9664/kube-controlle
tcp6 0 0 :::10252 :::* LISTEN 9664/kube-controlle kube-controller-manager 创建权限
ClusteRole system:kube-controller-manager的权限太小，只能创建secret、serviceaccount等资源,将controller的权限分散到ClusterRole system:controller:xxx中[root@k8s-01 work]# kubectl describe clusterrole system:kube-controller-manager
Name: system:kube-controller-manager
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:Resources Non-Resource URLs Resource Names Verbs--------- ----------------- -------------- -----secrets [] [] [create delete get update]endpoints [] [] [create get update]serviceaccounts [] [] [create get update]events [] [] [create patch update]tokenreviews.authentication.k8s.io [] [] [create]subjectaccessreviews.authorization.k8s.io [] [] [create]configmaps [] [] [get]namespaces [] [] [get]*.* [] [] [list watch]
需要在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数，这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。
[root@k8s-01 work]# kubectl get clusterrole|grep controller
system:controller:attachdetach-controller 12m
system:controller:certificate-controller 12m
system:controller:clusterrole-aggregation-controller 12m
system:controller:cronjob-controller 12m
system:controller:daemon-set-controller 12m
system:controller:deployment-controller 12m
system:controller:disruption-controller 12m
system:controller:endpoint-controller 12m
system:controller:expand-controller 12m
system:controller:generic-garbage-collector 12m
system:controller:horizontal-pod-autoscaler 12m
system:controller:job-controller 12m
system:controller:namespace-controller 12m
system:controller:node-controller 12m
system:controller:persistent-volume-binder 12m
system:controller:pod-garbage-collector 12m
system:controller:pv-protection-controller 12m
system:controller:pvc-protection-controller 12m
system:controller:replicaset-controller 12m
system:controller:replication-controller 12m
system:controller:resourcequota-controller 12m
system:controller:route-controller 12m
system:controller:service-account-controller 12m
system:controller:service-controller 12m
system:controller:statefulset-controller 12m
system:controller:ttl-controller 12m
system:kube-controller-manager 12m#以 deployment controller 为例：
[root@k8s-01 work]# kubectl describe clusterrole system:controller:deployment-controller
Name: system:controller:deployment-controller
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:Resources Non-Resource URLs Resource Names Verbs--------- ----------------- -------------- -----replicasets.apps [] [] [create delete get list patch update watch]replicasets.extensions [] [] [create delete get list patch update watch]events [] [] [create patch update]pods [] [] [get list update watch]deployments.apps [] [] [get list update watch]deployments.extensions [] [] [get list update watch]deployments.apps/finalizers [] [] [update]deployments.apps/status [] [] [update]deployments.extensions/finalizers [] [] [update]deployments.extensions/status [] [] [update]#查看当前的 leader
[root@k8s-01 work]# kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1
kind: Endpoints
metadata:annotations:control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"k8s-01_ede3a742-4001-11ee-b717-4201c0a8641a","leaseDurationSeconds":15,"acquireTime":"2023-08-21T09:05:52Z","renewTime":"2023-08-21T09:09:23Z","leaderTransitions":0}'creationTimestamp: "2023-08-21T09:05:52Z"name: kube-controller-managernamespace: kube-systemresourceVersion: "604"selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manageruid: ede66bf9-4001-11ee-b6ee-4201c0a8641a
部署高可用kube-scheduler
cd /opt/k8s/work
cat > kube-scheduler-csr.json <<EOF
{"CN": "system:kube-scheduler","hosts": ["127.0.0.1","192.168.100.26","192.168.100.27","192.168.100.28"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "system:kube-scheduler","OU": "4Paradigm"}]
}
EOF
hosts 列表包含所有 kube-scheduler 节点 IP；
CN 和 O 均为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限；
生成证书和私钥：cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
ls kube-scheduler*pemcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-scheduler*.pem root@${node_ip}:/etc/kubernetes/cert/done
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \--client-certificate=kube-scheduler.pem \--client-key=kube-scheduler-key.pem \--embed-certs=true \--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler \--cluster=kubernetes \--user=system:kube-scheduler \--kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-scheduler.kubeconfig root@${node_ip}:/etc/kubernetes/done#创建 kube-scheduler 配置文件cd /opt/k8s/work
cat >kube-scheduler.yaml.template <<EOF
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
bindTimeoutSeconds: 600
clientConnection:burst: 200kubeconfig: "/etc/kubernetes/kube-scheduler.kubeconfig"qps: 100
enableContentionProfiling: false
enableProfiling: true
hardPodAffinitySymmetricWeight: 1
healthzBindAddress: 127.0.0.1:10251
leaderElection:leaderElect: true
metricsBindAddress: ##NODE_IP##:10251
EOF--kubeconfig：指定 kubeconfig 文件路径，kube-scheduler 使用它连接和验证 kube-apiserver；
--leader-elect=true：集群运行模式，启用选举功能；被选为 leader 的节点负责处理工作，其它节点为阻塞状态；
替换模板文件中的变量：cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))dosed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-scheduler.yaml.template > kube-scheduler-${NODE_IPS[i]}.yamldone
ls kube-scheduler*.yamlcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-scheduler-${node_ip}.yaml root@${node_ip}:/etc/kubernetes/kube-scheduler.yamldonecd /opt/k8s/work
cat > kube-scheduler.service.template <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
WorkingDirectory=${K8S_DIR}/kube-scheduler
ExecStart=/opt/k8s/bin/kube-scheduler \\--config=/etc/kubernetes/kube-scheduler.yaml \\--bind-address=##NODE_IP## \\--secure-port=10259 \\--port=0 \\--tls-cert-file=/etc/kubernetes/cert/kube-scheduler.pem \\--tls-private-key-file=/etc/kubernetes/cert/kube-scheduler-key.pem \\--authentication-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\--client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-allowed-names="" \\--requestheader-client-ca-file=/etc/kubernetes/cert/ca.pem \\--requestheader-extra-headers-prefix="X-Remote-Extra-" \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User \\--authorization-kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\--logtostderr=true \\--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 3; i++ ))dosed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-scheduler.service.template > kube-scheduler-${NODE_IPS[i]}.service done
ls kube-scheduler*.servicecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"scp kube-scheduler-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-scheduler.servicedonesource /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-scheduler"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl restart kube-scheduler"
done
#检查服务运行状态
source /opt/k8s/bin/environment.sh
for node_ip in ${MASTER_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status kube-scheduler|grep Active"done
查看输出的 metrics注意：以下命令在 kube-scheduler 节点上执行。kube-scheduler 监听 10251 和 10251 端口：10251：接收 http 请求，非安全端口，不需要认证授权；10259：接收 https 请求，安全端口，需要认证授权；两个接口都对外提供 /metrics 和 /healthz 的访问。[root@k8s-01 work]# curl -s http://192.168.100.26:10251/metrics|head
# HELP apiserver_audit_event_total Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0#查看当前leader
kubectl get endpoints kube-scheduler --namespace=kube-system -o yamlwork节点安装
kubernetes work节点运行如下组件: >docker、kubelet、kube-proxy、flanneld、kube-nginx前面已经安装flanneld这就不在安装了
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "yum install -y epel-release"ssh root@${node_ip} "yum install -y conntrack ipvsadm ntp ntpdate ipset jq iptables curl sysstat libseccomp && modprobe ip_vs "done
部署Docker组件
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum -y install docker-cemkdir -p /etc/docker/
cat > /etc/docker/daemon.json <<EOF
{"exec-opts": ["native.cgroupdriver=systemd"],"registry-mirrors": ["https://hjvrgh7a.mirror.aliyuncs.com"],"log-driver": "json-file","log-opts": {"max-size": "100m"},"storage-driver": "overlay2"
}
EOF另一个版本mkdir -p /etc/docker/
cat > /etc/docker/daemon.json <<EOF
{"exec-opts": ["native.cgroupdriver=systemd"],"registry-mirrors": ["https://hub-mirror.c.163.com","https://mirror.baidubce.com"],"log-driver": "json-file","log-opts": {"max-size": "100m"},"storage-driver": "overlay2"
}
EOF#这里配置当时镜像加速器,可以不进行配置，但是建议配置
要添加我们harbor仓库需要在添加下面一行"insecure-registries": ["harbor.xxx.com"],
默认docker hub需要https协议，使用上面配置不需要配置https# 以下操作 每一台都要执行
>/usr/lib/systemd/system/docker.service
cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
EnvironmentFile=-/run/flannel/docker
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes# kill only the docker process, not all processes in the cgroup
KillMode=process[Install]
WantedBy=multi-user.targetsource /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable docker && systemctl restart docker"done
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status docker|grep Active"done# 检查docker网桥
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "/usr/sbin/ip addr show flannel.1 && /usr/sbin/ip addr show docker0"done
部署kubelet组件
kubelet运行在每个worker节点上，接收kube-apiserver发送的请求，管理Pod容器，执行交互命令kubelet启动时自动向kube-apiserver注册节点信息，内置的cAdivsor统计和监控节点的资源使用资源情况。为确保安全，部署时关闭了kubelet的非安全http端口，对请求进行认证和授权，拒绝未授权的访问创建kubelet bootstrap kubeconfig文件
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}doecho ">>> ${node_name}"# 创建 tokenexport BOOTSTRAP_TOKEN=$(kubeadm token create \--description kubelet-bootstrap-token \--groups system:bootstrappers:${node_name} \--kubeconfig ~/.kube/config)# 设置集群参数kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/cert/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig# 设置客户端认证参数kubectl config set-credentials kubelet-bootstrap \--token=${BOOTSTRAP_TOKEN} \--kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig# 设置上下文参数kubectl config set-context default \--cluster=kubernetes \--user=kubelet-bootstrap \--kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig# 设置默认上下文kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfigdone
向kubeconfig写入的是token，bootstrap结束后kube-controller-manager为kubelet创建client和server证书
查看kubeadm为各个节点创建的token[root@k8s-01 work]# kubeadm token list --kubeconfig ~/.kube/config
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
1r8xjv.oypmp64rmji9r4d1 23h 2023-08-22T17:28:35+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-01
7a6n62.yzd0y8rb2vnufqlc 23h 2023-08-22T17:28:36+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-04
fyw4n9.3wwgsgwgjyhxxw11 23h 2023-08-22T17:28:35+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-02
i3cd24.fbvpytiu9rliclke 23h 2023-08-22T17:28:36+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:k8s-03token有效期为1天，超期后将不能被用来bootstrap kubelet，且会被kube-controller-manager的token cleaner清理
kube-apiserver接收kubelet的bootstrap token后，将请求的user设置为system:bootstrap; group设置为system:bootstrappers，后续将为这个group设置ClusterRoleBinding
#查看各token关联的Secret
[root@k8s-01 work]# kubectl get secrets -n kube-system|grep bootstrap-token
bootstrap-token-1r8xjv bootstrap.kubernetes.io/token 7 73s
bootstrap-token-7a6n62 bootstrap.kubernetes.io/token 7 72s
bootstrap-token-fyw4n9 bootstrap.kubernetes.io/token 7 73s
bootstrap-token-i3cd24 bootstrap.kubernetes.io/token 7 72s#分发 bootstrap kubeconfig 文件到所有 worker 节点
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}doecho ">>> ${node_name}"scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfigdone#创建和分发kubelet
cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kubelet-config.yaml.template <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: "##NODE_IP##"
staticPodPath: ""
syncFrequency: 1m
fileCheckFrequency: 20s
httpCheckFrequency: 20s
staticPodURL: ""
port: 10250
readOnlyPort: 0
rotateCertificates: true
serverTLSBootstrap: true
authentication:anonymous:enabled: falsewebhook:enabled: truex509:clientCAFile: "/etc/kubernetes/cert/ca.pem"
authorization:mode: Webhook
registryPullQPS: 0
registryBurst: 20
eventRecordQPS: 0
eventBurst: 20
enableDebuggingHandlers: true
enableContentionProfiling: true
healthzPort: 10248
healthzBindAddress: "##NODE_IP##"
clusterDomain: "${CLUSTER_DNS_DOMAIN}"
clusterDNS:- "${CLUSTER_DNS_SVC_IP}"
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m
imageMinimumGCAge: 2m
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
volumeStatsAggPeriod: 1m
kubeletCgroups: ""
systemCgroups: ""
cgroupRoot: ""
cgroupsPerQOS: true
cgroupDriver: systemd
runtimeRequestTimeout: 10m
hairpinMode: promiscuous-bridge
maxPods: 220
podCIDR: "${CLUSTER_CIDR}"
podPidsLimit: -1
resolvConf: /etc/resolv.conf
maxOpenFiles: 1000000
kubeAPIQPS: 1000
kubeAPIBurst: 2000
serializeImagePulls: false
evictionHard:memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
evictionSoft: {}
enableControllerAttachDetach: true
failSwapOn: true
containerLogMaxSize: 20Mi
containerLogMaxFiles: 10
systemReserved: {}
kubeReserved: {}
systemReservedCgroup: ""
kubeReservedCgroup: ""
enforceNodeAllocatable: ["pods"]
EOFaddress：kubelet 安全端口（https，10250）监听的地址，不能为 127.0.0.1，否则 kube-apiserver、heapster 等不能调用 kubelet 的 API；
readOnlyPort=0：关闭只读端口(默认 10255)，等效为未指定；
authentication.anonymous.enabled：设置为 false，不允许匿名访问 10250 端口；
authentication.x509.clientCAFile：指定签名客户端证书的 CA 证书，开启 HTTP 证书认证；
authentication.webhook.enabled=true：开启 HTTPs bearer token 认证；
对于未通过 x509 证书和 webhook 认证的请求(kube-apiserver 或其他客户端)，将被拒绝，提示 Unauthorized；
authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查询 kube-apiserver 某 user、group 是否具有操作资源的权限(RBAC)；
featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自动 rotate 证书，证书的有效期取决于 kube-controller-manager 的 --experimental-cluster-signing-duration 参数；
需要 root 账户运行；
为各个节点创建和分发kubelet配置文件cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}do echo ">>> ${node_ip}"sed -e "s/##NODE_IP##/${node_ip}/" kubelet-config.yaml.template > kubelet-config-${node_ip}.yaml.templatescp kubelet-config-${node_ip}.yaml.template root@${node_ip}:/etc/kubernetes/kubelet-config.yamldonecd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kubelet.service.template <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=${K8S_DIR}/kubelet
ExecStart=/opt/k8s/bin/kubelet \\--allow-privileged=true \\--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\--cert-dir=/etc/kubernetes/cert \\--cni-conf-dir=/etc/cni/net.d \\--container-runtime=docker \\--container-runtime-endpoint=unix:///var/run/dockershim.sock \\--root-dir=${K8S_DIR}/kubelet \\--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\--config=/etc/kubernetes/kubelet-config.yaml \\--hostname-override=##NODE_NAME## \\--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 \\--image-pull-progress-deadline=15m \\--volume-plugin-dir=${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/ \\--logtostderr=true \\--v=2
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
EOF如果设置了 --hostname-override 选项，则 kube-proxy 也需要设置该选项，否则会出现找不到 Node 的情况；
--bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用该文件中的用户名和 token 向 kube-apiserver 发送 TLS Bootstrapping 请求；
K8S approve kubelet 的 csr 请求后，在 --cert-dir 目录创建证书和私钥文件，然后写入 --kubeconfig 文件；
--pod-infra-container-image 不使用 redhat 的 pod-infrastructure:latest 镜像，它不能回收容器的僵尸；
分发启动文件cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}do echo ">>> ${node_name}"sed -e "s/##NODE_NAME##/${node_name}/" kubelet.service.template > kubelet-${node_name}.servicescp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.servicedoneBootstrap Token Auth 和授予权限 kubelet 启动时查找 --kubeletconfig
参数对应的文件是否存在，如果不存在则使用 --bootstrap-kubeconfig 指定的 kubeconfig 文件向
kube-apiserver 发送证书签名请求 (CSR)。 kube-apiserver 收到 CSR 请求后，对其中的 Token
进行认证，认证通过后将请求的 user 设置为 system:bootstrap:，group 设置为
system:bootstrappers，这一过程称为 Bootstrap Token Auth。创建user和group的CSR权限，不创建kubelet会启动失败
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers#启动kubelet
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kubelet/kubelet-plugins/volume/exec/"ssh root@${node_ip} "/usr/sbin/swapoff -a"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"done关闭 swap 分区，否则 kubelet 会启动失败；
kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求，当这个CSR 被 approve 后，kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和--kubeletconfig 文件。 注意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --cluster-signing-key-file 参数，才会为 TLS Bootstrap 创建证书和私钥。[root@k8s-01 work]# kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-4fhqr 25s system:bootstrap:i3cd24 Pending
csr-85jzl 20s system:bootstrap:fyw4n9 Pending
csr-9lwfj 9s system:bootstrap:fyw4n9 Pending
csr-bzhvt 20s system:bootstrap:i3cd24 Pending
csr-gw4w4 4s system:bootstrap:fyw4n9 Pending
csr-j42zv 15s system:bootstrap:fyw4n9 Pending
csr-jc7mx 5s system:bootstrap:1r8xjv Pending
csr-lns5s 3s system:bootstrap:i3cd24 Pending
csr-pjczh 14s system:bootstrap:i3cd24 Pending
csr-qztj9 27s system:bootstrap:1r8xjv Pending
csr-s9c2x 16s system:bootstrap:1r8xjv Pending
csr-th5pc 9s system:bootstrap:i3cd24 Pending
csr-vvsdv 10s system:bootstrap:1r8xjv Pending
csr-wgj7x 26s system:bootstrap:fyw4n9 Pending
csr-xd8gb 21s system:bootstrap:1r8xjv Pending这里节点均处于pending(等待)状态自动approve CSR请求创建三个ClusterRoleBinding，分别用于自动approve client、renew client、renew server证cd /opt/k8s/work
cat > csr-crb.yaml <<EOF# Approve all CSRs for the group "system:bootstrappers"kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:name: auto-approve-csrs-for-groupsubjects:- kind: Groupname: system:bootstrappersapiGroup: rbac.authorization.k8s.ioroleRef:kind: ClusterRolename: system:certificates.k8s.io:certificatesigningrequests:nodeclientapiGroup: rbac.authorization.k8s.io
---# To let a node of the group "system:nodes" renew its own credentialskind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:name: node-client-cert-renewalsubjects:- kind: Groupname: system:nodesapiGroup: rbac.authorization.k8s.ioroleRef:kind: ClusterRolename: system:certificates.k8s.io:certificatesigningrequests:selfnodeclientapiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]resources: ["certificatesigningrequests/selfnodeserver"]verbs: ["create"]
---# To let a node of the group "system:nodes" renew its own server credentialskind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:name: node-server-cert-renewalsubjects:- kind: Groupname: system:nodesapiGroup: rbac.authorization.k8s.ioroleRef:kind: ClusterRolename: approve-node-server-renewal-csrapiGroup: rbac.authorization.k8s.io
EOF
kubectl apply -f csr-crb.yamlauto-approve-csrs-for-group 自动approve node的第一次CSR，注意第一次CSR时，请求的Group为system:bootstrappers
node-client-cert-renewal 自动approve node后续过期的client证书，自动生成的证书Group为system:nodes
node-server-cert-renewal 自动approve node后续过期的server证书，自动生成的证书Group查看kubelet等待1-10分钟，3个节点的CSR都会自动approvedPending的CSR用于创建kubelet serve证书，需要手动approve (后面步骤)目前所有节点均为ready状态NAME STATUS ROLES AGE VERSION
k8s-01 Ready 2m29s v1.14.2
k8s-02 Ready 2m28s v1.14.2
k8s-03 Ready 2m28s v1.14.2
k8s-04 Ready 2m27s v1.14.2kube-controller-manager为各node生成了kubeconfig文件和公钥
[root@k8s-01 work]# ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2316 Aug 21 17:34 /etc/kubernetes/kubelet.kubeconfig
[root@k8s-01 work]# ls -l /etc/kubernetes/cert/|grep kubelet
-rw------- 1 root root 1273 Aug 21 17:34 kubelet-client-2023-08-21-17-34-58.pem
lrwxrwxrwx 1 root root 59 Aug 21 17:34 kubelet-client-current.pem -> /etc/kubernetes/cert/kubelet-client-2023-08-21-17-34-58.pem手动approve server cert csr基于安全考虑，CSR approving controllers不会自动approve kubelet server证书签名请求，需要手动approve
kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approvekubelet API接口kubelet启动后监听多个端口，用于接受kube-apiserver或其他客户端发送的请求[root@k8s-02 work]# netstat -lntup|grep kubelet
tcp 0 0 127.0.0.1:40605 0.0.0.0:* LISTEN 27600/kubelet
tcp 0 0 192.168.100.27:10248 0.0.0.0:* LISTEN 27600/kubelet
tcp 0 0 192.168.100.27:10250 0.0.0.0:* LISTEN 27600/kubelet 10248: healthz http 服务；
10250: https 服务，访问该端口时需要认证和授权（即使访问 /healthz 也需要）；
未开启只读端口 10255；
从 K8S v1.10 开始，去除了 --cadvisor-port 参数（默认 4194 端口），不支持访问 cAdvisor UI & API
bear token认证和授权创建一个ServiceAccount，将它和ClusterRole system:kubelet-api-admin绑定，从而具有调用kubelet API的权限kubectl create sa kubelet-api-test
kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
echo ${TOKEN}[root@k8s-01 work]# kubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-test
clusterrolebinding.rbac.authorization.k8s.io/kubelet-api-test created
[root@k8s-01 work]# SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
[root@k8s-01 work]# TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
[root@k8s-01 work]# echo ${TOKEN}
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Imt1YmVsZXQtYXBpLXRlc3QtdG9rZW4tMndtbjYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoia3ViZWxldC1hcGktdGVzdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImI3NWEzMzE4LTQwMGMtMTFlZS1iNmVlLTQyMDFjMGE4NjQxYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0Omt1YmVsZXQtYXBpLXRlc3QifQ.nvB9YjjVclFqrMMmAVF2dMD5hHJ6UzW6ujNO-i1ijFTEoG1cD1FQTAmdYS-hdbY_1heB5O3sFNRO6hF-E-s_yKF3aEcaxtiyJX95WE2Wt52VxA4750WV9Gqyz5HxGpsluEwlkz7eLh8jPwQHJqAE__sO1zo4-68hVGjbuEm8vFh5Bz1iQUfWFPK6RXaIzqcnkMe3T4SlZrtzHvd5ttmA2s4fsIZTvM6T157NUem7POF0MwWMIhtO1F49bMTxf2fe90dGSqZFx0gtWKoe671sbAcf6emcRkWz93GCumBrxZw5bLsm3dX_NWFj8kkVAB9WG-RGGGyK7iRg-oizXrumiA部署kube-proxy组件
kube-proxy运行在所有worker节点上，它监听apiserver中service和endpoint的变化情况，创建路由规则提供服务IP和负载均衡功能。这里使用ipvs模式的kube-proxy进行部署在各个节点需要安装ipvsadm和ipset命令，加载ip_vs内核模块cd /opt/k8s/work
cat > kube-proxy-csr.json <<EOF
{"CN": "system:kube-proxy","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "4Paradigm"}]
}
EOFCN：指定该证书的 User 为 system:kube-proxy；
预定义的 RoleBinding system:node-proxier 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；
该证书只会被 kube-proxy 当做 client 证书使用，所以 hosts 字段为空；
生成证书和私钥：cd /opt/k8s/work
cfssl gencert -ca=/opt/k8s/work/ca.pem \-ca-key=/opt/k8s/work/ca-key.pem \-config=/opt/k8s/work/ca-config.json \-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
ls kube-proxy*cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
kubectl config set-cluster kubernetes \--certificate-authority=/opt/k8s/work/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \--client-certificate=kube-proxy.pem \--client-key=kube-proxy-key.pem \--embed-certs=true \--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \--cluster=kubernetes \--user=kube-proxy \--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig--embed-certs=true：将 ca.pem 和 admin.pem 证书内容嵌入到生成的kubectl-proxy.kubeconfig文件中(不加时，写入的是证书文件路径)；
分发 kubeconfig 文件：cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}doecho ">>> ${node_name}"scp kube-proxy.kubeconfig root@${node_name}:/etc/kubernetes/done创建kube-proxy配置文件从v1.10开始，kube-proxy部分参数可以配置在文件中，可以使用--write-config-to选项生成该配置文件cd /opt/k8s/work
cat > kube-proxy-config.yaml.template <<EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:burst: 200kubeconfig: "/etc/kubernetes/kube-proxy.kubeconfig"qps: 100
bindAddress: ##NODE_IP##
healthzBindAddress: ##NODE_IP##:10256
metricsBindAddress: ##NODE_IP##:10249
enableProfiling: true
clusterCIDR: ${CLUSTER_CIDR}
hostnameOverride: ##NODE_NAME##
mode: "ipvs"
portRange: ""
kubeProxyIPTablesConfiguration:masqueradeAll: false
kubeProxyIPVSConfiguration:scheduler: rrexcludeCIDRs: []
EOFbindAddress: 监听地址；
clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件；
-clusterCIDR: kube-proxy 根据 --cluster-cidr判断集群内部和外部流量，指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
mode: 使用 ipvs 模式；
分发和创建kube-proxy配置文件cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for (( i=0; i < 4; i++ ))do echo ">>> ${NODE_NAMES[i]}"sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" kube-proxy-config.yaml.template > kube-proxy-config-${NODE_NAMES[i]}.yaml.templatescp kube-proxy-config-${NODE_NAMES[i]}.yaml.template root@${NODE_NAMES[i]}:/etc/kubernetes/kube-proxy-config.yamldone#我这里一共有4个节点要运行，所以这里写4，这是整个集群的node节点的数量！ 这里一定要注意修改！！#创建和分发 kube-proxy systemd unit 文件cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
cat > kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=${K8S_DIR}/kube-proxy
ExecStart=/opt/k8s/bin/kube-proxy \\--config=/etc/kubernetes/kube-proxy-config.yaml \\--logtostderr=true \\--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOFcd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}do echo ">>> ${node_name}"scp kube-proxy.service root@${node_name}:/etc/systemd/system/done启动 kube-proxy 服务cd /opt/k8s/work
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "mkdir -p ${K8S_DIR}/kube-proxy"ssh root@${node_ip} "modprobe ip_vs_rr"ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy"donesource /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "systemctl status kube-proxy|grep Active"done[root@k8s-01 work]# netstat -lnpt|grep kube-prox
tcp 0 0 192.168.100.26:10249 0.0.0.0:* LISTEN 32089/kube-proxy
tcp 0 0 192.168.100.26:10256 0.0.0.0:* LISTEN 32089/kube-proxy source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh root@${node_ip} "/usr/sbin/ipvsadm -ln"done[root@k8s-01 work]# for node_ip in ${NODE_IPS[@]}
> do
> echo ">>> ${node_ip}"
> ssh root@${node_ip} "/usr/sbin/ipvsadm -ln"
> done
>>> 192.168.100.26
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr-> 192.168.100.26:6443 Masq 1 0 0 -> 192.168.100.27:6443 Masq 1 0 0 -> 192.168.100.28:6443 Masq 1 0 0
>>> 192.168.100.27
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr-> 192.168.100.26:6443 Masq 1 0 0 -> 192.168.100.27:6443 Masq 1 0 0 -> 192.168.100.28:6443 Masq 1 0 0
>>> 192.168.100.28
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.254.0.1:443 rr-> 192.168.100.26:6443 Masq 1 0 0 -> 192.168.100.27:6443 Masq 1 0 0 -> 192.168.100.28:6443 Masq 1 0 0
>>> 192.168.100.29
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn可见所有通过 https 访问 K8S SVC kubernetes 的请求都转发到 kube-apiserver 节点的 6443 端口；验证集群功能
cd /opt/k8s/work
cat > nginx-ds.yml <<EOF
apiVersion: v1
kind: Service
metadata:name: nginx-dslabels:app: nginx-ds
spec:type: NodePortselector:app: nginx-dsports:- name: httpport: 80targetPort: 80
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:name: nginx-dslabels:addonmanager.kubernetes.io/mode: Reconcile
spec:template:metadata:labels:app: nginx-dsspec:containers:- name: my-nginximage: nginxports:- containerPort: 80
EOFkubectl create -f nginx-ds.yml# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ds-29n8p 1/1 Running 0 116s 172.17.0.2 k8s-02 source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh ${node_ip} "ping -c 1 172.17.0.2"ssh ${node_ip} "ping -c 1 172.30.96.2"ssh ${node_ip} "ping -c 1 172.17.0.2"done#检查服务IP和端口可达性
kubectl get svc |grep nginx-ds
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho ">>> ${node_ip}"ssh ${node_ip} "curl -s 10.254.248.73"doneCoreDNS安装
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}doecho "$node_ip"ssh $node_ip "wget -P /opt/ http://down.i4t.com/coredns_v1.4.tar"ssh $node_ip "docker load -i /opt/coredns_v1.4.tar"donewget -P /opt/ http://down.i4t.com/k8s1.14/coredns.yamlkubectl create -f /opt/coredns.yamlkubectl get pod -n kube-system -l k8s-app=kube-dns参考文章
https://i4t.com/4253.html
相关文章：
飞天使-k8sv1.14二进制安装
文章目录 安装前准备安装前设置分发脚本 开始安装k8s集群cfssl 安装部署kubectl命令行工具创建admin证书和私钥创建kubeconfig文件部署ETCD集群部署Flannel网络kube-apiserver 高可用KeepLived 部署部署master节点部署高可用kube-controller-manager集群kube-controller-manage…...
TypeScript封装Axios
TypeScript封装Axios Axios的基本使用 因axios基础使用十分简单，可参考axios官方文档，这里不在介绍他基本用法，主要讲解拦截器。 拦截器主要分为两种，请求拦截器和响应拦截器。 请求拦截器：请求发送之前进行拦截&…...
指针（一）【C语言进阶版】
大家好，我是深鱼~ 【前言】： 指针的主题，在初阶指针章节已经接触过了，我们知道了指针的概念： 1.指针就是个变量，用来存放地址，地址的唯一标识一块内存空间（指针变量）&a…...
回归预测 | MATLAB实现SA-BP模拟退火算法优化BP神经网络多输入单输出回归预测（多指标，多图）
回归预测 | MATLAB实现SA-BP模拟退火算法优化BP神经网络多输入单输出回归预测（多指标，多图） 目录 回归预测 | MATLAB实现SA-BP模拟退火算法优化BP神经网络多输入单输出回归预测（多指标，多图）效果一览基本介…...
springMVC 已解密的登录请求
问题描述： 解决方案： 1.对用户所输入的密码在页面进行MD5加密并反馈至密码输入框。 2. 手动生成SSL安全访问证书；在此不做介绍，相关方法可通过网上查找； 3. 将产品HTTP访问方式改为SSL安全访问方式；在Ap…...
机器学习赋能乳腺癌预测：如何使用贝叶斯分级进行精确诊断？
一、引言 乳腺癌是女性最常见的恶性肿瘤之一，也会发生在男性身上。每年全球有数百万人被诊断出乳腺癌，对患者的生活和健康造成了巨大的影响。早期的乳腺癌检测和准确的诊断对于提高治疗的成功率至关重要。然而，乳腺癌的早期诊断面临着许多挑战…...
Java后端开发面试题——框架篇
Spring框架中的bean是单例的吗？Spring框架中的单例bean是线程安全的吗？ singleton : bean在每个Spring IOC容器中只有一个实例。 prototype：一个bean的定义可以有多个实例。 Spring bean并没有可变的状态(比如Service类和DAO类)，…...
【新版】系统架构设计师 - 系统测试与维护
个人总结，仅供参考，欢迎加好友一起讨论 文章目录 架构 - 系统测试与维护考点摘要软件测试软件测试 - 测试类型软件测试 - 静态测试软件测试 - 动态测试软件测试 - 测试阶段软件测试 - 测试阶段 - 单元测试软件测试 - 测试阶段 - 集成测试软件测试 - 测试…...
使用 useEffect 和 reactStrictMode：优化 React 组件的性能和可靠性
使用useEffect和React.StrictMode是一种优化React组件性能和可靠性的常见做法。下面是关于如何使用这两个特性的示例： import React, { useEffect } from react;function MyComponent() {useEffect(() > {// 在组件挂载/更新时执行的副作用代码// 可以进行数据获…...
商业智能BI是什么都不明白，如何实现数字化？
2021年下半年中国商业智能软件市场规模为4.8亿美元，2021年度市场规模达到7.8亿美元，同比增长34.9%，呈现飞速增长的趋势。数字化时代，商业智能BI对于企业的落地应用有着巨大价值，逐渐成为了现代企业信息化、数字化转型中…...
【五子棋】
五子棋 文章目录 五子棋前言一、登录功能二.哈希表管理用户的会话和房间三.基于Websocket连接开发的功能1.匹配功能2.游戏房间3.挑战功能4.人机对战5.聊天功能 前言 这篇博客主要详细介绍我的五子棋项目的核心功能的实现细节，也就是详细介绍五子棋各个功能是如何实…...
docker 01(初识docker)
一、docker概念 Docker是一个开源的应用容器引擎；诞生于2013年初，基于Go 语言实现，dotCloud公司出品(后改名为Dockerlnc);Docker 可以让开发者打包他们的应用以及依赖包到一个轻量级、可移植的容器中，然后发布到任何流行的Linux …...
爬虫逆向实战（十九）--某号站登录
一、数据接口分析 主页地址：某号站 1、抓包 通过抓包可以发现登录接口 2、判断是否有加密参数 请求参数是否加密？ 通过查看“载荷”模块可以发现有一个jsondata_rsa的加密参数 请求头是否加密？ 无响应是否加密？ 无cookie是否…...
linux环境docker安装mysql
1：docker拉取mysql镜像（可有自己选择mysql版本） [rootlocalhost ~]# docker pull mysql:8.02： Docker 中启动 MySQL 容器，可以使用以下命令： docker run -d --name mysql_container -v ./sql:/docker-…...
taro h5 formData上传图片的坑-Required request part ‘file‘ is not present
描述：用formData上传图片 1、生成formData const formData new FormData() formData.append(file, data) // data是file formData.append(xxx, xxx) // 添加其他参数2、用taro.request请求 Taro.request({url: xxxx,data: formData,header: {Content-Type: mult…...
GNU GRUB version 2.06 Minimal Bash-lke line editing is supported 问题修复
一、问题背景 博主喜欢折腾系统，电脑原来有一个windows系统，想整一个Linux双系统，结果开机时出现以下画面： GNU GRUB version 2.06 Minimal Bash-lke line editing is supported. TAB lists possible comand completions, Anywh…...
Embedding 向量生成GPT数据使用相关
如果使用python3.6的版本，使用pycharm创建工程，那么默认会使用 docx包，这样运行程序会爆异常，突然想起以前请教的一个大神，想当 初，这个问题困扰了我 两天时间，在此记录一下： pytho…...
Jenkins工具系列 —— 配置邮箱 每个job下动态设置临时发送人
文章目录 安装插件添加邮箱认证邮箱申请（以QQ邮箱网页为例）jenkins添加邮箱认证 jenkins设置邮箱相关信息配置全局邮件单个JOB邮箱配置 安装插件 点击 左侧的 Manage Jenkins —> Plugins ——> 左侧的 Available plugins 添加邮箱认证 邮箱申请…...
华纳云：ubuntu中怎么查看进程占用的端口
在Ubuntu中，你可以使用以下命令来查看进程占用的端口： 打开终端（Terminal）。 使用 netstat 命令来查看进程占用的端口。以下是几个常用的命令示例： 查看所有进程占用的端口和地址： netstat -tuln 查看TCP端…...
【学会动态规划】 最长递增子序列（26）
目录 动态规划怎么学？ 1. 题目解析 2. 算法原理 1. 状态表示 2. 状态转移方程 3. 初始化 4. 填表顺序 5. 返回值 3. 代码编写 写在最后： 动态规划怎么学？ 学习一个算法没有捷径，更何况是学习动态规划， 跟我…...
Azure虚拟网络对等互连
什么是Azure虚拟网络对等互联 Azure虚拟网络对等互联（Azure Virtual Network peering）是一种连接两个虚拟网络的方法，使得这两个虚拟网络能够在同一地理区域内进行通信。它通过私有IP地址在虚拟网络之间建立网络连接，不论是在同一…...
CTFhub-sql-整数注入
判断存在 sqli 注入 1 1 and 11 1 and 12 因为 11 为真，12 为假，且 11 与 1 显示的数据一样，那么就存在 sqli 注入 查询该数据表的字段数量 一、 2 3 1,2成功带出数据，3没有数据，所以有两个字段 二、 1 order by …...
管理类联考——逻辑——真题篇——按知识分类——汇总篇——二、论证逻辑——归纳——第三节 归纳论证有效性
文章目录 第三节 归纳论证有效性真题（2007-37）——归纳——归纳论证有效性——两面验证法真题（2000-60）——归纳——归纳论证有效性——构造对照组实验真题（2001-44）——归纳——归纳论证有效性——寻找针对该缺陷的选项第三节 归纳论证有效性 真题（2007-37）——归纳—…...
PaddleRS 1.0.0版本安装
PaddleRS 1.0.0版本安装 PaddleRS是百度飞桨、遥感科研院所及相关高校共同开发的基于飞桨的遥感影像智能解译开发套件， 支持图像分割、目标检测、场景分类、变化检测、图像复原等常见遥感任务。 PaddleRS致力于帮助遥感领域科研从业者快速完成算法的研发、验证和调…...
三维重建 PyQt Python MRP 四视图（横断面，冠状面，矢状面，3D）
本文实现了 Python MPR 的 四视图，横断面，冠状面，矢状面，3D MPR(multi-planner reformation)也称多平面重建，多重面重建是将扫描范围内所有的轴位图像叠加起来再对某些标线标定的重组线所指定的组织进行冠状、矢状位、…...
使用vscode编写插件-php语言
https://blog.csdn.net/qq_45701130/article/details/125206645 一、环境搭建 1、安装 Visual Studio Code 2、安装 Node.js 3、安装 Git 4、安装生产插件代码的工具：npm install -g yo generator-code 二、创建工程 yo code 选择项解释： 选择编写扩…...
【笔记】Spark3 AQE(Adaptive Query Execution)
提效 7 倍，Apache Spark 自适应查询优化在网易的深度实践及改进 Performance Tuning 配置Spark SQL开启Adaptive Execution特性 How To Use Spark Adaptive Query Execution (AQE) in Kyuubi 【spark系列3】spark 3.0.1 AQE(Adaptive Query Exection)分析 玩转Spark…...
【雷达】接收和去噪L波段雷达接收到的信号研究（Matlab代码实现）
💥💥💞💞欢迎来到本博客❤️❤️💥💥 🏆博主优势：🌞🌞🌞博客内容尽量做到思维缜密，逻辑清晰，为了方便读者。 ⛳️座右铭&a…...
【云原生】Docker Cgroups资源控制管理
目录 一、cgroups简介 cgroups有四大功能： 二、cpu时间片的概念 三、对CPU使用的限制 3.1 设置CPU使用率上限 （1）查看容器的默认CPU使用限制 （2）进行压力测试 （3）创建容器时设置CPU使用时…...
k8s部署prometheus
1、prometheus部署yml文件地址 github地址 2、下载yml文件 rootiZj6cd9joygowsf7am5hryZ:~# git clone https://github.com/redhatxl/k8s-prometheus-grafana.git Cloning into k8s-prometheus-grafana... remote: Enumerating objects: 21, done. remote: Total 21 (delta 0)…...
飞书小程序开发
1.tt.showModal后跳转页面 跳转路径要为绝对路径，相对路径跳转无响应。 2.手机息屏后将不再进入onload()生命周期，直接进入onshow()生命周期。 onLoad()在页面初始化的时候触发，一个页面只调用一次。 onShow()在切入前台时就会触发&#x…...
Revit 3D高效处理：cad exchanger sdk 3.21 Crack
3D 格式概述：Revit Revit 已成为寻求高效、准确的建筑信息建模的专业人士的首选解决方案。在这篇引人入胜的功能概述中了解 Revit 的特性和影响。 什么是Revit？ Autodesk Revit 是一款流行的 CAD 软件，重点关注 BIM，被建筑师、工…...
【已解决】Linux中启动docker 出现 ‘ Failed to start docker.service: Unit not found. ’ 错误
启动docker 出现 ‘ Failed to start docker.service: Unit not found. ’ 错误 这是因为缺少 rhel-push-plugin.socket 单元，该单元是rhel-push-plugin软件包的一部分。所以我们执行以下指令就可以成功解决： curl -sSL https://get.docker.com/ | sh 执…...
【学习日记】【FreeRTOS】时间片的实现
前言 本文以野火的教程和代码为基础，对 FreeRTOS 中时间片的概念作了解释，并且给出了实现方式，同时发现并解决了野火教程代码中的 bug。 一、时间片是什么 在前面的文章中，我们已经知道任务根据不同的优先级被放入就绪列表中不…...
CentOS Docker仓库和代理配置
无法直接访问外部网络时，除了Host自己的全局代理设置之外，需要单独给Docker Client和Instance设置代理。 如执行docker run时遇到下面的错误 docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp 3.216.…...
Lnton羚通算法算力云平台在环境配置中Windows10终端和VSCode下如何打开Anaconda-Prompt
在Windows 10的终端和VSCode中，可以直接打开Anaconda Prompt。下面是两种方法： Windows 10终端：在开始菜单中搜索"Anaconda Prompt"，然后点击打开。这将启动Anaconda Prompt终端，你可以在其中执行conda相关命…...
Python web实战之细说Django的集成测试
关键词： Python Web开发、Django、集成测试、实战、测试驱动开发、自动化测试、Selenium、测试框架、测试用例、代码覆盖率、持续集成 今天给大家分享一下Python Web开发——Django的集成测试，如何利用集成测试来提高代码质量、减少bug。 1. 什么是集成…...
Laravel 模型的作用域 模型的访问器和修改器 ⑨
作者 : SYFStrive 博客首页 : HomePage 📜： THINK PHP 📌：个人社区（欢迎大佬们加入） 👉：社区链接🔗 📌：觉得文章不错可以点点关注 ὄ…...
每日一学——交换机
交换机是一种网络设备，用于连接多台计算机和其他网络设备，以实现数据的交换和传输。它通过将数据包在不同端口之间转发，将数据从一个设备发送到目标设备。交换机可以提供高速、可靠和安全的局域网连接。 交换机的工作原理是根据目标MAC地址来…...
数学建模大全及优缺点解读
分类模型 1、距离聚类（系统聚类）（常用，需掌握） 优点： ①将一批样本数据按照他们在性质上的亲密程度在没有先验知识的情况下自动进行分类 ②是一种探索性的分析方法，分类结果不一定相同 例如&am…...
C++简介
文章目录 C简介C版本C11例子 C14例子 C17C20例子 C简介 C是一种高级编程语言，它是对C语言的扩展和增强。C由Bjarne Stroustrup于1980年发明，主要用于系统级编程、游戏开发、嵌入式系统等领域。 C具有许多特性，其中最重要的是面向对象编程&a…...
【广州华锐互动】3D空间编辑器：一款简洁易用的VR/3D在线编辑工具
随着虚拟现实技术的不断发展，数字孪生技术的应用已经被广泛应用于产品设计和制作中，能充分发挥企业应用3D建模的优势，凸显了三维设计的价值，在生产阶段也能够充分发挥3D模型的作用。 如今，广州华锐互动开发的3D空间编辑…...
golang云原生项目☞redis配置
配置redis适用与golang云原生架构。包括redis与数据库一致性等重要内容 1、编写redis配置文件、使用viper读取 配置文件 db.yml redis:addr: 127.0.0.1port: 6379password: tiktokRedisdb: 0 # 数据库编号读取配置文件 var (config viper.Init("db")zapL…...
C++ malloc/free/new/delete详解（内存管理）
C malloc/free/new/delete详解（内存管理） malloc/free典型用法内存分配实现过程brk和mmap申请小于128k的内存申请大于128k的内存释放内存brk和mmap的区别 new/delete典型用法 内存分配实现过程new/delete和malloc/free的区别malloc对于给每个进程分配的内…...
SpringBoot中Mapper.xml的入参方式
在SpringBoot开发过程中，我们使用 ***Mapper.xml***Mapper.java 来封装对数据库表的 CURD 操作，正常每张表会有一组对应的文件。 一、Mapper常见用法 下面例举一个查询操作： 数据表t_sap_customer，表中有字段id、code、name、c…...
回归预测 | MATLAB实现WOA-RBF鲸鱼优化算法优化径向基函数神经网络多输入单输出回归预测（多指标，多图）
回归预测 | MATLAB实现WOA-RBF鲸鱼优化算法优化径向基函数神经网络多输入单输出回归预测（多指标，多图） 目录 回归预测 | MATLAB实现WOA-RBF鲸鱼优化算法优化径向基函数神经网络多输入单输出回归预测（多指标，多图&#…...
浅析Python爬虫ip程序延迟和吞吐量影响因素
作为一名资深的爬虫程序员，今天我们很有必要来聊聊Python爬虫ip程序的延迟和吞吐量，这是影响我们爬取效率的重要因素。这里我们会提供一些实用的解决方案，让你的爬虫程序飞起来！ 网络延迟 首先，让我们来看看网络延迟对…...
【100天精通python】Day43：python网络爬虫开发_爬虫基础（urlib库、Beautiful Soup库、使用代理+实战代码）
目录 1 urlib 库 2 Beautiful Soup库 3 使用代理 3.1 代理种类 HTTP、HTTPS 和 SOCKS5 3.2 使用 urllib 和 requests 库使用代理 3.3 案例：自建代理池 4 实战 提取视频信息并进行分析 1 urlib 库 urllib 是 Python 内置的标准库，用于处理URL、发送…...
Linux:安全技术与防火墙
目录 一、安全技术 1.安全技术 2.防火墙的分类 3.防水墙 4.netfilter/iptables关系 二、防火墙 1、iptables四表五链 2、黑白名单 3.iptables命令 3.1查看filter表所有链 iptables -L 编辑3.2用数字形式(fliter)表所有链 查看输出结果 iptables -nL 3.3 清空所有链…...
Confluent kafka 异常退出rd_tmpabuf_alloc0: rd kafka topic info_new_with_rack
rd_tmpabuf_alloc0: rd kafka topic info_new_with_rack 根据网上的例子，做了一个测试程序。 C# 操作Kafka_c# kafka_Riven Chen的博客-CSDN博客 但是执行下面一行时，弹出上面的异常，闪退。 consumer.Subscribe(queueName) 解决方案&…...