在鲲鹏服务器搭建k8s高可用集群分享

高可用架构

本文采用kubeadm方式搭建k8s高可用集群，k8s高可用集群主要是对apiserver、etcd、controller-manager、scheduler做的高可用；高可用形式只要是为：

1. apiserver利用haproxy+keepalived做的负载，多apiserver节点同时工作；

2. etcd利用k8s内部提供的解决方案做的高可用，最多容忍一台etcd服务宕机

3. controller-manager、scheduler的高可用机制为如果多个服务同时存在，则会选举一个leader工作，另外两个处于sleep的状态

集群信息规划

主机名称	系统版本	主机IP	备注
K8s-master01	Centos7.6	192.168.0.101	Master01/registry
K8s-master02	Centos7.6	192.168.0.102	Master02
K8s-master03	Centos7.6	192.168.0.103	Master03
K8s-node01	Centos7.6	192.168.0.111	Node01
K8s-node02	Centos7.6	192.168.0.112	Node02
K8s-node03	Centos7.6	192.168.0.113	Node03
Apiserver-keepalived	Centos7.6	192.168.0.100	HA-apiserver

服务器的初始化

#此些操作根据实际需求所有服务器都需要操作 

修改hosts文件 

192.168.0.101 k8s-master01 192.168.0.102 k8s-master02  192.168.0.103 k8s-master03 192.168.0.111 k8s-node01 192.168.0.112 k8s-node02 192.168.0.113 k8s-node03

修改主机名称

hostnamectl set-hostname k8s-master01 hostname -b k8s-master01 #其余服务器按照此方式进行修改

关闭NetworkManager服务

[root@k8s-master01 ~]# systemctl stop NetworkManager
[root@k8s-master01 ~]# systemctl disable NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.

修改服务器的IP地址

[root@k8s-master01 ~]# sed -i '/IP/d' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# sed -i 's/BOOTPROTO=dhcp/BOOTPROTO=static/g' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# sed -i 's/ONBOOT=no/ONBOOT=yes/g' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# sed -i '/UUID/d' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# echo -e "IPADDR=192.168.0.101\nNETMASK=255.255.255.0\nGATEWAY=192.168.0.1" >> /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# systemctl restart network

安装依赖包

[root@k8s-master01 ~]# yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wgetvimnet-tools git

设置防火墙并且清空规则

[root@k8s-master01 ~]# systemctl stop firewalld && systemctl disable firewalld
[root@k8s-master01 ~]#yum -y install iptables-services && systemctl start iptables && systemctl enable iptables&& iptables -F && service iptables save

关闭selinux

[root@k8s-master02 ~]# swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[root@k8s-master01 ~]# setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

服务器之间免密配置

#生成密钥
[root@k8s-master01 ~]# ssh-keygen -t rsa#将密钥传到其他服务器上面
[root@k8s-master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.102

Docker依赖安装

[root@k8s-master01 ~]# yum install -y   yum-utils device-mapper-persistent-data lvm2

安装Docker源

[root@k8s-master01 ~]# yum-config-manager --add-repo  http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装Docker

[root@k8s-master01 ~]# yum update -y && yum install -y docker-ce

配置daemon配置文件

[root@k8s-master01 ~]# mkdir -p /etc/docker
[root@k8s-master01 ~]# mkdir -p  /etc/systemd/system/docker.service.d
[root@k8s-master01 ~]# cat > /etc/docker/daemon.json < {
> "exec-opts": ["native.cgroupdriver=systemd"], #着重注意标红字体
> "log-driver": "json-file",
> "log-opts": {
> "max-size": "100m"
> },
> "insecure-registries":["http://registry.k8s-test.com"] #后面会使用到私有镜像仓库
> }
> EOF

启动docker并且设置开机自启动

[root@k8s-master01 ~]# systemctl daemon-reload && systemctl restart docker && systemctl enable docker

安装kubelet

[root@k8s-master01 ~]# cat  <  /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=0
> repo_gpgcheck=0
> gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
> http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
> EOF[root@k8s-master01 ~]# yum install -y kubelet-1.16.4 kubeadm-1.16.4 kubectl-1.16.4

启动kubelet并且设置为开机启动

[root@k8s-master01 ~]#  systemctl enable kubelet && systemctl start kubelet

Keepalive安装

[root@k8s-master01 ~]# yum -y install keepalived

修改配置文件:

master01和master02和master03配置一样，只是router_id记得变一下

[root@k8s-master01 ~]# more /etc/keepa hived/keepalhived.Conf
! Configuration File for keepalived 
global_ defs{router_id master01
}
vrrp_instance VI_1 {state BACKUPinterface ens33virtual_router_id 5priority 90advert_int 1authentication {auth type PASSauth pass 1111}virtual ipaddress {192.168.0.100}
}

启动keepalived

[root@k8s-master02 ~]# systemctl enable keepalived && systemctl start keepalived

验证

master01

master02

master03

k8s安装

下载镜像

[root@k8s-master01 tools]# sh get_images.sh

部署镜像中心

[root@k8s-master01 images]# docker images | grep k8s-registry.com/registry[root@k8s-master01 images]# docker run -d -p 80:5000 -v /home/registry:/var/lib/registry --restart=always --name registry k8s-registry.com/registry:1.0

将镜像push到镜像中心

[root@k8s-master01 images]# docker images | awk '{print $1":"$2}' | xargs -i docker push {}

创建kubeadm-config.yaml的配置文件

[root@k8s-master01 install-master]# more  kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.4
apiServer:certSANs:- k8s-master01- k8s-master02- k8s-master03- k8s-node01- k8s-node02- k8s-node03- 192.168.0.100- 192.168.0.101- 192.168.0.102- 192.168.0.103- 192.168.0.111- 192.168.0.112- 192.168.0.113
controlPlaneEndpoint: "192.168.0.100:6443"
networking:podSubnet: "10.244.0.0/16"

初始化master

[root@k8s-master01 install-master]# kubeadm init --config=kubeadm-config.yaml

验证安装

创建flannel网络

下载flannel的yaml文件

[root@k8s-master01 flannel]# wget https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

创建flannel网络

[root@k8s-master01 flannel]# kubectl app -f kube-flannel.yml

验证安装

master节点添加至集群

master01向其他两个节点颁发证书

[root@k8s-master01 tools]# cat cert-amin-master.sh
[root@k8s-master01 tools]# sh cert-amin-master.sh

其他节点将证书放到指定目录

[root@k8s-master02 ~]# sh cert-other-master.sh
[root@k8s-master02 ~]# more cert-other-master.sh

加入集群

kubeadm join 192.168.0.100:6443 --token lllil4.2wm1u6ocuxmysn7l \--discovery-token-ca-cert-hash sha256:fa5075ba896b8dbfdaf19125dee28817fdd349b7c4cea9ab243ad4224eb90892 \--control-plane

查看布置的节点