kubekey 离线安装高可用 kubernetes 集群

1. 准备环境

版本：
kubernetes: v1.29.2
kubesphere: v3.4.1
kubekey: v3.1.1

说明：

• kubekey 只用于安装 kubernetes，因为 kubesphere 的配置在安装时经常需要变动，用 ks-installer 的 yaml 文件更好管理；
• ks-installer 用于安装 kubesphere，kubekey、ks-installer 分工明确；
• 本文在已有 harbor 仓库环境下，建议把镜像都放在公开仓库 library 中，如没有 harbor 仓库，按官方文档来即可；

1.1 机器准备

4 台机器，操作系统：Ubuntu 24.04/RHEL8/CentOS9

10.0.0.130 k8smaster-lv01
10.0.0.131 k8snode-lv01
10.0.0.132 k8snode-lv02
10.0.0.133 k8snode-lv03
10.0.0.140 lb.kubesphere.local

1.2 离线安装包准备

## kk create manifest --with-kubernetes v1.29.2
# kk create manifest  # 最好在已有集群创建

这里提供一个可以直接使用的，因为系统安装包，我们自己处理，所以这里不用管系统的 iso 啥的地址。
manifest-sample.yaml:

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Manifest
metadata:name: sample
spec:arches:- amd64operatingSystems:- arch: amd64type: linuxid: redversion: "Can't get the os version. Please edit it manually."osImage: Red Hat Enterprise Linux 8.10 (Ootpa)repository:iso:localPath: url:- arch: amd64type: linuxid: redversion: "Can't get the os version. Please edit it manually."osImage: Red Hat Enterprise Linux 8.9 (Ootpa)repository:iso:localPath: url:kubernetesDistributions:- type: kubernetesversion: v1.29.2components:helm: version: v3.14.3cni: version: v1.2.0etcd: version: v3.5.13containerRuntimes:- type: containerdversion: 1.7.13calicoctl:version: v3.27.3crictl: version: v1.29.0images:- docker.io/aledbf/kube-keepalived-vip:0.35- docker.io/bitnami/etcd:3.5.6-debian-11-r10- docker.io/bitnami/kubectl:1.29.2- docker.io/calico/cni:v3.27.3- docker.io/calico/node:v3.27.3- docker.io/coredns/coredns:1.9.3- docker.io/grafana/promtail:2.8.3- docker.io/kubesphere/examples-bookinfo-reviews-v1:1.16.2- docker.io/kubesphere/fluent-bit:v1.9.4- docker.io/kubesphere/k8s-dns-node-cache:1.22.20- docker.io/kubesphere/ks-apiserver:v3.4.1- docker.io/kubesphere/ks-installer:v3.4.1- docker.io/kubesphere/ks-installer:v3.4.1-patch.0- docker.io/kubesphere/ks-jenkins:v3.4.0-2.319.3-1- docker.io/kubesphere/kube-apiserver:v1.29.2- docker.io/kubesphere/kube-controller-manager:v1.29.2- docker.io/kubesphere/kube-proxy:v1.29.2- docker.io/kubesphere/kube-rbac-proxy:v0.11.0- docker.io/kubesphere/kube-scheduler:v1.29.2- docker.io/kubesphere/pause:3.9- docker.io/library/busybox:latest- docker.io/openebs/lvm-driver:1.5.0- docker.io/openebs/mayastor-agent-ha-node:v2.6.1- docker.io/openebs/mayastor-csi-node:v2.6.1- docker.io/openebs/mayastor-io-engine:v2.6.1- docker.io/openebs/zfs-driver:2.5.0- docker.io/opensearchproject/opensearch:2.6.0- docker.io/osixia/openldap:1.3.0- docker.io/prom/node-exporter:v1.3.1- docker.io/weaveworks/scope:1.13.0- quay.io/argoproj/argocd:v2.3.3- registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0- registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0registry:auths: {}

# 生成离线安装包
kk artifact export -m manifest-sample.yaml -o kubesphere.tar.gz

因为最近 docker hub 无法在大陆拉 image，可以使用我这个 Jenkinsfile 配合海外节点将 image 拉下来，传到私有 harbor 仓库。
mirror-images-to-harbor，当前直接在它上面提 issue 同步也可，只是一个 issue 只能同步一个 image。

当然，官方的这个 image 同步脚本 offline-installation-tool.sh 也可以，它是直接拉的 image，可能拉不动，你下面列表里的 image 得转成你拉得动的 image 地址：

以下为image列表，生成的 image 列表，并不完全，所以后面发现有漏的 image，随时使用 jenkins 把缺少的image同步下。

docker.io/aledbf/kube-keepalived-vip:0.35
docker.io/bitnami/etcd:3.5.6-debian-11-r10
docker.io/bitnami/kubectl:1.29.2
docker.io/calico/cni:v3.27.3
docker.io/calico/node:v3.27.3
docker.io/coredns/coredns:1.9.3
docker.io/grafana/promtail:2.8.3
docker.io/kubesphere/examples-bookinfo-reviews-v1:1.16.2
docker.io/kubesphere/fluent-bit:v1.9.4
docker.io/kubesphere/k8s-dns-node-cache:1.22.20
docker.io/kubesphere/ks-apiserver:v3.4.1
docker.io/kubesphere/ks-installer:v3.4.1
docker.io/kubesphere/ks-installer:v3.4.1-patch.0
docker.io/kubesphere/ks-jenkins:v3.4.0-2.319.3-1
docker.io/kubesphere/kube-apiserver:v1.29.2
docker.io/kubesphere/kube-controller-manager:v1.29.2
docker.io/kubesphere/kube-proxy:v1.29.2
docker.io/kubesphere/kube-rbac-proxy:v0.11.0
docker.io/kubesphere/kube-scheduler:v1.29.2
docker.io/kubesphere/pause:3.9
docker.io/library/busybox:latest
docker.io/openebs/lvm-driver:1.5.0
docker.io/openebs/mayastor-agent-ha-node:v2.6.1
docker.io/openebs/mayastor-csi-node:v2.6.1
docker.io/openebs/mayastor-io-engine:v2.6.1
docker.io/openebs/zfs-driver:2.5.0
docker.io/opensearchproject/opensearch:2.6.0
docker.io/osixia/openldap:1.3.0
docker.io/prom/node-exporter:v1.3.1
docker.io/weaveworks/scope:1.13.0
quay.io/argoproj/argocd:v2.3.3
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0

1.3 安装依赖和配置

系统源即可安装的依赖包，所有节点都需要执行

Ubuntu 系:

apt-get install -y socat conntrack ebtables ipset chrony

RHEL 系：

yum install -y socat conntrack ebtables ipset chrony 

因为使用私有 Harbor 仓库，所有节点配置 host：

10.25.23.102 harbor.ops.shenzhen.com

1.4 安装负载均衡 keepalived + haproxy

负载均衡节点安装 keepalived haproxy

apt-get -y install keepalived haproxy
# yum -y install keepalived haproxy

新建用户

useradd -r -u 139 -g 100 -s /sbin/nologin keepalived_script

配置 keepalived
/etc/keepalived/keepalived.conf （注意配置中注释，不同节点权重不同）内容：

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {script_user keepalived_scriptenable_script_security     router_id LVS_DEVELmax_auto_priority 99
}
vrrp_script check_apiserver {script "/etc/keepalived/check_apiserver.sh"interval 3weight -2fall 10rise 2
}vrrp_instance VI_1 {# 只配置一个 MASTER，其它的配置为 BACKUPstate MASTER# 注意网卡名interface ens192virtual_router_id 60# MASTER 权重最高，尽量全部设置为不同的权重priority 101authentication {auth_type PASSauth_pass k8s}virtual_ipaddress {10.0.0.140}track_script {check_apiserver}
}

/etc/keepalived/check_apiserver.sh 内容：

#!/bin/bash# if check error then repeat check for 12 times, else exit
err=0
for k in $(seq 1 12)
docheck_code=$(curl -k https://localhost:6443)if [[ $check_code == "" ]]; thenerr=$(expr $err + 1)sleep 5continueelseerr=0breakfi
doneif [[ $err != "0" ]]; then# if apiserver is down send SIG=1echo 'apiserver error!'exit 1
else# if apiserver is up send SIG=0echo 'apiserver normal!'exit 0
fi

chmod a+x /etc/keepalived/check_apiserver.sh

配置 haproxy
/etc/haproxy/haproxy.cfg 内容：

globallog 127.0.0.1 local0 errmaxconn 50000uid 138 gid 138#daemonpidfile haproxy.piddefaultsmode httplog 127.0.0.1 local0 errmaxconn 50000retries 3timeout connect 5stimeout client 30stimeout server 30stimeout check 2slisten admin_statsmode httpbind 0.0.0.0:1080log 127.0.0.1 local0 errstats refresh 30sstats uri     /haproxy-statusstats realm   Haproxy\ Statisticsstats auth    admin:k8sstats hide-versionstats admin if TRUEfrontend k8s-httpsbind 0.0.0.0:8443mode tcp#maxconn 50000default_backend k8s-httpsbackend k8s-httpsmode tcpbalance roundrobinserver master1 10.0.0.130:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3

systemctl start haproxy
systemctl start keepalived
systemctl enable haproxy
systemctl enable keepalived

2. kubernetes 集群安装

2.1 kk 安装

kk 命令是二进制的，从安装好的机器，直接拷贝到安装集群的 master 机器即可。

curl -sfL https://get-kk.kubesphere.io | VERSION=v3.1.1 sh -
mv kk /usr/local/sbin/
kk version --show-supported-k8s# 生成 ssh key
ssh-keygen -t ed25519 -C "master"
# 公钥添加到其它节点
ssh-copy-id -i ~/.ssh/id_ed25519.pub root@其它节点IP

2.2 kk 配置

mkdir k8s
cd k8s
# kk create config --with-kubernetes v1.29.2

直接使用如下配置文件 config-sample.yaml，然后修改下节点信息，如下：

内容：

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:name: sample
spec:hosts:- {name: k8smaster-lv01, address: 10.0.0.130, internalAddress: 10.0.0.130, user: root, privateKeyPath: "~/.ssh/id_ed25519"}- {name: k8snode-lv01, address: 10.0.0.131, internalAddress: 10.0.0.131, user: root, privateKeyPath: "~/.ssh/id_ed25519"}- {name: k8snode-lv02, address: 10.0.0.132, internalAddress: 10.0.0.132, user: root, privateKeyPath: "~/.ssh/id_ed25519"}- {name: k8snode-lv03, address: 10.0.0.133, internalAddress: 10.0.0.133 , user: root, privateKeyPath: "~/.ssh/id_ed25519"}roleGroups:etcd:- k8smaster-lv01- k8snode-lv01- k8snode-lv02control-plane: - k8smaster-lv01worker:- k8snode-lv01- k8snode-lv02- k8snode-lv03controlPlaneEndpoint:## Internal loadbalancer for apiservers # internalLoadbalancer: haproxydomain: lb.kubesphere.localaddress: "10.0.0.140"port: 8443kubernetes:version: v1.29.2clusterName: cluster.localautoRenewCerts: truecontainerManager: containerdetcd:type: kubekeynetwork:plugin: calicokubePodsCIDR: 10.233.64.0/18kubeServiceCIDR: 10.233.0.0/18## multus support. https://github.com/k8snetworkplumbingwg/multus-cnimultusCNI:enabled: falseregistry:privateRegistry: "harbor地址"namespaceOverride: ""registryMirrors: [""]# insecureRegistries: ["非httpsimage地址"]  # 如果是 http，请取消注释#auths:#  "harbor地址":#    username: ''#    password: ''#    skipTLSVerify: true # Allow contacting registries over HTTPS with failed TLS verification.#    plainHTTP: true # Allow contacting registries over HTTP.addons: []

2.3 kk 安装集群

检查时间，如果时间未同步，重启 chronyd 服务。

date
systemctl stop chronyd
systemctl start chronyd
chronyc tracking
# chronc -a makestep # 强制同步

集群安装：

kk create cluster -f config-sample.yaml -a kubesphere.tar.gz

如果使用的 harbor 是 http 的，kubekey 内部是使用的 https 的仓库，可能会中断安装。我们再次安装，前面二进制文件已经缓存至 kubekey 安装目录，然后不加 -a 参数，就是在线安装，只是使用内部仓库的在线安装方式。

再次集群安装：

kk create cluster -f config-sample.yaml

有可能需要处理的地方

1. /etc/containerd/config.toml，里面改成私有仓库地址

    sandbox_image = "harbor地址/library/kubesphere/pause:3.9"

2. 如果你需要 coredns 添加 hosts，注意 /etc/resolv.conf 里面要设置 DNS，否则 coredns 可能无法启动。
   kubectl edit cm coredns -n kube-system：

data:Corefile: |.:53 {errorshealth {lameduck 5s}hosts /etc/coredns/hosts {IP 域名fallthrough} 

kubectl edit cm nodelocaldns -n kube-system：

    }.:53 {errorscache 30reloadloopbind 169.254.25.10# forward . /etc/resolv.confforward . 10.233.0.3 {force_tcp}prometheus :9253}

2.4 kk 安装集群失败处理

在 master 节点执行，注意确认正确的节点和危险性！！！

kk delete cluster -f config-sample.yaml

3. kubesphere 安装

根据官方文档来即可。

curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml
curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yaml

spec:persistence:storageClass: ""authentication:jwtSecret: ""local_registry: 你的harbor地址/library  # 镜像都放在 library 下并保持原来目录结构

sed -i "s#^\s*image: kubesphere.*/ks-installer:.*#        image: 你的harbor地址/library/kubesphere/ks-installer:v3.4.1#" kubesphere-installer.yaml

安装：

kubectl apply -f kubesphere-installer.yaml
kubectl apply -f cluster-configuration.yaml

最容易出问题的就是镜像拉取，随时查看并解决镜像问题。

kubectl get pod -A
kubectl describe pod <pod名> -n <namespace>

4. 暴露 kubesphere 控制台

4.1 安装 metallb

推荐 helm 方式安装，这里略，详情参考 metallb官方文档。
安装好后配置一个全局 IP 池。

4.2 打开全局网关

使用 http://某一节点IP:30880 登录 ks 后，在集群设置中，打开全局网关，注意使用 Loadbalancer 类型暴露，因为有全局 IP 池可用，按理说随便选一个负载均衡厂商即可，因为没有 metallb 选项。

4.3 ingress 暴露 ks 控制台

console-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/upstream-vhost: ks-console.kubesphere-system.svc.cluster.localname: ks-consolenamespace: kubesphere-system
spec:rules:- host: ks域名http:paths:- backend:service:name: ks-consoleport:number: 80path: /pathType: ImplementationSpecific

kubectl apply -f console-ingress.yaml

参考资料：
[1] https://www.kubesphere.io/zh/docs/v3.4/installing-on-linux/introduction/air-gapped-installation/
[2] https://www.kubesphere.io/zh/docs/v3.4/installing-on-kubernetes/on-prem-kubernetes/install-ks-on-linux-airgapped/