K8S cluster with multi-masters on Azure VM

拓扑参考：

在 Azure VM 实例上部署 KubeSphere

• 基础模板
  需要修改 IP 地址和 VM Image的可以在模板中修改。

  {"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#","contentVersion": "1.0.0.0","parameters": {"vmNamePrefix": {"defaultValue": "master-","type": "String","metadata": {"description": "The name of your VM master node."}},"vmssName": {"defaultValue": "node","type": "String","metadata": {"description": "The name of your VMSS cluster."}},"location": {"defaultValue": "[resourceGroup().location]","type": "String","metadata": {"description": "Location for all resources."}},"adminUsername": {"type": "String","metadata": {"description": "Username for the Virtual Machine."}},"adminKey": {"type": "SecureString","metadata": {"description": "SSH Key for the Virtual Machine."}},"defaultMasterCount": {"defaultValue": 3,"type": "Int","metadata": {"description": "The default instances count of master"}},"defaultNodeCount": {"defaultValue": 3,"type": "Int","metadata": {"description": "The initial node size of your VMSS cluster."}},"minNodeCount": {"defaultValue": 1,"type": "Int","metadata": {"description": "The min node size of your VMSS cluster."}},"maxNodeCount": {"defaultValue": 20,"type": "Int","metadata": {"description": "The max node size of your VMSS cluster."}},"dnsLabelPrefix": {"defaultValue": "[toLower(concat('k8s-cluster-', uniqueString(resourceGroup().id)))]","type": "String","metadata": {"description": "Unique DNS Name for the Public IP used to access the Virtual Machine."}},"vmSize": {"defaultValue": "Standard_DS2_v2","type": "String","metadata": {"description": "The size of the VM"}},"virtualNetworkName": {"defaultValue": "vNetwork","type": "String","metadata": {"description": "Name of the Virtual Network"}},"subnetName": {"defaultValue": "Subnet","type": "String","metadata": {"description": "Name of the subnet in the virtual network"}},"vmssSubnetName": {"defaultValue": "nodeSubnet","type": "String","metadata": {"description": "Name of the VMSS subnet in the virtual network"}},"publicLBName": {"defaultValue": "publicLB","type": "String","metadata": {"description": "Internal Load Balancer name"}}},"variables": {"publicIPAddressName": "[concat(parameters('publicLBName'), 'IP' )]","availabilitySetName": "masterAvSet","networkInterfaceName": "[concat(parameters('vmNamePrefix'),'Interface')]","networkSecurityGroupName": "[concat(parameters('virtualNetworkName'),'nsg-default')]","subnetRef": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworkName'), parameters('subnetName'))]","vmssSubnetRef": "[resourceId('Microsoft.Network/virtualNetworks/subnets', parameters('virtualNetworkName'), parameters('vmssSubnetName'))]","osDiskType": "Standard_LRS","publicLBID": "[resourceId('Microsoft.Network/loadBalancers',parameters('publicLBName'))]"},"resources": [{"type": "Microsoft.Network/networkSecurityGroups","apiVersion": "2015-06-15","name": "[variables('networkSecurityGroupName')]","location": "[parameters('location')]","properties": {"securityRules": [{"name": "Port_SSH","properties": {"description": "SSH","protocol": "*","sourcePortRange": "*","destinationPortRange": "22","sourceAddressPrefix": "*","destinationAddressPrefix": "*","access": "Allow","priority": 100,"direction": "Inbound","sourcePortRanges": [],"destinationPortRanges": [],"sourceAddressPrefixes": [],"destinationAddressPrefixes": []}},{"name": "Port_API_Server","properties": {"description": "k8s API Server","protocol": "TCP","sourcePortRange": "*","destinationPortRange": "6443","sourceAddressPrefix": "*","destinationAddressPrefix": "*","access": "Allow","priority": 140,"direction": "Inbound","sourcePortRanges": [],"destinationPortRanges": [],"sourceAddressPrefixes": [],"destinationAddressPrefixes": []}},{"name": "Port_Http","properties": {"description": "Web","protocol": "TCP","sourcePortRange": "*","destinationPortRange": "80","sourceAddressPrefix": "*","destinationAddressPrefix": "*","access": "Allow","priority": 120,"direction": "Inbound","sourcePortRanges": [],"destinationPortRanges": [],"sourceAddressPrefixes": [],"destinationAddressPrefixes": []}},{"name": "Port_Https","properties": {"protocol": "TCP","sourcePortRange": "*","destinationPortRange": "443","sourceAddressPrefix": "*","destinationAddressPrefix": "*","access": "Allow","priority": 130,"direction": "Inbound","sourcePortRanges": [],"destinationPortRanges": [],"sourceAddressPrefixes": [],"destinationAddressPrefixes": []}}]}},{"type": "Microsoft.Network/virtualNetworks","apiVersion": "2019-11-01","name": "[parameters('virtualNetworkName')]","location": "[parameters('location')]","dependsOn": ["[resourceId('Microsoft.Network/networkSecurityGroups/', variables('networkSecurityGroupName'))]"],"properties": {"addressSpace": {"addressPrefixes": ["10.211.0.0/16"]},"subnets": [{"name": "[parameters('vmssSubnetName')]","properties": {"networkSecurityGroup": {"id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]"},"addressPrefix": "10.211.0.0/24"}},{"name": "[parameters('subnetName')]","properties": {"networkSecurityGroup": {"id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('networkSecurityGroupName'))]"},"addressPrefix": "10.211.1.0/24"}}]}},{"type": "Microsoft.Network/networkInterfaces","apiVersion": "2019-11-01","name": "[concat(variables('networkInterfaceName'), copyindex())]","location": "[parameters('location')]","dependsOn": ["[resourceId('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'))]","[resourceId('Microsoft.Network/loadBalancers/', parameters('publicLBName'))]"],"properties": {"ipConfigurations": [{"name": "ipconfig1","properties": {"subnet": {"id": "[variables('subnetRef')]"},"loadBalancerBackendAddressPools": [{"id": "[concat(variables('publicLBID'), '/backendAddressPools/BackendPoolMaster')]"}],"loadBalancerInboundNatRules": [{"id": "[concat(variables('publicLBID'), '/inboundNatRules/lbNAT-master',copyindex())]"}],"privateIPAllocationMethod": "Dynamic"}}]},"copy": {"name": "nicLoop","count": "[parameters('defaultMasterCount')]"}},{"type": "Microsoft.Network/publicIPAddresses","apiVersion": "2019-09-01","name": "[variables('publicIPAddressName')]","location": "[parameters('location')]","sku": {"name": "Standard"},"properties": {"publicIPAllocationMethod": "Static","publicIPAddressVersion": "IPv4","dnsSettings": {"domainNameLabel": "[parameters('dnsLabelPrefix')]"},"idleTimeoutInMinutes": 10}},{"type": "Microsoft.Network/loadBalancers","apiVersion": "2018-06-01","name": "[parameters('publicLBName')]","location": "[parameters('location')]","dependsOn": ["[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]"],"sku": {"name": "Standard"},"properties": {"frontendIPConfigurations": [{"name": "LoadBalancerFrontEnd","properties": {"publicIPAddress": {"id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"}}}],"backendAddressPools": [{"name": "BackendPoolNode"},{"name": "BackendPoolMaster"}],"loadBalancingRules": [{"name": "HttpLBRule","properties": {"frontendIPConfiguration": {"id": "[concat(variables('publicLBID'),'/frontendIPConfigurations/LoadBalancerFrontEnd')]"},"backendAddressPool": {"id": "[concat(variables('publicLBID'),'/backendAddressPools/BackendPoolNode')]"},"protocol": "Tcp","frontendPort": 80,"backendPort": 80,"enableFloatingIP": false,"disableOutboundSnat": false,"idleTimeoutInMinutes": 5,"probe": {"id": "[concat(variables('publicLBID'),'/probes/tcpProbe')]"}}},{"name": "APILBRule","properties": {"frontendIPConfiguration": {"id": "[concat(variables('publicLBID'),'/frontendIPConfigurations/LoadBalancerFrontEnd')]"},"backendAddressPool": {"id": "[concat(variables('publicLBID'),'/backendAddressPools/BackendPoolMaster')]"},"protocol": "Tcp","frontendPort": 6443,"backendPort": 6443,"enableFloatingIP": false,"disableOutboundSnat": false,"idleTimeoutInMinutes": 5,"probe": {"id": "[concat(variables('publicLBID'),'/probes/apitcpProbe')]"}}}],"probes": [{"name": "tcpProbe","properties": {"protocol": "Tcp","port": 80,"intervalInSeconds": 5,"numberOfProbes": 2}},{"name": "apitcpProbe","properties": {"protocol": "Tcp","port": 6443,"intervalInSeconds": 5,"numberOfProbes": 2}}],"inboundNatRules": [{"name": "lbNAT-master0","properties": {"frontendIPConfiguration": {"id": "[concat(variables('publicLBID'),'/frontendIPConfigurations/LoadBalancerFrontEnd')]"},"protocol": "Tcp","frontendPort": "50200","backendPort": "22"}},{"name": "lbNAT-master1","properties": {"frontendIPConfiguration": {"id": "[concat(variables('publicLBID'),'/frontendIPConfigurations/LoadBalancerFrontEnd')]"},"protocol": "Tcp","frontendPort": "50201","backendPort": "22"}},{"name": "lbNAT-master2","properties": {"frontendIPConfiguration": {"id": "[concat(variables('publicLBID'),'/frontendIPConfigurations/LoadBalancerFrontEnd')]"},"protocol": "Tcp","frontendPort": "50202","backendPort": "22"}}],"inboundNatPools": [{"name": "lbNAT-node","properties": {"frontendIPConfiguration": {"id": "[concat(variables('publicLBID'),'/frontendIPConfigurations/LoadBalancerFrontEnd')]"},"protocol": "Tcp","frontendPortRangeStart": 50100,"frontendPortRangeEnd": 50199,"backendPort": 22}}]}},{"type": "Microsoft.Compute/availabilitySets","apiVersion": "2016-04-30-preview","name": "[variables('availabilitySetName')]","location": "[parameters('location')]","properties": {"platformFaultDomainCount": 2,"platformUpdateDomainCount": 2,"managed": true}},{"type": "Microsoft.Compute/virtualMachines","apiVersion": "2019-07-01","name": "[concat(parameters('vmNamePrefix'), copyindex())]","location": "[parameters('location')]","dependsOn": ["[concat('Microsoft.Network/networkInterfaces/', variables('networkInterfaceName'), copyindex())]","[concat('Microsoft.Compute/availabilitySets/', variables('availabilitySetName'))]","[concat('Microsoft.Compute/virtualMachineScaleSets/', parameters('vmssName'))]"],"properties": {"availabilitySet": {"id": "[resourceId('Microsoft.Compute/availabilitySets',variables('availabilitySetName'))]"},"hardwareProfile": {"vmSize": "[parameters('vmSize')]"},"storageProfile": {"osDisk": {"createOption": "FromImage","managedDisk": {"storageAccountType": "[variables('osDiskType')]"}},"imageReference": {"publisher": "Canonical","offer": "0001-com-ubuntu-server-focal","sku": "20_04-lts-gen2","version": "latest"}},"networkProfile": {"networkInterfaces": [{"id": "[resourceId('Microsoft.Network/networkInterfaces',concat(variables('networkInterfaceName'),copyindex()))]"}]},"osProfile": {"computerName": "[concat(parameters('vmNamePrefix'), copyindex())]","adminUsername": "[parameters('adminUsername')]","adminPassword": "[parameters('adminKey')]","linuxConfiguration": {"disablePasswordAuthentication": true,"ssh": {"publicKeys": [{"path": "[concat('/home/', parameters('adminUsername'), '/.ssh/authorized_keys')]","keyData": "[parameters('adminKey')]"}]}}}},"copy": {"name": "virtualMachineLoop","count": "[parameters('defaultMasterCount')]"}},{"type": "Microsoft.Compute/virtualMachineScaleSets","apiVersion": "2019-07-01","name": "[parameters('vmssName')]","location": "[parameters('location')]","dependsOn": ["[concat('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'))]","[concat('Microsoft.Network/loadBalancers/', parameters('publicLBName'))]"],"tags": {"cluster-autoscaler-enabled": "true","cluster-autoscaler-name": "[resourceGroup().name]","min": "[parameters('minNodeCount')]","max": "[parameters('maxNodeCount')]","poolName": "[parameters('vmssName')]"},"sku": {"name": "[parameters('vmSize')]","tier": "Standard","capacity": "[parameters('defaultNodeCount')]"},"properties": {"overprovision": false,"upgradePolicy": {"mode": "Manual"},"virtualMachineProfile": {"storageProfile": {"osDisk": {"createOption": "FromImage","caching": "ReadWrite"},"imageReference": {"publisher": "Canonical","offer": "0001-com-ubuntu-server-focal","sku": "20_04-lts-gen2","version": "latest"}},"osProfile": {"computerNamePrefix": "[parameters('vmssName')]","adminUsername": "[parameters('adminUsername')]","adminPassword": "[parameters('adminKey')]","linuxConfiguration": {"disablePasswordAuthentication": true,"ssh": {"publicKeys": [{"path": "[concat('/home/', parameters('adminUsername'), '/.ssh/authorized_keys')]","keyData": "[parameters('adminKey')]"}]}}},"networkProfile": {"networkInterfaceConfigurations": [{"name": "[concat(parameters('vmssName'),'nic')]","properties": {"primary": true,"ipConfigurations": [{"name": "[concat('ipconfigVmss', parameters('vmssName'))]","properties": {"subnet": {"id": "[variables('vmssSubnetRef')]"},"loadBalancerBackendAddressPools": [{"id": "[concat(variables('publicLBID'), '/backendAddressPools/BackendPoolNode')]"}],"loadBalancerInboundNatPools": [{"id": "[concat(variables('publicLBID'), '/inboundNatPools/lbNAT-node')]"}]}}]}}]}}}}],"outputs": {"adminUsername": {"type": "String","value": "[parameters('adminUsername')]"},"hostname": {"type": "String","value": "[reference(variables('publicIPAddressName')).dnsSettings.fqdn]"},"sshCommand": {"type": "String","value": "[concat('ssh ', parameters('adminUsername'), '@', reference(variables('publicIPAddressName')).dnsSettings.fqdn)]"}}
  }
  

  • 可以修改 master 和 node 的名字前缀、部署区域、数量和 VM 类型

    

    • ssh 22 对外由 LB 配置 NAT 端口实现，如配置文件中 50200 → master-0

      • 已经包含的规则转换（不含 30880）

        服务	协议	规则	后端端口	前端端口	节点池
        ssh	TCP	入站 NAT	22	50200, 50201, 50202, 50100~50199	主节点, 普通节点
        api 服务器	TCP	负载均衡	6443	6443	主节点
        ks 控制台	TCP	负载均衡	30880	30880	主节点
        http	TCP	负载均衡	80	80	普通节点
        https	TCP	负载均衡	443	443	普通节点

    • node 使用 VMSS

• 部署 K8S cluster
  最简单的方式还是用 kk 完成，注意在 kubernetes 1.24 以后，psp 弃用，在 kk 中还有 psp 权限管理，安装的时候会报错。建议使用 1.23.10，也是 kk 现在默认的版本。

  • 证书传输 → Master-0

    scp  -i zyi.pem -P 50200 zyi.pem zyi@20.247.0.170:/home/zyi/.ssh/

  • 每一台安装 socat 等必须的软件

    ssh -i zyi.pem -p50200 zyi@20.247.0.170 'sudo apt install socat conntrack'
    

  • 登录到 Mater-0

    ssh -i zyi.pem -p50200 zyi@20.247.0.170
    

  • 下载 kk 并赋予可执行权限

  curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.10 sh -chmod +x kk
  

  • 创建配置文件模板

    ./kk create config --with-kubesphere v3.3.2 --with-kubernetes v1.22.12
    

    • KubeSphere 3.3 对应 Kubernetes 版本推荐：v1.20.x、v1.21.x、* v1.22.x、* v1.23.x 和 * v1.24.x。带星号的版本可能出现边缘节点部分功能不可用的情况。因此，如需使用边缘节点，推荐安装 v1.21.x。如果未指定 Kubernetes 版本，KubeKey 将默认安装 Kubernetes v1.23.10。有关支持的 Kubernetes 版本请参阅支持矩阵。

    • 如果在此步骤中的命令中未添加标志 -with-kubesphere，则不会部署 KubeSphere，除非您使用配置文件中的 addons 字段进行安装，或稍后使用 ./kk create cluster 时再次添加此标志。

    • 如果在未指定 KubeSphere 版本的情况下添加标志 --with kubesphere`，将安装 KubeSphere 的最新版本。

    • 修改的内容用红色标注

      apiVersion: kubekey.kubesphere.io/v1alpha2
      kind: Cluster
      metadata:name: kubeCluster
      spec:hosts:- {name: master-0, address: 20.210.0.156, port: 50200, internalAddress: 10.211.1.5, user: zyi, privateKeyPath: "~/.ssh/zyi.pem"}- {name: master-1, address: 20.210.0.156, port: 50201, internalAddress: 10.211.1.6, user: zyi, privateKeyPath: "~/.ssh/zyi.pem"}- {name: master-2, address: 20.210.0.156, port: 50202, internalAddress: 10.211.1.4, user: zyi, privateKeyPath: "~/.ssh/zyi.pem"}- {name: node000000, address: 20.210.0.156, port: 50100, internalAddress: 10.211.0.4, user: zyi, privateKeyPath: "~/.ssh/zyi.pem"}- {name: node000001, address: 20.210.0.156, port: 50101, internalAddress: 10.211.0.5, user: zyi, privateKeyPath: "~/.ssh/zyi.pem"}- {name: node000002, address: 20.210.0.156, port: 50102, internalAddress: 10.211.0.6, user: zyi, privateKeyPath: "~/.ssh/zyi.pem"}roleGroups:etcd:- master-0- master-1- master-2control-plane:- master-0- master-1- master-2worker:- node000000- node000001- node000002controlPlaneEndpoint:## Internal loadbalancer for apiservers# internalLoadbalancer: haproxydomain: lb.etaon.labaddress: "20.210.0.156"port: 6443kubernetes:version: v1.23.10clusterName: cluster.localautoRenewCerts: truecontainerManager: dockeretcd:type: kubekeynetwork:plugin: flannelkubePodsCIDR: 10.233.64.0/18kubeServiceCIDR: 10.233.0.0/18## multus support. https://github.com/k8snetworkplumbingwg/multus-cnimultusCNI:enabled: falseregistry:privateRegistry: ""namespaceOverride: ""registryMirrors: []insecureRegistries: []addons: []---
      apiVersion: installer.kubesphere.io/v1alpha1
      kind: ClusterConfiguration
      metadata:name: ks-installernamespace: kubesphere-systemlabels:version: v3.3.2
      spec:persistence:storageClass: ""authentication:jwtSecret: ""zone: ""local_registry: ""namespace_override: ""# dev_tag: ""etcd:monitoring: falseendpointIps: localhostport: 2379tlsEnable: truecommon:core:console:enableMultiLogin: trueport: 30880type: NodePort# apiserver:#  resources: {}# controllerManager:#  resources: {}redis:enabled: falsevolumeSize: 2Giopenldap:enabled: falsevolumeSize: 2Giminio:volumeSize: 20Gimonitoring:# type: externalendpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090GPUMonitoring:enabled: falsegpu:kinds:- resourceName: "nvidia.com/gpu"resourceType: "GPU"default: truees:# master:#   volumeSize: 4Gi#   replicas: 1#   resources: {}# data:#   volumeSize: 20Gi#   replicas: 1#   resources: {}logMaxAge: 7elkPrefix: logstashbasicAuth:enabled: falseusername: ""password: ""externalElasticsearchHost: ""externalElasticsearchPort: ""alerting:enabled: false# thanosruler:#   replicas: 1#   resources: {}auditing:enabled: false# operator:#   resources: {}# webhook:#   resources: {}devops:enabled: false# resources: {}jenkinsMemoryLim: 8GijenkinsMemoryReq: 4GijenkinsVolumeSize: 8Gievents:enabled: false# operator:#   resources: {}# exporter:#   resources: {}# ruler:#   enabled: true#   replicas: 2#   resources: {}logging:enabled: falselogsidecar:enabled: truereplicas: 2# resources: {}metrics_server:enabled: falsemonitoring:storageClass: ""node_exporter:port: 9100# resources: {}# kube_rbac_proxy:#   resources: {}# kube_state_metrics:#   resources: {}# prometheus:#   replicas: 1#   volumeSize: 20Gi#   resources: {}#   operator:#     resources: {}# alertmanager:#   replicas: 1#   resources: {}# notification_manager:#   resources: {}#   operator:#     resources: {}#   proxy:#     resources: {}gpu:nvidia_dcgm_exporter:enabled: false# resources: {}multicluster:clusterRole: nonenetwork:networkpolicy:enabled: falseippool:type: nonetopology:type: noneopenpitrix:store:enabled: falseservicemesh:enabled: falseistio:components:ingressGateways:- name: istio-ingressgatewayenabled: falsecni:enabled: falseedgeruntime:enabled: falsekubeedge:enabled: falsecloudCore:cloudHub:advertiseAddress:- ""service:cloudhubNodePort: "30000"cloudhubQuicNodePort: "30001"cloudhubHttpsNodePort: "30002"cloudstreamNodePort: "30003"tunnelNodePort: "30004"# resources: {}# hostNetWork: falseiptables-manager:enabled: truemode: "external"# resources: {}# edgeService:#   resources: {}terminal:timeout: 600
      

  • 安装部署

    ./kk create cluster -f config-sample.yaml
    

    • 安装日志

    kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f
    

    #####################################################
    ###              Welcome to KubeSphere!           ###
    #####################################################Console: http://10.211.1.5:30880
    Account: admin
    Password: P@88w0rd
    NOTES：1. After you log into the console, please check themonitoring status of service components in"Cluster Management". If any service is notready, please wait patiently until all components are up and running.2. Please change the default password after login.#####################################################
    https://kubesphere.io             2023-08-11 03:31:56
    #####################################################
    

  • 在 LBer 上为 30880端口 配置规则并在 ASG 上 permit

    

    

  • http://hostip：30880

    

• 测试

  apiVersion: v1
  kind: Service
  metadata:name: hello-kubernetes
  spec:type: NodePortports:- port: 80targetPort: 8080selector:app: hello-kubernetes
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:name: hello-kubernetes
  spec:replicas: 3selector:matchLabels:app: hello-kubernetestemplate:metadata:labels:app: hello-kubernetesspec:containers:- name: hello-kubernetesimage: paulbouwer/hello-kubernetes:1.5ports:- containerPort: 8080env:- name: MESSAGEvalue: I just deployed a PodVM on the Azure VM Cluster!!
  

  • 配置外部访问30596 → 80

    kubectl get svc
    NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
    hello-kubernetes   NodePort    10.233.31.158   <none>        80:30596/TCP   167m
    

  • 负载均衡上配置

    

  • 访问前端 公网 IP 或 DNS 名称